Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This Office action is in response to the amendment filed on 03/03/2026. Claims 1-2, 4-8, and 10-15 are currently pending with claim 1 being amended.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 02/20/2026 has been received. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Amendment
The amendments to the claims submitted on 03/03/2026 overcome the claim objections set forth in the previous Office action except for those set forth in the claim objection section.
Response to Arguments
Applicant's arguments filed 03/03/2026 have been fully considered but they are not persuasive. The Applicant has asserted that the cited prior art does not anticipate the newly amended claim limitation of the plurality of gripping positions changing based on the posture of the workpiece to be grasped in combination with the rest of independent claim 1. The Examiner disagrees for at least the following reasons. Wellman teaches a method of determine a grasping strategy which is based on “a size, shape, position, orientation, weight, or other attributes of the item” (Wellman Paragraph 0077). This demonstrates that it is well known to adjust the grasping strategy (at least the approach) based on these attributes. Nagarajan, discloses a method of determining grasping regions on a target object. They perform the determination of grasping regions based on the identified approach vector to be performed. (Nagarajan Paragraph 0004). The combination of these references would suggest a method of identifying a grasp strategy including a direction of approach based on the position/orientation of the object to be grasped and to further identify a plurality of possible grasping regions based on the direction of approach.
Examiner notes wherein Applicant argues the newly amended limitations, which have not been addressed by the prior art of record. As such, Examiner has augmented the below rejection(s) in view of the prior art of record to address the newly amended limitations.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 6-8, and 11-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wellman et al. (US 20170106532 A1), hereinafter Wellman in view of Nagarajan et al. (US 20210347040 A1), hereinafter Nagarajan.
Regarding claim 1, Wellman further teaches:
[Claim 1] (Currently Amended) A controller that performs a control in which a robot autonomously performs a given work, the controller comprising:
a first processor and a first storage, the first processor performs processing including: (Paragraph 0041, " Processor 90 is operable to execute instructions associated with the functionality provided by management module 15. Processor 90 may comprise one or more general purpose computers, dedicated microprocessors, or other processing devices capable of communicating electronic information. Examples of processor 90 include one or more application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs) and any other suitable specific or general purpose processors." And Paragraph 0042, “Memory 91 stores processor instructions, inventory requests, reservation information, state information for the various components of inventory system 10 and/or any other appropriate values, parameters, or information utilized by management module 15 during operation. Memory 91 may represent any collection and arrangement of volatile or nonvolatile, local or remote devices suitable for storing data. Examples of memory 91 include, but are not limited to, random access memory (RAM) devices, read only memory (ROM) devices, magnetic storage devices, optical storage devices or any other suitable data storage devices.”)
acquiring state information including a state of a workpiece that is a work target while performing the given work; (Paragraph 0072, "The attribute detection module 710 receives information from sensors, such as the sensor package 16 of FIG. 1, and determines attributes of inventory items 40 detected by the sensors. The database query module 715 can access information from the item database 37 and/or the item gripping database 36 of FIG. 1, such as based on the information from the attribute detection module 710 or other modules described herein.")
determining candidates of a work position of the workpiece based on the state information; (Paragraph 0077, " In some embodiments, the database query module 715 may access grasping strategy information based on detected physical attributes of the item (e.g., a size, shape, position, orientation, weight, or other attributes of the item regardless of a SKU of the item). The database query module 715 may receive information about grasping strategies based on the recognized item and/or characteristics.")
transmitting a selection request for requesting a selection of the work position from the candidates of the work position to an operation terminal, (Paragraph 0072, "The human-based grasping strategy module 720 handles requests and receipt of any human input for grasping strategy, such as at 54 in FIG. 1.") the operation terminal being connected data-communicably with the first processor via a communication network; and (Paragraph 0097, "FIG. 13 illustrates aspects of an example environment 1300 for implementing aspects in accordance with various embodiments. As will be appreciated, although a Web-based environment is used for purposes of explanation, different environments may be used, as appropriate, to implement various embodiments. The environment includes an electronic client device 1302, which can include any appropriate device operable to send and receive requests, messages, or information over an appropriate network 1304 and convey information back to a user of the device. Examples of such client devices include personal computers, cell phones, handheld messaging devices, laptop computers, set-top boxes, personal data assistants, electronic book readers, and the like. The network can include any appropriate network, including an intranet, the Internet, a cellular network, a local area network or any other such network or combination thereof. Components used for such a system can depend at least in part upon the type of network and/or environment selected. Protocols and components for communicating via such a network are well known and will not be discussed herein in detail. Communication over the network can be enabled by wired or wireless connections and combinations thereof. In this example, the network includes the Internet, as the environment includes a Web server 1306 for receiving requests and serving content in response thereto, although for other networks an alternative device serving a similar purpose could be used as would be apparent to one of ordinary skill in the art.")
when information on a selected position that is the selected work position is received from the operation terminal, (Paragraph 0089, "FIG. 8 illustrates an example user interface for obtaining human input for a grasping strategy. The user interface includes a screen showing various selection options that a human user can select to provide input about a grasping strategy for a particular item, e.g., a coffee cup 802. In a first row at 804, the user has options for different end effectors, such as a mechanical pincher 806, soft robotics 808, vacuum 810, or electro-adhesion 812. The user can select one or more of the end effectors as illustrated by the selection boxes 814. In a second row at 820, the user can select an approach for picking up the desired item 802. For example, the presented options may include an angled or lateral approach 822, a top-down approach 824, an underneath approach 826, or a multiple arm approach 828 (in which multiple arms are used for grasping and moving the item 828). The user can select an approach (or more than one approach in the case of multiple robotic arms) as illustrated by the selection box 818. At 830, a visualization of the selected grasping strategy is presented in which the coffee cup is to be grasped at an angle in a lateral approach by a combination of a mechanical pincher and vacuum based on the selections made at 814 and 818.") causing the robot to operate autonomously (Paragraph 0020, "The controller 32 can provide instructions to the robotic arm 12 for gripping the item 40 based on the gripping strategy accessed from the gripping database at 36 (e.g., at 52).") according to the selected position, (Paragraph 0021, "The grasping strategy instructed by the controller 32 to the robotic arm 12 (e.g., at 52) can be based on a combination of the human input for grasping strategy communicated at 54, the detected attributes communicated at 46, the record accessed from the item database 37 at 49, and/or the information accessed from the item gripping database 36 at 48.")
wherein, in the processing of transmitting the selection request to the operation terminal, (Paragraph 0072, "The human-based grasping strategy module 720 handles requests and receipt of any human input for grasping strategy, such as at 54 in FIG. 1.") the first processor acquires first image data that is data of an image of the workpiece and a peripheral environment of the workpiece, (Paragraph 0018, "A camera or other optical imaging device scans the mug, providing information about the mug's orientation, e.g., in this case that the mug is on its side with the open top facing the camera and the handle to the right. The weight, shape, and orientation of the mug are collectively used with any other relevant and/or available information (e.g., about the size and orientation of the box into which the mug is to be placed to fulfill the order) to query a grasping strategy database for a grasping strategy appropriate for this situation." as well as Paragraph 0074, "In some embodiments, the attribute detection module 710 can additionally, or alternatively, receive active sensing information from other sensors (e.g., force sensing, tactile sensing, pressure sensing, voltage sensing, conductance sensing, ultrasonic sensing, x-ray sensing, or other sensing), such as to determine physical attributes of a detected item to be grasped or its surroundings, such as structural integrity, deformability, weight, surface characteristics (e.g., how slippery the item is), or other physical attributes of a detected object.") and carries out image processing of the first image data (Paragraph 0089, "FIG. 8 illustrates an example user interface for obtaining human input for a grasping strategy. The user interface includes a screen showing various selection options that a human user can select to provide input about a grasping strategy for a particular item, e.g., a coffee cup 802. In a first row at 804, the user has options for different end effectors, such as a mechanical pincher 806, soft robotics 808, vacuum 810, or electro-adhesion 812. The user can select one or more of the end effectors as illustrated by the selection boxes 814. In a second row at 820, the user can select an approach for picking up the desired item 802. For example, the presented options may include an angled or lateral approach 822, a top-down approach 824, an underneath approach 826, or a multiple arm approach 828 (in which multiple arms are used for grasping and moving the item 828). The user can select an approach (or more than one approach in the case of multiple robotic arms) as illustrated by the selection box 818. At 830, a visualization of the selected grasping strategy is presented in which the coffee cup is to be grasped at an angle in a lateral approach by a combination of a mechanical pincher and vacuum based on the selections made at 814 and 818.") to generate a virtual model that is a model including at least 3-D data from the image of the workpiece and the peripheral environment of the workpiece, (Paragraph 0073, "The attribute detection module 710 can interact with any number and/or type of sensors to determine attributes of an item to be grasped. For example, the attribute detection module 710 can receive information from imaging devices or optical sensors to determine physical characteristics, such as size, shape, position, orientation, and/or surface characteristics (e.g., how porous and/or slippery the item is based on the surface appearance). Any suitable optical technology can be utilized, including, but not limited to, two-dimensional cameras, depth sensors, time of flight sensing (e.g., broadcasting a source of light and determining a time of reflection for each pixel to determine a distance from the sensor for each pixel to determine a three-dimensional array of data points representing a virtual model of the sensed item and environment), structured light sensing (e.g., projecting a known image from a light source, observing the image as distorted by variations in the surface of the detected item, and analyzing the distortions with respect to the projected image to determine positioning of the features that caused the distortion), stereo sensing (e.g., analyzing differences in images collected from multiple cameras arranged at known offsets from one another to generate a point cloud or digital model), active stereo sensing (e.g., projecting a pattern of light to improve precision of detection of features while using stereo sensing), any other optically-based methodology of observing light for generating a digital representation of a physical object, or any combination thereof.") and the first processor transmits the selection using the virtual model to the operation terminal for an operator to select … from a plurality of … presented to the operator, (Paragraph 0089, "FIG. 8 illustrates an example user interface for obtaining human input for a grasping strategy. The user interface includes a screen showing various selection options that a human user can select to provide input about a grasping strategy for a particular item, e.g., a coffee cup 802. In a first row at 804, the user has options for different end effectors, such as a mechanical pincher 806, soft robotics 808, vacuum 810, or electro-adhesion 812. The user can select one or more of the end effectors as illustrated by the selection boxes 814. In a second row at 820, the user can select an approach for picking up the desired item 802. For example, the presented options may include an angled or lateral approach 822, a top-down approach 824, an underneath approach 826, or a multiple arm approach 828 (in which multiple arms are used for grasping and moving the item 828). The user can select an approach (or more than one approach in the case of multiple robotic arms) as illustrated by the selection box 818. At 830, a visualization of the selected grasping strategy is presented in which the coffee cup is to be grasped at an angle in a lateral approach by a combination of a mechanical pincher and vacuum based on the selections made at 814 and 818.") … in which the plurality of gripping … change based on a posture of the workpiece, (Paragraph 0037, “Components of inventory system 10 may provide information to management module 15 regarding their current state, other components of inventory system 10 with which they are interacting, and/or other conditions relevant to the operation of inventory system 10. This may allow management module 15 to utilize feedback from the relevant components to update algorithm parameters, adjust policies, or otherwise modify its decision-making to respond to changes in operating conditions or the occurrence of particular events.” And Paragraph 0077, “A database query module 715, as mentioned above, may access information about item grasping strategy from a database, such as the item grasping database 36 of FIG. 1. The database query module 715 may additionally or alternatively access records about items from a database, such as from the item database 37 of FIG. 1. In some embodiments, the database query module 715 may receive grasping strategy information based on a detected item being a recognized item (e.g., corresponding to a particular SKU). In some embodiments, the database query module 715 may access grasping strategy information based on detected physical attributes of the item (e.g., a size, shape, position, orientation, weight, or other attributes of the item regardless of a SKU of the item). The database query module 715 may receive information about grasping strategies based on the recognized item and/or characteristics.”)
wherein the first storage stores second attribute information (Paragraph 0072, "The attribute detection module 710 receives information from sensors, such as the sensor package 16 of FIG. 1, and determines attributes of inventory items 40 detected by the sensors. The database query module 715 can access information from the item database 37 and/or the item gripping database 36 of FIG. 1, such as based on the information from the attribute detection module 710 or other modules described herein.") including information on characteristics of the peripheral environment of the workpiece (Paragraph 0073, "The attribute detection module 710 can interact with any number and/or type of sensors to determine attributes of an item to be grasped. For example, the attribute detection module 710 can receive information from imaging devices or optical sensors to determine physical characteristics, such as size, shape, position, orientation, and/or surface characteristics (e.g., how porous and/or slippery the item is based on the surface appearance). Any suitable optical technology can be utilized, including, but not limited to, two-dimensional cameras, depth sensors, time of flight sensing (e.g., broadcasting a source of light and determining a time of reflection for each pixel to determine a distance from the sensor for each pixel to determine a three-dimensional array of data points representing a virtual model of the sensed item and environment), structured light sensing (e.g., projecting a known image from a light source, observing the image as distorted by variations in the surface of the detected item, and analyzing the distortions with respect to the projected image to determine positioning of the features that caused the distortion), stereo sensing (e.g., analyzing differences in images collected from multiple cameras arranged at known offsets from one another to generate a point cloud or digital model), active stereo sensing (e.g., projecting a pattern of light to improve precision of detection of features while using stereo sensing), any other optically-based methodology of observing light for generating a digital representation of a physical object, or any combination thereof.") of the given work set to the peripheral environment, (Paragraphs 0022-0023, "FIG. 2 illustrates the contents of an inventory system 10 according to some embodiments of the present disclosure. Inventory system 10 includes a management module 15, one or more mobile drive units 20, one or more inventory holders 30, and one or more inventory stations 50. Mobile drive units 20 transport inventory holders 30 between points within a workspace 70 in response to commands communicated by management module 15. Each inventory holder 30 stores one or more types of inventory items. As a result, inventory system 10 is capable of moving inventory items between locations within workspace 70 to facilitate the entry, processing, and/or removal of inventory items from inventory system 10 and the completion of other tasks involving inventory items.
Management module 15 assigns tasks to appropriate components of inventory system 10 and coordinates operation of the various components in completing the tasks. These tasks may relate not only to the movement and processing of inventory items, but also to the management and maintenance of the components of inventory system 10. For example, management module 15 may assign portions of workspace 70 as parking spaces for mobile drive units 20, the scheduled recharge or replacement of mobile drive unit batteries, the storage of empty inventory holders 30, or any other operations associated with the functionality supported by inventory system 10 and its various components. Management module 15 may select components of inventory system 10 to perform these tasks and communicate appropriate commands and/or data to the selected components to facilitate completion of these operations. Although shown in FIG. 2 as a single, discrete component, management module 15 may represent multiple components and may represent or include portions of mobile drive units 20 or other elements of inventory system 10. As a result, any or all of the interaction between a particular mobile drive unit 20 and management module 15 that is described below may, in particular embodiments, represent peer-to-peer communication between that mobile drive unit 20 and one or more other mobile drive units 20. The contents and operation of an example embodiment of management module 15 are discussed further below with respect to FIG. 3.") and information on a position of the workpiece with respect to the peripheral environment, (Paragraph 0020, "The sensor package 16 includes one or more sensors (of like or varying type) arranged to detect the item 40 while the item 40 is being maintained by the grasping environment 14. The sensor package 16 communicates detected attributes (as at 46), such as weight, geometric characteristics (e.g., size, position, or orientation), electrical conductivity, magnetic properties, surface characteristics (e.g., how slippery or porous the item is), deformability, and/or structural integrity of the item 40, to the controller 32. The detected attributes may also include a unique identifier of the item 40, such as a barcode- or RFID-encoded serial number. Based on the detected attributes, the controller 32 may access (as at 49) the item database 37, such as to access a record for the inventory item 40. The record can include information about attributes of the item, such as weight, size, shape, or other physical characteristics of the item. Based on the record from the item database 37 and/or the detected attributes from the sensor package 16, the controller 32 may access (as at 48) an item gripping database 36 to access an item grasping strategy stored for that item or items with similar characteristics. The controller 32 can provide instructions to the robotic arm 12 for gripping the item 40 based on the gripping strategy accessed from the gripping database at 36 (e.g., at 52).") and
wherein, in the processing of determining the candidates of the work position, the first processor determines candidates of the position of the workpiece with respect to the peripheral environment (Paragraph 0083, "The constraints module 725 can handle information about factors that may affect or constrain a grasping strategy independent of the identity of the item to be grasped. As an illustrative example, the constraints module 725 may access information about a receptacle into which a grasped item is to be placed so that end effectors that are too large to fit in the receptacle will be eliminated as options for the grasping strategy. In general, the constraints module may provide information about a grasping or receiving environment for the grasped item, such as tray types available for initial presentation for grasping, a size of a receptacle or a receiving zone, a location of a receptacle or receiving zone (such as on the ground, three feet off the ground, five feet from the grasping environment, etc.), a type of receiving environment (such as a tray, shipping box, inventory holder, etc.), a designated orientation of the grasped item when released in the receiving environment (such as if the grasped item is to be presented to a human operator in a particular ergonomically compliant orientation or if the item is to be placed in a particular orientation to facilitate subsequent automated actions).") based on the second attribute information, as the candidates of the work position, (Paragraph 0085, "The grasping strategy selection module 730 can determine a grasping strategy for a particular item. For example, the grasping strategy selection module 730 may utilize information from any or all of the attribute detection module 710, the database query module 715, the human-based grasping strategy module 720, the constraints module 725, and the grasping strategy evaluation module 740 to determine a grasping strategy for a particular item and the environments in which the item is to be grasped, moved, and/or released. In addition to determining how an item is to be grasped, or as an alternative, the grasping strategy selection module 730 may be involved in determining whether to grasp something using a robotic arm 12. For example, if the attribute detection module 710 detects damage to an item 40, the grasping strategy selection module 730 may instruct an appropriate response, such as selecting a grasping strategy that includes refraining from grasping the damaged item and locating another item of the same type that is undamaged instead.") and the first processor transmits the candidates of the work position to the operation terminal. (Paragraph 0089, "FIG. 8 illustrates an example user interface for obtaining human input for a grasping strategy. The user interface includes a screen showing various selection options that a human user can select to provide input about a grasping strategy for a particular item, e.g., a coffee cup 802. In a first row at 804, the user has options for different end effectors, such as a mechanical pincher 806, soft robotics 808, vacuum 810, or electro-adhesion 812. The user can select one or more of the end effectors as illustrated by the selection boxes 814. In a second row at 820, the user can select an approach for picking up the desired item 802. For example, the presented options may include an angled or lateral approach 822, a top-down approach 824, an underneath approach 826, or a multiple arm approach 828 (in which multiple arms are used for grasping and moving the item 828). The user can select an approach (or more than one approach in the case of multiple robotic arms) as illustrated by the selection box 818. At 830, a visualization of the selected grasping strategy is presented in which the coffee cup is to be grasped at an angle in a lateral approach by a combination of a mechanical pincher and vacuum based on the selections made at 814 and 818.")
Wellman does not specifically teach the grasping approach including contact positions on the object and this being part of the communication/display to the user for determining an overall grasping strategy. However, Nagarajan, in the same field of endeavor of robotics, teaches:
… a gripping position of the workpiece … gripping positions … the gripping position being at least one of the plurality of gripping positions identified for possible selection on the workpiece itself, … positions identified for possible selection on the workpiece itself … (Paragraph 0056, “In some implementations, the approach vector engine 132 generates the end effector approach vector based on a spatial region determined by the model engine 120, based on the spatial region corresponding to an object to be grasped. The spatial region can be, for example, a grasp region defined by output from a “grasp regions and semantic indications model” as described herein, or a spatial region defined by an object detection and classification model as described herein. In some of those implementations, the approach vector engine 132 selects one or more particular 3D points, from a group of 3D points of 3D vision data, based on the 3D point(s) corresponding to the pixel(s) encapsulated by the spatial region. Further, the approach vector engine 132 determines a surface normal for each of one or more of the selected 3D point(s), and determines an end effector approach vector based on one or more of the surface normal(s). The end effector approach vector can have a direction component that is opposite from one of the surface normals, but otherwise strictly conforms to that surface normal. In some implementations, the approach vector engine 132 utilizes a grasp direction (e.g., top, side) of a selected grasp strategy to select a surface normal utilized in determining the approach vector. For example, if a “top” grasp is to be performed, a surface normal that extends “up” can be selected in lieu of one that extends to the “side” or “down”. As one example, and referring to FIG. 10, a first surface normal 1002A and a second surface normal 1002B can both be determined based on separate 3D points, of 3D vision data, of a spatial region of a coffee mug. If a “top” grasp is to be performed, first surface normal 1002A can be selected in lieu of surface normal 1002B, as surface normal 1002A extends in a “top” direction, whereas surface normal 1002B extends in a “side” direction. Additionally, in some implementations multiple grasp poses can be considered for each of one or more end effector approach vectors, and one grasp pose selected based on it avoiding collisions (with the object and/or with other objects in the environment), satisfying inverse kinematic constraints, and/or based on other criterion/criteria.” Please also see Paragraphs 0004, “As one particular example, the vision data can be a two-dimensional (2D) image generated by a vision component of a robot and can be processed over a trained machine learning model to generate one or more grasp regions and corresponding semantic indications. For instance, each grasp region can indicate a bounding rectangle (or other bounding shape) that encapsulates one or more contiguous pixels of the 2D image. Also, for instance, the corresponding semantic indications can each indicate a grasp direction for the grasp (e.g., side, top, etc.). At least one grasp region can be selected based on it corresponding to an object to be grasped. For example, a given grasp region can be selected based in it corresponding to a region having a classification that corresponds to an object to be grasped, where the classification of the region is based on output generated over a separate object detection and classification machine learning model. Further, one or more particular three-dimensional (3D) points can be selected, from a group of 3D points, based on the 3D point(s) corresponding to the pixel(s) encapsulated by the selected grasp region. The group of 3D points can be generated by the same vision component that generated the 2D image (e.g., the 2D image can be the same as the 3D points, except for lacking a depth channel) or can be generated by an additional vision component (e.g., the 2D image can be generated by a camera and the 3D points can be a point cloud from a separate laser scanner) and mapped to the pixels of the 2D image. A surface normal can be determined for each of one or more of the selected 3D point(s), and an end effector approach vector determined based on one or more of the surface normal(s). The end effector approach vector can have a direction component that is opposite from one of the surface normals, but otherwise strictly conforms to that surface normal. In some implementations, the grasp direction indicated by the semantic indication (e.g., top, side) can be utilized to select a surface normal utilized in determining the approach vector. For example, if a “top” grasp is to be performed, a surface normal that extends “up” can be selected in lieu of one that extends to the “side” or “down”. In this manner, the grasp direction that defines at least part of the grasp strategy is utilized in determining the approach vector (and resultantly the grasp pose determined based on the approach vector). An end effector grasp pose that conforms to the end effector approach vector can be determined, and one or more control commands provided to actuators of the robot to cause the end effector to traverse to the grasp pose and attempt a grasp of the object subsequent to traversing to the grasp pose. Additionally, in some implementations multiple grasp poses can be considered for each of one or more end effector approach vectors, and one grasp pose selected based on it avoiding collisions (with the object and/or with other objects in the environment), satisfying inverse kinematic constraints, and/or based on other criterion/criteria.” and 0010, “In some implementations, a method is provided that includes applying sensor data as input to at least one trained machine learning model. The sensor data is generated by one or more sensor components of a robot and captures features of an object in an environment of the robot. The method further includes processing the sensor data using the at least one trained machine learning model to generate output defining a spatial region for interacting with the object to grasp the object, and defining a semantic indication associated with the object. The method further includes selecting, based on the semantic indication, a particular grasp strategy of plurality of candidate grasp strategies. The method further includes determining an end effector pose, for interacting with the object to grasp the object, based on the spatial region defined by the output and based on the particular grasp strategy selected based on the semantic indication defined by the output. The method further includes providing, to actuators of the robot, commands that cause an end effector of the robot to traverse to the end effector pose in association with attempting a grasp of the object.” As well as paragraphs 0016 and 0053)
While Wellman teaches a user selecting a grasping approach from a plurality of grasping approaches via an interface, they do not specifically teach using contact points/positions to determine which approach to utilize. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the methods of grasp strategy planning as taught by Wellman with the ability to identify and utilize contact points/regions in order to determine a detailed and effective grasping strategy as taught by Nagarajan. Providing the user with the higher level of detail during their planning process would ensure they select the optimal grasping strategy and the object is grasped appropriately without damage.
Regarding claim 2, where all the limitations of claim 1 are discussed above, Wellman further teaches:
[Claim 2] (Previously Presented) The controller of claim 1, wherein, in the processing of acquiring the state information, the first processor acquires the first image data, (Paragraph 0018, "A camera or other optical imaging device scans the mug, providing information about the mug's orientation, e.g., in this case that the mug is on its side with the open top facing the camera and the handle to the right. The weight, shape, and orientation of the mug are collectively used with any other relevant and/or available information (e.g., about the size and orientation of the box into which the mug is to be placed to fulfill the order) to query a grasping strategy database for a grasping strategy appropriate for this situation." as well as Paragraph 0073, "The attribute detection module 710 can interact with any number and/or type of sensors to determine attributes of an item to be grasped. For example, the attribute detection module 710 can receive information from imaging devices or optical sensors to determine physical characteristics, such as size, shape, position, orientation, and/or surface characteristics (e.g., how porous and/or slippery the item is based on the surface appearance). Any suitable optical technology can be utilized, including, but not limited to, two-dimensional cameras, depth sensors, time of flight sensing (e.g., broadcasting a source of light and determining a time of reflection for each pixel to determine a distance from the sensor for each pixel to determine a three-dimensional array of data points representing a virtual model of the sensed item and environment), structured light sensing (e.g., projecting a known image from a light source, observing the image as distorted by variations in the surface of the detected item, and analyzing the distortions with respect to the projected image to determine positioning of the features that caused the distortion), stereo sensing (e.g., analyzing differences in images collected from multiple cameras arranged at known offsets from one another to generate a point cloud or digital model), active stereo sensing (e.g., projecting a pattern of light to improve precision of detection of features while using stereo sensing), any other optically-based methodology of observing light for generating a digital representation of a physical object, or any combination thereof.") and detects the state information by carrying out image processing of the first image data. (Paragraph 0020, "The sensor package 16 includes one or more sensors (of like or varying type) arranged to detect the item 40 while the item 40 is being maintained by the grasping environment 14. The sensor package 16 communicates detected attributes (as at 46), such as weight, geometric characteristics (e.g., size, position, or orientation), electrical conductivity, magnetic properties, surface characteristics (e.g., how slippery or porous the item is), deformability, and/or structural integrity of the item 40, to the controller 32. The detected attributes may also include a unique identifier of the item 40, such as a barcode- or RFID-encoded serial number. Based on the detected attributes, the controller 32 may access (as at 49) the item database 37, such as to access a record for the inventory item 40. The record can include information about attributes of the item, such as weight, size, shape, or other physical characteristics of the item. Based on the record from the item database 37 and/or the detected attributes from the sensor package 16, the controller 32 may access (as at 48) an item gripping database 36 to access an item grasping strategy stored for that item or items with similar characteristics. The controller 32 can provide instructions to the robotic arm 12 for gripping the item 40 based on the gripping strategy accessed from the gripping database at 36 (e.g., at 52).")
Regarding claim 6, where all the limitations of claim 1 are discussed above, Wellman further teaches:
[Claim 6] (Previously Presented) The controller of claim 1, further comprising the first storage, (Paragraph 0042, "Memory 91 stores processor instructions, inventory requests, reservation information, state information for the various components of inventory system 10 and/or any other appropriate values, parameters, or information utilized by management module 15 during operation. Memory 91 may represent any collection and arrangement of volatile or nonvolatile, local or remote devices suitable for storing data. Examples of memory 91 include, but are not limited to, random access memory (RAM) devices, read only memory (ROM) devices, magnetic storage devices, optical storage devices or any other suitable data storage devices.")
wherein the first storage stores first attribute information including information on characteristics of the workpiece and the given work set to the workpiece, (Paragraph 0072, "The attribute detection module 710 receives information from sensors, such as the sensor package 16 of FIG. 1, and determines attributes of inventory items 40 detected by the sensors. The database query module 715 can access information from the item database 37 and/or the item gripping database 36 of FIG. 1, such as based on the information from the attribute detection module 710 or other modules described herein.") and wherein the first processor causes the robot to operate autonomously (Paragraph 0020, "The controller 32 can provide instructions to the robotic arm 12 for gripping the item 40 based on the gripping strategy accessed from the gripping database at 36 (e.g., at 52).") according to the first attribute information and the selected position. (Paragraph 0021, "The grasping strategy instructed by the controller 32 to the robotic arm 12 (e.g., at 52) can be based on a combination of the human input for grasping strategy communicated at 54, the detected attributes communicated at 46, the record accessed from the item database 37 at 49, and/or the information accessed from the item gripping database 36 at 48.")
Regarding claim 7, where all the limitations of claim 6 are discussed above, Wellman further teaches:
[Claim 7] (Original) The controller of claim 6, wherein the first processor further performs processing including transmitting the first attribute information corresponding to the selected position to the operation terminal to cause the operation terminal to present the first attribute information, (Paragraph 0021, "In some scenarios—which may include when a grasping strategy from the item gripping database 36 is not available or not complete—human input for a grasping strategy can be received by the controller 32, as at 54. To this end, the human operator 34 may provide a suggestion regarding how the inventory item 40 should be grasped by the robotic arm 12, and make the suggestion via the human input device 38. As non-limiting examples, the human input device 38 may include a computer interface by which the human operator 34 can input instructions, may observe a human action for grasping an item to learn and/or determine information for forming a grasping strategy, and/or may provide a virtual environment in which the human operator can perform a simulated grasping of the item to obtain information for learning and/or determining a grasping strategy. The grasping strategy instructed by the controller 32 to the robotic arm 12 (e.g., at 52) can be based on a combination of the human input for grasping strategy communicated at 54, the detected attributes communicated at 46, the record accessed from the item database 37 at 49, and/or the information accessed from the item gripping database 36 at 48. The controller 32 may update the item gripping database 36 (e.g., at 48) and/or the item database 37 based on the human input grasping strategy received at 54 and/or detected attributes communicated at 46, either of which may include feedback about the success of the grasping strategy implemented. Such updating and accessing of the item database 37 and/or the item gripping database 36 can allow robotic arms 12 throughout the inventory system 10 (and throughout other inventory systems having access to the item gripping database 36) to be used to effectively move inventory items 40 between elements within the inventory system so as to increase efficiency and throughput.")
wherein the first processor accepts a change in the first attribute information through the operation terminal, and changes the first attribute information according to the accepted change, (Paragraph 0017, "Embodiments herein are directed to an inventory system having multiple inventory holders and drive units for moving the inventory holders. Specifically, features herein are directed to robotic arms or manipulators and associated components that may facilitate the movement of inventory items and other features of the inventory system among and between elements of the inventory system. The robotic arms or manipulators may be controlled so as to grasp items in a manner particularly suited for grasping of a target item. For example, a target item, or characteristics thereof, may be identified, such as by optical or other sensors, in order to determine a grasping strategy for the item. The grasping strategy may be based at least in part upon a database containing information about the item, characteristics of the item, and/or similar items, such as information indicating grasping strategies that have been successful or unsuccessful for such items in the past. Entries or information in the database may be originated and/or updated based on human input for grasping strategies, determined characteristics of a particular item, and/or machine learning related to grasping attempts of other items sharing characteristics with the particular item. Embodiments herein include aspects directed to generating and/or accessing such databases.) and wherein the first processor causes the robot to operate autonomously (Paragraph 0020, "The controller 32 can provide instructions to the robotic arm 12 for gripping the item 40 based on the gripping strategy accessed from the gripping database at 36 (e.g., at 52).") according to the changed first attribute information and the selected position. (Paragraph 0021, "The grasping strategy instructed by the controller 32 to the robotic arm 12 (e.g., at 52) can be based on a combination of the human input for grasping strategy communicated at 54, the detected attributes communicated at 46, the record accessed from the item database 37 at 49, and/or the information accessed from the item gripping database 36 at 48.")
Regarding claim 8, where all the limitations of claim 6 are discussed above, Wellman further teaches:
[Claim 8] (Previously Presented) The controller of claim 6, wherein the first attribute information includes information on a (Paragraph 0054, "Position sensor 140 represents one or more sensors, detectors, or other components suitable for determining the location of mobile drive unit 20 in any appropriate manner. For example, in particular embodiments, the workspace 70 associated with inventory system 10 includes a number of fiducial marks that mark points on a two-dimensional grid that covers all or a portion of workspace 70. In such embodiments, position sensor 140 may include a camera and suitable image- and/or video-processing components, such as an appropriately-programmed digital signal processor, to allow position sensor 140 to detect fiducial marks within the camera's field of view. Control module 170 may store location information that position sensor 140 updates as position sensor 140 detects fiducial marks. As a result, position sensor 140 may utilize fiducial marks to maintain an accurate indication of the location mobile drive unit 20 and to aid in navigation when moving within workspace 70." as well as Paragraph 0059, "Control module 170 monitors and/or controls operation of drive module 120 and docking actuator 130. Control module 170 may also receive information from sensors such as position sensor 140 and holder sensor 150 and adjust the operation of drive module 120, docking actuator 130, and/or other components of mobile drive unit 20 based on this information. Additionally, in particular embodiments, mobile drive unit 20 may be configured to communicate with a management device of inventory system 10 and control module 170 may receive commands transmitted to mobile drive unit 20 and communicate information back to the management device utilizing appropriate communication components of mobile drive unit 20. Control module 170 may include any appropriate hardware and/or software suitable to provide the described functionality. In particular embodiments, control module 170 includes a general-purpose microprocessor programmed to provide the described functionality. Additionally, control module 170 may include all or portions of docking actuator 130, drive module 120, position sensor 140, and/or holder sensor 150, and/or share components with any of these elements of mobile drive unit 20." Please also see paragraphs 0035-0036 which demonstrate the robotic system monitors the space/position in which it is operating and may move between regions based on the task and the known position.) robot area for the robot, (Paragraph 0020, "The sensor package 16 includes one or more sensors (of like or varying type) arranged to detect the item 40 while the item 40 is being maintained by the grasping environment 14. The sensor package 16 communicates detected attributes (as at 46), such as weight, geometric characteristics (e.g., size, position, or orientation), electrical conductivity, magnetic properties, surface characteristics (e.g., how slippery or porous the item is), deformability, and/or structural integrity of the item 40, to the controller 32. The detected attributes may also include a unique identifier of the item 40, such as a barcode- or RFID-encoded serial number. Based on the detected attributes, the controller 32 may access (as at 49) the item database 37, such as to access a record for the inventory item 40. The record can include information about attributes of the item, such as weight, size, shape, or other physical characteristics of the item. Based on the record from the item database 37 and/or the detected attributes from the sensor package 16, the controller 32 may access (as at 48) an item gripping database 36 to access an item grasping strategy stored for that item or items with similar characteristics. The controller 32 can provide instructions to the robotic arm 12 for gripping the item 40 based on the gripping strategy accessed from the gripping database at 36 (e.g., at 52).") and
wherein, in the process of determining the candidates of the work position, (Paragraph 0077, " In some embodiments, the database query module 715 may access grasping strategy information based on detected physical attributes of the item (e.g., a size, shape, position, orientation, weight, or other attributes of the item regardless of a SKU of the item). The database query module 715 may receive information about grasping strategies based on the recognized item and/or characteristics.") the first processor determines candidates of a position at which the robot applies an action to the workpiece in the robot area (Paragraph 0054, "Position sensor 140 represents one or more sensors, detectors, or other components suitable for determining the location of mobile drive unit 20 in any appropriate manner. For example, in particular embodiments, the workspace 70 associated with inventory system 10 includes a number of fiducial marks that mark points on a two-dimensional grid that covers all or a portion of workspace 70. In such embodiments, position sensor 140 may include a camera and suitable image- and/or video-processing components, such as an appropriately-programmed digital signal processor, to allow position sensor 140 to detect fiducial marks within the camera's field of view. Control module 170 may store location information that position sensor 140 updates as position sensor 140 detects fiducial marks. As a result, position sensor 140 may utilize fiducial marks to maintain an accurate indication of the location mobile drive unit 20 and to aid in navigation when moving within workspace 70." as well as Paragraph 0059, "Control module 170 monitors and/or controls operation of drive module 120 and docking actuator 130. Control module 170 may also receive information from sensors such as position sensor 140 and holder sensor 150 and adjust the operation of drive module 120, docking actuator 130, and/or other components of mobile drive unit 20 based on this information. Additionally, in particular embodiments, mobile drive unit 20 may be configured to communicate with a management device of inventory system 10 and control module 170 may receive commands transmitted to mobile drive unit 20 and communicate information back to the management device utilizing appropriate communication components of mobile drive unit 20. Control module 170 may include any appropriate hardware and/or software suitable to provide the described functionality. In particular embodiments, control module 170 includes a general-purpose microprocessor programmed to provide the described functionality. Additionally, control module 170 may include all or portions of docking actuator 130, drive module 120, position sensor 140, and/or holder sensor 150, and/or share components with any of these elements of mobile drive unit 20." Please also see paragraphs 0035-0036 which demonstrate the robotic system monitors the space/position in which it is operating and may move between regions based on the task and the known position.) based on the first attribute information, as the candidates of the work position. (Paragraph 0078-0079, "A grasping strategy can include any information regarding the manner in which a robotic arm 12 is to attempt to grasp a particular item 40 or group of items. For example, a grasping strategy may include an indication of how the robotic arm is to approach the item to be grasped, an indication of one or more end effectors to be utilized by the robotic arm, and/or an indication of a level of intensity (e.g., amount of force, pressure, voltage, current, etc.) with which the robotic arm is to operate the end effector(s). In some embodiments, the grasping strategy may also include a number of items to be simultaneously grasped.
The approach identified in a grasping strategy may include a direction from which the robotic arm is to approach the item (e.g., from above, from a side, from an angle) and/or a sequence of motions by which the robotic arm is to perform a particular grasping operation, which may include reaching the target item, grasping the target item, moving the target item to a target location, and/or releasing the target item in the target location.")
Regarding claim 11, where all the limitations of claim 9 are discussed above, Wellman further teaches:
[Claim 11] (Previously Presented) The controller of claim 1, wherein … and
wherein, in the processing of determining the candidates of the work position, (Paragraph 0077, " In some embodiments, the database query module 715 may access grasping strategy information based on detected physical attributes of the item (e.g., a size, shape, position, orientation, weight, or other attributes of the item regardless of a SKU of the item). The database query module 715 may receive information about grasping strategies based on the recognized item and/or characteristics.") the first processor determines candidates of the position of the workpiece with respect to the peripheral environment based on the second attribute information, as the candidates of the work position. (Paragraph 0093, "FIG. 10 illustrates a box 1022 (such as may correspond to a box 922 of FIG. 9) having corner pieces 1030 having fiducial markers 1032. The corner pieces 1030 can retain flaps of the box open for loading by the robotic arm 912. For example, the corner pieces 1030 may be T- or L-shaped to retain perpendicular flaps in position relative to one another. Corner pieces may be constructed of any material. The corner pieces 1030 can include any type of fiducial marker, including, but not limited to barcodes, colors, patterns, alphanumeric text or codes, symbols, other indicia. Such fiducial markers 1032 may provide additional points of reference for sensors used in conjunction with the robotic arm 912 to identify particular boxes in accordance with instructions for moving the robotic arm 912.")
Wellman does not specifically teach determining the position of a workpiece with respect to the environment. However, Nagarajan, in the same field of endeavor of robotic control, teaches:
… the second attribute information includes information on the position of the workpiece with respect to the peripheral environment, (Paragraph 0050, "The sensor data engine 112 of grasp system 110 receives instance(s) of sensor data, from sensor(s) of a robot, and provides the instance(s) to one or more other components of the grasp system 110, for use in selecting a grasp strategy and/or in determining how to perform a grasp attempt in accordance with a selected grasp strategy. In some implementations, the sensor data includes vision data, such as 2D vision data and/or 3D vision data. 2D vision data can include 2D images generated based on image data captured by camera(s) associated with a robot, and each of the 2D images can include a plurality of pixels and values defined for each of one or more channels of each of the pixels. For example, a 2D image can include a plurality of pixels each having red, green, and blue channels and may define, for each of the channels for each of the pixels, a value (e.g., from 0 to 255). 3D vision data, as used herein, can include so-called 2.5D images that include a depth channel (in addition to one or more color channels) and/or can include 3D point cloud data that includes X, Y, and Z position values of detected surfaces (optionally with “intensity” values).") …
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic control system as taught by Wellman with the ability to track the position of the workpiece with respect to the environment as taught by Nagarajan. This would ensure a higher level of accuracy within the system as well as increasing the chance of a successful grasping action.
Regarding claim 12, where all the limitations of claim 1 are discussed above, Wellman further teaches:
[Claim 12] (Previously Presented) A robot system, comprising:
the controller of claim 1; and
the robot controlled by the controller. (Paragraph 0019, "The system 10 may include the robotic arm 12, a grasping environment 14, an inventory item 40, a sensor package 16, a controller 32, an item gripping database 36, an item database 37, a human input device 38, and a human operator 34.")
Regarding claim 13, where all the limitations of claim 12 are discussed above, Wellman further teaches:
[Claim 13] (Previously Presented) The robot system of claim 12, further comprising:
robot groups including combinations of the controller and the robot controlled by the controller; (Paragraph 0090, " FIG. 9 illustrates an example of a station having a robotic arm 912. The station 900 can represent a station 50 described elsewhere herein. The station 900 can include a picking area 902, a grasping area 904, a release area 906, and a packing area 908. The mobile drive unit 910 (e.g., mobile drive unit 20 described elsewhere herein) can bring an inventory holder 914 (e.g., inventory holder 30 described elsewhere herein), to the picking area 902. An operator 916 in the picking area 902 can remove an inventory item 940 from the inventory holder 914 and transfer the inventory item 940 to the grasping area 904 (e.g., to a tray 918). The tray 918 may be one of a plurality of trays in the grasping area 904. In some embodiments, the plurality of trays 918 are moveable so as to bring empty trays into position to be filled by the operator 916 and trays bearing items 40 into a position to be accessed by the robotic arm 912. Multiple and/or movable trays may allow the robotic arm 912 and the operator and 916 to work at different rates and may reduce wait times in a workflow at the station 900. Multiple trays may also facilitate simultaneous, independent operation of multiple robotic arms 912 and/or picking operators 916. In some embodiments, different trays may be provided having differing characteristics that facilitate grasping different types of items (e.g., one tray type with soft lining for grasping fruit and another tray type with features more suited to grasping books).") and
a mediator data-communicably connected to the communication network, the mediator mediating a connection between an operation terminal that is selected by the operator from a plurality of operation terminals, (Paragraph 0102, "The various embodiments further can be implemented in a wide variety of operating environments, which in some cases can include one or more user computers, computing devices or processing devices which can be used to operate any of a number of applications. User or client devices can include any of a number of general purpose personal computers, such as desktop or laptop computers running a standard operating system, as well as cellular, wireless and handheld devices running mobile software and capable of supporting a number of networking and messaging protocols. Such a system also can include a number of workstations running any of a variety of commercially-available operating systems and other known applications for purposes such as development and database management. These devices also can include other electronic devices, such as dummy terminals, thin-clients, gaming systems and other devices capable of communicating via a network.") and the controller of the robot group that is selected, by the select operation terminal, from the robot groups. (Paragraph 0097, "FIG. 13 illustrates aspects of an example environment 1300 for implementing aspects in accordance with various embodiments. As will be appreciated, although a Web-based environment is used for purposes of explanation, different environments may be used, as appropriate, to implement various embodiments. The environment includes an electronic client device 1302, which can include any appropriate device operable to send and receive requests, messages, or information over an appropriate network 1304 and convey information back to a user of the device. Examples of such client devices include personal computers, cell phones, handheld messaging devices, laptop computers, set-top boxes, personal data assistants, electronic book readers, and the like. The network can include any appropriate network, including an intranet, the Internet, a cellular network, a local area network or any other such network or combination thereof. Components used for such a system can depend at least in part upon the type of network and/or environment selected. Protocols and components for communicating via such a network are well known and will not be discussed herein in detail. Communication over the network can be enabled by wired or wireless connections and combinations thereof. In this example, the network includes the Internet, as the environment includes a Web server 1306 for receiving requests and serving content in response thereto, although for other networks an alternative device serving a similar purpose could be used as would be apparent to one of ordinary skill in the art.")
Regarding claim 14, where all the limitations of claim 1 are discussed above, Wellman further teaches:
[Claim 14] (Previously Presented) A learning device, comprising:
a second processor; (Paragraph 0041, "Processor 90 is operable to execute instructions associated with the functionality provided by management module 15. Processor 90 may comprise one or more general purpose computers, dedicated microprocessors, or other processing devices capable of communicating electronic information. Examples of processor 90 include one or more application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs) and any other suitable specific or general purpose processors.") and
a second storage, (Paragraph 0042, "Memory 91 stores processor instructions, inventory requests, reservation information, state information for the various components of inventory system 10 and/or any other appropriate values, parameters, or information utilized by management module 15 during operation. Memory 91 may represent any collection and arrangement of volatile or nonvolatile, local or remote devices suitable for storing data. Examples of memory 91 include, but are not limited to, random access memory (RAM) devices, read only memory (ROM) devices, magnetic storage devices, optical storage devices or any other suitable data storage devices.")
wherein the second storage stores the state information and the selected position corresponding to the state information that are acquired by one or more controllers of claim 1, …
wherein the selected position corresponding to the state information is the selected position selected from the candidates of the work position based on the state information, and (Paragraph 0020, "The detected attributes may also include a unique identifier of the item 40, such as a barcode- or RFID-encoded serial number. Based on the detected attributes, the controller 32 may access (as at 49) the item database 37, such as to access a record for the inventory item 40. The record can include information about attributes of the item, such as weight, size, shape, or other physical characteristics of the item. Based on the record from the item database 37 and/or the detected attributes from the sensor package 16, the controller 32 may access (as at 48) an item gripping database 36 to access an item grasping strategy stored for that item or items with similar characteristics. The controller 32 can provide instructions to the robotic arm 12 for gripping the item 40 based on the gripping strategy accessed from the gripping database at 36 (e.g., at 52).")
… and
accepting input state information that is the state information including the state of the workpiece as input data, (Paragraph 0072, "The database query module 715 can access information from the item database 37 and/or the item gripping database 36 of FIG. 1, such as based on the information from the attribute detection module 710 or other modules described herein. The human-based grasping strategy module 720 handles requests and receipt of any human input for grasping strategy, such as at 54 in FIG. 1. The constraints module 725 can provide information about constraints to available grasping strategies from various sources, such as information about where an item grasped by a robotic arm 12 is to be placed. The grasping strategy selection module 730 may select and/or determine a grasping strategy to be provided to a robotic arm 12 for a particular grasping action, such as based on information from other modules described herein. The grasping strategy instruction module 735 can provide the instructions embodying the grasping strategy selected by the grasping strategy selection module 730 to the robotic arm 12 for a particular grasping action. The grasping strategy evaluation module 740 can receive information about a grasping action performed in response to instructions provided by the grasping strategy instruction module 735 and evaluate a success of the grasping strategy for the item or attributes of the item involved in the particular grasping action. The database update module 745 may update the item database 37 and/or the item gripping database 36, such as utilizing information from other modules described herein.") and outputting information on an optimal work position among the candidates of the work position of the workpiece corresponding to the input state information as output data. (Paragraph 0072, "The database query module 715 can access information from the item database 37 and/or the item gripping database 36 of FIG. 1, such as based on the information from the attribute detection module 710 or other modules described herein. The human-based grasping strategy module 720 handles requests and receipt of any human input for grasping strategy, such as at 54 in FIG. 1. The constraints module 725 can provide information about constraints to available grasping strategies from various sources, such as information about where an item grasped by a robotic arm 12 is to be placed. The grasping strategy selection module 730 may select and/or determine a grasping strategy to be provided to a robotic arm 12 for a particular grasping action, such as based on information from other modules described herein. The grasping strategy instruction module 735 can provide the instructions embodying the grasping strategy selected by the grasping strategy selection module 730 to the robotic arm 12 for a particular grasping action. The grasping strategy evaluation module 740 can receive information about a grasping action performed in response to instructions provided by the grasping strategy instruction module 735 and evaluate a success of the grasping strategy for the item or attributes of the item involved in the particular grasping action. The database update module 745 may update the item database 37 and/or the item gripping database 36, such as utilizing information from other modules described herein.")
Wellman does not specifically teach utilizing these instances of control as training data for training a machine learning model to perform the selection of a grasping strategy. However, Nagarajan, in the same field of endeavor of robotic control, teaches:
… as learning data, … wherein the second processor performs processing including:
learning, while using the state information on the learning data as learning input data, and the information on the selected position of the learning data corresponding to the state information as teaching data; (Paragraph 0092, "At optional block 566, the system trains additional machine learning model(s) based on stored instances of an end effector pose and vision data and/or a group of 3D points, including the instance stored at optional block 564, and additional instances stored at block 564 in additional iterations of method 500. For example, the stored instances can be training instances that each include corresponding vision data and/or corresponding 3D points as training instance input, and that include a corresponding end effector pose (e.g., a grasp pose) as training instance output. In this manner, an additional machine learning model can be trained that predicts an end effector pose (e.g., a grasp pose) directly based on vision data (e.g., a 2D image) and/or a group of 3D points.") …
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic control system as taught by Wellman with the ability to utilize the control data and information for training a machine learning model to perform the selection of a grasping strategy as taught by Nagarajan. This would allow for a system to operate at a higher accuracy level with less human intervention needed.
Claim(s) 4-5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wellman in view of Nagarajan and in further view of Dubrovsky et al. (US 20210204787 A1), hereinafter Dubrovsky.
Regarding claim 4, where all the limitations of claim 1 are discussed above, Wellman further teaches:
[Claim 4] (Previously Presented) The controller of claim 1,wherein the first processor further performs processing including:
when the information on the selected position is received from the operation terminal, (Paragraph 0089, "FIG. 8 illustrates an example user interface for obtaining human input for a grasping strategy. The user interface includes a screen showing various selection options that a human user can select to provide input about a grasping strategy for a particular item, e.g., a coffee cup 802. In a first row at 804, the user has options for different end effectors, such as a mechanical pincher 806, soft robotics 808, vacuum 810, or electro-adhesion 812. The user can select one or more of the end effectors as illustrated by the selection boxes 814. In a second row at 820, the user can select an approach for picking up the desired item 802. For example, the presented options may include an angled or lateral approach 822, a top-down approach 824, an underneath approach 826, or a multiple arm approach 828 (in which multiple arms are used for grasping and moving the item 828). The user can select an approach (or more than one approach in the case of multiple robotic arms) as illustrated by the selection box 818. At 830, a visualization of the selected grasping strategy is presented in which the coffee cup is to be grasped at an angle in a lateral approach by a combination of a mechanical pincher and vacuum based on the selections made at 814 and 818.") detecting a scheduled operation of the robot according to the selected position; and (Paragraph 0034, "With respect to mobile drive units 20 specifically, management module 15 may, in particular embodiments, communicate task assignments 18 to selected mobile drive units 20 that identify one or more destinations for the selected mobile drive units 20. Management module 15 may select a mobile drive unit 20 to assign the relevant task based on the location or state of the selected mobile drive unit 20, an indication that the selected mobile drive unit 20 has completed a previously-assigned task, a predetermined schedule, and/or any other suitable consideration. These destinations may be associated with an inventory request the management module 15 is executing or a management objective the management module 15 is attempting to fulfill. For example, the task assignment may define the location of an inventory holder 30 to be retrieved, an inventory station 50 to be visited, a storage location where the mobile drive unit 20 should park until receiving another task, or a location associated with any other task appropriate based on the configuration, characteristics, and/or state of inventory system 10, as a whole, or individual components of inventory system 10. For example, in particular embodiments, such decisions may be based on the popularity of particular inventory items, the staffing of a particular inventory station 50, the tasks currently assigned to a particular mobile drive unit 20, and/or any other appropriate considerations.")
…
Wellman does not specifically teach displaying the operating schedule information on a terminal. However, Dubrovsky, in the same field of endeavor of robotic control, teaches:
… transmitting information on the scheduled operation to the operation terminal to cause the operation terminal to present the information. (Paragraph 0052, "The communication device shown in FIG. 2B also includes a display 46. This display 46 can for example be a liquid crystal display (LCD), allowing the user to see the information being inputted into the communication device 40, showing which configuration information, scheduling information or control information a robotic device is being sent, or for showing information sent from the robotic device to the communication device 40.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic control system as taught by Wellman with the ability to display scheduling information to a user via an interface as taught by Dubrovsky. This would allow the user to quickly make adjustments to the operating schedule. Thereby increasing the efficiency of the system.
Regarding claim 5, where all the limitations of claim 4 are discussed above, Wellman further teaches:
[Claim 5] (Original) The controller of claim 4, wherein … and causes the robot to operate autonomously according to the changed scheduled operation. (Paragraph 0099, "The data store 1310 can include several separate data tables, databases or other data storage mechanisms and media for storing data relating to a particular aspect. For example, the data store illustrated includes mechanisms for storing information which can be used by modules described herein, such as resource scheduling information 1312, route planning information 1314, segment reservation information 1316, and/or inventory information 1318. Furthermore, the management module 15 of FIG. 2 and/or FIG. 7, and all modules included therein, can be stored in data store 1310. It should be understood that there can be many other aspects that may need to be stored in the data store, such as for page image information and to access right information, which can be stored in any of the above listed mechanisms as appropriate or in additional mechanisms in the data store 1310. The data store 1310 is operable, through logic associated therewith, to receive instructions from the application server 1308 and obtain, update or otherwise process data in response thereto.")
Wellman does not specifically teach displaying the operating schedule information on a terminal. However, Dubrovsky, in the same field of endeavor of robotic control, teaches:
… the first processor accepts a change in the scheduled operation through the operation terminal, (Paragraph 0042, "In one embodiment of the invention, scheduling information 16 can be input into the communication device 12 through a user interface of the device 12. This information can then be communicated to a stationary 24 or mobile 26 robotic device through a wireless connection between the communication device 12 and the robotic device. The robotic device stores this information and runs according to the stored scheduling information 16 without the need for any other input from a user, controller or communication device 12. Changes in the scheduling information 16 stored in the robotic device can be made by simply inputting new scheduling information 16 into the communication device 12 and communicating it to the robotic device. In an alternative embodiment, a further step, such as but not limited to clearing the stored scheduling information 16 from the robotic device's memory or inputting a code (either into the communication device 12 or directly into the robotic device), may be required before new scheduling information 16 can be loaded into the robotic device.") …
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic control system as taught by Wellman with the ability to display scheduling information to a user via an interface as taught by Dubrovsky. This would allow the user to quickly make adjustments to the operating schedule. Thereby increasing the efficiency of the system.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wellman in view of Nagarajan and in further view of Aloisio et al. (US 20210023710 A1), hereinafter Aloisio.
Regarding claim 10, where all the limitations of claim 9 are discussed above, Wellman further teaches:
[Claim 10] (Previously Presented) The controller of claim 1, wherein the first processor further performs processing including …
wherein the first processor accepts a change in the second attribute information (Paragraph 0017, "Embodiments herein are directed to an inventory system having multiple inventory holders and drive units for moving the inventory holders. Specifically, features herein are directed to robotic arms or manipulators and associated components that may facilitate the movement of inventory items and other features of the inventory system among and between elements of the inventory system. The robotic arms or manipulators may be controlled so as to grasp items in a manner particularly suited for grasping of a target item. For example, a target item, or characteristics thereof, may be identified, such as by optical or other sensors, in order to determine a grasping strategy for the item. The grasping strategy may be based at least in part upon a database containing information about the item, characteristics of the item, and/or similar items, such as information indicating grasping strategies that have been successful or unsuccessful for such items in the past. Entries or information in the database may be originated and/or updated based on human input for grasping strategies, determined characteristics of a particular item, and/or machine learning related to grasping attempts of other items sharing characteristics with the particular item. Embodiments herein include aspects directed to generating and/or accessing such databases.) … and
wherein the first processor causes the robot to operate autonomously (Paragraph 0020, "The controller 32 can provide instructions to the robotic arm 12 for gripping the item 40 based on the gripping strategy accessed from the gripping database at 36 (e.g., at 52).") according to the changed second attribute information and the selected position. (Paragraph 0023, "Management module 15 assigns tasks to appropriate components of inventory system 10 and coordinates operation of the various components in completing the tasks. These tasks may relate not only to the movement and processing of inventory items, but also to the management and maintenance of the components of inventory system 10. For example, management module 15 may assign portions of workspace 70 as parking spaces for mobile drive units 20, the scheduled recharge or replacement of mobile drive unit batteries, the storage of empty inventory holders 30, or any other operations associated with the functionality supported by inventory system 10 and its various components. Management module 15 may select components of inventory system 10 to perform these tasks and communicate appropriate commands and/or data to the selected components to facilitate completion of these operations. Although shown in FIG. 2 as a single, discrete component, management module 15 may represent multiple components and may represent or include portions of mobile drive units 20 or other elements of inventory system 10. As a result, any or all of the interaction between a particular mobile drive unit 20 and management module 15 that is described below may, in particular embodiments, represent peer-to-peer communication between that mobile drive unit 20 and one or more other mobile drive units 20. The contents and operation of an example embodiment of management module 15 are discussed further below with respect to FIG. 3.")
Wellman does not specifically teach displaying attribute information on an operating terminal. However, Aloisio, in the same field of endeavor of robotic control, teaches:
… transmitting the second attribute information corresponding to the selected position to the operation terminal to cause the operation terminal to present the second attribute information, … through the operation terminal, and changes the second attribute information according to the accepted change, (Paragraph 0073, "In some embodiments, robotic bin picking process 10 may include determining a pose of the workpiece relative to the robot, based upon, at least in part, the scanning. As is shown in FIG. 19, the pose of the workpiece relative to the robot or end effector may be displayed to the user at the graphical user interface. This may allow for both a visual verification of the workpiece and the manner in which it is associated with the end effector as well as the X, Y, and Z offsets from the TCP as shown in FIGS. 19-20. The X, Y, and Z coordinate system may be shown at the graphical user interface and may be adjusted based upon user preferences. The process may allow a user to adjust the pose of the workpiece relative to the robot by entering and/or editing the coordinates. The adjusted pose of the workpiece may be displayed at the graphical user interface for verification.") …
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic control system as taught by Wellman with the display capabilities and methods as taught by Aloisio. This would provide a user with a better understanding of the operating environment as a whole before inputting a selection for the grasping strategy thereby increasing the effectiveness of the system.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wellman in view of Nagarajan and in further view of Murphy et al. (US 20210178576 A1), hereinafter Murphy.
Regarding claim 15, where all the limitations of claim 1 are discussed above, Wellman further teaches:
[Claim 15] (Previously Presented) A learning device, comprising:
a second processor; (Paragraph 0041, "Processor 90 is operable to execute instructions associated with the functionality provided by management module 15. Processor 90 may comprise one or more general purpose computers, dedicated microprocessors, or other processing devices capable of communicating electronic information. Examples of processor 90 include one or more application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs) and any other suitable specific or general purpose processors.") and
a second storage, (Paragraph 0042, "Memory 91 stores processor instructions, inventory requests, reservation information, state information for the various components of inventory system 10 and/or any other appropriate values, parameters, or information utilized by management module 15 during operation. Memory 91 may represent any collection and arrangement of volatile or nonvolatile, local or remote devices suitable for storing data. Examples of memory 91 include, but are not limited to, random access memory (RAM) devices, read only memory (ROM) devices, magnetic storage devices, optical storage devices or any other suitable data storage devices.")
wherein the second storage stores the state information and the selected position corresponding to the state information that are acquired by one or more controllers of claim 1, …
wherein the selected position corresponding to the state information is the selected position selected from the candidates of the work position based on the state information, and (Paragraph 0020, "The detected attributes may also include a unique identifier of the item 40, such as a barcode- or RFID-encoded serial number. Based on the detected attributes, the controller 32 may access (as at 49) the item database 37, such as to access a record for the inventory item 40. The record can include information about attributes of the item, such as weight, size, shape, or other physical characteristics of the item. Based on the record from the item database 37 and/or the detected attributes from the sensor package 16, the controller 32 may access (as at 48) an item gripping database 36 to access an item grasping strategy stored for that item or items with similar characteristics. The controller 32 can provide instructions to the robotic arm 12 for gripping the item 40 based on the gripping strategy accessed from the gripping database at 36 (e.g., at 52).")
…
accepting input state information that is the state information including the state of the workpiece as input data, (Paragraph 0072, "The database query module 715 can access information from the item database 37 and/or the item gripping database 36 of FIG. 1, such as based on the information from the attribute detection module 710 or other modules described herein. The human-based grasping strategy module 720 handles requests and receipt of any human input for grasping strategy, such as at 54 in FIG. 1. The constraints module 725 can provide information about constraints to available grasping strategies from various sources, such as information about where an item grasped by a robotic arm 12 is to be placed. The grasping strategy selection module 730 may select and/or determine a grasping strategy to be provided to a robotic arm 12 for a particular grasping action, such as based on information from other modules described herein. The grasping strategy instruction module 735 can provide the instructions embodying the grasping strategy selected by the grasping strategy selection module 730 to the robotic arm 12 for a particular grasping action. The grasping strategy evaluation module 740 can receive information about a grasping action performed in response to instructions provided by the grasping strategy instruction module 735 and evaluate a success of the grasping strategy for the item or attributes of the item involved in the particular grasping action. The database update module 745 may update the item database 37 and/or the item gripping database 36, such as utilizing information from other modules described herein.") and …
determining an optimal work position from arbitrary positions of the workpiece, and outputting information on the optimal work position. (Paragraph 0072, "The database query module 715 can access information from the item database 37 and/or the item gripping database 36 of FIG. 1, such as based on the information from the attribute detection module 710 or other modules described herein. The human-based grasping strategy module 720 handles requests and receipt of any human input for grasping strategy, such as at 54 in FIG. 1. The constraints module 725 can provide information about constraints to available grasping strategies from various sources, such as information about where an item grasped by a robotic arm 12 is to be placed. The grasping strategy selection module 730 may select and/or determine a grasping strategy to be provided to a robotic arm 12 for a particular grasping action, such as based on information from other modules described herein. The grasping strategy instruction module 735 can provide the instructions embodying the grasping strategy selected by the grasping strategy selection module 730 to the robotic arm 12 for a particular grasping action. The grasping strategy evaluation module 740 can receive information about a grasping action performed in response to instructions provided by the grasping strategy instruction module 735 and evaluate a success of the grasping strategy for the item or attributes of the item involved in the particular grasping action. The database update module 745 may update the item database 37 and/or the item gripping database 36, such as utilizing information from other modules described herein.")
Wellman does not specifically teach utilizing these instances of control as training data for training a machine learning model to perform the selection of a grasping strategy or determining a confidence score for the output of the model. However, Nagarajan, in the same field of endeavor of robotic control, teaches:
… as learning data, … wherein the second processor performs processing including:
learning, while using the state information on the learning data as learning input data, and the information on the selected position of the learning data corresponding to the state information as teaching data; (Paragraph 0092, "At optional block 566, the system trains additional machine learning model(s) based on stored instances of an end effector pose and vision data and/or a group of 3D points, including the instance stored at optional block 564, and additional instances stored at block 564 in additional iterations of method 500. For example, the stored instances can be training instances that each include corresponding vision data and/or corresponding 3D points as training instance input, and that include a corresponding end effector pose (e.g., a grasp pose) as training instance output. In this manner, an additional machine learning model can be trained that predicts an end effector pose (e.g., a grasp pose) directly based on vision data (e.g., a 2D image) and/or a group of 3D points.") … outputting a … of the candidate of the work position of the workpiece corresponding to the input state information as output data; and
based on the … , (Paragraph 0071, "An error module 142A, of the training engine 140A, generates an error 143A1 based on comparing the predicted regions with predicted semantic indications 140A1 to the region(s) with semantic indication(s) indicated by the training instance output 165A1B. The error module 142A further updates the grasp regions and semantic indication model 160A1 based on the determined error 143A1. For example, in non-batch techniques, a gradient can be determined based on only the error 143A1, and backpropagated over the model 160A1 to update various weights of the model 160A1. Also, for example, in batch techniques, the error 143A1 can be combined with additional errors determined based on additional training instances, and utilized to update various weights of the model 160A1. Although only the training instance 165A1 is illustrated in FIG. 3, it is understood that many (e.g., thousands) of additional training instances will be utilized during training. Through training, the grasp regions and semantic indications model 160A1 is trained to enable prediction, using the model 160A1 and based on a 2D image, of grasp regions of the 2D image and corresponding semantic indications.") …
However, Murphy, in the same field of endeavor of robotic control, teaches:
… reliability score … reliability score … (Paragraph 0033, "When a machine learning model is used to identify objects in visual data, along with predicted object identifications, the model may output a confidence score which is higher the more certain the model is that a predicted object identity is correct. A robot may fail to identify an object, or return an object identification with a low confidence score, based on the visual data it has collected from visual sensors. The robot could then alert a human operator of the failed identification. The human operator could then review the alert and failed identification and determine whether or not to have the robot start a further learning process to collect additional data. Having the human operator decide to start the process rather than having the robot start the process automatically may save time that could unnecessarily be spent by the robot.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic control system as taught by Wellman with the ability to utilize the control data and information for training a machine learning model to perform the selection of a grasping strategy and monitoring the error rate of the model as taught by Nagarajan and using a confidence value to determine how to operate as taught by Murphy. Incorporating a confidence score into the output of the machine learning model would allow the system to request human assistance only when necessary and increase the efficiency of the decision making process. This would allow for a system to operate at a higher accuracy level with less human intervention needed.
Conclusion
The Examiner has cited particular paragraphs or columns and line numbers in the referencesapplied to the claims above for the convenience of the Applicant. Although the specified citations arerepresentative of the teachings of the art and are applied to specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested of the Applicant in preparing responses, to fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the Examiner. See MPEP 2141.02 [R-07.2015] VI. A prior art reference must be considered in its entirety, i.e., as a whole, including portions that would lead away from the claimed Invention. W.L. Gore & Associates, Inc. v. Garlock, Inc., 721 F.2d 1540, 220 USPQ 303 (Fed. Cir. 1983), cert, denied, 469 U.S. 851 (1984). See also MPEP §2123.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
/H.J.K./Examiner, Art Unit 3657
/ADAM R MOTT/Supervisory Patent Examiner, Art Unit 3657