DETAILED ACTION
Status of Claims
Claims 1, 5-11 and 13-20 are currently pending and have been examined in this application. This Non-final communication is in response to the amendment submitted on 1/8/2026.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments and Amendments
Applicant’s arguments, filed on 1/8/2026, with respect to the rejection of Claims 1, 3-11 and 13-20 under 35 USC 103 have been fully considered but they are moot in view of the new grounds of rejection provided below, which was necessitated based on Applicant’s amendments to the claims, which changed the scope of the claims. Examiner notes wherein Applicant’s arguments are directed towards the newly amended claim limitation(s), which are addressed by the newly found prior art, as indicated below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 5-7, 10, 11, 13, 14, 16, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Wellman (US 20170021499 A1) as modified by Asatani (US 20250065495 A1) in view of Sun (US 9321176 B1)
Claim 1:
Wellman teaches the following limitations:
A device comprising processing circuitry coupled to storage, the processing circuitry configured to: receive sensor data of an observed grasping of an object by a hand
(Wellman – [0021] … As non-limiting examples, the human input device 38 may include a computer interface by which the human operator 34 can input instructions, may observe a human action for grasping an item to learn and/or determine information for forming a grasping strategy, and/or may provide a virtual environment in which the human operator can perform a simulated grasping of the item to obtain information for learning and/or determining a grasping strategy. …)
determine, based on the sensor data of the observed grasping of the object,
(Wellman – [0082] … For example, sensors (such as in the sensor package 16 of FIG. 1) can be configured to detect information while a human is grasping a target item. In an illustrative example, a human wearing a glove or other garment with fiducial markers and/or sensors grasps a particular item. The motion of the human's approach is tracked based on the fiducial markers (e.g., as detected by optical sensors), and other characteristics are tracked based on the sensors (e.g., pressure sensors, tactile sensors, or other sensors may provide data regarding details about an amount and location of pressure that the human exerts on the item throughout the process, shear forces or other indicia translatable into an amount that the object slips in the human's hand, or other information that may be useful in determining a grasping strategy) …)
Examiner’s Note:
Database Query Module corresponds to Data Label
Wellman does not explicitly teach the following limitations, however Asatani teaches:
at least one manipulation point in relation to the object determine, based on the sensor data, a corresponding force applied to the object at the manipulation point
(Asatani - {0022] … The holding mode determination device 10 determines a mode to be used by the robot 2 to hold the holding target object 80, and outputs the mode to the robot control device 110. The mode in which the robot 2 holds the holding target object 80 is also simply referred to as a holding mode. The holding mode includes a position at which the robot 2 touches the holding target object 80, a force to be applied to the holding target object 80 by the robot 2, or the like. ; [0045] … The controller 12 acquires an image captured by photographing the holding target object 80 from the camera 4, inputs the image to the trained model, and acquires an inference result of a holding mode for the holding target object 80 from the trained model. …)
wherein the respective data label indicates the at least one manipulation point and the corresponding force for the object and
(Asatani - [0079] In the above embodiment, the configuration in which the holding mode determination device 10 determines the holding position 82 as a holding mode has been described. The holding mode determination device 10 is capable of determining not only the holding position(s) 82 but also another mode as the holding mode. ; [0080] For example, the holding mode determination device 10 may determine, as the holding mode, a force to be applied by the robot 2 to hold the holding target object 80. In this case, the holding mode inference model 50 infers and outputs a force to be applied to hold the holding target object 80.)
control a robot to grasp an item at a grasping point based on the at least one manipulation point and the corresponding force in the respective data label
(Asatani - [0026] … In the configuration illustrated in FIG. 1, the robot 2 causes the end effector 2B to hold the holding target object 80 at the work start point 6, and moves the end effector 2B to the work target point 7. The robot 2 causes the end effector 2B to release the holding target object 80 at the work target point 7. In this way, the robot 2 is capable of moving the holding target object 80 from the work start point 6 to the work target point 7.)
Examiner’s Note:
Holding Mode corresponds to Data Label
Examiner’s Note:
Based on the instant specification in at least paragraph [0017], the term “Manipulation Point” has been defined as an ideal or reference grasping point stored in memory as opposed to a “Grasping Point” which in the examiner’s opinion refers to the actual point at which an object is grasped during a task.
Wellman in combination with Asatani does not explicitly teach the following limitations, however Sun teaches:
for each of the plurality of tasks;
(Sun – [column 5; lines 41-46] Many grasp classifications are defined based on Cutkosky's grasp taxonomy, which classifies user performed grasps into 16 classes that vary by task requirement and dexterities. To recognize the demonstrated grasp to be a type from the Cutkosky's taxonomy, pattern classification techniques can be applied.)
for each of the plurality of tasks
(Sun – [column 5; lines 41-46] Many grasp classifications are defined based on Cutkosky's grasp taxonomy, which classifies user performed grasps into 16 classes that vary by task requirement and dexterities. To recognize the demonstrated grasp to be a type from the Cutkosky's taxonomy, pattern classification techniques can be applied.)
create, for each of the plurality of tasks, a respective data label for the object for a corresponding task of the plurality of tasks,
(Sun - [column 14; lines 50-54] In some embodiments, a shape-matching algorithm can be combined with the presented approach for an unknown object. Grasp planning can be performed by matching a similar shape in the database, which collects a set of objects with a labeled thumb contact point and task-specific grasp type.)
that is specific to its corresponding task;
(Sun - [column 14; lines 29-33] … Grasp types import important task-oriented experience to the robot because a human usually applies different grasp types according to the task properties. Grasp types also provide a good referenced hand posture to the robot. …)
during execution of a planned task from among the plurality of tasks
(Sun - [column 14; lines 50-54] In some embodiments, a shape-matching algorithm can be combined with the presented approach for an unknown object. Grasp planning can be performed by matching a similar shape in the database, which collects a set of objects with a labeled thumb contact point and task-specific grasp type.)
that corresponds to the planned task.
(Sun - [column 14; lines 50-54] In some embodiments, a shape-matching algorithm can be combined with the presented approach for an unknown object. Grasp planning can be performed by matching a similar shape in the database, which collects a set of objects with a labeled thumb contact point and task-specific grasp type.)
Therefore, prior to the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Wellman to provide a database of quality grasping points and applied forces in the form of holding modes as taught in Asatani and to also develop and store grasping techniques that are task specific as taught in Sun. Having stored reference grasping points that are aligned with specific tasks in a database provides a way for a robot to associate a particular set of high quality grasping options and applied forces with a specific target object and task, therefore increasing the speed and accuracy with which an object can be controlled and manipulated.
Claim 5:
Wellman teaches the following limitations:
The device of claim 1, wherein the processing circuitry is further configured to store the respective data labels in a memory that associates respective data labels with object identifiers, wherein the respective data label and an identifier for the object comprise a record in a database of the memory.
(Wellman – [0076] … For example, an item having a unique identifier may have an associated database entry (e.g., from a record in the item database 37 of FIG. 1) regarding anticipated physical attributes of the item that may be used to locate and grasp the item within a particular environment (e.g., identifying a shape of the item based on the unique identifier so that that shape may be located in a detected environment to provide direction to the robotic arm to select the correct item out of a group of items). In some embodiments, a unique identifier may provide confirmation that a grasped item is the item that was intended to be grasped. …)
Claim 6:
Wellman teaches the following limitations:
The device of claim 1, wherein the processing circuitry is configured to identify the object based on a barcode label on the object that provides a unique identifier for the object. (Wellman - [0076] In some embodiments, a unique identifier may be an attribute detected by the attribute detection module 710. For example, the attribute detection module 710 may determine a UPC or SKU (universal product code or stock keeping unit) of an inventory item based on a barcode detected by an optical scanner or from an RFID tag detected by a RFID reader. Determining a unique identifier of an item may permit or facilitate certain other uses of detected attributes. For example, an item having a unique identifier may have an associated database entry (e.g., from a record in the item database 37 of FIG. 1) …)
Claim 7:
Wellman teaches the following limitations:
The device of claim 1, wherein the processing circuitry is configured to determine the observed grasping of the object by the hand based on the sensor data, (Wellman – [0082] … For example, sensors (such as in the sensor package 16 of FIG. 1) can be configured to detect information while a human is grasping a target item. In an illustrative example, a human wearing a glove or other garment with fiducial markers and/or sensors grasps a particular item. The motion of the human's approach is tracked based on the fiducial markers (e.g., as detected by optical sensors), and other characteristics are tracked based on the sensors (e.g., pressure sensors, tactile sensors, or other sensors may provide data regarding details about an amount and location of pressure that the human exerts on the item throughout the process, shear forces or other indicia translatable into an amount that the object slips in the human's hand, or other information that may be useful in determining a grasping strategy) …) wherein the processing circuitry is further configured to segregate the hand from the object based on the sensor data and (Wellman – [0082] … For example, sensors (such as in the sensor package 16 of FIG. 1) can be configured to detect information while a human is grasping a target item. In an illustrative example, a human wearing a glove or other garment with fiducial markers and/or sensors grasps a particular item. The motion of the human's approach is tracked based on the fiducial markers (e.g., as detected by optical sensors), and other characteristics are tracked based on the sensors …) determine a pose of the hand with respect to the object. (Wellman [0082] … For example, an illustrative user interface is described or shown in FIG. 8. In some embodiments, the human-based grasping strategy module 720 can provide a mechanism for a human to pose the robotic arm 12 in an orientation or position so as to instruct the robotic arm 12 how to grasp a target item. In some embodiments, the human-based grasping strategy module 720 can provide a virtual environment in which a human can perform or direct a grasping action for an item to facilitate machine learning of information for learning, developing, and/or determining a grasping strategy for the robotic arm 12 to grasp a target item.)
Claim 10:
Wellman teaches the following limitations:
The device of claim 1, wherein the processing circuitry is further configured to determine a multidimensional shape of the object based on the sensor data.
(Wellman- [0073] The attribute detection module 710 can interact with any number and/or type of sensors to determine attributes of an item to be grasped. For example, the attribute detection module 710 can receive information from imaging devices or optical sensors to determine physical characteristics, such as size, shape, position, orientation, and/or surface characteristics (e.g., how porous and/or slippery the item is based on the surface appearance). Any suitable optical technology can be utilized, including, but not limited to, two-dimensional cameras, depth sensors, time of flight sensing …)
Claim 11:
Wellman teaches the following limitations:
The device of claim 10, wherein the multidimensional shape of the object comprises a three-dimensional shape of the object. (Wellman- [0073] … (e.g., broadcasting a source of light and determining a time of reflection for each pixel to determine a distance from the sensor for each pixel to determine a three-dimensional array of data points representing a virtual model of the sensed item and environment) …)
Claim 13:
Wellman teaches the following limitations:
The device of claim 1, wherein the sensor data is received from a camera, a depth camera, a barcode scanner, a pressure sensor, and/or a force sensor.
(Wellman - [0073] … For example, the attribute detection module 710 can receive information from imaging devices or optical sensors to determine physical characteristics, such as size, shape, position, orientation, and/or surface characteristics (e.g., how porous and/or slippery the item is based on the surface appearance). Any suitable optical technology can be utilized, including, but not limited to, two-dimensional cameras, depth sensors, time of flight sensing … ; [0076] In some embodiments, a unique identifier may be an attribute detected by the attribute detection module 710. For example, the attribute detection module 710 may determine a UPC or SKU (universal product code or stock keeping unit) of an inventory item based on a barcode detected by an optical scanner or from an RFID tag detected by a RFID reader. )
Claim 14:
Wellman teaches the following limitations:
The device of claim 1, wherein the sensor data about the object comprises an observation of a human interaction with the object according to a task, wherein the human interaction comprises grasping the object with the hand.
(Wellman – [0082] … In an illustrative example, a human wearing a glove or other garment with fiducial markers and/or sensors grasps a particular item. The motion of the human's approach is tracked based on the fiducial markers (e.g., as detected by optical sensors), and other characteristics are tracked based on the sensors (e.g., pressure sensors, tactile sensors, or other sensors may provide data regarding details about an amount and location of pressure that the human exerts on the item throughout the process, shear forces or other indicia translatable into an amount that the object slips in the human's hand, or other information that may be useful in determining a grasping strategy) …)
Claim 16:
Wellman teaches the following limitations:
A non-transitory, computer-readable medium comprising instructions that, when executed, (Wellman – [0100] Each server typically will include an operating system that provides executable program instructions for the general administration and operation of that server and typically will include a computer-readable storage medium (e.g., a hard disk, random access memory, read only memory, etc.) storing instructions that, when executed by a processor of the server, allow the server to perform its intended functions. …) cause one or more processors to: receive sensor data of an observed grasping of an object by a hand; (Wellman – [0021] … As non-limiting examples, the human input device 38 may include a computer interface by which the human operator 34 can input instructions, may observe a human action for grasping an item to learn and/or determine information for forming a grasping strategy, and/or may provide a virtual environment in which the human operator can perform a simulated grasping of the item to obtain information for learning and/or determining a grasping strategy. …)determine, based on the sensor data of the observed grasping of the object, (Wellman – [0082] … For example, sensors (such as in the sensor package 16 of FIG. 1) can be configured to detect information while a human is grasping a target item. In an illustrative example, a human wearing a glove or other garment with fiducial markers and/or sensors grasps a particular item. The motion of the human's approach is tracked based on the fiducial markers (e.g., as detected by optical sensors), and other characteristics are tracked based on the sensors (e.g., pressure sensors, tactile sensors, or other sensors may provide data regarding details about an amount and location of pressure that the human exerts on the item throughout the process, shear forces or other indicia translatable into an amount that the object slips in the human's hand, or other information that may be useful in determining a grasping strategy) …)
Wellman does not explicitly teach the following limitations, however Asatani teaches:
at least one manipulation point in relation to the object; determine, based on the sensor data, a corresponding force applied to the object at the manipulation point;
(Asatani - {0022] … The holding mode determination device 10 determines a mode to be used by the robot 2 to hold the holding target object 80, and outputs the mode to the robot control device 110. The mode in which the robot 2 holds the holding target object 80 is also simply referred to as a holding mode. The holding mode includes a position at which the robot 2 touches the holding target object 80, a force to be applied to the holding target object 80 by the robot 2, or the like. ; [0045] … The controller 12 acquires an image captured by photographing the holding target object 80 from the camera 4, inputs the image to the trained model, and acquires an inference result of a holding mode for the holding target object 80 from the trained model. …)
wherein the respective data label indicates the at least one manipulation point for the object and the corresponding force and
(Asatani - [0079] In the above embodiment, the configuration in which the holding mode determination device 10 determines the holding position 82 as a holding mode has been described. The holding mode determination device 10 is capable of determining not only the holding position(s) 82 but also another mode as the holding mode. ; [0080] For example, the holding mode determination device 10 may determine, as the holding mode, a force to be applied by the robot 2 to hold the holding target object 80. In this case, the holding mode inference model 50 infers and outputs a force to be applied to hold the holding target object 80.
control a robot ] to grasp an item at a grasping point based on the at least one manipulation point and the corresponding force in the respective data label
(Asatani - [0026] … In the configuration illustrated in FIG. 1, the robot 2 causes the end effector 2B to hold the holding target object 80 at the work start point 6, and moves the end effector 2B to the work target point 7. The robot 2 causes the end effector 2B to release the holding target object 80 at the work target point 7. In this way, the robot 2 is capable of moving the holding target object 80 from the work start point 6 to the work target point 7.)
Wellman in combination with Asatani does not explicitly teach the following limitations, however Sun teaches:
create, for each of a plurality of tasks associated with the observed grasping, a respective data label for the object for a corresponding task of the plurality of tasks,
(Sun – [column 14; lines 29-33] … Grasp types import important task-oriented experience to the robot because a human usually applies different grasp types according to the task properties. Grasp types also provide a good referenced hand posture to the robot. …) ; [column 14; lines 50-54] In some embodiments, a shape-matching algorithm can be combined with the presented approach for an unknown object. Grasp planning can be performed by matching a similar shape in the database, which collects a set of objects with a labeled thumb contact point and task-specific grasp type.)
that is specific to its corresponding task;
(Sun - [column 14; lines 29-33] … Grasp types import important task-oriented experience to the robot because a human usually applies different grasp types according to the task properties. Grasp types also provide a good referenced hand posture to the robot. …)
during execution of a current task
(Sun - [column 14; lines 50-54] In some embodiments, a shape-matching algorithm can be combined with the presented approach for an unknown object. Grasp planning can be performed by matching a similar shape in the database, which collects a set of objects with a labeled thumb contact point and task-specific grasp type.)
from the plurality of tasks that corresponds to the current task.
(Sun – [column 14; lines 29-33] … Grasp types import important task-oriented experience to the robot because a human usually applies different grasp types according to the task properties. Grasp types also provide a good referenced hand posture to the robot. … ; [column 14; lines 50-54] In some embodiments, a shape-matching algorithm can be combined with the presented approach for an unknown object. Grasp planning can be performed by matching a similar shape in the database, which collects a set of objects with a labeled thumb contact point and task-specific grasp type.)
Therefore, prior to the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Wellman to provide a database of quality grasping points and applied forces in the form of holding modes as taught in Asatani and to also develop and store grasping techniques that are task specific as taught in Sun. Having stored reference grasping points that are aligned with specific tasks in a database provides a way for a robot to associate a particular set of high quality grasping options and applied forces with a specific target object and task, therefore increasing the speed and accuracy with which an object can be controlled and manipulated.
Claim 18:
Wellman teaches the following limitations:
The non-transitory, computer-readable medium of claim 16, wherein the sensor data about the object comprises a series of images of the object as it is being manipulated by the hand.
(Wellman – [0018] … As another example, the human operator may provide input about how the mug may be effectively grasped by the robotic arm, such as by selecting from different options presented on a screen or by donning a glove and grasping the mug so that a grasping strategy for the robotic arm may be generated using information from features on the glove (e.g., pressure sensors, tactile sensors, or fiducial markers used to track the motion of the glove with an optical imaging device).
Claim 19:
Wellman teaches the following limitations:
A robot comprising: a sensor system configured identify an object based on sensor data about the object; ; (Wellman - [0020] The sensor package 16 includes one or more sensors (of like or varying type) arranged to detect the item 40 while the item 40 is being maintained by the grasping environment 14. The sensor package 16 communicates detected attributes (as at 46), such as weight, geometric characteristics (e.g., size, position, or orientation), electrical conductivity, magnetic properties, surface characteristics (e.g., how slippery or porous the item is), deformability, and/or structural integrity of the item 40, to the controller 32. …) an end-effector for performing a task in relation to the object; and a manipulation system configured to determine a grasping point and a grasping force by which the end-effector is to grasp the object when performing the task, (Wellman – [0019] Referring now to the drawings, in which like reference numerals and/or names may refer to like elements, FIG. 1 illustrates an inventory system 10 having a robotic arm or manipulator 12 configured to grasp inventory items 40. Although the description herein primarily refers to a robotic arm 12, any other mechatronic or robotic device may be used in lieu of or in addition to an arm. Additionally, the terms “grasping,” “gripping,” or the like as used herein should be understood to include any physical manipulation of objects, including, but not limited to, picking up, pushing, pulling, compressing, stretching, and moving. The system 10 may include the robotic arm 12, a grasping environment 14, an inventory item 40, a sensor package 16, a controller 32, an item gripping database 36, an item database 37, a human input device 38, and a human operator 34. The item database 37 and the item gripping database 36, although depicted as separate in FIG. 1, may share structure and/or content. …)
Wellman does not explicitly teach the following limitations, however Asatani teaches:
wherein the grasping point is determined based on manipulation points for the task and the grasping force is determined based on corresponding forces at the manipulation points
(Asatani – [0022] … The holding mode determination device 10 determines a mode to be used by the robot 2 to hold the holding target object 80, and outputs the mode to the robot control device 110. The mode in which the robot 2 holds the holding target object 80 is also simply referred to as a holding mode. The holding mode includes a position at which the robot 2 touches the holding target object 80, a force to be applied to the holding target object 80 by the robot 2, or the like. ; [0045] … The controller 12 acquires an image captured by photographing the holding target object 80 from the camera 4, inputs the image to the trained model, and acquires an inference result of a holding mode for the holding target object 80 from the trained model. …)
wherein the manipulation points are determined based on a learning model that associates objects with their manipulation points and corresponding forces
(Asatani - [0055] The controller 12 performs learning by using a target object image as learning data, thereby generating the class inference model 40 and the holding mode inference model 50 as a trained model. The controller 12 requires correct answer data to generate the trained model. For example, to generate the class inference model 40, the controller 12 requires learning data associated with information indicating a correct answer about a class into which an image used as learning data is classified.)
Wellman in combination with Asatani does not explicitly teach the following limitations, however Sun teaches:
from among a plurality of tasks
(Sun - [column 5; lines 41-46] Many grasp classifications are defined based on Cutkosky's grasp taxonomy, which classifies user performed grasps into 16 classes that vary by task requirement and dexterities. To recognize the demonstrated grasp to be a type from the Cutkosky's taxonomy, pattern classification techniques can be applied.)
forces at the manipulation points for the task,
(Sun – [column 5; lines 2 -8] … As the jar lid gets tighter, one may switch to a power grasp to apply larger force on the jar lid. Thus, one useful task-specific strategy that can be extracted from a demonstration is the grasp type, which can be recognized by observing humans demonstrating grasps. In this context, the grasp type refers to the manner in which the demonstrator grips an object.)
for each of the plurality of tasks,
(Sun - [column 5; lines 41-46] (32) Many grasp classifications are defined based on Cutkosky's grasp taxonomy, which classifies user performed grasps into 16 classes that vary by task requirement and dexterities. To recognize the demonstrated grasp to be a type from the Cutkosky's taxonomy, pattern classification techniques can be applied.)
specific to each of the plurality of tasks.
(Sun – [column 14; lines 29-33] … Grasp types import important task-oriented experience to the robot because a human usually applies different grasp types according to the task properties. Grasp types also provide a good referenced hand posture to the robot. …)
Therefore, prior to the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Wellman to provide a database of quality grasping points and applied forces in the form of holding modes as taught in Asatani and to also develop and store grasping techniques that are task specific as taught in Sun. Having stored reference grasping points that are aligned with specific tasks in a database provides a way for a robot to associate a particular set of high quality grasping options and applied forces with a specific target object and task, therefore increasing the speed and accuracy with which an object can be controlled and manipulated.
Claim 20:
Wellman teaches the following limitations:
The robot of claim 19, wherein the manipulation system is further configured to control the robot to grasp the object at the grasping point
(Wellman – [0019] Referring now to the drawings, in which like reference numerals and/or names may refer to like elements, FIG. 1 illustrates an inventory system 10 having a robotic arm or manipulator 12 configured to grasp inventory items 40. Although the description herein primarily refers to a robotic arm 12, any other mechatronic or robotic device may be used in lieu of or in addition to an arm. Additionally, the terms “grasping,” “gripping,” or the like as used herein should be understood to include any physical manipulation of objects, including, but not limited to, picking up, pushing, pulling, compressing, stretching, and moving. The system 10 may include the robotic arm 12, a grasping environment 14, an inventory item 40, a sensor package 16, a controller 32, an item gripping database 36, an item database 37, a human input device 38, and a human operator 34. The item database 37 and the item gripping database 36, although depicted as separate in FIG. 1, may share structure and/or content. …)
Wellman does not explicitly teach the following limitations, however Asatani teaches:
based on the manipulation points for the task.
(Asatani – [0022] … The holding mode determination device 10 determines a mode to be used by the robot 2 to hold the holding target object 80, and outputs the mode to the robot control device 110. The mode in which the robot 2 holds the holding target object 80 is also simply referred to as a holding mode. The holding mode includes a position at which the robot 2 touches the holding target object 80, a force.)
Therefore, prior to the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Wellman to provide a database of quality grasping points and applied forces in the form of holding modes as taught in Asatani. Having stored reference grasping points in a database provides a way for a robot to associate a particular set of high quality grasping options and applied forces with a specific target object, therefore increasing the speed and accuracy with which an object can be controlled and manipulated.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Wellman (US 20170021499 A1) as modified by Asatani ((US 20250065495 A1) in view of Sun (US 9321176 B1) and in further view of Zhou (US 20220277580 A1)
Claim 8:
Wellman in combination with Asatani and Sun does not explicitly teach the following limitations, however Zhou teaches:
The device of claim 7, wherein the sensor data comprises an image of the object and the hand, (Zhou – [0029] In particular, the successful development of depth cameras in recent years has led to greater progress in reconstructing motions of the hand. Here, the depth cameras include structured light cameras, laser scanning cameras, TOF cameras, etc. Herein, the widely applied TOF cameras are currently adopted.) wherein the processing circuitry is configured to determine the pose of the hand based on an interference extraction on the image, a peak extraction from the image, (Zhou – [0007] In a first aspect, embodiments of the present disclosure provide a hand pose estimation method. The method includes: obtaining a feature map corresponding to a hand depth image; inputting the feature map into an image feature extraction network to obtain an image information set feature map corresponding to the hand depth image; …) and a rendering, based on the image, of a set of key points that define a multidimensional stick model of the pose. (Zhou – [0007] … inputting the target resolution feature map into a predetermined depth classification network to obtain depth maps corresponding to hand key points in the hand depth image, wherein the predetermined depth classification network is configured to distinguish hand key points of different depths; and determining, based on the depth maps, depth values corresponding to the hand key points to perform hand pose estimation.)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Wellman, Asatani and Sun to provide a method for extracting the image of a hand grasping an object and further convert that image data into a digital key point model as taught in Zhou. Having the ability to extract an accurate model of a grasping hand allows the user to more accurately train a grasping robot by transforming the movements of a grasping hand to the task motion of a grasping robot.
Claims 9 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Wellman (US 20170021499 A1) as modified by Asatani (US 20250065495 A1) in view of Sun (US 9321176 B1) and in further view of Lucas (US 20240070992 A1)
Claim 9:
Wellman in combination with Asatani and Sun does not explicitly teach the following limitations, however Lucas teaches:
The device of claim 8, wherein the set of key points comprises
multiple points that together define the multidimensional stick model.
(Lucas - [0049] FIG. 3A and FIG. 3B are illustrations of an AR user interface, FIG. 3C is collaboration diagram of components of an AR system providing the AR user interface, and FIG. 3D is an activity diagram of a gesture and hand pose detection method used by the AR system to provide user inputs to the AR user interface, in accordance with some examples. As shown in FIG. 3C, an AR system 352 includes a hand-tracking pipeline 348 that captures video frame tracking data 330 of gestures and hand poses 334 being made by a user 332 as the user 332 interacts with an AR application 328. ; [see also Fig. 3A and 3B] )
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Wellman, Asatani and Sun to provide a method for modeling a human hand as digital key point or skeletal model as taught in Lucas. Having the ability to apply a digital key point or skeletal model of a grasping hand allows the user to more accurately train a grasping robot by transforming the movements of a grasping hand to the task motion of a grasping robot.
Claim 17:
Wellman in combination with Asatani and Sun does not explicitly teach the following limitations, however Lucas teaches:
The non-transitory, computer-readable medium of claim 16, wherein the instructions further cause the one or more processors to render a three-dimensional stick model of the hand based on the sensor data and (Lucas – [0052] … In some examples, the skeletal 3D model data 338 includes landmark data such as landmark identification, a physical location of the landmark, segments between joints of the user's fingers, and categorization information of one or more landmarks associated with the forearm, wrist, and hand 322. For example, the skeletal 3D model inference component 324 generates the skeletal 3D model data 338 based on the video frame tracking data 330 using artificial intelligence methodologies and an ML hand-tracking model 336 previously generated using machine learning methodologies.) to determine the observed grasping of the object by the hand based on the three-dimensional stick model. (Lucas – [0064] … The user 332 interacts with the one or more virtual 3D objects using gestures and hand poses 334 to select and manipulate the virtual 3D objects. The one or more virtual 3D objects are represented in the AR user interface model 358 using 3D coordinates in a 3D coordinate system of the AR user interface model 358. In a similar manner, the filtered skeletal 3D model data 362 includes a temporal sequence of skeletal 3D models comprising skeletal 3D model features where each skeletal 3D model feature has a set of 3D coordinates in a common 3D coordinate system of the AR user interface model 358.)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Wellman, Asatani and Sun to provide a method for modeling a human hand as digital key point or skeletal model as taught in Lucas. Having the ability to apply a digital key point or skeletal model of a grasping hand allows the user to more accurately train a grasping robot by transforming the movements of a grasping hand to the task motion of a grasping robot.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Wellman (US 20170021499 A1) as modified by Asatani (US 20250065495 A1) in view of Sun (US 9321176 B1) and in further view of Huang (US 20200331709 A1)
Claim 15:
Wellman in combination with Asatani and Sun does not explicitly teach the following limitations, however Huang teaches:
The device of claim 1, wherein the processing circuitry is configured to segregate the object from the hand based on a depth image comprising depth information, wherein the sensor data comprises the depth image.
(Huang - [0026] More specifically, the system 100 includes both onboard and fixed sensors relative to the robotic arm assembly 102. In the exemplary embodiment shown in FIG. 1, fixed sensors include a vision sensor or camera 124 mounted on a solid surface or wall 126 in an area close to a grasping location for the object … ; [0047] Similar to the fixed camera, a mounted camera 132 (FIG. 1) can provide filtered images of the object at different instances, such that the positional and orientational consistency of the object relative to the gripper can be evaluated and monitored. …)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Wellman, Asatani and Sun to utilize depth images from cameras in order to separate the images of the grasping hand from the grasped object as taught in Huang. Having the ability to extract an accurate image of a grasping hand from the grasped object allows the user to more accurately train a grasping robot by transforming the movements of a grasping hand to the task motion of a grasping robot.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure or directed to the state of the art is listed on the enclosed PTO-892.
The following is a brief description for relevant prior art that was cited but not applied:
Cronie (US 20170008172 A1) describes a technology for grasp point selection for robotic grippers through machine vision and ultrasound beam based deformation. The grasp point selection may aim to increase a probability that the grasp points on an object behave in a substantially similar way when a robotic gripper executes a corresponding grasp on the object. According to some examples, an outline of an object may be extracted from a three-dimensional (3D) image of the object and a set of points may be selected as candidate grasp points from the outline based on the candidate grasp points.
Horowitz (US 20230191608 A1) describes using machine learning to recognize variant objects, including: identifying an object as a variant of an object type by inputting sensed data associated with the object into a modified machine learning model corresponding to the variant of the object type. The system generates a control signal to perform a sorting operation on the object, wherein the sorting operation on the object is determined based at least in part on the object type and stored identifiers.
Wozniak (US 11922368 B1) describes techniques for classifying and processing physical objects. A machine learning model of the computer system may determine a cluster of physical objects that includes an identifier of the physical object, whereby the identifier is included in the cluster. The computer system may then determine data for processing physical objects that have the common attribute.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALAN LINDSAY OSTROW whose telephone number is (703)756-1854. The examiner can normally be reached M-F 8 - 5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Adam Mott can be reached on (571) 270 5376. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALAN LINDSAY OSTROW/Examiner, Art Unit 3657
/ADAM R MOTT/Supervisory Patent Examiner, Art Unit 3657
Prosecution Insights