DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This Office Action is in response to Amendments filed on 01/27/2026, wherein Claims 1-22 are pending. Claims 1-2, 4-5, 8-9, 13, 15, 18-19, and 21-22 have been amended.
Response to Arguments
Regarding 35 USC 103 rejection: Applicant’s arguments with respect to claims 1-22, have been considered but are moot because of the new ground of rejection necessitated by the amendments.
Regarding Examiner’s objections: Applicant's arguments, see Remarks (p. 9), filed on 01/27/2026, with respect to the objections to the Claims have been fully considered. In view of the amendments to the Claims addressing the informalities raised in the previous office action, the objections to the Claims have been withdrawn
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 8, 12-15, and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over US20200130192 to Ogawa et al. (hereinafter Ogawa) in view of US20140107843 to Okazaki (hereinafter Okazaki) and in further view of NPL Costanzo et al. Design and Calibration of a Force/Tactile Sensor for Dexterious Manipulation, Sensors, 2019, 19, 966 (hereinafter Costanzo).
Regarding Claim 1: Ogawa discloses:
“A method, comprising: causing a robotic arm to manipulate an object” (para 0068 – “The manipulator 20 includes an arm 2 and a hand 200. The arm 2 is what is called an articulated or multi-axis robot arm, and is driven by a plurality of servomotors, for example. The arm 2 may be, for example, a robot of a type combining a plurality of a vertical articulated-type robot, a horizontal articulated-type robot, and a linear motion robot. Various sensors (not illustrated) such as a force sensor are provided to the arm 2. The hand 200 is, for example, a mechanism for grasping the object P (target object) with a multi-finger mechanism that performs sucking, jamming, pinching, and the like (i.e. manipulating an object, added by examiner). ”);
“obtaining a signal from a force sensor” (para 0074 – “The signal processing unit 101 b performs, on various pieces of sensor information, signal processing such as signal amplification processing, analog-to-digital conversion processing, noise removal, extraction of feature information, and processing of determining a state change”; para 0068 – “Various sensors (not illustrated) such as a force sensor are provided to the arm 2… various sensors (not illustrated) such as a force sensor are provided to the hand 200”);
“comprising motion information of the object resulting from an applied force; to generate one or more instructions to adjust the applied force to the object by the robotic arm based, at least in part, on the estimated force and the motion information of the object resulting from the applied force” (para 0236 – “the manipulator control unit 110 controls the manipulator 20 so that the distal end of the hand 200 applies a predetermined force to the contactable position... a force sensor 200 h, and the manipulator control unit 110 can control magnitude of the force to be applied (interpreted as the force can be adjusted, added by examiner), acceleration, and the like, so that the predetermined force can be applied without damaging the component (S902)… the positional relation (i.e. motion information, added by examiner) between the object handling device 1 and the article to be grasped is varied, so that the route (i.e. motion information, added by examiner) to the article grasping position may be generated with high possibility. ”).
Ogawa does not specifically disclose:
“estimating a force on the robotic arm based on the motion of the object and a model of the object, wherein the model of the object comprises mass of the object and a measure of friction between the object and a planar surface; and using one or more machine learning models to calibrate the force sensor”.
However, Okazaki discloses:
“estimating a force on the robotic arm based on the motion of the object and a model of the object, wherein the model of the object comprises mass of the object and a measure of friction between the object and a planar surface” (para 0174 – “in a case where a representative coefficient of kinetic friction between the object 38 and the floor surface 90 (i.e. planar surface, added by examiner) is set as the setting coefficient of kinetic friction μ, a resistance force 60 is increased when the coefficient of kinetic friction during the conveyance motion is larger than the setting coefficient of kinetic friction p.)”; para 0176 – “The desired resistance force FRgd may be determined by an experiment considering the maneuverability at the time of manipulating the robot arm 5. However, μmax(mg−FAmax)<FRgd<μminmg may be considered as the setting range. In the case where mg<FAmax, (when transportability sufficient for lifting the object 38 is provided to the robot arm 5), the lower limit value μmax(mg−FAmax) is 0. Here, m is the mass of the object 38, g is a gravity acceleration thereof, μmax is a maximum coefficient of kinetic friction that can be generated, μmin is a minimum coefficient of kinetic friction that can be generated, and FAmax is a maximum force that can be generated as the lifting force by the robot arm 5 (i.e. force estimated based on mass of the object and a measure of friction between the object and a planar surface, added by examiner). The necessary environment information such as the maximum coefficient of kinetic friction μmax, the minimum coefficient of kinetic friction μmin, the mass m of the object 38, and the gravity acceleration g is stored in a database of the assist force correcting section 34 in advance”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method, disclosed by Ogawa, as taught by Okazaki, in order to estimate a force on the robotic arm based on the motion of the object and a model of the object and a measure of friction between the object and a planar surface and make the operation of a robotic arm safe.
Ogawa/Okazaki combination does not specifically disclose:
“using one or more machine learning models to calibrate the force sensor”.
However, Costanzo discloses:
“using one or more machine learning models to calibrate the force sensor” (page 2 of 23 – “In the present paper the complete design and calibration of a new and upgraded version of that force/tactile sensor is presented. The new sensor design starts from the main requirement to manipulate objects of generic shape avoiding both linear and rotational slippage… The novel calibration procedure is aimed at enhancing the estimation accuracy especially of the torsional moment component of the contact wrench. It is based on a machine learning approach and, in detail, on the training of a multi-layer feed-forward neural network (FF-NN), and special attention had to be given to the construction of the training set”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method, disclosed by Ogawa/Okazaki combination, as taught by Costanzo, in order to use the machine learning models to calibrate the force sensor and make the operation of a robotic arm safe.
Regarding Claim 8: Ogawa discloses:
“One or more processors comprising: circuitry to: cause a robotic arm to manipulate an object” (para 0072 – “The arithmetic processing unit 101 includes a manipulator control unit 110, … and the like. The manipulator control unit 110 includes an arm control unit 120 (i.e. circuitry, added by examiner), … and the like”; Claim 18 – “the processor is configured to control, when a motion trajectory of the effector to a target object to be grasped cannot be calculated, the manipulator to vary a positional relationship between the target object and the object handling device” (i.e. manipulate an object, added by examiner));
“obtain a signal from a force sensor comprising motion information of the object resulting from an applied force; to generate one or more instructions to adjust the applied force to the object by the robotic arm based, at least in part, on the estimated force and the motion information of the object resulting from the applied force” (para 0236 – “the manipulator control unit 110 controls the manipulator 20 so that the distal end of the hand 200 applies a predetermined force to the contactable position... a force sensor 200 h, and the manipulator control unit 110 can control magnitude of the force to be applied (interpreted as the force can be adjusted, added by examiner), acceleration, and the like, so that the predetermined force can be applied without damaging the component (S902)… the positional relation (i.e. motion information, added by examiner) between the object handling device 1 and the article to be grasped is varied, so that the route (i.e. motion information, added by examiner) to the article grasping position may be generated with high possibility ”).
Ogawa does not specifically disclose:
“estimate a force on the robotic arm based on the motion of the object and a model of the object, wherein the model of the object comprises mass of the object and a measure of friction between the object and a planar surface; and using one or more machine learning models to calibrate the force sensor”.
However, Okazaki discloses:
“estimate a force on the robotic arm based on the motion of the object and a model of the object, wherein the model of the object comprises mass of the object and a measure of friction between the object and a planar surface” (para 0174 – “in a case where a representative coefficient of kinetic friction between the object 38 and the floor surface 90 (i.e. planar surface, added by examiner) is set as the setting coefficient of kinetic friction μ, a resistance force 60 is increased when the coefficient of kinetic friction during the conveyance motion is larger than the setting coefficient of kinetic friction p.)”; para 0176 – “The desired resistance force FRgd may be determined by an experiment considering the maneuverability at the time of manipulating the robot arm 5. However, μmax(mg−FAmax)<FRgd<μminmg may be considered as the setting range. In the case where mg<FAmax, (when transportability sufficient for lifting the object 38 is provided to the robot arm 5), the lower limit value μmax(mg−FAmax) is 0. Here, m is the mass of the object 38, g is a gravity acceleration thereof, μmax is a maximum coefficient of kinetic friction that can be generated, μmin is a minimum coefficient of kinetic friction that can be generated, and FAmax is a maximum force that can be generated as the lifting force by the robot arm 5 (i.e. force estimated based on mass of the object and a measure of friction between the object and a planar surface, added by examiner). The necessary environment information such as the maximum coefficient of kinetic friction μmax, the minimum coefficient of kinetic friction μmin, the mass m of the object 38, and the gravity acceleration g is stored in a database of the assist force correcting section 34 in advance”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method, disclosed by Ogawa, as taught by Okazaki, in order to estimate a force on the robotic arm based on the motion of the object and a model of the object and a measure of friction between the object and a planar surface and make the operation of a robotic arm safe.
Ogawa/Okazaki combination does not specifically disclose:
“using one or more machine learning models to calibrate the force sensor”.
However, Costanzo discloses:
“using one or more machine learning models to calibrate the force sensor” (page 2 of 23 – “In the present paper the complete design and calibration of a new and upgraded version of that force/tactile sensor is presented. The new sensor design starts from the main requirement to manipulate objects of generic shape avoiding both linear and rotational slippage… The novel calibration procedure is aimed at enhancing the estimation accuracy especially of the torsional moment component of the contact wrench. It is based on a machine learning approach and, in detail, on the training of a multi-layer feed-forward neural network (FF-NN), and special attention had to be given to the construction of the training set”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method, disclosed by Ogawa/Okazaki combination, as taught by Costanzo, in order to use the machine learning models to calibrate the force sensor and make the operation of a robotic arm safe.
Regarding Claim 12: Ogawa/Okazaki/Costanzo combination discloses the one or more processors of Claim 8 (see the rejection for Claim 8).
Regarding the limitation “wherein the circuitry is further to estimate the force by solving a least squared minimization of linear acceleration and angular acceleration of the object”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the least squares minimization approach, widely used in science and in the industry, to obtain the most reliable value for the force estimation.
Regarding Claim 13: Ogawa/Okazaki/Costanzo combination discloses the one or more processors of claim 8 (see the rejection for Claim 8).
Ogawa does not specifically disclose:
“wherein the circuitry is further to: determine the one or more machine learning models based at least in part on the estimated force and the signal”.
However, Costanzo discloses:
“wherein the circuitry is further to: determine the one or more machine learning models based at least in part on the estimated force and the signal” (page 2 of 23 – “In the present paper the complete design and calibration of a new and upgraded version of that force/tactile sensor is presented. The new sensor design starts from the main requirement to manipulate objects of generic shape avoiding both linear and rotational slippage… The novel calibration procedure is aimed at enhancing the estimation accuracy especially of the torsional moment component of the contact wrench. It is based on a machine learning approach and, in detail, on the training of a multi-layer feed-forward neural network (FF-NN), and special attention had to be given to the construction of the training set”; page 3 of 23 – “The major objective that can be pursued is to design a sensor as similar as possible to human touch, i.e., able to supply different types of information about the manipulated objects at the same time: A tactile map, an estimate of contact forces and moments… ”; page 4 of 23 – “The whole tactile map allows, after a calibration procedure, to estimate contact force and moment together with information about the orientation of the contact surface and object properties”; page 17 of 23 – “The estimated normal force of a finger is then compared
to the normal force measured by the reference sensor. The signals and the corresponding error are reported in Figure 15, which shows a maximum error of about 0.7N for a maximum force of 16 N.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method, disclosed by Ogawa/Okazaki/Costanzo combination, as taught by Costanzo, in order to make the operation of a robotic arm safer and not to destroy the object being manipulated.
Regarding Claim 14: Ogawa/Okazaki/Costanzo combination discloses the one or more processors of claim 8 (see the rejection for Claim 8).
Ogawa does not specifically disclose:
“wherein the circuitry is further to: use the one or more machine learning models to estimate an amount of force represented by the signal”.
However, Costanzo discloses:
“wherein the circuitry is further to: use the one or more machine learning models to estimate an amount of force represented by the signal” (page 2 of 23 – “In the present paper the complete design and calibration of a new and upgraded version of that force/tactile sensor is presented. The new sensor design starts from the main requirement to manipulate objects of generic shape avoiding both linear and rotational slippage… The novel calibration procedure is aimed at enhancing the estimation accuracy especially of the torsional moment component of the contact wrench. It is based on a machine learning approach and, in detail, on the training of a multi-layer feed-forward neural network (FF-NN), and special attention had to be given to the construction of the training set”; page 3 of 23 – “The major objective that can be pursued is to design a sensor as similar as possible to human touch, i.e., able to supply different types of information about the manipulated objects at the same time: A tactile map, an estimate of contact forces and moments… ”; page 4 of 23 – “The whole tactile map allows, after a calibration procedure, to estimate contact force and moment together with information about the orientation of the contact surface and object properties”; page 17 of 23 – “The estimated normal force of a finger is then compared to the normal force measured by the reference sensor. The signals and the corresponding error are reported in Figure 15, which shows a maximum error of about 0.7N for a maximum force of 16 N.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method, disclosed by Ogawa/Okazaki/Costanzo combination, as taught by Costanzo, in order to make the operation of a robotic arm safer and not to destroy the object being manipulated.
Regarding Claim 15: Ogawa discloses:
“A non-transitory computer-readable medium having stored thereon instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to at least: cause a robotic arm to manipulate an object” (para 0101 – “The command generation unit 140 a generates an operation procedure required for each work process as an operation command in response to an operation instruction input … The operation instruction is an instruction related to a series of operations of the arm 2 and the hand 200 of the manipulator 20, and held as a computer program, for example”; para 0072 – “The arithmetic processing unit 101 includes a manipulator control unit 110, … and the like. The manipulator control unit 110 includes an arm control unit 120, … and the like”; Claim 18 – “the processor is configured to control, when a motion trajectory of the effector to a target object to be grasped cannot be calculated, the manipulator to vary a positional relationship between the target object and the object handling device” (i.e. manipulate an object, added by examiner));
“obtain a signal from a force sensor comprising motion information of the object resulting from an applied force; to generate one or more instructions to adjust the applied force to the object by the robotic arm based, at least in part, on the estimated force and the motion information of the object resulting from the applied force” (para 0236 – “the manipulator control unit 110 controls the manipulator 20 so that the distal end of the hand 200 applies a predetermined force to the contactable position... a force sensor 200 h, and the manipulator control unit 110 can control magnitude of the force to be applied (interpreted as the force can be adjusted, added by examiner), acceleration, and the like, so that the predetermined force can be applied without damaging the component (S902)… the positional relation (i.e. motion information, added by examiner) between the object handling device 1 and the article to be grasped is varied, so that the route (i.e. motion information, added by examiner) to the article grasping position may be generated with high possibility ”).
Ogawa does not specifically disclose:
“estimate a force on the robotic arm based on the motion of the object and a model of the object, wherein the model of the object comprises mass of the object and a measure of friction between the object and a planar surface; and using one or more machine learning models to calibrate the force sensor”.
However, Okazaki discloses:
“estimate a force on the robotic arm based on the motion of the object and a model of the object, wherein the model of the object comprises mass of the object and a measure of friction between the object and a planar surface” (para 0174 – “in a case where a representative coefficient of kinetic friction between the object 38 and the floor surface 90 (i.e. planar surface, added by examiner) is set as the setting coefficient of kinetic friction μ, a resistance force 60 is increased when the coefficient of kinetic friction during the conveyance motion is larger than the setting coefficient of kinetic friction p.)”; para 0176 – “The desired resistance force FRgd may be determined by an experiment considering the maneuverability at the time of manipulating the robot arm 5. However, μmax(mg−FAmax)<FRgd<μminmg may be considered as the setting range. In the case where mg<FAmax, (when transportability sufficient for lifting the object 38 is provided to the robot arm 5), the lower limit value μmax(mg−FAmax) is 0. Here, m is the mass of the object 38, g is a gravity acceleration thereof, μmax is a maximum coefficient of kinetic friction that can be generated, μmin is a minimum coefficient of kinetic friction that can be generated, and FAmax is a maximum force that can be generated as the lifting force by the robot arm 5 (i.e. force estimated based on mass of the object and a measure of friction between the object and a planar surface, added by examiner). The necessary environment information such as the maximum coefficient of kinetic friction μmax, the minimum coefficient of kinetic friction μmin, the mass m of the object 38, and the gravity acceleration g is stored in a database of the assist force correcting section 34 in advance”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method, disclosed by Ogawa, as taught by Okazaki, in order to estimate a force on the robotic arm based on the motion of the object and a model of the object and a measure of friction between the object and a planar surface and make the operation of a robotic arm safe.
Ogawa/Okazaki combination does not specifically disclose:
“using one or more machine learning models to calibrate the force sensor”.
However, Costanzo discloses:
“using one or more machine learning models to calibrate the force sensor” (page 2 of 23 – “In the present paper the complete design and calibration of a new and upgraded version of that force/tactile sensor is presented. The new sensor design starts from the main requirement to manipulate objects of generic shape avoiding both linear and rotational slippage… The novel calibration procedure is aimed at enhancing the estimation accuracy especially of the torsional moment component of the contact wrench. It is based on a machine learning approach and, in detail, on the training of a multi-layer feed-forward neural network (FF-NN), and special attention had to be given to the construction of the training set”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method, disclosed by Ogawa/Okazaki combination, as taught by Costanzo, in order to use the machine learning models to calibrate the force sensor and make the operation of a robotic arm safe.
Regarding Claim 21: Ogawa/Okazaki/Costanzo combination discloses the non-transitory computer-readable medium of claim 15.
Ogawa further discloses:
“wherein the executable instructions further cause the computer system to: cause a robot instrumented with one or more force sensors to manipulate the object” (para 0068 – “The manipulator 20 includes an arm 2 and a hand 200… Various sensors (not illustrated) such as a force sensor are provided to the arm 2. ”; para 0101 – “The command generation unit 140 a (i.e. computer system, added by examiner) generates an operation procedure required for each work process as an operation command in response to an operation instruction input from the grasp/operation plan generation unit 101 d (FIG. 2) via the plan generation unit 101 c (FIG. 2). ... The operation instruction is an instruction related to a series of operations of the arm 2 and the hand 200 of the manipulator 20, and held as a computer program, for example.”; para 0065 – “in a case in which the manipulator 20 carries (i.e. manipulates, added by examiner) the object P in the object handling device 1, after the object P is carried in, the conveying mechanism 40 can carry the object P out of the object handling device 1.”);
“receive the signal from the one or more force sensors” (para 0107 – “ the arm control unit 120 controls the driving unit of the arm 2 based on a plan planned by the plan generation unit 101 c… detection results (i.e. signals, added by examiner) obtained by various sensors”); and
“determine a force on the object using the one or more machine learning models and the signal” (para 0247 – “the manipulator control unit 110 controls the manipulator 20 so that the distal end of the hand 200 applies a predetermined force to the contactable position. In this case, the hand 200 includes the elastic mechanism 200 e and a force sensor 200 h, and the manipulator control unit 110 can control magnitude of the force to be applied, acceleration, and the like, so that the predetermined force can be applied without damaging the component (S902). Next, the manipulator control unit 110 returns the hand 200 to the initial position p0 (S903). The process then returns to S601 in FIG. 19A (S904). In this procedure, a placement state of the article is varied when the force is applied to the component or the surroundings thereof at S902”).
Regarding Claim 22: Ogawa/Okazaki/Costanzo combination discloses the non-transitory computer-readable medium of claim 15.
Ogawa further discloses:
“wherein the executable instructions further cause the computer system to:
cause the robotic arm to adjust a manipulation of the object” (para 0068 – “The manipulator 20 includes an arm 2 and a hand 200… Various sensors (not illustrated) such as a force sensor are provided to the arm 2. ”; para 0101 – “The command generation unit 140 a (i.e. computer system, added by examiner) generates an operation procedure required for each work process as an operation command in response to an operation instruction input from the grasp/operation plan generation unit 101 d (FIG. 2) via the plan generation unit 101 c (FIG. 2). ... The operation instruction is an instruction related to a series of operations of the arm 2 and the hand 200 of the manipulator 20, and held as a computer program, for example.”; para 0065 – “in a case in which the manipulator 20 carries (i.e. manipulates, added by examiner) the object P in the object handling device 1, after the object P is carried in, the conveying mechanism 40 can carry the object P out of the object handling device 1”; para 0139 – “The controller 320 can add, to the grasp information, a grasp score indicating a position and a posture in which the object P can be grasped, ease of grasp, and the like, a pressable amount in grasping (i.e. adjusting manipulation of the object, added by examiner), a threshold for grasp determination, and the like to be stored, based on the input, received information”);
“based at least in part on the force applied using the one or more machine learning models and the signal from one or more force sensors” (para 0247 – “the manipulator control unit 110 controls the manipulator 20 so that the distal end of the hand 200 applies a predetermined force to the contactable position. In this case, the hand 200 includes the elastic mechanism 200 e and a force sensor 200 h, and the manipulator control unit 110 can control magnitude of the force to be applied, acceleration, and the like, so that the predetermined force can be applied without damaging the component (S902). Next, the manipulator control unit 110 returns the hand 200 to the initial position p0 (S903). The process then returns to S601 in FIG. 19A (S904). In this procedure, a placement state of the article is varied when the force is applied to the component or the surroundings thereof at S902”).
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Ogawa/Okazaki/Costanzo combination, in view of US20200130178A1 to Colasanto et al. (hereinafter Colasanto).
Regarding Claim 2: Ogawa/Okazaki/Costanzo combination discloses the method of Claim 1 (see the rejection for Claim 1).
Ogawa further discloses:
“further comprising: manipulating the object with a probe that is instrumented with the force sensor” (para 0101 – “The command generation unit 140 a generates an operation procedure required for each work process as an operation command in response to an operation instruction input … The operation instruction is an instruction related to a series of operations of the arm 2 and the hand 200 of the manipulator 20, and held as a computer program, for example”; para 0072 – “The arithmetic processing unit 101 includes a manipulator control unit 110, … and the like. The manipulator control unit 110 includes an arm control unit 120, … and the like”; Claim 18 – “the processor is configured to control, when a motion trajectory of the effector to a target object to be grasped cannot be calculated, the manipulator to vary a positional relationship between the target object and the object handling device” (i.e. manipulate an object with a probe, added by examiner));”);
Ogawa does not specifically disclose:
“wherein the object is to be manipulated by pushing the object on a planar surface”.
However, Colasanto discloses:
“wherein the object is to be manipulated by pushing the object on a planar surface” (para 0024 – “The operator 102 uses his or her finger, hand, arm, or palm with the wearable computer or device 104 thereon for teaching the robot how to process an object 112 on a base or surface 114 of a counter-top, table (i.e. planar surface, added by examiner), or the like”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method disclosed by Ogawa/Okazaki/Costanzo combination, as taught by Colasanto, in order to improve the calibration of the force applied to the object since the use of planar surface makes calibration simpler and more reliable.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Ogawa/Okazaki/Costanzo combination in view of US20120253360 to White et al. (hereinafter White).
Regarding Claim 3: Ogawa/Okazaki/Costanzo combination discloses the method of Claim 1.
Ogawa does not specifically disclose:
“further comprising: receiving video of the object; and determining the motion information of the object based at least in part on the video”.
However, White discloses:
“further comprising: receiving video of the object; and determining the motion information of the object based at least in part on the video” (para 0017 – “The surgical tracker 110 can detect, measure, or otherwise capture information associated with one or more physical variables. In a particular embodiment, the surgical tracker 100 captures video and motion data (i.e. motion information, added by examiner) relating to the surgical tool 102 using a video capture device 112 and a motion capture device 114, respectively. As will be described in further detail below, the motion capture device 114 can capture positional state information (e.g., absolute and relative position, orientation, velocity, acceleration, and jerk in both linear and angular coordinates) and electrical contact information for the surgical tool 102.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method disclosed by Ogawa/Okazaki/Costanzo combination, as taught by White, in order to obtain the independent view of object’s movement and to have the ability to track the object’s motion from different sides of view.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Ogawa/Okazaki/Costanzo combination in view of WO2010136961A1 to Verhaar et al. (hereinafter Verhaar).
Regarding Claim 4: Ogawa/Okazaki/Costanzo combination discloses the method of Claim 1 (see the rejection for Claim 1).
Ogawa is silent on:
“wherein: the motion information of the object includes a linear component and an angular component; and the force on the object is determined to be a vector parallel to the planar surface that the object is being pushed on”.
However, Verhaar discloses:
“wherein: the motion information of the object includes a linear component and an angular component” (page 7 - “at least one of said actuators is adapted for conducting a rotational movement, wherein said position sensor is further adapted for sensing an angular position (i.e. motion information, added by examiner) ”. ); and
“the force on the object is determined to be a vector parallel to the planar surface that the object is being pushed on” (page 14 - “according to the present invention, a motion constraint or physical constraints occurring during the training movement is detected by correlating force vector variations and/or torque vector variations with variations in position, velocity and acceleration vectors or variations in angle, angular velocity and angular acceleration vectors.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method disclosed by Ogawa/Okazaki/Costanzo combination, as taught by Verhaar, in order to predict the force acting on an object, more accurately.
Claims 5 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Ogawa/Okazaki/Costanzo combination in view of Blenkmann et al., 2017, “iElectrodes: A Comprehensive Open-Source Toolbox for Depth and Subdural Grid Electrode Localization”, Front. Neuroinform. Vol.11, Art.14, pp.11-16 (hereinafter Blenkmann).
Regarding Claim 5: Ogawa/Okazaki/Costanzo combination discloses the method of Claim 1 (see the rejection for Claim 1).
Ogawa further discloses:
“further comprising: receiving the signal from the force sensor disposed on the robotic arm” (para 0074 – “The signal processing unit 101 b performs, on various pieces of sensor information, signal processing (i.e. signal receiving, added by examiner) such as signal amplification processing, analog-to-digital conversion processing, noise removal, extraction of feature information, and processing of determining a state change”; para 0068 – “Various sensors (not illustrated) such as a force sensor are provided to the arm 2… various sensors (not illustrated) such as a force sensor are provided to the hand 200”).
Ogawa does not specifically disclose:
“wherein the signal includes a plurality of voxelized electrode signals; and a contact point on the force sensor is determined based at least in part on the voxelized electrode signals; and the one or more machine learning models are to be trained using the signal and an estimate of the force based on the motion information of the object and a model of the object”.
However, Blenkmann discloses:
“wherein: the signal includes a plurality of voxelized electrode signals; and a contact point on the force sensor is determined based at least in part on the voxelized electrode signals” (page 14 – “all voxels corresponding to intracranial electrodes were obtained by masking and thresholding the CT. Then, electrodes coordinates were determined as the center of mass of each cluster of voxels”; page 5 – “We used the selection drawing tool to select the electrode voxels for each electrode array (interpreted as the contact point, added by examiner) in a 3D space (Figure 3A). This procedure is facilitated by rotations, zoom, and pan in the 3D space plot. After selection, all electrode voxels (N voxels) were clustered using a K-means algorithm”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method disclosed by Ogawa/Costanzo combination, as taught by Blenkmann, in order to improve the accuracy of the estimation of force direction.
Ogawa/Okazaki/Blenkmann combination does not specifically disclose:
“the one or more machine learning models are to be trained using the signal and an estimate of the force based on the motion information of the object and a model of the object”.
However, Costanzo discloses:
“the one or more machine learning models are to be trained using the signal and an estimate of the force based on the motion information of the object and a model of the object” (page 2 of 23 – “In the present paper the complete design and calibration of a new and upgraded version of that force/tactile sensor is presented. The new sensor design starts from the main requirement to manipulate objects of generic shape avoiding both linear and rotational slippage… The novel calibration procedure is aimed at enhancing the estimation accuracy especially of the torsional moment component of the contact wrench. It is based on a machine learning approach and, in detail, on the training of a multi-layer feed-forward neural network (FF-NN), and special attention had to be given to the construction of the training set… a suitable trade-off between surface curvature (i.e. model of the object, added by examiner) and extension of the contact surface must be achieved to ensure proper estimation of force tangential components and torsional moment”; page 3 of 23 – “The major objective that can be pursued is to design a sensor as similar as possible to human touch, i.e., able to supply different types of information about the manipulated objects at the same time: A tactile map, an estimate of contact forces and moments… ”; page 4 of 23 – “The whole tactile map allows, after a calibration procedure, to estimate contact force and moment together with information about the orientation of the contact surface and object properties”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method, disclosed by Ogawa/Okazaki/Costanzo/Blenkmann combination, as taught by Costanzo, in order to make the operation of a robotic arm safer and not to destroy the object being manipulated.
Regarding Claim 16: Ogawa/Okazaki/Costanzo combination discloses the non-transitory computer-readable medium of Claim 15 (see the rejection for Claim 15).
Ogawa does not specifically disclose:
“wherein the executable instructions further cause the computer system to: receive, from the force sensor disposed on a probe on the robotic arm, a set of signals that represent a force produced as a result of an interaction with the object, wherein the set of signals is collected from a set of electrodes on the force sensor; and the set of signals is stored as a voxel grid”.
However, Costanzo discloses:
“wherein the executable instructions further cause the computer system to: receive, from the force sensor disposed on a probe on the robotic arm, a set of signals that represent a force produced as a result of an interaction with the object” (page 17 of 23 – “A first validation experiment is aimed at assessing the quality of the reconstruction of the normal force component. The reference force sensor is grasped by the gripper applying a chirp force signal from 0.05Hz to 0.1Hz in 40 s to the fingers. The estimated normal force of a finger (i.e. probe, added by examiner) is then compared to the normal force measured by the reference sensor. The signals and the corresponding error are reported in Figure 15, which shows a maximum error of about 0.7N for a maximum force of 16 N”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method disclosed by Ogawa/Okazaki/Costanzo combination, as taught by Costanzo, in order to make a force prediction more accurate.
Ogawa/Okazaki/Costanzo combination does not specifically disclose:
“the set of signals is collected from a set of electrodes on the force sensor; and the set of signals is stored as a voxel grid”.
However, Blenkmann discloses:
“the set of signals is collected from a set of electrodes on the force sensor; and the set of signals is stored as a voxel grid” (page 15 – “the center of mass coordinate Lk of each electrode (k = 1,2,...,K) is calculated as a weighted average of all voxel coordinates inside each cluster, using the voxels signal intensities as weights (Figure 3C)”; page 6 – “Naming electrodes was the last step for both grid and strips. Each electrode in the array was named using a prefix—number concatenation, for example Grid1, Grid2, Grid3,... GridK in a grid of K electrodes, or an ordered list of labels obtained from the iEEG recording files.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method disclosed by Ogawa/Okazaki/Costanzo combination, as taught by Blenkmann, in order to make a force prediction more accurate.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Ogawa/Okazaki/Costanzo combination in view of Wang, S., et al., 2018, October. 3d shape perception from monocular vision, touch, and shape priors. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 1606-1613). IEEE (hereinafter Wang).
Regarding Claim 6: Ogawa/Okazaki/Costanzo combination discloses the method of Claim 1 (see the rejection for Claim 1).
Ogawa does not specifically disclose:
“further comprising: estimating the force based at least in part on a loss function, wherein the loss function is a function of an angular distance between an imparted force on the object and a surface normal of the object at a contact point”.
However, Wang discloses:
“further comprising: estimating the force based at least in part on a loss function, wherein the loss function is a function of an angular distance between an imparted force on the object and a surface normal of the object at a contact point” (page 1606 – “Touch is another way to perceive 3D shapes. The majority of tactile sensors measure the force distribution or geometry over a small contact area. A robot can use multiple touches, combined with the position and pose of the sensor in each touch, to reconstruct an object’s shape”; page 1609 – “We then present how we update the model’s prediction with tactile signals, after converting them into surface normals, and registering them into the system coordinates. The key observation here is to design a differentiable loss function that enables fine-tuning with back-propagation”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method disclosed by Ogawa/Okazaki/Costanzo combination, as taught by Wang, in order to refine the force data obtained from the sensor used in training.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Ogawa/Okazaki/Costanzo combination in view of Colasanto.
Regarding Claim 7: Ogawa/Okazaki/Costanzo combination discloses the method of Claim 1 (see the rejection for Claim1).
Ogawa further discloses:
“using the force data to generate the one or more machine learning” (para 0084 – “The learning control unit 101 g controls learning functions such as robot model learning for improving operation accuracy such as … grasp control parameter learning for improving a grasp performance (i.e. force data, added by examiner) of the object P, grasp database learning, error detection learning for improving an execution performance of the work plan, and the like.”).
Ogawa does not specifically disclose:
“further comprising: collecting force data as a result of performing planar pushing, ball manipulation, and rigid-object pressing tasks; using the one or more machine learning models to estimate a force on the force sensor produced when performing a different task”
However, Costanzo discloses:
“using the one or more machine learning models to estimate a force on a force sensor produced when performing a different task” (page 2 of 23 – “In the present paper the complete design and calibration of a new and upgraded version of that force/tactile sensor is presented. The new sensor design starts from the main requirement to manipulate objects of generic shape avoiding both linear and rotational slippage… The novel calibration procedure is aimed at enhancing the estimation accuracy especially of the torsional moment component of the contact wrench. It is based on a machine learning approach and, in detail, on the training of a multi-layer feed-forward neural network (FF-NN), and special attention had to be given to the construction of the training set”; page 3 of 23 – “The major objective that can be pursued is to design a sensor as similar as possible to human touch, i.e., able to supply different types of information about the manipulated objects at the same time: A tactile map, an estimate of contact forces and moments… ”; page 4 of 23 – “The whole tactile map allows, after a calibration procedure, to estimate contact force and moment together with information about the orientation of the contact surface and object properties”; page 17 of 23 – “The estimated normal force of a finger is then compared
to the normal force measured by the reference sensor. The signals and the corresponding error are reported in Figure 15, which shows a maximum error of about 0.7N for a maximum force of 16 N.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method, disclosed by Ogawa/Costanzo combination, as taught by Costanzo, in order to make the operation of a robotic arm safer and not to destroy the object being manipulated.
Ogawa/Okazaki/Costanzo combination does not specifically disclose:
“further comprising: collecting force data as a result of performing planar pushing, ball manipulation, and rigid-object pressing tasks”.
However, Colasanto discloses:
“further comprising: collecting force data as a result of performing planar pushing, ball manipulation, and rigid-object pressing tasks” (para 0005 – “The data of teaching commands is selected from a group consisting of an end-effector motion, a grasping motion, strength of grasp, an approaching motion to the object, a lifting motion, a holding motion, a throwing motion, and a waving motion. Examples of sensors are … force sensing”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method disclosed by Ogawa/Okazaki/Costanzo combination, as taught by Colasanto, in order to improve the calibration of the force applied to the object and avoid damaging the object.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Ogawa/Okazaki/Costanzo combination in view of Colasanto and in further view of Okazaki.
Regarding Claim 9: Ogawa/Okazaki/Costanzo combination discloses the one or more processors of claim 8 (see the rejection for Claim 8).
Ogawa further discloses:
“wherein the circuitry is further to: manipulate the object with a probe that is instrumented with the force sensor” (para 0101 – “The command generation unit 140 a generates an operation procedure required for each work process as an operation command in response to an operation instruction input … The operation instruction is an instruction related to a series of operations of the arm 2 and the hand 200 of the manipulator 20, and held as a computer program, for example”; para 0072 – “The arithmetic processing unit 101 includes a manipulator control unit 110, … and the like. The manipulator control unit 110 includes an arm control unit 120, … and the like”; Claim 18 – “the processor is configured to control, when a motion trajectory of the effector to a target object to be grasped cannot be calculated, the manipulator to vary a positional relationship between the target object and the object handling device” (i.e. manipulate an object with a probe, added by examiner));”);
“train the one or more machine learning models using the signal, from the force sensor, generated in response to the force applied to the object, and the estimated force; and use the trained one or more machine learning models to convert the signal to predict a force estimate” (para 0068 – “The manipulator 20 includes an arm 2 and a hand 200… Various sensors (not illustrated) such as a force sensor are provided to the arm 2. ”; para 0101 – “The command generation unit 140 a (i.e. computer system, added by examiner) generates an operation procedure required for each work process as an operation command in response to an operation instruction input from the grasp/operation plan generation unit 101 d (FIG. 2) via the plan generation unit 101 c (FIG. 2). ... The operation instruction is an instruction related to a series of operations of the arm 2 and the hand 200 of the manipulator 20, and held as a computer program, for example.”; para 0065 – “in a case in which the manipulator 20 carries (i.e. manipulates, added by examiner) the object P in the object handling device 1, after the object P is carried in, the conveying mechanism 40 can carry the object P out of the object handling device 1”; para 0139 – “The controller 320 can add, to the grasp information, a grasp score indicating a position and a posture in which the object P can be grasped, ease of grasp, and the like, a pressable amount in grasping (i.e. adjusting manipulation of the object, added by examiner), a threshold for grasp determination, and the like to be stored, based on the input, received information”; Fig. 25A; para 0084 – “The learning control unit 101 g controls learning functions such as robot model learning for improving operation accuracy such as … grasp control parameter learning for improving a grasp performance of the object P, grasp database learning,”).
Ogawa/Okazaki/Costanzo combination does not specifically disclose:
“wherein the object is to be manipulated by moving the object on a planar surface”.
However, Colasanto discloses:
“wherein the object is to be manipulated by moving the object on a planar surface” (para 0024 – “The operator 102 uses his or her finger, hand, arm, or palm with the wearable computer or device 104 thereon for teaching the robot how to process an object 112 on a base or surface 114 of a counter-top, table, or the like”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method disclosed by Ogawa/Costanzo combination, as taught by Colasanto, in order to improve the calibration of the force applied to the object since the use of planar surface makes calibration simpler and more reliable.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Ogawa/Okazaki/Costanzo combination in view of US20170106542 to Wolf et al. (hereinafter Wolf).
Regarding Claim 10: Ogawa/Okazaki/Costanzo combination discloses the one or more processors of claim 8.
Ogawa does not specifically disclose:
“wherein the circuitry is to further predict a force comprising a force magnitude and a force direction relative to the orientation of the force sensor”.
However, Wolf discloses:
“wherein the circuitry is to further predict a force comprising a force magnitude and a force direction relative to the orientation of the force sensor” (para 0021 – “a robot having a plurality of sensors, in particular joint torque and angle sensors, and at least two neural networks receiving the output of said sensors and providing in response a first signal indicative of a force applied to the robot, and a second signal indicative of the direction of said force.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method disclosed by Ogawa/Costanzo combination, as taught by Wolf, in order to improve the efficiency of neural network training operation.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Ogawa/Okazaki/Costanzo combination in view of Gao et al., 2016, Deep Learning for Tactile Understanding From Visual and Haptic Data, 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 536-543 (hereinafter Gao) and in further view of Verhaar.
Regarding Claim 11: Ogawa/Okazaki/Costanzo combination discloses the one or more processors of Claim 8 (see the rejection for Claim 8).
Ogawa does not specifically disclose:
“wherein: the object is a rectangular object; and the motion information of the object includes planar motion along a surface and angular motion of the object”.
However, Gao discloses:
“wherein: the object is a rectangular object” (see Fig. 5B, where the object is of a rectangular shape; page 539 - “The dataset also contains high resolution (3000 x 2000) images of each object from eight different viewpoints (see Fig 5)… For example, the toothpaste box (i.e. rectangular object, added by examiner) is given the classification smooth.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method disclosed by Ogawa/Costanzo combination, as taught by Gao, in order to explore the variety of shapes of objects handled by a robot, to ensure the safe handling of an object.
Both Ogawa/Costanzo combination and Gao are silent on:
“the motion information of the object includes planar motion along a surface and angular motion of the object“.
However, Verhaar discloses:
“the motion information of the object includes planar motion along a surface and angular motion of the object” (page 7 - “at least one of said actuators is adapted for conducting a rotational movement, wherein said position sensor is further adapted for sensing an angular position).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method disclosed by Ogawa/Okazaki/Costanzo/Gao combination, as taught by Verhaar, in order to predict the force acting on an object, more accurately.
Claims 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Ogawa/Okazaki/Costanzo combination in view of Blenkmann and in further view of US20190366539 to Arisoy et al. (hereinafter Arisoy).
Regarding Claim 17: Ogawa/Costanzo combination discloses the non-transitory computer-readable medium of Claim 15 (see the rejection for Claim 15).
Ogawa does not specifically disclose:
“wherein: a contact point on the force sensor disposed on a probe on the robotic arm is stored in a voxel grid; and the voxel grid passes through two layers of 3D convolutions”.
However, Blenkmann discloses:
“wherein: a contact point on the force sensor disposed on a probe on the robotic arm is stored in a voxel grid” (page 14 – “all voxels corresponding to intracranial electrodes were obtained by masking and thresholding the CT. Then, electrodes coordinates were determined as the center of mass of each cluster of voxels”; page 5 – “We used the selection drawing tool to select the electrode voxels for each electrode array (interpreted as the contact point, added by examiner) in a 3D space (Figure 3A). This procedure is facilitated by rotations, zoom, and pan in the 3D space plot. After selection, all electrode voxels (N voxels) were clustered using a K-means algorithm”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method disclosed by Ogawa/Costanzo combination, as taught by Blenkmann, in order to improve the accuracy of the estimation of force direction.
Ogawa/Okazaki/Costanzo/Blenkmann combination is silent on:
“the voxel grid passes through two layers of 3D convolutions”.
However, Arisoy discloses:
“the voxel grid passes through two layers of 3D convolutions” (para 0035 – “the user provided grasping data 146 for each 3D model may similarly be represented in a 3D voxel grids (e.g., having a bounding box with the same shape and orientation as the 3D voxel grid for the corresponding 3D model).”; para 0036 – “in order to train the convolutional neural network 138, an example embodiment may use a deep learning model software tool such as the Caffe open source toolbox. Such a software tool may take as input the training 3D voxel grids 142 comprising the 3D voxel grids of the 3D models 144 of the training objects 206 and the corresponding 3D voxel grids of the grasping data 146 that were collected from multiple users.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method disclosed by Ogawa/Okazaki/Costanzo/Blenkmann combination, as taught by Arisoy, in order to control the force applied by the robotic arm, with the higher level of accuracy.
Regarding Claim 18: Ogawa/Okazaki/Costanzo combination discloses the non-transitory computer-readable medium of Claim 15 (see the rejection for Claim 15).
Ogawa does not specifically disclose:
“wherein the executable instructions further cause the computer system to: apply 3D convolutions to a voxel grid comprising a contact point on the force sensor disposed on the robotic arm, wherein one or more features of the 3D convolutions are flattened and passed through a layer of 2D convolutions; and wherein features of the 2D convolutions are flattened into a vector which is used to estimate the force”.
However, Blenkmann discloses:
“wherein the executable instructions further cause the computer system to: apply 3D convolutions to a voxel grid comprising a contact point on the force sensor disposed on the robotic arm” (page 14 – “all voxels corresponding to intracranial electrodes were obtained by masking and thresholding the CT. Then, electrodes coordinates were determined as the center of mass of each cluster of voxels”; page 5 – “We used the selection drawing tool to select the electrode voxels for each electrode array (interpreted as the contact point, added by examiner) in a 3D space (Figure 3A). This procedure is facilitated by rotations, zoom, and pan in the 3D space plot. After selection, all electrode voxels (N voxels) were clustered using a K-means algorithm”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method disclosed by Ogawa/Costanzo combination, as taught by Blenkmann, in order to improve the accuracy of the estimation of force direction.
Ogawa/Okazaki/Costanzo/Blenkmann combination is silent on:
“wherein one or more features of the 3D convolutions are flattened and passed through a layer of 2D convolutions; and wherein features of the 2D convolutions are flattened into a vector which is used to estimate the force”.
However, Arisoy discloses:
“wherein one or more features of the 3D convolutions are flattened and passed through a layer of 2D convolutions; and wherein features of the 2D convolutions are flattened into a vector which is used to estimate the force” (para 0021 – “3D models 150 may be used by PLM software applications such as Tecnomatix Jack to display through a display device 114 3D visual representations 122 of target objects 120 corresponding to the 3D models 150 in a simulated 3D environment (see view A in FIG. 1). It should also be understood that a 3D representation or 3D visualization of a 3D model of a target object or other components in a 3D environment may be carried out via a 2D display screen by illustrating components in a perspective or orthogonal view. Also, it should be appreciated that alternative embodiments may employ 3D display screens and/or a virtual reality headset to enhance the 3D perception of the 3D visual representation of the 3D model.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method disclosed by Ogawa/Okazaki/Costanzo/Blenkmann combination, as taught by Arisoy, in order to control the force applied by the robotic arm, with the higher level of accuracy.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Ogawa/Okazaki/Costanzo combination in view of Dwivedi et al., 2018, Design, Modeling, and Validation of a Soft Magnetic 3-D Force Sensor, IEEE Sensors Journal, Vol. 18, No.9, pp.3852 – 3863 (hereinafter Dwivedi).
Regarding Claim 19: Ogawa/Okazaki/Costanzo combination discloses the non-transitory computer-readable medium of Claim 15 (see the rejection for Claim 15).
Ogawa further discloses:
“wherein the executable instructions further cause the computer system to: train the one or more machine learning models using a set of signals received from the force sensor and an estimate of the force” (para 0068 – “The manipulator 20 includes an arm 2 and a hand 200… Various sensors (not illustrated) such as a force sensor are provided to the arm 2. ”; para 0101 – “The command generation unit 140 a (i.e. computer system, added by examiner) generates an operation procedure required for each work process as an operation command in response to an operation instruction input from the grasp/operation plan generation unit 101 d (FIG. 2) via the plan generation unit 101 c (FIG. 2). ... The operation instruction is an instruction related to a series of operations of the arm 2 and the hand 200 of the manipulator 20, and held as a computer program, for example.”; para 0065 – “in a case in which the manipulator 20 carries (i.e. manipulates, added by examiner) the object P in the object handling device 1, after the object P is carried in, the conveying mechanism 40 can carry the object P out of the object handling device 1”; para 0139 – “The controller 320 can add, to the grasp information, a grasp score indicating a position and a posture in which the object P can be grasped, ease of grasp, and the like, a pressable amount in grasping (i.e. adjusting manipulation of the object, added by examiner), a threshold for grasp determination, and the like to be stored, based on the input, received information”; Fig. 25A; para 0084 – “The learning control unit 101 g controls learning functions such as robot model learning for improving operation accuracy such as … grasp control parameter learning for improving a grasp performance of the object P, grasp database learning,”).
Ogawa/Okazaki/Costanzo combination does not specifically disclose:
“validate a force magnitude produced by the one or more machine learning models by performing a task of grasping a deformable object; and validate a force vector direction produced by the one or more machine learning models by placing the deformable object onto a planar surface”.
However, Dwivedi discloses:
“validate a force magnitude produced by the one or more machine learning models by performing a task of grasping a deformable object” (p.3853 – “Any force applied on the soft matrix produces a deformation on it”; p.3854 – “We use a neural network (interpreted as a machine learning model, added by examiner) to calibrate the force sensor in 3-D and demonstrate its generalization ability to other materials and loading conditions. We show that the network is able to learn to respond to a large range of force directions whereas only small number of training forces are applied at pre-determined directions”) and
“validate a force vector direction produced by the one or more machine learning models by placing the deformable object onto a planar surface” (p.3861 – “Shown by the period t2 in Figure 10, here the gripper initiates contact with the egg as seen by the increase in FZ . The fingers continue closing until a predefined threshold is reached to produce a normal grasping force between contact surfaces to hold on to the object due to friction, with negligible deformation... Shown by the period t3 in Figure 10, here the arm moves the gripper along with the egg over to a destination location while keeping the gripper position 10 cm above the desk level… Shown by the period t4 in Figure 10, the gripper moves down towards the table. As the egg makes contact with the table top, the shear force decreases. This decrease in shear force allows the arm to recognize that the egg has been placed at the destination spot.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the non-transitory computer-readable storage media, disclosed by Ogawa/Okazaki/Costanzo combination, as taught by Dwivedi, in order to improve the adjustment of the amount of force applied to an object, so the object is not being destroyed.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Ogawa/Okazaki/Costanzo combination in view of Lin et al., 2013, Estimating Point of Contact, Force and Torque in a Biomimetic Tactile Sensor with Deformable Skin, Tech. Rep., pp.123-128 (hereinafter Lin).
Regarding Claim 20: Ogawa/Okazaki/Costanzo combination discloses the non-transitory computer-readable medium of claim 15.
Ogawa/Costanzo combination does not specifically disclose:
“wherein: the force sensor, disposed on a probe on the robotic arm, provides a plurality of force measurements from a plurality of different points on a surface of the sensor; and the force sensor provides information that identifies a location on the surface of the sensor corresponding to each force measurement of the plurality of force measurements”.
However, Lin discloses:
“wherein: the force sensor, disposed on a probe on the robotic arm, provides a plurality of force measurements from a plurality of different points on a surface of the sensor” (p.124 – “Data presenting the location and orientation of the 19 sensing electrodes and four excitation electrodes in Cartesian coordinates are provided in (Table 1).”) ; and
“the force sensor provides information that identifies a location on the surface of the sensor corresponding to each force measurement of the plurality of force measurements” (p.126 – “Results of determining point of contact from the electrode poking test are shown in Figure 7. PoC estimates from the BioTac (blue circles) are clustered around actual electrode locations, while the contact points estimated from the force plate (green circles) are less precise. Table 2 shows the two different trials with their standard deviations from these two methods. In general, using electrode data from the BioTac to determine contact location was found to be more accurate and significantly more precise than using the force plate.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to modify the non-transitory computer-readable storage medium, disclosed by Ogawa/Costanzo combination, as taught by Lin, in order to use the data from different points of such sensor to be able to distinguish the tactile reaction on the force for each area and hence to improve the accuracy of the force determination using the location on a sensor.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Lyudmila Zaykova-Feldman whose telephone number is (469)295-9269. The examiner can normally be reached 8:30am - 5:30pm, Monday through Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Arleen M. Vazquez can be reached on 571-272-2619. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LYUDMILA ZAYKOVA-FELDMAN/Examiner, Art Unit 2857
/LINA CORDERO/Primary Examiner, Art Unit 2857
Prosecution Insights