DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Joint Inventors
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 08/21/2025 has been entered.
Response to Amendment
Claims 1, 13, and 14 have been amended. No claims have been added or cancelled.
Response to Arguments
Applicant's arguments filed 08/21/2025 have been fully considered but they are not persuasive.
Without conceding to Applicant’s characterization of the disclosure in Lee, Examiner has updated the 35 U.S.C. 103 rejection to better discuss the features Applicant considers inventive over Lee in the broader context of methods known in the art of model-based predictive control. As such Applicant’s arguments regarding the current 35 U.S.C. 103 rejection are rendered moot with consideration to the updated 35 U.S.C. 103 rejection presented below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-6, 9-14, 17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (US2014046128, referred to as Lee) in view of Vijayanarasimhan et al. (US9914213, referred to as Vijayanarasimhan) and further in view of Pekar et al. (US8145329, referred to as Pekar)
Regarding claim 1: Lee discloses: A controller ([0048] The master interface 110 of the master robot 100 may include master manipulators 112 and a display unit 114.) for positioning an interventional device ([0060] The slave robot 200 may include the robot arms 202 and the endoscope 210.) the controller comprising: a memory that stores at least one of: ([0062] The storage unit 130 may include a memory device that stores a database 130) a forward predictive model configured with embedded kinematics ([0069] The predictor 143 predicts basic motions to be performed by the operator based on the data regarding a motion detected by the motion data detector 120 and the results of learning performed by the learner 142. In addition, the predictor 143 sequentially connects the predicted basic motions to one another, thereby predicting a surgical task that the operator wishes to perform.) of the interventional device to receive [commanded] positioning motion ([0065] The first control signal generator 141 generates a first control signal to control motions of the slave robot 200, i.e. motions of the main surgical tools 206 and/or the auxiliary surgical tools 208 and 210 based on the data regarding a motion of the master manipulator 112 acquired via the motion data detector 120. [0098] When the operator performs the given surgical motion via the master manipulator 112, the motion data detector 120 detects data regarding the motion of the master manipulator 112, and transmits the detected data to the controller 140 (312).) of the interventional device and to output data ([0104] the predictor 143 transmits the operator motion predicted as described above to the second control signal generator 144 included in the controller 140. [0070] The second control signal generator 144 generates a second control signal to control motions of the slave robot 200 and/or the master manipulators 112 having a redundant degree of freedom (DOF) based on the basic motions or the surgical task predicted by the predictor 143.) related to a prediction of a navigated pose of a device portion of the interventional device based on the [commanded] positioning motion [and the embedded kinematics,] or a ([0076] data regarding the motion of the master manipulator 112 is acquired. A first control signal to control motions of the main surgical tools 206 and/or the auxiliary surgical tools 208 and 210 that the operator wishes to manipulate is generated based on the acquired motion data. [0078] A second control signal to control motions of the slave robot 200 (the main surgical tools 206 and/or the auxiliary surgical tools 208 and 210) and/or the master manipulators 112, may be generated based on the predicted basic motion or surgical task.) [a control predictive model configured with kinematics of the interventional device to receive target pose data of the device portion of the interventional device and to output data related to a prediction of a positioning motion of the interventional device based on the target pose data and the kinematics;] and at least one processor in communication with the memory, ([0064] The controller 140 may include a processor to control general motions of the surgical robot system.) wherein the at least one processor is configured to at least one of: (i) apply the forward predictive model to the [commanded] positioning motion of the interventional device ([0065] The first control signal generator 141 generates a first control signal to control motions of the slave robot 200, i.e. motions of the main surgical tools 206 and/or the auxiliary surgical tools 208 and 210 based on the data regarding a motion of the master manipulator 112 acquired via the motion data detector 120. [0098] When the operator performs the given surgical motion via the master manipulator 112, the motion data detector 120 detects data regarding the motion of the master manipulator 112, and transmits the detected data to the controller 140 (312). The data regarding the motion of the master manipulator 112 may include interactive force data F between the master manipulator 112 and the operator, position data x of the master manipulator 112, and speed data v of the master manipulator 112.) to predict a navigated pose of the device portion, and ([0070] The second control signal generator 144 generates a second control signal to control motions of the slave robot 200 and/or the master manipulators 112 having a redundant degree of freedom (DOF) based on the basic motions or the surgical task predicted by the predictor 143.) generate positioning data to control positioning of the device portion to a target pose [based on the predicted navigated pose of the device portion,] and controlling positioning of the device portion to the target pose; or ([0076] if surgery is initiated and the operator of the master manipulator 112 (e.g., a doctor using the surgical robot system) manipulates the master manipulator 112 to perform a surgical motion, data regarding the motion of the master manipulator 112 is acquired. A first control signal to control motions of the main surgical tools 206 and/or the auxiliary surgical tools 208 and 210 that the operator wishes to manipulate is generated based on the acquired motion data. [0078] A second control signal to control motions of the slave robot 200 (the main surgical tools 206 and/or the auxiliary surgical tools 208 and 210) and/or the master manipulators 112, may be generated based on the predicted basic motion or surgical task. [0082] to control the main surgical tools 206 and/or the auxiliary surgical tools 208 and 210 so as not to invade the particular region of the human body, a corrected first control signal may be generated by combining the first control signal to control motions of the main surgical tools 206 and/or the auxiliary surgical tools 208 and 210 that the operator wishes to manipulate with the second signal generated based on the predicted motion, such that the slave robot 200 is controlled by the corrected first control signal.) [(ii) apply the control predictive model to the target pose data of the device portion to predict a positioning motion of the interventional device, and generate positioning commands to control the positioning of the device portion to the target pose based on the predicted positioning motion of the interventional device, and controlling positioning of the device portion to the target pose.]
Lee does not explicitly disclose: [a control predictive model configured with kinematics of the interventional device to receive target pose data of the device portion of the interventional device and to output data related to a prediction of a positioning motion of the interventional device based on the target pose data and the kinematics;] [(ii) apply the control predictive model to the target pose data of the device portion to predict a positioning motion of the interventional device, and generate positioning commands to control the positioning of the device portion to the target pose based on the predicted positioning motion of the interventional device, and controlling positioning of the device portion to the target pose.]
Lee does not disclose the following limitations, however, Vijayanarasimhan, in an analogous field of endeavor, teaches: a control predictive model configured with kinematics of the interventional device to receive target pose data of the device portion of the interventional device and ([col. 2, lines 36-45] a method is provided that includes: generating a candidate end effector motion vector defining motion to move a grasping end effector of a robot from a current pose to an additional pose; and identifying a current image captured by a vision sensor associated with the robot. The current image captures the grasping end effector and at least one object in an environment of the robot. The method further includes: applying the current image and the candidate end effector motion vector as input to a trained grasp convolutional neural network;) to output data related to a prediction of a positioning motion of the interventional device based on the target pose data and the kinematics; ([col. 2, lines 58-63] The method further includes generating an end effector command based on the measure of successful grasp and the additional measure that indicates whether the desired object semantic feature is present; and providing the end effector command to one or more actuators of the robot.) (ii) apply the control predictive model to the target pose data of the device portion to predict a positioning motion of the interventional device, and ([col. 2, lines 58-63] The method further includes generating an end effector command based on the measure of successful grasp and the additional measure that indicates whether the desired object semantic feature is present; and providing the end effector command to one or more actuators of the robot.) generate positioning commands to control the positioning of the device portion to the target pose based on the predicted positioning motion of the interventional device, and controlling positioning of the device portion to the target pose ([col. 2, lines 36-45] a method is provided that includes: generating a candidate end effector motion vector defining motion to move a grasping end effector of a robot from a current pose to an additional pose; and identifying a current image captured by a vision sensor associated with the robot. The current image captures the grasping end effector and at least one object in an environment of the robot. The method further includes: applying the current image and the candidate end effector motion vector as input to a trained grasp convolutional neural network;)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to modify the positioning controller of Lee to enable the predictive model of Vijayanarasimhan for the purpose of improving positioning command accuracy.
The motivation for modification would have been to provide a predictive model of the end effector positioning for the purpose of improving positioning command accuracy through adjustment or training of the predictive model.
Examiner’s Note: While the Lee reference relies on predictions of operator motion tracking, it would have been obvious to one of ordinary skill in the art to provide the model predictive control of Lee with commanded motions and embedded kinematics as inputs, as is taught in the Pekar reference presented below.
Lee does not explicitly disclose: [commanded] … [embedded kinematics]
Lee does not disclose the following limitations, however Pekar, from an analogous field of endeavor, further teaches: commanded … ([col. 6-7, lines ] The feedforward control 220 can generate a feedforward signal 샃, which is more accurate and reliable and can be utilized in association with a feedback signal μMPC. Note that the feedforward signal 샃 can be externally computed by a nonlinear function or a look-up table. The relations between the system parameters can be determined in order to compute the feedforward signals 샃 for a set of actuators in the on-line part of the control system 200. A state observer 240 can be configured as an unknown input observer to estimate the effect of the feedforward signal 샃. The strategy for manipulating the constraints of the feedback MPC 250 utilizing the constraint manipulation module 300 can be implemented.) embedded kinematics ([col. 7, lines 38-47] The state observer 240 provides an estimated system state and disturbances based on the feedback model based predictive control signal μMPC and the control action μ. In the preferred embodiment, the state observer 240 can be configured as the unknown input observer in order to estimate the effect of the feedforward signal 샃. The state observer 240 generally receives present and/or past values for a number of inputs from the sensors 110, a number of control outputs, and a number of internal variables from the actuators 130 associated with the non-linear plant 230.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to further modify the positioning controller of Lee to enable the specific model inputs taught in Pekar.
The motivation for modification would have been to provide common MPC inputs to the predictive control in Lee to better predict the interventional device positioning.
Regarding claim 2: The combination of Lee, Vijayanarasimhan, and Pekar teaches: The controller of claim 1,
Lee does not explicitly disclose: [wherein the forward predictive model is trained on forward kinematics of the interventional device and the control predictive model is an inverse predictive model trained on inverse kinematics of the interventional device.]
Lee does not disclose the following limitations, however, Vijayanarasimhan further teaches: wherein the forward predictive model is trained on forward kinematics of the interventional device and the control predictive model is an inverse predictive model trained on inverse kinematics of the interventional device. ([col. 2, lines 36-45] a method is provided that includes: generating a candidate end effector motion vector defining motion to move a grasping end effector of a robot from a current pose to an additional pose; and identifying a current image captured by a vision sensor associated with the robot. The current image captures the grasping end effector and at least one object in an environment of the robot. The method further includes: applying the current image and the candidate end effector motion vector as input to a trained grasp convolutional neural network; [col. 23, lines 25-35] At block 752, the system generates a candidate end effector motion vector. The candidate end effector motion vector may be defined in task-space, joint-space, or other space, depending on the input parameters of the trained CNN to be utilized in further blocks. In some implementations, the system generates a candidate end effector motion vector that is random within a given space, such as the work-space reachable by the end effector, a restricted space within which the end effector is confined for the grasp attempts, and/or a space defined by position and/or torque limits of actuator(s) that control the pose of the end effector. [col. 14, lines 16-28] At block 358, the system stores: (1) an image that captures the end effector and the environment at the current instance of the grasp attempt and (2) the pose of the end effector at the current instance. For example, the system may store a current image generated by a vision sensor associated with the robot and associate the image with the current instance (e.g., with a timestamp). Also, for example the system may determine the current pose of the end effector based on data from one or more joint position sensors of joints of the robot whose positions affect the pose of the robot, and the system may store that pose. The system may determine and store the pose of the end effector in task-space, joint-space, or another space. [col. 14, lines 29-48] At block 360, the system determines whether the current instance is the final instance for the grasp attempt. In some implementations, the system may increment an instance counter at block 352, 354, 356, or 358 and/or increment a temporal counter as time passes—and determine if the current instance is the final instance based on comparing a value of the counter to a threshold. For example, the counter may be a temporal counter and the threshold may be 3 seconds, 4 seconds, 5 seconds, and/or other value. In some implementations, the threshold may vary between one or more iterations of the method 300. If the system determines at block 360 that the current instance is not the final instance for the grasping attempt, the system returns to block 356, where it determines and implements another end effector movement, then proceeds to block 358 where it stores an image and the pose at the current instance. Through multiple iterations of blocks 356, 358, and 360 for a given grasp attempt, the pose of the end effector will be altered by multiple iterations of block 356, and an image and the pose stored at each of those instances.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to further modify the positioning controller of Lee with the forward kinematics neural network training of Vijayanarasimhan to improve the positioning accuracy over time with machine learning and neural network training.
The motivation for modification would have been to provide additional training data to the neural network for the purpose of improving the accuracy of the end effector positioning by improving the training of the model.
Regarding claim 3: The combination of Lee, Vijayanarasimhan, and Pekar teaches: The controller of claim 1,
Lee further discloses: wherein the device portion is an end-effector of the interventional device. ([0018] a control method of a surgical robot system, the surgical robot system including a slave robot having a robot arm to which a main surgical tool and an auxiliary surgical tool are coupled, and … adjusting the auxiliary surgical tool so as to correspond to the operator basic motion based on the predicted basic motion.)
Regarding claim 4: The combination of Lee, Vijayanarasimhan, and Pekar teaches: The controller of claim 1,
Lee further discloses: wherein the at least one processor is further configured to generate at least one of the positioning data continuously or the positioning commands continuously ([0018] a control method of a surgical robot system, the surgical robot system including a slave robot having a robot arm to which a main surgical tool and an auxiliary surgical tool are coupled, and … a controller that predicts a basic motion to be performed by an operator based on the motion data detected via the motion data detector and results of learning a plurality of motions constituting a surgical task, and controls the auxiliary surgical tool so as to correspond to the operator basic motion based on the predicted basic motion.)
Regarding claim 5: The combination of Lee, Vijayanarasimhan, and Pekar teaches: The controller of claim 1,
Lee does not explicitly disclose: [wherein the forward predictive model includes: a neural network base having an input layer configured to input joint variables of an interventional robot representative of the commanded positioning motion of the interventional device, and an output layer configured to output at least one of a translation, a rotation, or a pivoting of the device portion derived from a regression of the joint variables of the interventional robot, wherein the at least one processor is further configured to infer the at least one of the translation, the rotation or the pivoting of the device portion based on the predicted navigated pose of the device portion.]
Lee does not disclose the following limitations, however, Vijayanarasimhan further teaches: wherein the forward predictive model includes: a neural network base having an input layer configured to input joint variables ([col.11, lines 5-7] The current pose and the end effector motion vector from the current pose to the final pose of the grasp attempt may be represented in task-space, in joint-space, or in another space ) of an interventional robot representative of the commanded positioning motion of the interventional device, and ([col. 2, lines 36-45] a method is provided that includes: generating a candidate end effector motion vector defining motion to move a grasping end effector of a robot from a current pose to an additional pose; and identifying a current image captured by a vision sensor associated with the robot. The current image captures the grasping end effector and at least one object in an environment of the robot. The method further includes: applying the current image and the candidate end effector motion vector as input to a trained grasp convolutional neural network;) an output layer configured to output ([col. 21, lines 21-24] a convolutional neural network is a multilayer learning framework that includes an input layer, one or more convolutional layers, optional weight and/or other layers, and an output layer.) at least one of a translation, a rotation, or a pivoting of the device portion derived from a regression of the joint variables of the interventional robot, wherein the at least one processor is further configured to infer the at least one of the translation, the rotation or the pivoting of the device portion based on the predicted navigated pose of the device portion. ([col. 2, lines 58-63] The method further includes generating an end effector command based on the measure of successful grasp and the additional measure that indicates whether the desired object semantic feature is present; and providing the end effector command to one or more actuators of the robot. [col. 20, lines 1-18] At block 566, the system performs backpropagation of the semantic CNN and/or the STN based on the grasped object label(s) of the training example. For example, the system may generate, over the semantic CNN based on the applying of block 564, predicted semantic object feature(s) of object(s) present in the spatially transformed image, determine an error based on comparison of the predicted semantic object feature(s) and the grasped object label(s) of the training example, and backpropagate the error through one or more layers of the semantic CNN. In some implementations, the error may further be backpropagated through the STN parameter layer(s) of the grasp CNN (but optionally not through any other layers of the grasp CNN). Backpropagation of the STN and/or the STN parameter layer(s) may enable, over multiple iterations during training, the STN parameters and/or the STN to adapt to cause spatially transformed images to be generated that are of a location/area to be grasped. [col. 10, lines 37-52] each training example includes at least the image observed at that time step (It i), the end effector motion vector (pT i−pt i) from the pose at that time step to the one that is eventually reached (the final pose of the grasp attempt), and the grasp success label (li) and/or grasped object label(s) (oli) of the grasp attempt. Each end effector motion vector may be determined by the end effector motion vector engine 114 of training example generation system 110. For example, the end effector motion vector engine 114 may determine a transformation between the current pose and the final pose of the grasp attempt and use the transformation as the end effector motion vector. The training examples for the plurality of grasp attempts of a plurality of robots are stored by the training example generation system 110 in training examples database 117.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to further modify the positioning controller of Lee to enable the neural network training of the joint position backpropagation training model in Vijayanarasimhan.
The motivation for modification would have been to provide the predictive model with machine learning training to more accurately position the end effector.
Regarding claim 6: The combination of Lee, Vijayanarasimhan, and Pekar teaches: The controller of claim 1,
Lee does not explicitly disclose: [wherein the control predictive model includes: a neural network base having an input layer configured to input at least one of a translation, a rotation or a pivoting of the device portion, and an output layer configured to output joint variables of the interventional robot derived from a regression of least one of the translation, the rotation, or the pivoting of the device portion, wherein the at least one processor is configured to infer the joint variables of the interventional robot infer the predicted positioning motion of the interventional device.]
Lee does not disclose the following limitations, however, Vijayanarasimhan further teaches: wherein the control predictive model includes: a neural network base having an input layer configured to input ([col. 2, lines 43-45] The method further includes: applying the current image and the candidate end effector motion vector as input to a trained grasp convolutional neural network;) at least one of a translation, a rotation, or a pivoting of the device portion, and ([col. 10, lines 37-52] each training example includes at least the image observed at that time step (It i), the end effector motion vector (pT i−pt i) from the pose at that time step to the one that is eventually reached (the final pose of the grasp attempt), and the grasp success label (li) and/or grasped object label(s) (oli) of the grasp attempt. Each end effector motion vector may be determined by the end effector motion vector engine 114 of training example generation system 110. For example, the end effector motion vector engine 114 may determine a transformation between the current pose and the final pose of the grasp attempt and use the transformation as the end effector motion vector. The training examples for the plurality of grasp attempts of a plurality of robots are stored by the training example generation system 110 in training examples database 117.) an output layer configured to output ([col. 21, lines 21-24] a convolutional neural network is a multilayer learning framework that includes an input layer, one or more convolutional layers, optional weight and/or other layers, and an output layer.) joint variables of the interventional robot derived from a regression at least one of the translation, the rotation or the pivoting of the device portion, wherein the at least one processor is configured to infer the joint variables of the interventional robot based on the predicted positioning motion of the interventional device. ([col. 23, lines 25-35] At block 752, the system generates a candidate end effector motion vector. The candidate end effector motion vector may be defined in task-space, joint-space, or other space, depending on the input parameters of the trained CNN to be utilized in further blocks. In some implementations, the system generates a candidate end effector motion vector that is random within a given space, such as the work-space reachable by the end effector, a restricted space within which the end effector is confined for the grasp attempts, and/or a space defined by position and/or torque limits of actuator(s) that control the pose of the end effector. [col. 2, lines 58-63] The method further includes generating an end effector command based on the measure of successful grasp and the additional measure that indicates whether the desired object semantic feature is present; and providing the end effector command to one or more actuators of the robot. [col. 20, lines 1-18] At block 566, the system performs backpropagation of the semantic CNN and/or the STN based on the grasped object label(s) of the training example. For example, the system may generate, over the semantic CNN based on the applying of block 564, predicted semantic object feature(s) of object(s) present in the spatially transformed image, determine an error based on comparison of the predicted semantic object feature(s) and the grasped object label(s) of the training example, and backpropagate the error through one or more layers of the semantic CNN. In some implementations, the error may further be backpropagated through the STN parameter layer(s) of the grasp CNN (but optionally not through any other layers of the grasp CNN). Backpropagation of the STN and/or the STN parameter layer(s) may enable, over multiple iterations during training, the STN parameters and/or the STN to adapt to cause spatially transformed images to be generated that are of a location/area to be grasped. [col. 10, lines 37-52] each training example includes at least the image observed at that time step (It i), the end effector motion vector (pT i−pt i) from the pose at that time step to the one that is eventually reached (the final pose of the grasp attempt), and the grasp success label (li) and/or grasped object label(s) (oli) of the grasp attempt. Each end effector motion vector may be determined by the end effector motion vector engine 114 of training example generation system 110. For example, the end effector motion vector engine 114 may determine a transformation between the current pose and the final pose of the grasp attempt and use the transformation as the end effector motion vector. The training examples for the plurality of grasp attempts of a plurality of robots are stored by the training example generation system 110 in training examples database 117.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to further modify the positioning controller of Lee to enable the neural network input layer of movement of the end effector and output layer of joint position vectors of Vijayanarasimhan.
The motivation for modification would have been to provide machine learning model training to the predictive model to more accurately position the end effector.
Regarding claim 9: The combination of Lee, Vijayanarasimhan, and Pekar teaches: The positioning controller of claim 1,
Lee does not explicitly disclose: [wherein the forward predictive model includes: a neural network base having an input layer configured to input a preceding sequence of shapes of the interventional device representative of the commanded positioning motion of the interventional device, and an output layer configured to output a succeeding sequence of shapes of the interventional device derived from a time series prediction of the preceding sequence of shapes of the interventional device, wherein the at least one processor is configured to infer the succeeding sequence of shapes of the interventional device based on the predicted navigated pose of the device portion.]
Lee does not disclose the following limitations, however, Vijayanarasimhan further teaches: wherein the forward predictive model includes: a neural network base having an input layer configured to input ([col. 2, lines 43-45] The method further includes: applying the current image and the candidate end effector motion vector as input to a trained grasp convolutional neural network;) a preceding sequence of shapes of the interventional device representative of the commanded positioning motion of the interventional device, and ([col. 23, lines 25-35] At block 752, the system generates a candidate end effector motion vector. The candidate end effector motion vector may be defined in task-space, joint-space, or other space, depending on the input parameters of the trained CNN to be utilized in further blocks. In some implementations, the system generates a candidate end effector motion vector that is random within a given space, such as the work-space reachable by the end effector, a restricted space within which the end effector is confined for the grasp attempts, and/or a space defined by position and/or torque limits of actuator(s) that control the pose of the end effector.) an output layer configured to output ([col. 21, lines 21-24] a convolutional neural network is a multilayer learning framework that includes an input layer, one or more convolutional layers, optional weight and/or other layers, and an output layer.) a succeeding sequence of shapes of the interventional device derived from a time series prediction of the preceding sequence of shapes of the interventional device, wherein the at least one processor is configured to infer the succeeding sequence of shapes of the interventional device based on the predicted navigated pose of the device portion. ([col. 14, lines 29-48] At block 360, the system determines whether the current instance is the final instance for the grasp attempt. In some implementations, the system may increment an instance counter at block 352, 354, 356, or 358 and/or increment a temporal counter as time passes—and determine if the current instance is the final instance based on comparing a value of the counter to a threshold. For example, the counter may be a temporal counter and the threshold may be 3 seconds, 4 seconds, 5 seconds, and/or other value. In some implementations, the threshold may vary between one or more iterations of the method 300. If the system determines at block 360 that the current instance is not the final instance for the grasping attempt, the system returns to block 356, where it determines and implements another end effector movement, then proceeds to block 358 where it stores an image and the pose at the current instance. Through multiple iterations of blocks 356, 358, and 360 for a given grasp attempt, the pose of the end effector will be altered by multiple iterations of block 356, and an image and the pose stored at each of those instances. [col. 23, lines 25-35] At block 752, the system generates a candidate end effector motion vector. The candidate end effector motion vector may be defined in task-space, joint-space, or other space, depending on the input parameters of the trained CNN to be utilized in further blocks. In some implementations, the system generates a candidate end effector motion vector that is random within a given space, such as the work-space reachable by the end effector, a restricted space within which the end effector is confined for the grasp attempts, and/or a space defined by position and/or torque limits of actuator(s) that control the pose of the end effector.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to further modify the positioning controller of Lee to enable the neural network motion training of Vijayanarasimhan.
The motivation for modification would have been to provide additional training data to the neural network for the purpose of improving the accuracy of the end effector positioning by improving the training of the model.
Regarding claim 10: The combination of Lee, Vijayanarasimhan, and Pekar teaches: The positioning controller of claim 1,
Lee does not explicitly disclose: [wherein the forward predictive model includes: a neural network base having an input layer configured to input a preceding sequence of shapes of the interventional device representative of the commanded positioning motion of the interventional device, and an output layer configured to output a succeeding shape of the interventional device derived from a time series prediction of the preceding sequence of shapes of the interventional device, wherein the at least one processor is configured to infer the succeeding shape of the interventional device based on the predicted navigated pose of the device portion.]
Lee does not disclose the following limitations, however, Vijayanarasimhan further teaches: wherein the forward predictive model includes: a neural network base having an input layer configured to input ([col. 2, lines 43-45] The method further includes: applying the current image and the candidate end effector motion vector as input to a trained grasp convolutional neural network;) a preceding sequence of shapes of the interventional device representative of the commanded positioning motion of the interventional device, and ([col. 23, lines 25-35] At block 752, the system generates a candidate end effector motion vector. The candidate end effector motion vector may be defined in task-space, joint-space, or other space, depending on the input parameters of the trained CNN to be utilized in further blocks. In some implementations, the system generates a candidate end effector motion vector that is random within a given space, such as the work-space reachable by the end effector, a restricted space within which the end effector is confined for the grasp attempts, and/or a space defined by position and/or torque limits of actuator(s) that control the pose of the end effector.) an output layer configured to output ([col. 21, lines 21-24] a convolutional neural network is a multilayer learning framework that includes an input layer, one or more convolutional layers, optional weight and/or other layers, and an output layer.) a succeeding shape of the interventional device derived from a time series prediction of the preceding sequence of shapes of the interventional device, wherein the at least one processor is configured to infer the succeeding shape of the interventional device based on the predicted navigated pose of the device portion. ([col. 14, lines 29-48] At block 360, the system determines whether the current instance is the final instance for the grasp attempt. In some implementations, the system may increment an instance counter at block 352, 354, 356, or 358 and/or increment a temporal counter as time passes—and determine if the current instance is the final instance based on comparing a value of the counter to a threshold. For example, the counter may be a temporal counter and the threshold may be 3 seconds, 4 seconds, 5 seconds, and/or other value. In some implementations, the threshold may vary between one or more iterations of the method 300. If the system determines at block 360 that the current instance is not the final instance for the grasping attempt, the system returns to block 356, where it determines and implements another end effector movement, then proceeds to block 358 where it stores an image and the pose at the current instance. Through multiple iterations of blocks 356, 358, and 360 for a given grasp attempt, the pose of the end effector will be altered by multiple iterations of block 356, and an image and the pose stored at each of those instances. [col. 23, lines 25-35] At block 752, the system generates a candidate end effector motion vector. The candidate end effector motion vector may be defined in task-space, joint-space, or other space, depending on the input parameters of the trained CNN to be utilized in further blocks. In some implementations, the system generates a candidate end effector motion vector that is random within a given space, such as the work-space reachable by the end effector, a restricted space within which the end effector is confined for the grasp attempts, and/or a space defined by position and/or torque limits of actuator(s) that control the pose of the end effector.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to further modify the positioning controller of Lee to enable the neural network motion training of Vijayanarasimhan.
The motivation for modification would have been to provide additional training data to the neural network for the purpose of improving the accuracy of the end effector positioning by improving the training of the model.
Regarding claim 11: The combination of Lee, Vijayanarasimhan, and Pekar teaches: The positioning controller of claim 2,
Lee does not explicitly disclose: [wherein at least one of: the forward predictive model is further trained on at least one navigation parameter of the interventional device auxiliary to the forward kinematics of the interventional device predictive of the pose of the device portion, wherein the at least one navigation parameter include at least one of a navigation command communicated to the interventional device, an actuation signal communicated to the interventional device, and a navigation force imposed onto the interventional device, or the control predictive model is further trained on the at least one navigation parameter of the interventional device auxiliary to the inverse kinematics of the interventional device predictive of the positioning motion of the interventional device; and wherein the at least one processor is further configured to at least one of: (i') apply the forward predictive model to both the commanded positioning motion of the interventional device and the at least one navigation parameter auxiliary to the forward kinematics of the interventional device to predict the navigated pose of the device portion; or (ii') apply the control predictive model to both the target pose of the device portion and the at least one navigation parameter auxiliary to the inverse kinematics of the interventional device to predict the positioning motion of the interventional device.]
Lee does not disclose the following limitations, however, Vijayanarasimhan further teaches: wherein at least one of: the forward predictive model is further trained on at least one navigation parameter of the interventional device auxiliary to the forward kinematics of the interventional device predictive of the pose of the device portion, wherein the at least one navigation parameter include at least one of a navigation command communicated to the interventional device, an actuation signal communicated to the interventional device, and a navigation force imposed onto the interventional device, or the control predictive model is further trained on the at least one navigation parameter of the interventional device auxiliary to the inverse kinematics of the interventional device predictive of the positioning motion of the interventional device; and ([col. 2, lines 36-45] a method is provided that includes: generating a candidate end effector motion vector defining motion to move a grasping end effector of a robot from a current pose to an additional pose; and identifying a current image captured by a vision sensor associated with the robot. The current image captures the grasping end effector and at least one object in an environment of the robot. The method further includes: applying the current image and the candidate end effector motion vector as input to a trained grasp convolutional neural network; [col. 23, lines 25-35] At block 752, the system generates a candidate end effector motion vector. The candidate end effector motion vector may be defined in task-space, joint-space, or other space, depending on the input parameters of the trained CNN to be utilized in further blocks. In some implementations, the system generates a candidate end effector motion vector that is random within a given space, such as the work-space reachable by the end effector, a restricted space within which the end effector is confined for the grasp attempts, and/or a space defined by position and/or torque limits of actuator(s) that control the pose of the end effector. [col. 14, lines 29-48] At block 360, the system determines whether the current instance is the final instance for the grasp attempt. In some implementations, the system may increment an instance counter at block 352, 354, 356, or 358 and/or increment a temporal counter as time passes—and determine if the current instance is the final instance based on comparing a value of the counter to a threshold. For example, the counter may be a temporal counter and the threshold may be 3 seconds, 4 seconds, 5 seconds, and/or other value. In some implementations, the threshold may vary between one or more iterations of the method 300. If the system determines at block 360 that the current instance is not the final instance for the grasping attempt, the system returns to block 356, where it determines and implements another end effector movement, then proceeds to block 358 where it stores an image and the pose at the current instance. Through multiple iterations of blocks 356, 358, and 360 for a given grasp attempt, the pose of the end effector will be altered by multiple iterations of block 356, and an image and the pose stored at each of those instances. [col. 14, lines 16-28] At block 358, the system stores: (1) an image that captures the end effector and the environment at the current instance of the grasp attempt and (2) the pose of the end effector at the current instance. For example, the system may store a current image generated by a vision sensor associated with the robot and associate the image with the current instance (e.g., with a timestamp). Also, for example the system may determine the current pose of the end effector based on data from one or more joint position sensors of joints of the robot whose positions affect the pose of the robot, and the system may store that pose. The system may determine and store the pose of the end effector in task-space, joint-space, or another space.) wherein the at least one processor is further configured to at least one of: (i') apply the forward predictive model to both the commanded positioning motion of the interventional device and the at least one navigation parameter auxiliary to the forward kinematics of the interventional device to predict the navigated pose of the device portion; or (ii') apply the control predictive model to both the target pose of the device portion and the at least one navigation parameter auxiliary to the inverse kinematics of the interventional device to predict the positioning motion of the interventional device. ([col. 5, lines 1-4] The method further includes training, by one or more processors, a semantic convolutional neural network based on the training examples. [col. 2, lines 36-45] a method is provided that includes: generating a candidate end effector motion vector defining motion to move a grasping end effector of a robot from a current pose to an additional pose; and identifying a current image captured by a vision sensor associated with the robot. The current image captures the grasping end effector and at least one object in an environment of the robot. The method further includes: applying the current image and the candidate end effector motion vector as input to a trained grasp convolutional neural network; [col. 14, lines 16-28] At block 358, the system stores: (1) an image that captures the end effector and the environment at the current instance of the grasp attempt and (2) the pose of the end effector at the current instance. For example, the system may store a current image generated by a vision sensor associated with the robot and associate the image with the current instance (e.g., with a timestamp). Also, for example the system may determine the current pose of the end effector based on data from one or more joint position sensors of joints of the robot whose positions affect the pose of the robot, and the system may store that pose. The system may determine and store the pose of the end effector in task-space, joint-space, or another space.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to further modify the positioning controller of Lee with the forward kinematics neural network training of Vijayanarasimhan to improve the positioning accuracy over time with machine learning and neural network training.
The motivation for modification would have been to provide additional training data to the neural network for the purpose of improving the accuracy of the end effector positioning by improving the training of the model.
Regarding claim 12: The combination of Lee, Vijayanarasimhan, and Pekar teaches: The positioning controller of claim 1,
Lee does not explicitly disclose: [wherein at least one of: the forward predictive model is configured to further receive at least one auxiliary navigation parameter of the interventional device, and further process it to output the prediction of navigated pose of the device portion, wherein the at least one auxiliary navigation parameter include at least one of an image of the interventional device, shape of the interventional device, strain of the interventional device, twist of the interventional device, or temperature of the interventional device, or the control predictive model is configured to further receive at least one auxiliary navigation parameter of the interventional device, and further process it to output the prediction of positioning motion of the interventional device; and wherein the at least one processor is configured to at least one of: (i') apply the forward predictive model to both the commanded positioning motion of the interventional device and at least one auxiliary navigation parameter to render the predicted navigated pose of the device portion; or (ii') apply the control predictive model to both the target pose of the device portion and at least one auxiliary navigation parameter to render the predicted positioning motion of the interventional device.]
Lee does not disclose the following limitations, however, Vijayanarasimhan further teaches: wherein at least one of: the forward predictive model is configured to further receive at least one auxiliary navigation parameter of the interventional device, and further process it to output the prediction of navigated pose of the device portion, wherein the at least one auxiliary navigation parameter include at least one of an image of the interventional device, shape of the interventional device, strain of the interventional device, twist of the interventional device, or temperature of the interventional device, or the control predictive model is configured to further receive at least one auxiliary navigation parameter of the interventional device, and further process it to output the prediction of positioning motion of the interventional device; and ([col. 2, lines 36-45] a method is provided that includes: generating a candidate end effector motion vector defining motion to move a grasping end effector of a robot from a current pose to an additional pose; and identifying a current image captured by a vision sensor associated with the robot. The current image captures the grasping end effector and at least one object in an environment of the robot. The method further includes: applying the current image and the candidate end effector motion vector as input to a trained grasp convolutional neural network; [col. 23, lines 25-35] At block 752, the system generates a candidate end effector motion vector. The candidate end effector motion vector may be defined in task-space, joint-space, or other space, depending on the input parameters of the trained CNN to be utilized in further blocks. In some implementations, the system generates a candidate end effector motion vector that is random within a given space, such as the work-space reachable by the end effector, a restricted space within which the end effector is confined for the grasp attempts, and/or a space defined by position and/or torque limits of actuator(s) that control the pose of the end effector. [col. 14, lines 29-48] At block 360, the system determines whether the current instance is the final instance for the grasp attempt. In some implementations, the system may increment an instance counter at block 352, 354, 356, or 358 and/or increment a temporal counter as time passes—and determine if the current instance is the final instance based on comparing a value of the counter to a threshold. For example, the counter may be a temporal counter and the threshold may be 3 seconds, 4 seconds, 5 seconds, and/or other value. In some implementations, the threshold may vary between one or more iterations of the method 300. If the system determines at block 360 that the current instance is not the final instance for the grasping attempt, the system returns to block 356, where it determines and implements another end effector movement, then proceeds to block 358 where it stores an image and the pose at the current instance. Through multiple iterations of blocks 356, 358, and 360 for a given grasp attempt, the pose of the end effector will be altered by multiple iterations of block 356, and an image and the pose stored at each of those instances.) wherein the at least one processor is configured to at least one of: (i') apply the forward predictive model to both the commanded positioning motion of the interventional device and at least one auxiliary navigation parameter to predict the navigated pose of the device portion; or (ii') apply the control predictive model to both the target pose of the device portion and at least one auxiliary navigation parameter to predict the positioning motion of the interventional device. ([col. 5, lines 1-4] The method further includes training, by one or more processors, a semantic convolutional neural network based on the training examples. [col. 2, lines 36-45] a method is provided that includes: generating a candidate end effector motion vector defining motion to move a grasping end effector of a robot from a current pose to an additional pose; and identifying a current image captured by a vision sensor associated with the robot. The current image captures the grasping end effector and at least one object in an environment of the robot. The method further includes: applying the current image and the candidate end effector motion vector as input to a trained grasp convolutional neural network;)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to further modify the positioning controller of Lee to enable the neural network motion parameter training of Vijayanarasimhan.
The motivation for modification would have been to provide additional training data to the neural network for the purpose of improving the accuracy of the end effector positioning by improving the training of the model.
Regarding claim 13: Rejected using the same rationale as claim 1.
Regarding claim 14: Rejected using the same rationale as claim 1.
Regarding claim 17: Rejected using the same rationale as claim 9.
Regarding claim 20: Rejected using the same rationale as claim 9 and 17.
Claims 7-8, 15-16, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (US2014046128, referred to as Lee) in view of Vijayanarasimhan et al. (US9914213, referred to as Vijayanarasimhan), further in view of Pekar et al. (US8145329, referred to as Pekar), and further in view of Phung et al. (‘Data based Kinematic Model of a Multi-Flexible-Link Robot Arm for Varying Payloads’, referred to as Phung).
Regarding claim 7: The combination of Lee, Vijayanarasimhan, and Pekar teaches: The positioning controller of claim 1,
Lee does not explicitly disclose: [wherein the forward predictive model includes: a neural network base having an input layer configured to input joint velocities of the interventional device representative of the commanded positioning motion of the interventional device, and an output layer configured to output at least one of a linear velocity or an angular velocity of the device portion from a regression of the joint velocities of the interventional device, wherein at least one of a linear velocity or an angular velocity of the device portion infers the predicted navigated pose of the interventional device.]
Lee does not disclose the following limitations, however, Vijayanarasimhan further teaches: wherein the forward predictive model includes: a neural network base having an input layer configured to input ([col. 2, lines 43-45] The method further includes: applying the current image and the candidate end effector motion vector as input to a trained grasp convolutional neural network;) joint [velocities] of the interventional device representative of the commanded positioning motion of the interventional device, ([col. 2, lines 36-45] a method is provided that includes: generating a candidate end effector motion vector defining motion to move a grasping end effector of a robot from a current pose to an additional pose; and identifying a current image captured by a vision sensor associated with the robot. The current image captures the grasping end effector and at least one object in an environment of the robot. The method further includes: applying the current image and the candidate end effector motion vector as input to a trained grasp convolutional neural network;) and an output layer configured to output ([col. 21, lines 21-24] a convolutional neural network is a multilayer learning framework that includes an input layer, one or more convolutional layers, optional weight and/or other layers, and an output layer.) [at least one of a linear velocity or an angular velocity] of the device portion from a regression ([col. 2, lines 58-63] The method further includes generating an end effector command based on the measure of successful grasp and the additional measure that indicates whether the desired object semantic feature is present; and providing the end effector command to one or more actuators of the robot. [col. 20, lines 1-18] At block 566, the system performs backpropagation of the semantic CNN and/or the STN based on the grasped object label(s) of the training example. For example, the system may generate, over the semantic CNN based on the applying of block 564, predicted semantic object feature(s) of object(s) present in the spatially transformed image, determine an error based on comparison of the predicted semantic object feature(s) and the grasped object label(s) of the training example, and backpropagate the error through one or more layers of the semantic CNN. In some implementations, the error may further be backpropagated through the STN parameter layer(s) of the grasp CNN (but optionally not through any other layers of the grasp CNN). Backpropagation of the STN and/or the STN parameter layer(s) may enable, over multiple iterations during training, the STN parameters and/or the STN to adapt to cause spatially transformed images to be generated that are of a location/area to be grasped. [col. 10, lines 37-52] each training example includes at least the image observed at that time step (It i), the end effector motion vector (pT i−pt i) from the pose at that time step to the one that is eventually reached (the final pose of the grasp attempt), and the grasp success label (li) and/or grasped object label(s) (oli) of the grasp attempt. Each end effector motion vector may be determined by the end effector motion vector engine 114 of training example generation system 110. For example, the end effector motion vector engine 114 may determine a transformation between the current pose and the final pose of the grasp attempt and use the transformation as the end effector motion vector. The training examples for the plurality of grasp attempts of a plurality of robots are stored by the training example generation system 110 in training examples database 117.) of the joint [velocities] of the interventional device, wherein [at least one of a linear velocity or an angular velocity] of the device portion infers the predicted navigated pose of the interventional device. ([col. 2, lines 36-45] a method is provided that includes: generating a candidate end effector motion vector defining motion to move a grasping end effector of a robot from a current pose to an additional pose; and identifying a current image captured by a vision sensor associated with the robot. The current image captures the grasping end effector and at least one object in an environment of the robot. The method further includes: applying the current image and the candidate end effector motion vector as input to a trained grasp convolutional neural network;)
Examiners Note: The combination of Lee, Vijayanarasimhan, and Pekar teaches an input layer and output layer that exactly resemble that of the instant application with the exception of the kinematic values used as input. The combination of Lee, Vijayanarasimhan, and Pekar teaches a neural network that trains a translational motion vector, the training of a neural network with linear/angular joint velocities instead is taught by Phung.
The combination of Lee, Vijayanarasimhan, and Pekar teaches: does not explicitly teach: velocities; at least one of a linear velocity and an angular velocity; velocities; at least one of a linear velocity and an angular velocity;
The combination of Lee, Vijayanarasimhan, and Pekar teaches: does not teach the following limitations, however, Phung, in an analogous field of endeavor, teaches: velocities; at least one of a linear velocity and an angular velocity; velocities; at least one of a linear velocity or an angular velocity; ([pg. 1259, col. 1, lines 4-18] MLP models with strain-based regression and nonlinear features are chosen for the both forward and inverse kinematic model because they achieve the lowest remapping errors of the inverse kinematic problem shown in table II. In open loop pose control the joint reference angles for a dynamic load are predicted by the inverse kinematic model according to the desired end effector pose in task space and the strain gauge signal. Errors in the inverse kinematic model are not detected, thus the overall task space accuracy of about 2 − 4 mm is determined by the remapping error reported in table II. Fig. 4 illustrates the structure of the open loop controller. The joint axis controller exhibits a cascaded structure in which the inner PI controller regulates the joint velocity and the outer loop PD controller regulates the joint position. [pg. 1257, col 1, lines 40-42] The linear and nonlinear models are trained to predict the end-effector positions of the flexible link arm subject to gravitation from its joint values (forward kinematics).)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to further modify Lee so that the joint position neural network (following the previous modification using Vijayanarasimhan and Pekar) includes the forward kinematics neural network training utilizing joint velocities of Phung to improve the positioning accuracy over time with machine learning and neural network training by providing another kinematic variable to the calculations.
The motivation for modification would have been to provide additional training data to the neural network for the purpose of improving the accuracy of the end effector positioning by improving the training of the model.
Regarding claim 8: The combination of Lee, Vijayanarasimhan, and Pekar teaches: The positioning controller of claim 1,
Lee does not explicitly disclose: [wherein the control predictive model includes: a neural network base having an input layer configured to input at least one of a linear velocity or an angular velocity of the device portion to the target pose and an output layer configured to output joint velocities of the interventional device from a regression at least one of a linear velocity or an angular velocity of the device portion to the target pose, wherein the joint velocities of the interventional device infer the predicted positioning motion of the interventional device.]
Lee does not disclose the following limitations, however Vijayanarasimhan teaches: wherein the control predictive model includes: a neural network base having an input layer configured to input ([col. 2, lines 43-45] The method further includes: applying the current image and the candidate end effector motion vector as input to a trained grasp convolutional neural network;) at least one of a [linear velocity and an angular velocity] of the device portion to the target pose ([col. 2, lines 36-45] a method is provided that includes: generating a candidate end effector motion vector defining motion to move a grasping end effector of a robot from a current pose to an additional pose; and identifying a current image captured by a vision sensor associated with the robot. The current image captures the grasping end effector and at least one object in an environment of the robot. The method further includes: applying the current image and the candidate end effector motion vector as input to a trained grasp convolutional neural network;) and an output layer configured to output ([col. 21, lines 21-24] a convolutional neural network is a multilayer learning framework that includes an input layer, one or more convolutional layers, optional weight and/or other layers, and an output layer.) joint [velocities] of the interventional device from a regression of at least one of a [linear velocity or an angular velocity] of the device portion to the target pose, ([col. 2, lines 58-63] The method further includes generating an end effector command based on the measure of successful grasp and the additional measure that indicates whether the desired object semantic feature is present; and providing the end effector command to one or more actuators of the robot. [col. 20, lines 1-18] At block 566, the system performs backpropagation of the semantic CNN and/or the STN based on the grasped object label(s) of the training example. For example, the system may generate, over the semantic CNN based on the applying of block 564, predicted semantic object feature(s) of object(s) present in the spatially transformed image, determine an error based on comparison of the predicted semantic object feature(s) and the grasped object label(s) of the training example, and backpropagate the error through one or more layers of the semantic CNN. In some implementations, the error may further be backpropagated through the STN parameter layer(s) of the grasp CNN (but optionally not through any other layers of the grasp CNN). Backpropagation of the STN and/or the STN parameter layer(s) may enable, over multiple iterations during training, the STN parameters and/or the STN to adapt to cause spatially transformed images to be generated that are of a location/area to be grasped. [col. 10, lines 37-52] each training example includes at least the image observed at that time step (It i), the end effector motion vector (pT i−pt i) from the pose at that time step to the one that is eventually reached (the final pose of the grasp attempt), and the grasp success label (li) and/or grasped object label(s) (oli) of the grasp attempt. Each end effector motion vector may be determined by the end effector motion vector engine 114 of training example generation system 110. For example, the end effector motion vector engine 114 may determine a transformation between the current pose and the final pose of the grasp attempt and use the transformation as the end effector motion vector. The training examples for the plurality of grasp attempts of a plurality of robots are stored by the training example generation system 110 in training examples database 117.) wherein the joint [velocities] of the interventional device infer the predicted positioning motion of the interventional device. ([col. 10, lines 45-53] That is, each training example includes at least the image observed at that time step (It i), the end effector motion vector (pT i−pt i) from the pose at that time step to the one that is eventually reached (the final pose of the grasp attempt), and the grasp success label (li) and/or grasped object label(s) (oli) of the grasp attempt. Each end effector motion vector may be determined by the end effector motion vector engine 114 of training example generation system 110. For example, the end effector motion vector engine 114 may determine a transformation between the current pose and the final pose of the grasp attempt and use the transformation as the end effector motion vector)
Lee does not explicitly teach: velocities; at least one of a linear velocity and an angular velocity; velocities; at least one of a linear velocity and an angular velocity;
Lee does not teach the following limitations, however, Phung, in an analogous field of endeavor, teaches: velocities; at least one of a linear velocity and an angular velocity; velocities; at least one of a linear velocity or an angular velocity; ([pg. 1259, col. 1, lines 4-18] MLP models with strain-based regression and nonlinear features are chosen for the both forward and inverse kinematic model because they achieve the lowest remapping errors of the inverse kinematic problem shown in table II. In open loop pose control the joint reference angles for a dynamic load are predicted by the inverse kinematic model according to the desired end effector pose in task space and the strain gauge signal. Errors in the inverse kinematic model are not detected, thus the overall task space accuracy of about 2 − 4 mm is determined by the remapping error reported in table II. Fig. 4 illustrates the structure of the open loop controller. The joint axis controller exhibits a cascaded structure in which the inner PI controller regulates the joint velocity and the outer loop PD controller regulates the joint position. [pg. 1257, col 1, lines 40-42] The linear and nonlinear models are trained to predict the end-effector positions of the flexible link arm subject to gravitation from its joint values (forward kinematics).)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to further modify Lee so that the joint position neural network (following the previous modification using Vijayanarasimhan and Pekar) includes the forward kinematics neural network training utilizing joint velocities of Phung to improve the positioning accuracy over time with machine learning and neural network training by providing another kinematic variable to the calculations.
The motivation for modification would have been to provide additional training data to the neural network for the purpose of improving the accuracy of the end effector positioning by improving the training of the model.
Regarding claim 15: Rejected using the same rationale as claim 7.
Regarding claim 16: Rejected using the same rationale as claim 8.
Regarding claim 18: Rejected using the same rationale as claim 7 and 15.
Regarding claim 19: Rejected using the same rationale as claim 8 and 16.
Conclusion
The prior art made of record, and not relied upon, considered pertinent to applicant' s disclosure or directed to the state of art is listed on the enclosed PTO-892.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ATTICUS A CAMERON whose telephone number is 703-756-4535. The examiner can normally be reached M-F 8:30 am - 4:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Thomas Worden can be reached on 571-272-4876. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ATTICUS A CAMERON/ /JASON HOLLOWAY/ Primary Examiner, Art Unit 3658 Examiner, Art Unit 3658A
Prosecution Insights