DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1-4 and 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Fujitsu et al. (JP05088721), hereinafter referred to as ‘Fujitsu’ and in further view of Tanabiki et al. (US20130230211), hereinafter referred to as ‘Tanabiki’ and Tremblay et al. (US 20200311855), hereinafter referred to as ‘Tremblay’.
Regarding Claim 1, Fujitsu discloses a training data generation device for generating training data for a machine learning device to generate a trained model (In the case of controlling an articulated robot, information obtained in a visual coordinate system is often used. This visual coordinate is generally represented in 2 or 3 dimensional orthogonal coordinates. However, in order to control the articulated robot using this rectangular coordinate, it is necessary to convert the orthogonal coordinate into the angular coordinate of each joint. [0002];FIG. 5 is a diagram showing another example of learning performed by the learning apparatus. In FIG. 5, the input signal of the learning data indicates the spatial coordinates and the spatial coordinates of the 2 cameras. [0028]), receiving input of a two-dimensional image of a robot captured by a camera (In Figure 1(a), the learning device 1 is provided with position information output by the work area measuring device 2 as an input signal and angle information output by the joint angle measuring device 3 as a teacher signal [0021]) and outputting a two-dimensional posture indicating positions of joint axes included in the robot at a time when the two- dimensional image was captured (In the case of controlling an articulated robot, information obtained in a visual coordinate system is often used. This visual coordinate is generally represented in 2 or 3 dimensional orthogonal coordinates. However, in order to control the articulated robot using this rectangular coordinate, it is necessary to convert the orthogonal coordinate into the angular coordinate of each joint. [0002]; By repeating the above learning process, the learning device 1 will learn to output angle information similar to the angle information from the joint angle measuring device 3 in response to input position information [0021]) and receiving input of the two- dimensional posture (FIG. 1 is an explanatory view of the principle of the present invention. In FIG. 1 a, the learning device 1 is given as an input signal as the position information output from the working site measuring device 2, and is given the angle information output from the joint angle measuring device 3 as a teacher signal. [0021]; …The learning devices 19-22 are trained to output angle information that each joint of the articulated robot should take when given positional information about the corresponding space. Given the Cartesian coordinates of the end-effector position, a learning device adapted to that end-effector position performs a conversion to angular coordinates [0029]), the training data generation device comprising (FIG. 3 is a diagram showing an example of learning performed by the learning apparatus. In FIG. 3, an input signal of learning data is generated based on image information of 2 cameras. Further, the teacher signal of the learning data is each joint angle of a robot having joints. By image processing of image information of the camera, an X coordinate and a Y coordinate at the center of gravity of an object and an inclination angle a and an area S are obtained [0025]), an input data acquisition unit configured to acquire a plurality of the two-dimensional image of the robot captured by the camera at various distances by changing the posture and direction of the robot, as input data of the training data (The position and orientation of the 2 cameras 15,16 and are measured by an angle encoder, i.e., label acquisition unit, or the like, and the position information of the camera, i.e., input data acquisition unit, and the position information of the object 17 based on the image information obtained by the camera are combined to obtain the input signal of the learning data. On the other hand, a teacher signal of the learning data is obtained from the information of the joint angle obtained by the measuring device such as an angle encoder provided at the joint part of the articulated robot 14 [0027]); and a label acquisition unit configured to acquire angles of the plurality of joint axes from the robot at the time when each of the plurality of the two-dimensional image was captured, and a plurality of the two-dimensional posture as label data of the training data (The position and orientation of the 2 cameras 15,16 and are measured by an angle encoder, i.e., label acquisition unit, or the like, and the position information of the camera, i.e., input data acquisition unit, and the position information of the object 17 based on the image information obtained by the camera are combined to obtain the input signal of the learning data. On the other hand, a teacher signal of the learning data is obtained from the information of the joint angle obtained by the measuring device such as an angle encoder provided at the joint part of the articulated robot 14 [0027]).
However, Fujitsu does not explicitly disclose the trained model including a two-dimensional skeleton estimation model receiving input of a two-dimensional image of a robot captured by a camera, and outputting a two-dimensional posture indicating positions of centers of a plurality of joint axes included in the robot at a time when the two- dimensional image was captured, and an estimation model receiving input of the two- dimensional posture outputted from the two-dimensional skeleton estimation model, a joint angle estimation model receiving input of the two- dimensional posture outputted from the two-dimensional skeleton estimation model and a distance and a tilt between the camera and the robot and outputting angles of the plurality of joint axes and an input data acquisition unit configured to acquire a plurality of the two-dimensional image of the robot captured by the camera at various distances and with various tilts in advance by changing the posture and direction of the robot, and the distances and the tilts between the camera and the robot as input data of the training data.
Nevertheless, Tanabiki discloses a two-dimensional skeleton estimation model receiving input of a two-dimensional image of a robot captured by a camera (Similar skeleton model storing section 254 generates a 2D skeleton model from a 3D skeleton model, using the camera parameters stored in image storing section 220. The camera parameters include camera exterior parameters and camera interior parameters. The camera exterior parameters are the position coordinates of the camera in a world coordinate system (the position of the origin point of the camera coordinate system), and the pose (xyz rotation angle) of the camera based on the world coordinate system. The camera interior parameters include the focal length of the camera, the vertical angle of view of the camera, the width of a 2D image projected on a screen, and the height of the 2D image projected on the screen [0109]), and outputting a two-dimensional posture indicating positions of centers of a plurality of joint axes included in the robot at a time when the two- dimensional image was captured (A method for estimating a joint position will be described in detail later. Skeleton estimating section 251 outputs skeleton information, i.e., information on the estimated positions of the joints to skeleton model description converting section 252. If the joint position can be determined, the position of the bone joining the joints, i.e., plurality of joint axes, can also be determined [0094]), and an estimation model receiving input of the two- dimensional posture outputted from the two-dimensional skeleton estimation model (Skeleton estimating section 251 outputs skeleton information, i.e., information on the estimated positions of the joints to skeleton model description converting section 252. If the joint position can be determined, the position of the bone joining the joints can also be determined. Estimation of the joint position is the same as the estimation of the skeleton model [0094]), a joint angle estimation model receiving input of the two- dimensional posture outputted from the two-dimensional skeleton estimation model (Skeleton estimating section 251 outputs skeleton information, i.e., information on the estimated positions of the joints to skeleton model description converting section 252. If the joint position can be determined, the position of the bone joining the joints can also be determined. Estimation of the joint position is the same as the estimation of the skeleton model [0094]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Fujitsu with the teachings of Tanabiki to enable an accurate determination of camera-to-robot distance and orientation and improve accuracy.
However, Fujitsu and Tanabiki does not explicitly disclose a joint angle estimation model receiving input of the two- dimensional posture outputted from the two-dimensional skeleton estimation model and a distance and a tilt between the camera and the robot and outputting angles of the plurality of joint axes and an input data acquisition unit configured to acquire a plurality of the two-dimensional image of the robot captured by the camera at various distances and with various tilts in advance by changing the posture and direction of the robot, and the distances and the tilts between the camera and the robot as input data of the training data.
Nevertheless, Tremblay discloses receiving input of distance and a tilt between the camera and the robot and outputting angles of the plurality of joint axes ( In another embodiment, the first pose of the target object with respect to the camera may include a three-dimensional (3D) rotation, i.e., tilt, and translation, i.e., distance, of the target object with respect to the camera [0021]) and an input data distance and with tilt in advance by changing the posture and direction of the robot, and the distance and the tilt between the camera and the robot as input data (In another embodiment, the first pose of the target object with respect to the camera may include a three-dimensional (3D) rotation, i.e., tilt, and translation, i.e., position, of the target object with respect to the camera [0021]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Fujitsu and Tanabiki with the teachings of Tremblay to enable an accurate determination of camera-to-robot distance and orientation and improve accuracy.
Regarding Claim 2, Fujitsu, Tanabiki, and Tremblay disclose the claimed invention discussed in claim 1.
Fujitsu discloses to execute supervised learning based on input data and label data of training data generated by the training data generation device generating a trained model (The position and attitude of the 2 cameras 15,16 and are measured by an angle encoder, i.e., label data, or the like, and the position information of the camera, i.e., input data, and the position information of the object 17 based on the image information obtained by the camera are combined to obtain the input signal of the learning data. On the other hand, a teacher signal of the learning data is obtained from the information of the joint angle obtained by the measuring device such as an angle encoder provided at the joint part of the articulated robot 14 [0027]).
Regarding Claim 3, Fujitsu, Tanabiki, and Tremblay disclose the claimed invention discussed in claim 2.
Fujitsu discloses a training data generation device (as discussed above), the training data generation device being for generating training data for the machine learning device to generate a trained model (as discussed above), the training data generation device comprising: an input data acquisition unit configured to acquire a plurality of the two-dimensional image of the robot captured by the camera at various distances(The position and orientation of the 2 cameras 15,16 and are measured by an angle encoder, i.e., label acquisition unit, or the like, and the position information of the camera, i.e., input data acquisition unit, and the position information, i.e., distances, of the object 17 based on the image information obtained by the camera are combined to obtain the input signal of the learning data. On the other hand, a teacher signal of the learning data is obtained from the information of the joint angle obtained by the measuring device such as an angle encoder provided at the joint part of the articulated robot 14 [0027]); and a label acquisition unit configured to acquire a plurality of the angles of the plurality of joint axes from the robot at the time when each of the plurality of the two-dimensional image was captured, and a plurality of the two-dimensional posture as label data of the training data (The position and orientation of the 2 cameras 15,16 and are measured by an angle encoder, i.e., label acquisition unit, or the like, and the position information of the camera, i.e., input data acquisition unit, and the position information of the object 17 based on the image information obtained by the camera are combined to obtain the input signal of the learning data. On the other hand, a teacher signal of the learning data is obtained from the information of the joint angle obtained by the measuring device such as an angle encoder provided at the joint part of the articulated robot 14 [0027]).
However, Fujitsu does not explicitly disclose the trained model including a two- dimensional skeleton, estimation model receiving input of a two-dimensional image of a robot captured by a camera and outputting a two-dimensional posture indicating positions of centers of a plurality of joint axes included in the robot at a time when the two-dimensional image was captured, and a joint angle estimation model receiving input of the two- dimensional posture outputted from the two-dimensional skeleton estimation model and a distance and a tilt between the camera and the robot and outputting estimating angles of the plurality of joint axes, the training data generation device comprising: an input data acquisition unit configured to acquire a plurality of the two-dimensional image of the robot captured by the camera at various distances and with various tilts in advance by changing the posture and direction of the robot, and the distances and the tilts between the camera and the robot as input data of the training data; and a label acquisition unit configured to acquire a plurality of the angles of the plurality of joint axes from the robot at the time when each of the plurality of the two-dimensional image was captured, and a plurality of the two-dimensional posture as label data of the training data.
Nevertheless, Tanabiki discloses the trained model including a two- dimensional skeleton estimation model receiving input of a two-dimensional image captured by a camera (Similar skeleton model storing section 254 generates a 2D skeleton model from a 3D skeleton model, using the camera parameters stored in image storing section 220. The camera parameters include camera exterior parameters and camera interior parameters. The camera exterior parameters are the position coordinates of the camera in a world coordinate system (the position of the origin point of the camera coordinate system), and the pose (xyz rotation angle) of the camera based on the world coordinate system. The camera interior parameters include the focal length of the camera, the vertical angle of view of the camera, the width of a 2D image projected on a screen, and the height of the 2D image projected on the screen [0109]) and outputting a two-dimensional posture indicating positions of centers of a plurality of joint axes (A method for estimating a joint position will be described in detail later. Skeleton estimating section 251 outputs skeleton information, i.e., information on the estimated positions of the joints to skeleton model description converting section 252. If the joint position can be determined, the position of the bone joining the joints can also be determined [0094]), and a joint angle estimation model receiving input of the two-dimensional posture outputted from the two-dimensional skeleton estimation model (Skeleton estimating section 251 outputs skeleton information, i.e., information on the estimated positions of the joints to skeleton model description converting section 252. If the joint position can be determined, the position of the bone joining the joints can also be determined. Estimation of the joint position is the same as the estimation of the skeleton model [0094]), the training data generation device comprising: an input data acquisition unit configured to acquire a plurality of the two-dimensional image captured by the camera (Similar skeleton model storing section 254 generates a 2D skeleton model from a 3D skeleton model, using the camera parameters stored in image storing section 220. The camera parameters include camera exterior parameters and camera interior parameters. The camera exterior parameters are the position coordinates of the camera in a world coordinate system (the position of the origin point of the camera coordinate system), and the pose (xyz rotation angle) of the camera based on the world coordinate system. The camera interior parameters include the focal length of the camera, the vertical angle of view of the camera, the width of a 2D image projected on a screen, and the height of the 2D image projected on the screen [0109]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Fujitsu with the teachings of Tanabiki to enable an accurate determination of camera-to-robot distance and orientation and improve accuracy.
However, Fujitsu and Tanabiki do not explicitly disclose a joint angle estimation model receiving input of the two- dimensional posture outputted from the two-dimensional skeleton estimation model and a distance and a tilt between the camera and the robot and outputting estimating angles of the plurality of joint axes and an input data acquisition unit configured to acquire a plurality of the two-dimensional image of the robot captured by the camera at various distances and with various tilts in advance by changing the posture and direction of the robot, and the distances and the tilts between the camera and the robot as input data of the training data.
Nevertheless, Tremblay discloses input of the two-dimensional posture and a distance and a tilt between the camera (In another embodiment, the first pose of the target object with respect to the camera may include a three-dimensional (3D) rotation, i.e., tilt, and translation of the target object with respect to the camera [0021];In at least one embodiment, a single image captured by camera 502 can be analyzed to determine features of robot 504 that can be used to determine a pose of robot 504. The features can correspond to joints or locations at which a robot can move or make adjustments in position or orientation. Further, since dimensions and kinematics of robot 504 are known, determining a pose of robot 504 from a perspective of camera 502 can enable an accurate determination of camera-to-robot distance and orientation [0066]) and to acquire a tilt and the distances and the tilt between the camera and the robot as input data of the training data (In another embodiment, the first pose of the target object with respect to the camera may include a three-dimensional (3D) rotation, i.e., tilt, and translation of the target object with respect to the camera [0021]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Fujitsu and Tanabiki with the teachings of Tremblay to enable an accurate determination of camera-to-robot distance and orientation and improve accuracy.
Regarding Claim 4, Fujitsu, Tanabiki, and Tremblay disclose the claimed invention discussed in claim 2.
Fujitsu discloses an input unit configured to input a two-dimensional image of a robot captured by a camera, (The position and orientation of the 2 cameras 15,16 and are measured by an angle encoder, i.e., label acquisition unit, or the like, and the position information of the camera, i.e., input data acquisition unit, and the position information of the object 17 based on the image information obtained by the camera are combined to obtain the input signal of the learning data. On the other hand, a teacher signal of the learning data is obtained from the information of the joint angle obtained by the measuring device such as an angle encoder provided at the joint part of the articulated robot 14 [0027]); and estimate angles of a plurality of joint axes included in the robot at the time when the two-dimensional image was captured (The position and orientation of the 2 cameras 15,16 and are measured by an angle encoder, i.e., label acquisition unit, or the like, and the position information of the camera, i.e., input data acquisition unit, and the position information of the object 17 based on the image information obtained by the camera are combined to obtain the input signal of the learning data. On the other hand, a teacher signal of the learning data is obtained from the information of the joint angle obtained by the measuring device such as an angle encoder provided at the joint part of the articulated robot 14 [0027]), and a two-dimensional posture indicating positions of centers of the plurality of joint axes in the two-dimensional image (The position and orientation of the 2 cameras 15,16 and are measured by an angle encoder, i.e., label acquisition unit, or the like, and the position information of the camera, i.e., input data acquisition unit, and the position information of the object 17 based on the image information obtained by the camera are combined to obtain the input signal of the learning data. On the other hand, a teacher signal of the learning data is obtained from the information of the joint angle obtained by the measuring device such as an angle encoder provided at the joint part of the articulated robot 14 [0027]).
However, Fujitsu does not explicitly disclose an input unit configured to input a two-dimensional image of a robot captured by a camera, and a distance and a tilt between the camera and the robot; and an estimation unit configured to input the two-dimensional image, and the distance and tilt between the camera and the robot, which have been inputted by the input unit, to the trained model, and estimate angles of a plurality of joint axes included in the robot at the time when the two-dimensional image was captured, and a two-dimensional posture indicating positions of centers of the plurality of joint axes in the two-dimensional image.
Nevertheless, Tanabiki discloses an input unit configured to input a two-dimensional image of a robot captured by a camera (Similar skeleton model storing section 254 generates a 2D skeleton model from a 3D skeleton model, using the camera parameters stored in image storing section 220. The camera parameters include camera exterior parameters and camera interior parameters. The camera exterior parameters are the position coordinates of the camera in a world coordinate system (the position of the origin point of the camera coordinate system), and the pose (xyz rotation angle) of the camera based on the world coordinate system. The camera interior parameters include the focal length of the camera, the vertical angle of view of the camera, the width of a 2D image projected on a screen, and the height of the 2D image projected on the screen [0109]); an estimation unit configured to input the two-dimensional image, which have been inputted by the input unit, to the trained model (Similar skeleton model storing section 254 generates a 2D skeleton model from a 3D skeleton model, using the camera parameters stored in image storing section 220. The camera parameters include camera exterior parameters and camera interior parameters. The camera exterior parameters are the position coordinates of the camera in a world coordinate system (the position of the origin point of the camera coordinate system), and the pose (xyz rotation angle), i.e., tilt, of the camera based on the world coordinate system. The camera interior parameters include the focal length of the camera, the vertical angle of view of the camera, the width of a 2D image projected on a screen, and the height of the 2D image projected on the screen [0109]), and estimate angles of a plurality of joint axes included in the robot at the time when the two-dimensional image was captured (Skeleton estimating section 251 outputs skeleton information, i.e., information on the estimated positions of the joints to skeleton model description converting section 252. If the joint position can be determined, the position of the bone joining the joints can also be determined, i.e., plurality of joint axes. Estimation of the joint position is the same as the estimation of the skeleton model [0094]), and a two-dimensional posture indicating positions of centers of the plurality of joint axes in the two-dimensional image (Skeleton estimating section 251 outputs skeleton information, i.e., information on the estimated positions of the joints to skeleton model description converting section 252. If the joint position can be determined, the position of the bone joining the joints can also be determined. Estimation of the joint position is the same as the estimation of the skeleton model [0094]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Fujitsu with the teachings of Tanabiki to enable an accurate determination of camera-to-robot distance and orientation and improve accuracy.
However, Fujitsu and Tanabiki do not explicitly disclose an input unit configured to input a two-dimensional image of a robot captured by a camera, and a distance and a tilt between the camera and the robot; and an estimation unit configured to input the two-dimensional image, and the distance and tilt between the camera and the robot, which have been inputted by the input unit, to the trained model.
Nevertheless, Tremblay discloses a distance and a tilt between the camera and the robot (In another embodiment, the first pose of the target object with respect to the camera may include a three-dimensional (3D) rotation, i.e., tilt, and translation of the target object with respect to the camera [0021]); the distance and tilt between the camera and the robot (In another embodiment, the first pose of the target object with respect to the camera may include a three-dimensional (3D) rotation, i.e., tilt, and translation of the target object with respect to the camera [0021]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Fujitsu and Tanabiki with the teachings of Tremblay to enable an accurate determination of camera-to-robot distance and orientation and improve accuracy.
Regarding Claim 6, Fujitsu, Tanabiki, and Tremblay disclose the claimed invention discussed in claim 4.
Fujitsu discloses the trained model (as discussed above).
However, Fujitsu and Tanabiki does not explicitly disclose the trained model is provided in a server that is connected to be accessible from the robot joint angle estimation device via a network.
Nevertheless, Tremblay discloses the trained model is provided in a server that is connected to be accessible from the robot joint angle estimation device via a network (As described herein, a method, computer readable medium, and system are disclosed for estimating an object-to-object pose using an externally captured image of the objects. In accordance with FIGS. 1-4, an embodiment may provide neural networks usable for performing inferencing operations and for providing inferenced data, where the neural networks are stored (partially or wholly) in one or both of data storage 601 and 605 in inference and/or training logic 615 as depicted in FIGS. 6A and 6B. Training and deployment of the neural networks may be performed as depicted in FIG. 7 and described herein. Distribution of the neural networks may be performed using one or more servers in a data center 800 as depicted in FIG. 8 and described herein [0096]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Fujitsu and Tanabiki with the teachings of Tremblay to provide sufficient memory for processing sensor data and execute real-time angle estimation.
Regarding Claim 7, Fujitsu, Tanabiki, and Tremblay disclose the claimed invention discussed in claim 1.
Fujitsu discloses a machine learning device, the machine learning device including a learning unit configured to execute supervised learning based on input data and label data of training data generated by a training data generation device to generate a trained model (Figure 3 shows an example of the learning process performed by the learning device. In Figure 3, the input signal for the training data is generated based on image information from two cameras…The learning device 13 uses this positional information as input signals to perform learning [0025]), the training data generation device being for generating training data for the machine learning device to generate a trained model (Here, the space in which the end-effector of the articulated robot moves is divided into multiple sections, and learning devices 19 to 22 corresponding to each section are provided. In this case, compared to using a single learning device, the learning time can be reduced and the accuracy of interpolation can be improved. The learning devices 19-22 are trained to output angle information that each joint of the articulated robot should take when given positional information about the corresponding space [0029]), receiving input of a two-dimensional image of a robot captured by a camera and outputting a two-dimensional posture indicating positions of centers of a plurality of joint axes included in the robot at a time when the two-dimensional image was captured (In Figure 4, the articulated robot 14 holds object 17 at its end-hand by bending the angles of its three joints to angles θ1, θ2, and θ3, respectively. The angles θ1, θ2, and θ3 are accurately measured by installing angle encoders or similar devices at each joint. The two cameras 15 and 16 are mounted on movable tables that can change their position and orientation in three dimensions, and they capture the end-effectors of the articulated robot 14 [0026]), and receiving input of the two-dimensional posture outputted from the two-dimensional skeleton estimation model and a tilt (The invention of claim 2 is a control device for an articulated robot according to claim 1, which includes a coordinate transformation device that performs the learning using position information obtained by a device for measuring the position of an object, which comprises a plurality of television cameras that photograph an object, and an image processing device that generates position information such as the X coordinate and Y coordinate of the object's center of gravity, the tilt angle of the object, and the area of the object from the image information of the television cameras [0010]), the training data generation device comprising: an input data acquisition unit configured to acquire a plurality of the two-dimensional image of the robot captured by the camera (The position and attitude of the 2 cameras 15,16 and are measured by an angle encoder, i.e., label acquisition unit, or the like, and the position information of the camera, i.e., input data acquisition unit, and the position information of the object 17 based on the image information obtained by the camera are combined to obtain the input signal of the learning data. On the other hand, a teacher signal of the learning data is obtained from the information of the joint angle obtained by the measuring device such as an angle encoder provided at the joint part of the articulated robot 14 [0027]); and a label acquisition unit configured to acquire a plurality of the angles of the plurality of joint axes from the robot at the time when each of the plurality of the two-dimensional image was captured, and a plurality of the two-dimensional posture as label data of the training data (The position and attitude of the 2 cameras 15,16 and are measured by an angle encoder, i.e., label acquisition unit, or the like, and the position information of the camera, i.e., input data acquisition unit, and the position information of the object 17 based on the image information obtained by the camera are combined to obtain the input signal of the learning data. On the other hand, a teacher signal of the learning data is obtained from the information of the joint angle obtained by the measuring device such as an angle encoder provided at the joint part of the articulated robot 14 [0027]).
However, Fujitsu does not explicitly disclose the trained model including a two- dimensional skeleton estimation model receiving input of a two-dimensional image of a robot captured by a camera and outputting a two-dimensional posture indicating positions of centers of a plurality of joint axes included in the robot at a time when the two-dimensional image was captured, and a joint angle estimation model receiving input of the two-dimensional posture outputted from the two-dimensional skeleton estimation model and a distance and a tilt between the camera and the robot and outputting angles of the plurality of joint axes, the training data generation device comprising: an input data acquisition unit configured to acquire a plurality of the two-dimensional image of the robot captured by the camera at various distances and with various tilts in advance by changing the posture and direction of the robot, and the distances and the tilts between the camera and the robot as input data of the training data.
Nevertheless, Tanabiki discloses a two- dimensional skeleton estimation model receiving input of a two-dimensional image captured by a camera and outputting a two-dimensional posture indicating positions of centers of a plurality of joint axes (Similar skeleton model storing section 254 generates a 2D skeleton model from a 3D skeleton model, using the camera parameters stored in image storing section 220. The camera parameters include camera exterior parameters and camera interior parameters. The camera exterior parameters are the position coordinates of the camera in a world coordinate system (the position of the origin point of the camera coordinate system), and the pose (xyz rotation angle) of the camera based on the world coordinate system. The camera interior parameters include the focal length of the camera, the vertical angle of view of the camera, the width of a 2D image projected on a screen, and the height of the 2D image projected on the screen [0109]), and a joint angle estimation model receiving input of the two-dimensional posture outputted from the two-dimensional skeleton estimation model (Skeleton estimating section 251 outputs skeleton information, i.e., information on the estimated positions of the joints to skeleton model description converting section 252. If the joint position can be determined, the position of the bone joining the joints can also be determined. Estimation of the joint position is the same as the estimation of the skeleton model [0094]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Fujitsu with the teachings of Tanabiki to enable an accurate determination of camera-to-robot distance and orientation and improve accuracy.
However, Fujitsu and Tanabiki do not explicitly disclose a joint angle estimation model receiving input of the two-dimensional posture outputted from the two-dimensional skeleton estimation model and a distance and a tilt between the camera and the robot and outputting angles of the plurality of joint axes, the training data generation device comprising: an input data acquisition unit configured to acquire a plurality of the two-dimensional image of the robot captured by the camera at various distances and with various tilts in advance by changing the posture and direction of the robot, and the distances and the tilts between the camera and the robot as input data of the training data.
Nevertheless, Tremblay discloses a distance and a tilt between the camera and the robot (In another embodiment, the first pose of the target object with respect to the camera may include a three-dimensional (3D) rotation, i.e., tilt, and translation of the target object with respect to the camera [0021]), the tilt between the camera and the robot as input data (In another embodiment, the first pose of the target object with respect to the camera may include a three-dimensional (3D) rotation, i.e., tilt, and translation of the target object with respect to the camera [0021]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Fujitsu and Tanabiki with the teachings of Tremblay to enable an accurate determination of camera-to-robot distance and orientation and improve accuracy.
Response to Arguments
USC § 112
Applicant’s arguments, filed 03/23/2026, with respect to claims 1-4 and 6-7 have been fully considered and are persuasive. The rejection of claims 1-4 and 6-7 have been withdrawn.
35 USC § 103
Applicant's arguments filed 03/23/2026 have been fully considered but they are not persuasive.
Applicant argues (pg. ) “However, Applicant respectfully submits that Tremblay does not disclose the process of generating and processing a training data set 702. Applicant respectfully submits further that there is no objective basis for the assertion that an apparatus corresponding to a "teacher data generation apparatus" includes both an "input data acquisition unit" and a "label acquisition unit”.
Examiner disagrees and submits that the "teacher data generation apparatus" includes both an "input data acquisition unit" and a "label acquisition unit” were addressed using Fujitsu [0027].
Applicant argues (pg. 11-12) “Although Tremblay discloses that a single image captured by the camera 502 can be analyzed to determine a plurality of joints and positions of the robot 504, the positions of the joints and the "positions of centers of the joint axes" of the invention as recited in independent claim 1 of the present application are not necessarily the same. Therefore, Applicant respectfully submits that even assuming, strictly arguendo, that the disclosure of Tremblay can be properly applied to the disclosure of Fujitsu, it would not have been easy for a person having ordinary skill in the subject art to resolve the above-discussed Differences 1 and 2 and thus conceive of the configuration and effects of the invention as recited in independent claim 1 of the present application”
The Examiner submits that “Therefore, Applicant respectfully submits that even assuming, strictly arguendo, that the disclosure of Tremblay can be properly applied to the disclosure of Fujitsu, it would not have been easy for a person having ordinary skill in the subject art to resolve the above-discussed Differences 1 and 2 and thus conceive of the configuration and effects of the invention as recited in independent claim 1 of the present application” is an argument where the applicant admits the combination of references is a reasonable combination.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHARAH ZAAB whose telephone number is (571)272-4973. The examiner can normally be reached Monday - Friday 7:00 am - 4:30 pm.
/SHARAH ZAAB/Examiner, Art Unit 2857
/Catherine T. Rastovski/Supervisory Primary Examiner, Art Unit 2857
Prosecution Insights