DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Joint Inventors
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). A certified copy of this document has been placed in the file wrapper. As such, the effective filing date of the instant application is considered 12/20/2022, coinciding with the filing date of the People’s Republic of China application to which foreign priority was requested.
Response to Amendment
Claims 1, 5, 8, 12, 14, and 18 have been amended. Claims 2, 9, and 15 have been canceled, and claims 21-23 have been added. The 35 U.S.C. 103 rejection has been updated in view of amendment.
Response to Argument
Applicant’s arguments filed 11/13/2025 have been considered, but are not persuasive.
Applicant first contends that Kalouche does not acquire calibration data using the IMU, and instead merely relies on visual image analysis. Examiner finds this argument warrantless, as at least col. 10, lines 3-20 as cited in the previous office action describe the potential integration of spatial information from an IMU in the calibration procedure.
Applicant next contends that Kalouche does not disclose that a specific calibration action needs to be performed before operation to eliminate wearing errors of sensors. Examiner contends that Applicant is overstating the claimed language, which does not state that a specific calibration action needs to be performed before operation, but merely that it happens prior to operation. Examiner is unsure where Applicant believes this limitation is found in the claimed language, but both “controlling a target robot to act according to the target action data to enable the target robot to complete an action corresponding to the target action data” and “wherein before acquiring the target action data of the target object, the method further comprises: acquiring, by the inertial motion capture device” represent only chronological limitations, which remain disclosed by the calibration procedure described in Kalouche.
Applicant finally contends that Kalouche does not disclose directly controlling the robot based on the acquired data. Examiner disagrees, and points to at least col. 10 lines 56-57 “The pose estimation of the subject is transmitted to the corresponding robotic system controller 145. This enables a robot of a corresponding robotic system to take a pose in accordance with the subject in real-time”, or additionally col. 2, lines 15-21 “The robotic system controller controls one or more actuators of the robot according to the plurality of kinematic parameters, causing the robot to take a pose corresponding to the pose of the subject in the captured image.”
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 3-8, 10-14 and 16-23 are rejected under 35 U.S.C. 103 as being unpatentable over Kalouche (US10919152, referred to as Kalouche) in view of Ge et al. (US12275152, referred to as Ge).
Regarding claim 1: Kalouche discloses: A method for robot teleoperation control, comprising: acquiring, by an inertial motion capture device worn on a target object, target action data and displacement data of the target object, ([col. 3, lines 12-17] A subject herein refers to any moving objects that have more than one pose. The moving objects include, among other objects, animals, people, and robots. Although embodiments herein are described with reference to humans as the subject, note that the present invention can be applied essentially in the same manner to any other object or animal having more than one pose. [col. 10, lines 42-57] The tracking module 270 tracks the poses of the subject in subsequent images captured by the image capturing device 125. The tracking module 270 receives one or more processed images from the image processor 260, and uses it to estimate pose information of the subject in the processed images. In some embodiments, the one or more processed images may be images that were captured subsequent to the captured images used to generate the 3D skeleton model. In this configuration, the pose estimation module 215 is able to estimate a pose of a subject in real-time as images are captured by the image capturing device 125. The pose estimation of the subject is transmitted to the corresponding robotic system controller 145. This enables a robot of a corresponding robotic system to take a pose in accordance with the subject in real-time.) wherein the target action data includes head action data and arm action data; ([col. 3, lines 19-29] The localized body parts herein refer to any portion of the subject that can be conceptually identified as one or more joints and links. For example, in a human subject, the localized body parts include, among other parts, a head, a torso, a left arm, a right arm, a left hand, a right hand, a left leg, and a right leg. The localized body parts can be subdivided into other parts (e.g., a left arm has a left upper arm and a left forearm, a left hand has a left thumb and left fingers). The one or more body parts may be localized relative to a camera, an external landmark, or another point on the subject's body.) controlling a target robot to act according to the target action data to enable the target robot to complete an action corresponding to the target action data; and ([col. 9, lines 5-14] the data collected from each robot-teleoperator pair can be shared collectively in a database that enables data sharing for parallelized learning such that a first robot in a first environment performs a task, and, once the task is learned by the imitation learning engine 150, a second robot in a second environment may also learn the motions to perform the same task (as well as a third robot in a third environment, a fourth robot in a fourth environment, and so on, until an Nth robot in an Nth environment).) performing centroid trajectory planning on the target robot ([col. 10, lines 3-20] The machine learning model may use spatial motion information from an IMU on the mobile device from the relationship between a changing image perspective and the 6-axis motion of the image capturing device 125 (in an embodiment in which the image capturing device and the IMU are embedded in the same device and do not move relative to one another). In alternative embodiments, the operator may manually set the subject's body part dimensions. In some embodiments, the machine learning model may track certain body parts, joints, or segments relative to other body joints, parts, or segments, relative to an external landmark, or relative to the image capturing device 140. The skeletal model mapper 265 projects the two-dimensional localized body parts to a three-dimensional skeleton model of the operator. In the embodiment of FIG. 2, the skeletal model mapper 265 executes an algorithm that enhances the alignment between a 2D pixel location of each body part in the captured image and the 3D skeleton model.) [based on a model predictive control (MPC) algorithm according to the displacement data to obtain a target centroid trajectory, and establishing a spring-damping system to track the target centroid trajectory so as to enable the target robot to move to a position corresponding to the displacement data,] wherein before acquiring the target action data of the target object, the method further comprises: acquiring, by the inertial motion capture device, head calibration action data, arm calibration action data and calibration position data of the target object performing calibration actions, wherein the calibration actions are performed to eliminate wearing errors of the inertial motion capture device on a body of the target object controlling a head action of the target robot according to the head calibration action data to enable a head of the target robot to complete the head action corresponding to the head calibration action data controlling an arm action of the target robot according to the arm calibration action data to enable arms of the target robot to complete the arm action corresponding to the arm calibration action data establishing a human body coordinate system with the calibration position data as an origin; and establishing a robot coordinate system with calibration position data of the target robot as an origin. ([col. 3, lines 19-29] The localized body parts herein refer to any portion of the subject that can be conceptually identified as one or more joints and links. For example, in a human subject, the localized body parts include, among other parts, a head, a torso, a left arm, a right arm, a left hand, a right hand, a left leg, and a right leg. The localized body parts can be subdivided into other parts (e.g., a left arm has a left upper arm and a left forearm, a left hand has a left thumb and left fingers). The one or more body parts may be localized relative to a camera, an external landmark, or another point on the subject's body. [col. 10, lines 3-20] The machine learning model may use spatial motion information from an IMU on the mobile device from the relationship between a changing image perspective and the 6-axis motion of the image capturing device 125 (in an embodiment in which the image capturing device and the IMU are embedded in the same device and do not move relative to one another). In alternative embodiments, the operator may manually set the subject's body part dimensions. In some embodiments, the machine learning model may track certain body parts, joints, or segments relative to other body joints, parts, or segments, relative to an external landmark, or relative to the image capturing device 140. The skeletal model mapper 265 projects the two-dimensional localized body parts to a three-dimensional skeleton model of the operator. In the embodiment of FIG. 2, the skeletal model mapper 265 executes an algorithm that enhances the alignment between a 2D pixel location of each body part in the captured image and the 3D skeleton model.) [col. 5, lines 50-55] Parameters may relate to motion scaling or sensitivity, pause functionality, origin reset, Carrtesian or joint axis locking and unlocking, bounding volumes, ‘home’ positions and orientations…)
Kalouche does not explicitly disclose: based on a model predictive control (MPC) algorithm according to the displacement data to obtain a target centroid trajectory, and establishing a spring-damping system to track the target centroid trajectory so as to enable the target robot to move to a position corresponding to the displacement data.
Kalouche does not disclose the following limitations, however Ge, from a similar field of endeavor, teaches: based on a model predictive control (MPC) algorithm according to the displacement data to obtain a target centroid trajectory, and establishing a spring-damping system to track the target centroid trajectory so as to enable the target robot to move to a position corresponding to the displacement data. ([col. 2, lines 29-44] in the process of robot stability control, there will be a phase lag between the actual motion trajectory and the desired motion trajectory of the body of the robot, and this characteristic of phase lag can be approximately described by the above-mentioned springmass-damping model, as shown in FIG. 3, in which x/ represents the desired position of the center of mass, and xcrepresents the actual position of center of mass. Because the spring-mass-damper model is used for analysis and control,
the problem of phase lag cannot be solved. The embodiments of the present disclosure propose a new physical model, that is, a spring-mass-damping-acceleration model. By introducing acceleration, the model can realize the lead input of the desired trajectory of the target object, which can greatly improve the tracking and response performance of the controller of a robot. [col. 10-11, lines 64-12] The centroid state acquisition module 120 is to, based on a ZMP tracking control model, the desired ZMP and the actual ZMP, calculate a desired value of a motion state of a center of mass of the robot at the current moment. The desired value of the motion state of the center of mass includes a correction amount of the position of the center of mass. The lead input calculation module 130 is to, based on a spring-massdamping-acceleration model and the desired value of the motion state of the center of mass, calculate a lead control input amount for the correction amount of the position of the
center of mass. The tracking control module 140 is to control motion of the robot according to the lead control input amount and a planned value of the position of the center of mass at the current moment, so as to realize tracking of the desired ZMP by the robot.)
Kalouche and Ge are analogous art to the claimed invention since they are from the similar field of robotic limb tracking and control. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to modify the mapping and control system of Kalouche to enable the spring-mass-damping based model predictive control taught in Ge.
The motivation for modification would have been to provide the mapping and control method disclosed in Kalouche with the method applied to a spring-mass-damping MPC system as taught in Ge.
Regarding claim 3: The combination of Kalouche and Ge teaches: The method according to claim 1,
Kalouche further discloses: wherein acquiring the displacement data of the target object includes: acquiring motion posture data and skeleton data of the target object; calculating a joint rotation matrix of the target object according to a data fusion algorithm, a filtering algorithm and the motion posture data; calculating a skeleton vector of the target object according to the skeleton data; and calculating a product of the joint rotation matrix and the skeleton vector to obtain the displacement data. ([col. 9, lines 5-14] the data collected from each robot-teleoperator pair can be shared collectively in a database that enables data sharing for parallelized learning such that a first robot in a first environment performs a task, and, once the task is learned by the imitation learning engine 150, a second robot in a second environment may also learn the motions to perform the same task (as well as a third robot in a third environment, a fourth robot in a fourth environment, and so on, until an Nth robot in an Nth environment [col. 10, lines 3-20] The machine learning model may use spatial motion information from an IMU on the mobile device from the relationship between a changing image perspective and the 6-axis motion of the image capturing device 125 (in an embodiment in which the image capturing device and the IMU are embedded in the same device and do not move relative to one another). In alternative embodiments, the operator may manually set the subject's body part dimensions. In some embodiments, the machine learning model may track certain body parts, joints, or segments relative to other body joints, parts, or segments, relative to an external landmark, or relative to the image capturing device 140. The skeletal model mapper 265 projects the two-dimensional localized body parts to a three-dimensional skeleton model of the operator. In the embodiment of FIG. 2, the skeletal model mapper 265 executes an algorithm that enhances the alignment between a 2D pixel location of each body part in the captured image and the 3D skeleton model.)
Regarding claim 4: The combination of Kalouche and Ge teaches: The method according to claim 1,
Kalouche further discloses: wherein controlling the target robot to act according to the target action data includes: controlling the target robot to act according to the head action data to enable a head of the target robot to complete an action corresponding to the head action data; and controlling the target robot to act according to the arm action data to enable arms of the target robot to complete an action corresponding to the arm action data. ([col. 3, lines 19-29] The localized body parts herein refer to any portion of the subject that can be conceptually identified as one or more joints and links. For example, in a human subject, the localized body parts include, among other parts, a head, a torso, a left arm, a right arm, a left hand, a right hand, a left leg, and a right leg. The localized body parts can be subdivided into other parts (e.g., a left arm has a left upper arm and a left forearm, a left hand has a left thumb and left fingers). The one or more body parts may be localized relative to a camera, an external landmark, or another point on the subject's body. [col. 9, lines 5-14] the data collected from each robot-teleoperator pair can be shared collectively in a database that enables data sharing for parallelized learning such that a first robot in a first environment performs a task, and, once the task is learned by the imitation learning engine 150, a second robot in a second environment may also learn the motions to perform the same task (as well as a third robot in a third environment, a fourth robot in a fourth environment, and so on, until an Nth robot in an Nth environment [col. 10, lines 3-20] The machine learning model may use spatial motion information from an IMU on the mobile device from the relationship between a changing image perspective and the 6-axis motion of the image capturing device 125 (in an embodiment in which the image capturing device and the IMU are embedded in the same device and do not move relative to one another). In alternative embodiments, the operator may manually set the subject's body part dimensions. In some embodiments, the machine learning model may track certain body parts, joints, or segments relative to other body joints, parts, or segments, relative to an external landmark, or relative to the image capturing device 140. The skeletal model mapper 265 projects the two-dimensional localized body parts to a three-dimensional skeleton model of the operator. In the embodiment of FIG. 2, the skeletal model mapper 265 executes an algorithm that enhances the alignment between a 2D pixel location of each body part in the captured image and the 3D skeleton model.)
Regarding claim 5: The combination of Kalouche and Ge teaches: The method according to claim 1,
Kalouche further discloses: wherein performing centroid trajectory planning on the target robot [based on the MPC algorithm] according to the displacement data to obtain the target centroid trajectory and [establishing the spring-damping system to track the target centroid trajectory] so as to enable the target robot to move to the position corresponding to the displacement data includes: mapping the displacement data into a human body coordinate system to obtain mapped displacement data; mapping the mapped displacement data into the robot coordinate system to obtain the target centroid trajectory; and controlling the target robot to act according to the target centroid trajectory to enable the target robot to move to the position corresponding to the displacement data. ([col. 3, lines 19-29] The localized body parts herein refer to any portion of the subject that can be conceptually identified as one or more joints and links. For example, in a human subject, the localized body parts include, among other parts, a head, a torso, a left arm, a right arm, a left hand, a right hand, a left leg, and a right leg. The localized body parts can be subdivided into other parts (e.g., a left arm has a left upper arm and a left forearm, a left hand has a left thumb and left fingers). The one or more body parts may be localized relative to a camera, an external landmark, or another point on the subject's body. [col. 9, lines 5-14] the data collected from each robot-teleoperator pair can be shared collectively in a database that enables data sharing for parallelized learning such that a first robot in a first environment performs a task, and, once the task is learned by the imitation learning engine 150, a second robot in a second environment may also learn the motions to perform the same task (as well as a third robot in a third environment, a fourth robot in a fourth environment, and so on, until an Nth robot in an Nth environment [col. 10, lines 3-20] The machine learning model may use spatial motion information from an IMU on the mobile device from the relationship between a changing image perspective and the 6-axis motion of the image capturing device 125 (in an embodiment in which the image capturing device and the IMU are embedded in the same device and do not move relative to one another). In alternative embodiments, the operator may manually set the subject's body part dimensions. In some embodiments, the machine learning model may track certain body parts, joints, or segments relative to other body joints, parts, or segments, relative to an external landmark, or relative to the image capturing device 140. The skeletal model mapper 265 projects the two-dimensional localized body parts to a three-dimensional skeleton model of the operator. In the embodiment of FIG. 2, the skeletal model mapper 265 executes an algorithm that enhances the alignment between a 2D pixel location of each body part in the captured image and the 3D skeleton model.)
Kalouche does not explicitly disclose: based on the MPC algorithm; establishing the spring-damping system to track the target centroid trajectory
Kalouche does not disclose the following limitations, however Ge, from a similar field of endeavor, teaches: based on the MPC algorithm; establishing the spring-damping system to track the target centroid trajectory ([col. 2, lines 29-44] in the process of robot stability control, there will be a phase lag between the actual motion trajectory and the desired motion trajectory of the body of the robot, and this characteristic of phase lag can be approximately described by the above-mentioned springmass-damping model, as shown in FIG. 3, in which x/ represents the desired position of the center of mass, and xcrepresents the actual position of center of mass. Because the spring-mass-damper model is used for analysis and control, the problem of phase lag cannot be solved. The embodiments of the present disclosure propose a new physical model, that is, a spring-mass-damping-acceleration model. By introducing acceleration, the model can realize the lead input of the desired trajectory of the target object, which can greatly improve the tracking and response performance of the controller of a robot. [col. 10-11, lines 64-12] The centroid state acquisition module 120 is to, based on a ZMP tracking control model, the desired ZMP and the actual ZMP, calculate a desired value of a motion state of a center of mass of the robot at the current moment. The desired value of the motion state of the center of mass includes a correction amount of the position of the center of mass. The lead input calculation module 130 is to, based on a spring-massdamping-acceleration model and the desired value of the motion state of the center of mass, calculate a lead control input amount for the correction amount of the position of the center of mass. The tracking control module 140 is to control motion of the robot according to the lead control input amount and a planned value of the position of the center of mass at the current moment, so as to realize tracking of the desired ZMP by the robot.)
Kalouche and Ge are analogous art to the claimed invention since they are from the similar field of robotic limb tracking and control. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to modify the mapping and control system of Kalouche to enable the spring-mass-damping based model predictive control taught in Ge.
The motivation for modification would have been to provide the mapping and control method disclosed in Kalouche with the method applied to a spring-mass-damping MPC system as taught in Ge.
Regarding claim 6: The combination of Kalouche and Ge teaches: The method according to claim 1,
Kalouche further discloses: further comprising: controlling a head camera of the target robot to photograph to obtain field of view data; and sending the field of view data to a target device. ([col. 2-3, lines 65-4] In the step of generating body pose information, an algorithm is used to localize an array of body parts of the subject in the captured image. The algorithm then projects the localized body parts of the subject onto a three-dimensional (3D) skeleton model of the subject. The 3D skeleton model is output as an estimate of the pose)
Regarding claim 7: The combination of Kalouche and Ge teaches: The method according to claim 6,
Kalouche further discloses: further comprising: obtaining information about an environment where the target robot is located according to the field of view data; and adjusting the target robot according to the information. ([col. 2-3, lines 65-4] In the step of generating body pose information, an algorithm is used to localize an array of body parts of the subject in the captured image. The algorithm then projects the localized body parts of the subject onto a three-dimensional (3D) skeleton model of the subject. The 3D skeleton model is output as an estimate of the pose and is used for estimating and tracking the poses of the subject in a next captured image.)
Regarding claim 8: Rejected using the same rationale as claim 1.
Regarding claim 10: Rejected using the same rationale as claim 3.
Regarding claim 11: The combination of Kalouche and Ge teaches: The robot according to claim 8,
Kalouche further discloses: wherein the processor is further configured to: control a head action according to the head action data to enable the head to complete the head action corresponding to the head action data; and control an arm action according to the arm action data to enable the arm to complete the arm action corresponding to the arm action data. ([col. 3, lines 19-29] The localized body parts herein refer to any portion of the subject that can be conceptually identified as one or more joints and links. For example, in a human subject, the localized body parts include, among other parts, a head, a torso, a left arm, a right arm, a left hand, a right hand, a left leg, and a right leg. The localized body parts can be subdivided into other parts (e.g., a left arm has a left upper arm and a left forearm, a left hand has a left thumb and left fingers). The one or more body parts may be localized relative to a camera, an external landmark, or another point on the subject's body.)
Regarding claim 12: The combination of Kalouche and Ge teaches: The robot according to claim 8,
Kalouche further discloses: wherein the processor is further configured to: map the displacement data into the human body coordinate system to obtain mapped displacement data; map the mapped displacement data into a robot coordinate system to obtain the target centroid trajectory; and move to the target position corresponding to the displacement data according to the target centroid trajectory. ([col. 3, lines 19-29] The localized body parts herein refer to any portion of the subject that can be conceptually identified as one or more joints and links. For example, in a human subject, the localized body parts include, among other parts, a head, a torso, a left arm, a right arm, a left hand, a right hand, a left leg, and a right leg. The localized body parts can be subdivided into other parts (e.g., a left arm has a left upper arm and a left forearm, a left hand has a left thumb and left fingers). The one or more body parts may be localized relative to a camera, an external landmark, or another point on the subject's body. [col. 9, lines 5-14] the data collected from each robot-teleoperator pair can be shared collectively in a database that enables data sharing for parallelized learning such that a first robot in a first environment performs a task, and, once the task is learned by the imitation learning engine 150, a second robot in a second environment may also learn the motions to perform the same task (as well as a third robot in a third environment, a fourth robot in a fourth environment, and so on, until an Nth robot in an Nth environment [col. 10, lines 3-20] The machine learning model may use spatial motion information from an IMU on the mobile device from the relationship between a changing image perspective and the 6-axis motion of the image capturing device 125 (in an embodiment in which the image capturing device and the IMU are embedded in the same device and do not move relative to one another). In alternative embodiments, the operator may manually set the subject's body part dimensions. In some embodiments, the machine learning model may track certain body parts, joints, or segments relative to other body joints, parts, or segments, relative to an external landmark, or relative to the image capturing device 140. The skeletal model mapper 265 projects the two-dimensional localized body parts to a three-dimensional skeleton model of the operator. In the embodiment of FIG. 2, the skeletal model mapper 265 executes an algorithm that enhances the alignment between a 2D pixel location of each body part in the captured image and the 3D skeleton model.)
Regarding claim 13: Rejected using the same rationale as claim 6.
Regarding claim 14: Rejected using the same rationale as claims 1 and 8.
Regarding claim 16: Rejected using the same rationale as claims 3 and 10.
Regarding claim 17: Rejected using the same rationale as claim 4.
Regarding claim 18: Rejected using the same rationale as claim 5.
Regarding claim 19: Rejected using the same rationale as claims 6 and 13.
Regarding claim 20: Rejected using the same rationale as claim 7.
Regarding claim 21: The combination of Kalouche and Ge teaches: The method according to claim 3,
Kalouche further discloses: wherein the inertial motion capture device collects data via inertial posture sensors, and each inertial posture sensor is internally provided with a high-dynamic triaxial accelerometer, a high-dynamic triaxial gyroscope and a high-dynamic triaxial magnetometer; and wherein the inertial posture sensors are worn on corresponding joints of the target object respectively. ([col. 3, lines 19-29] The localized body parts herein refer to any portion of the subject that can be conceptually identified as one or more joints and links. For example, in a human subject, the localized body parts include, among other parts, a head, a torso, a left arm, a right arm, a left hand, a right hand, a left leg, and a right leg. The localized body parts can be subdivided into other parts (e.g., a left arm has a left upper arm and a left forearm, a left hand has a left thumb and left fingers). The one or more body parts may be localized relative to a camera, an external landmark, or another point on the subject's body. [col. 9, lines 5-14] the data collected from each robot-teleoperator pair can be shared collectively in a database that enables data sharing for parallelized learning such that a first robot in a first environment performs a task, and, once the task is learned by the imitation learning engine 150, a second robot in a second environment may also learn the motions to perform the same task (as well as a third robot in a third environment, a fourth robot in a fourth environment, and so on, until an Nth robot in an Nth environment [col. 10, lines 3-20] The machine learning model may use spatial motion information from an IMU on the mobile device from the relationship between a changing image perspective and the 6-axis motion of the image capturing device 125 (in an embodiment in which the image capturing device and the IMU are embedded in the same device and do not move relative to one another). In alternative embodiments, the operator may manually set the subject's body part dimensions. In some embodiments, the machine learning model may track certain body parts, joints, or segments relative to other body joints, parts, or segments, relative to an external landmark, or relative to the image capturing device 140. The skeletal model mapper 265 projects the two-dimensional localized body parts to a three-dimensional skeleton model of the operator. In the embodiment of FIG. 2, the skeletal model mapper 265 executes an algorithm that enhances the alignment between a 2D pixel location of each body part in the captured image and the 3D skeleton model.)
Regarding claim 22: The combination of Kalouche and Ge teaches: The method according to claim 1,
Kalouche further discloses: wherein the calibration position data comprises a position of a waist of the target object performing calibration actions. ([col. 3, lines 19-29] The localized body parts herein refer to any portion of the subject that can be conceptually identified as one or more joints and links. For example, in a human subject, the localized body parts include, among other parts, a head, a torso, a left arm, a right arm, a left hand, a right hand, a left leg, and a right leg. The localized body parts can be subdivided into other parts (e.g., a left arm has a left upper arm and a left forearm, a left hand has a left thumb and left fingers). The one or more body parts may be localized relative to a camera, an external landmark, or another point on the subject's body.)
Regarding claim 23: The combination of Kalouche and Ge teaches: The method according to claim 4,
Kalouche further discloses: wherein the step of controlling the target robot to act according to the head action data comprises: mapping the head action data from the human body coordinate system to the robot coordinate system, and obtaining the head action data located in the robot coordinate system; and controlling the head of the target robot through a cubic interpolation curve according to the head action data located in the robot coordinate system; and wherein the step of controlling the target robot to act according to the arm action data comprises: mapping the arm action data from the human body coordinate system to the robot coordinate system, and obtaining the arm action data located in the robot coordinate system; and controlling the arms of the target robot through the cubic interpolation curve according to the arm action data located in the robot coordinate system. ([col. 3, lines 19-29] The localized body parts herein refer to any portion of the subject that can be conceptually identified as one or more joints and links. For example, in a human subject, the localized body parts include, among other parts, a head, a torso, a left arm, a right arm, a left hand, a right hand, a left leg, and a right leg. The localized body parts can be subdivided into other parts (e.g., a left arm has a left upper arm and a left forearm, a left hand has a left thumb and left fingers). The one or more body parts may be localized relative to a camera, an external landmark, or another point on the subject's body. [col. 9, lines 5-14] the data collected from each robot-teleoperator pair can be shared collectively in a database that enables data sharing for parallelized learning such that a first robot in a first environment performs a task, and, once the task is learned by the imitation learning engine 150, a second robot in a second environment may also learn the motions to perform the same task (as well as a third robot in a third environment, a fourth robot in a fourth environment, and so on, until an Nth robot in an Nth environment [col. 10, lines 3-20] The machine learning model may use spatial motion information from an IMU on the mobile device from the relationship between a changing image perspective and the 6-axis motion of the image capturing device 125 (in an embodiment in which the image capturing device and the IMU are embedded in the same device and do not move relative to one another). In alternative embodiments, the operator may manually set the subject's body part dimensions. In some embodiments, the machine learning model may track certain body parts, joints, or segments relative to other body joints, parts, or segments, relative to an external landmark, or relative to the image capturing device 140. The skeletal model mapper 265 projects the two-dimensional localized body parts to a three-dimensional skeleton model of the operator. In the embodiment of FIG. 2, the skeletal model mapper 265 executes an algorithm that enhances the alignment between a 2D pixel location of each body part in the captured image and the 3D skeleton model.)
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ATTICUS A CAMERON whose telephone number is 703-756-4535. The examiner can normally be reached M-F 8:30 am - 4:30 pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Thomas Worden can be reached on 571-272-4876. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ATTICUS A CAMERON/ /JASON HOLLOWAY/ Primary Examiner, Art Unit 3658 Examiner, Art Unit 3658A
Prosecution Insights