DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the amendment filed on Dec. 26th, 2022. The amendments are linked to the original application filed on Feb. 10th, 2026.
Response to Amendment
The Examiner thanks the applicant for the remarks, edits and arguments.
Regarding Claim Rejections – 35 U.S.C. 101
After each amendment submitted by the applicant, the examiner must evaluate the claims to ensure they comply with 35 U.S.C. 101. This is performed by applying the Alice/Mayo test to the current amended claims. The examiner has noted the remarks submitted by the applicant and the interview summary. During the evaluation of the claims at Step 2A, Prong 1, the examiner noted, along with the remarks, the independent claims 1, 11, and 12 do not disclose abstract ideas or mental concepts. The examiner has found the applicants argument persuasive and believes that a human cannot perform the claimed steps as mental concepts or abstract ideas. Therefore, the current claims pass the Alice/Mayo test and the current claims recite patent eligible subject matter. Therefore, the examiner has withdrawn the rejection under 35 U.S.C. 101.
Regarding Claim Rejections – 35 U.S.C. 102
The applicant has amended the claims and argues that the prior art Li fails to disclose the amended elements. The applicant believes that Li fails to disclose the use of training a system in a simulated environment or using data from a simulated environment and that Li fails to disclose the internal state parameters of a machine. Finally, the applicant believes that Li fails to teach the improvements proposed by the claimed invention. The examiner has reviewed the art Li and the amended claims. The examiner has found the arguments against Li persuasive. The examiner would like to note that Li does teach similar subject matter, however, the examiner believes Li fails to explicitly disclose each and every element of the independent claims as claimed and per 35 U.S.C. 102. After each amendment the examiner must review the remarks from the applicant and perform a complete search to ensure the claims comply with 35 U.S.C. 102/103. The examiner has performed this search and would like to note that a single art was not found which was able to anticipate the claimed subject matter in accordance with 35 U.S.C. 102. Therefore, the examiner has withdrawn the rejection under 35 U.S.C. 102
Regarding Claim Rejections – 35 U.S.C. 103
The applicant argues that the art Kutsuzawa fails to teach the use of a parameter distribution diagram as claimed. Further, the applicant has amended claim 5 to refine the parameter distribution diagram and what it depicts. The applicant believes that Kutsuzawa discloses different predetermined trajectories and not parameters and it does not disclose a parameter distribution diagram as it is known in the art. The examiner has reviewed the current amended claims and the information from the interview with the applicant. The examiner has found the applicants argument against Kutsuzawa persuasive. The examiner believes that Kutsuzawa fig. 2 is depicting the motions of a robotic system and not a distribution of movement parameters. Therefore, the examiner no longer relies on Kutsuzawa as prior art at this time. As stated above, the examiner has performed a complete search of the claims to ensure they comply with 35 U.S.C. 102/103. While performing this search, the examiner was able to locate new subject matter that the examiner believes is able to disclose the elements of claim 5. The new proposed art teaches the use of using a range of motion and monitoring the movement parameters within constraints to find new movement data. The examiner believes that the combination of proposed arts and the new subject matter is able to disclose the elements of claim 5 and the rejection under 35 U.S.C. 103 is upheld for claim 5, see 103 rejection below.
Next, the applicant argues that Li fails to disclose or teach the elements of claim 6. The applicant believes that Li fails to teach the use of training a system using a GAN and the simulation data generated. The applicant believes Li fails to disclose the use of training in a simulated environment. As stated above, the examiner also agrees with the applicant and believes that Li fails to teach the use of training a system in a simulated environment as stated in claims 1, 11, and 12. However, as stated above, a complete search was performed after reviewing the amendments and the remarks from the applicant. During this search, the examiner was able to locate new subject matter which the examiner believes is able to disclose or teach the elements of claim 6. The new proposed art uses GAN architecture, using a generator and discriminator models, with information from a simulated environment. The examiner believes that this art is able to teach the elements of claim 6 and therefore, with the combination of proposed arts, the examiner believes the rejection under 35 U.S.C. 103 is upheld, see 103 rejection below.
Finally, after consideration of the amendments, remarks and notes from the interview with the applicant, the examiner has found some argument to be persuasive and the rejections under 35 U.S.C. 101 and 102 have been withdrawn at this time. However, as stated above, the examiner has completed a search of the arts and has located new subject matter that is able to, in combination of the previously disclosed arts, teach or disclose the elements of the current claims. The examiner believes that combination of these arts would available at the time of filing this application and a person of ordinary skill in the art would have the motivation to combine, or use the teaches of the articles, to disclose the current claimed invention. The examiner would also like to note that the combination of arts also teaches or discloses the independent claims 1, 11, and 12. Therefore, the examiner has upheld the rejection under 35 U.S.C. 103, see 103 rejection below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 6-9, and 11-14 are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al, (Liu et al, “Real–Sim–Real Transfer for Real-World Robot Control Policy Learning with Deep Reinforcement Learning”, 2020, hereinafter “Liu”) in view of Zhang et al, (Zhang et al, “Adversarial discriminative sim-to-real transfer of visuo-motor policies”, 2019, hereinafter “Zhang”).
Regarding claim 1, Liu discloses, “A machine learning data generation device, comprising: at least one processor; and at least one memory device that stores a plurality of instructions which, when executed by the at least one processor, causes the at least one processor to execute:” (Policy Learning, pp. 9-10; “The task-relevant simulated environment was generated automatically based on our proposed method in the Mujoco physics engine [48] interfaced with OpenAI Gym [49], as is shown in Figure 6. For the manipulation task, the action dimension is 3, which moves the gripper in 3D space. For the navigation task, the action dimension is 2, which controls the TurtleBot’s navigation in 2D space. … For the Transfer-RGB, Transfer-Depth, DR, and DA methods, we built a simulated environment similar to our real-world robot working scenario in Gazebo simulator (https://gazebosim.org). The domain randomization technique refers to [9] (https://github.com/neka-nat/gazebo_domain_ randomization). All of the methods share the same neural networks. We initialized the convolutional layers of the policy networks by the weights pre-trained on ImageNet. The policy networks are trained with TensorFlow (https://www.tensorflow.org) on NVIDIA GTX1080. Table 1 summarizes the parameters used in our experiments.” This article discloses the use of different computer-based physic simulators. This also discloses the use of hardware which contains processors which are connected to memory systems to execute computer instructions.)
“acquiring, in association with a predetermined label, actual time series information indicating an operation state of a subject machine being one of a machine or an electric circuit;” (Policy Network, pp. 6; “The policy network takes simulated image
I
s
,
t
captured from the simulated environment at current time step t together with simulated images
I
s
,
t
-
1
and
I
s
,
t
-
2
at the time steps t - 1 and t - 2 as input.” This model will use temporal image data which contains a sequence of event to train a robotic system.) And (Policy Training, pp. 6; “At time step t, the robot agent takes an action at according to current state
I
s
,
t
and policy
π
θ
, receives reward
r
t
, and moves to the next state
I
s
,
t
+
1
.” The model will evaluate the input data as a state of the machine and will evaluate the robotic movement at each state.)
“executing physical simulation of generating a plurality of pieces of virtual time series information by sequentially calculating a virtual state after a unit time based on each of a plurality of parameters representing an internal state of the subject machine;” (Policy Network, pp. 6-7; “At time step t, the robot agent takes an action at according to current state
I
s
,
t
and policy
π
θ
, receives reward
r
t
, and moves to the next state
I
s
,
t
+
1
. Repeating the above procedure, an episode trajectory is obtained
τ
:
{
I
s
,
0
,
a
0
,
r
0
,
…
I
s
,
t
,
a
t
,
r
t
,
…
,
I
S
,
T
}
. The reward function
r
t
is set to be [See Equation (6)] where
d
=
x
-
x
*
is the Euclidean distance between the robot (or gripper) position x and the target object center
x
*
,
δ
is the threshold to determine whether the agent reaches the target position
(
δ
=
1
p
i
x
e
l
)
When encountering obstacles, the agent receives a reward of -10. We adopt the DRL method of proximal policy optimization (PPO) [31] to maximize a surrogate objective
L
c
l
i
p
(
θ
)
: [See Equation (7)] Where [See Equation (8)]
ε
=
0.2
, t specifies the time index in [0, T],
π
θ
o
l
d
denotes the old policy before the update, and
A
^
t
is the estimated advantage function
A
t
, [See Equation (9)] where
V
φ
(
I
s
,
t
)
is the estimated value function for state
I
s
,
t
, and the parameter
φ
is updated by regression on mean-squared error, [See Equation (10)]” This article uses Reinforcement Learning to take the current state of an environment and execute actions in that training space. This will perform these training actions in a simulated space using a simulation-to-real training policy. It will take in timed data, a subset of time stamped frames in a set of frames, and calculate the different states of the machine based on the different images and the previous state of the environment.)
“wherein physical simulation is executed using a physics engine;” (Figure 2, pp. 6; “Illustration of generating a task-relevant simulated environment. The target object is simplified to its geometry center of the corresponding regions in the semantic image. The robot (or gripper) is equivalent to a solid ball, the center and diameter of which are calculated from its semantic pixel region contour. The obstacles are completely preserved, and other irrelevant objects that cannot be objects are ignored.” This figure shows the simulated environment that the machine uses to train different polices. This will focus on giving negative rewards for collisions in this simulated environment.)
PNG
media_image1.png
202
764
media_image1.png
Greyscale
“wherein the plurality of parameters comprise values indicating characteristics of the subject machine;” (Policy Training, pp. 6; “Repeating the above procedure, an episode trajectory is obtained
τ
:
{
I
s
,
0
,
a
0
,
r
0
,
…
I
s
,
t
,
a
t
,
r
t
,
…
,
I
S
,
T
}
. The reward function
r
t
is set to be [See Equation (6)] where
d
=
x
-
x
*
is the Euclidean distance between the robot (or gripper) position x and the target object center
x
*
,
δ
is the threshold to determine whether the agent reaches the target position
(
δ
=
1
p
i
x
e
l
)
When encountering obstacles, the agent receives a reward of -10.” This system uses reinforcement learning concepts to train their model in a simulated environment. This will train a machine the parameters or movements of the policy after training is completed.)
“generating virtual time series information corresponding to a new internal state by executing the physical simulation through use of the new parameter value; and” (Deploying the Training Policy, pp. 7; “The sketch of deploying the policy trained from the simulated environment to the real-world scenario is shown in Figure 1b. Similar to the policy training period, semantic image
I
s
e
m
is segmented from the captured RGB image. Our method then synthesizes simulated-like images
I
s
y
n
in low-fidelity based on a segmented image respecting the following rules: the target object is simplified to be its geometry center; the robot (or gripper) is equivalent to a solid circle, the center and diameter of which are calculated from the semantic pixel region contour; the obstacles are completely preserved, and other irrelevant objects that cannot be obstacles are ignored. Similar to the training phase, at time step t of the policies employed period, the trained policy takes synthetic image
I
s
y
n
,
t
together with synthetic images
I
s
y
n
,
t
-
1
and
I
s
y
n
,
t
-
2
at time t - 1 and t - 2 as input and outputs actions that directly control the real-world robot.” This system will use the simulation to help train a robotic system. This article discloses experiments that focus on generating training data and policies for pick and place actions and movement through a simulated environment.)
“generating new machine learning data by associating the virtual time series information corresponding to the new internal state with the predetermined label corresponding to the new parameter value.” (Algorithm 1, RSR transfer method, pp. 8; This algorithm discloses the transfer learning method used in this article. This system will generate trained polices by taking real world data and embed it into a simulated environment. The model will then take this simulation a data and produce trained policies.)
Liu fails to explicitly discloses, “identifying one or more parameter values from the plurality of parameter values based on the plurality of pieces of virtual time series information and the actual time series information, and to associate the identified one or more parameter values with the predetermined label;” and “generating a new parameter value and the predetermined label corresponding to the new parameter value based on the one or more identified parameter values;”
However, Zhang discloses, “identifying one or more parameter values from the plurality of parameter values based on the plurality of pieces of virtual time series information and the actual time series information, and to associate the identified one or more parameter values with the predetermined label;” (ADT, pp. 1232; “ADT makes use of both adversarial and supervised losses to adapt a perception module with fewer labeled real images. In ADT, the perception module is divided into two parts: encoder and regressor. As shown in Figure 3, the encoder includes all the convolutional layers in a perception module; the regressor represents all the fully connected layers of the perception module. A perception module (source encoder + source regressor) is first pre-trained with simulated images (
I
S
) and their target object position labels (
x
*
S
), using the supervised loss [See Equation (4)] where
y
p
(
I
j
)
is the prediction of
x
*
j
for
I
j
. Here in the pretraining
I
=
I
S
,
x
*
=
x
*
S
The physical meaning of
x
*
guarantees the convenience of collecting labeled training data.” This model is able to take simulated data and real data and use them to train a machine learning system. This system uses a loss system to evaluate the domain label output from the Adversarial Discriminative Transfer module.) and (Figure. 3, pp. 1233; The Adversarial Discriminative Transfer section includes the sources and target encoders. This will take data from real and simulated data and use a discriminator to produce a domain label.)
PNG
media_image2.png
703
710
media_image2.png
Greyscale
“generating a new parameter value and the predetermined label corresponding to the new parameter value based on the one or more identified parameter values;” (The Training method, pp. 1232; “Perception The perception module is first pre-trained using labeled simulated data with a supervised loss
L
p
S
u
p
. Then it is adapted using both simulated and real data with a compound loss [See Equation (1)] where
L
p
A
d
is an adversarial loss. Definitions of the loss functions are introduced in Section 3.2. We call this perception training approach ADT. Control The control module is trained using supervised learning with only simulated data [See Equation (2)] where
y
c
(
S
j
)
is the prediction of joint velocity
v
j
for state
S
j
, here
S
=
Θ
(ground-truth); m is the number of samples.” This system is also trained using labeled data. This will learning the different joint angles based on the labeled data. This is seen in the control modules in Fig. 2 below.)
PNG
media_image1.png
202
764
media_image1.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zhang and Liu. Zhang teaches a machine learning system that is able to use simulated motion data and real data to train movements of a robotic system. Liu teaches A machine learning method that uses reinforcement learning and simulated training data to train movements of a robotic system. One of ordinary skill would have motivation to combine a machine learning model that is able to use generated adversarial data and a discriminator with art that teaches reinforcement learning methods to train robotic policies using simulated data. Further, both methods solve the issue of generating data to increase the amount of limited training samples, “The modular approach can also be used in more general ways. Although we explicitly equated the bottleneck layer with the target object position in this work, the bottleneck in general could be any explicit or latent low-dimensional features (as in an auto-encoder). The perception and control modules can also be trained with other methods such as unsupervised learning and reinforcement learning. The effectiveness of the modular approach for reinforcement learning (DQN) has been validated in a planar reaching task (Zhang et al., 2017a,b).” (Zhang, Value of a modular structure and end-to-end fine-tuning, pp. 1243).
Regarding claim 2, Zhang discloses, “the at least one memory device further stores the parameter value that is possible for each type of the predetermined label in association with the predetermined label,” (Task setup, pp. 1233; “The real-world task employs a Baxter robot’s left arm (7 DoFs) to reach a blue cuboid in clutter. All objects are arbitrarily placed in the operational area (50 cm × 60 cm) on a table, as shown in Figure 5(A). The blue cuboid has a side length of 6.5 cm. The robot observes environments through a monocular RGB camera in its right hand (Figure 1(A)), providing RGB images with a resolution of 256×256 (cropped from 640×400 images). The left arm is controlled in velocity mode. A reach is deemed successful, if the Euclidean distance between the top center of the target cuboid and the bottom center of the suction gripper (‘‘Top Center’’ and ‘‘Bottom Center’’ in Figure 5) is smaller than 4.6 cm (half of the diagonal length of any side of the cuboid” This system uses a Baxter arm to perform the learned motions. This is connected to computing device which contains the movement parameters for the arm.) and (Datasets collection, pp. 1234; “Perception datasets contain a number of image–position (
I
-
x
*
) pairs. In this work, we label the position of the target cuboid top center as the target position
x
*
rather than its center of mass. Figure 6 shows some samples of the collected simulated and real images for the benchmark task.” This uses labeled data, that is previously labeled, to train the robotic system. This data would need to be stored in some form in order for the system to process it.)
“wherein the new parameter value is generated based on the parameter stored in the at least one memory device.” (Datasets collection, pp. 1234; “Perception datasets contain a number of image–position (
I
-
x
*
) pairs. In this work, we label the position of the target cuboid top center as the target position
x
*
rather than its center of mass. Figure 6 shows some samples of the collected simulated and real images for the benchmark task.” This system will use the labeled data to train the robotic system.) and (Fig. 14, pp. 1233; This figure shows the Baxter art and the environment it is acting in. This system uses a Baxter arm and processing system which stores movement parameters for the arm.)
Regarding claim 3, Zhang discloses, “wherein the predetermined label corresponding to the new parameter value is determined based on a relationship between the new parameter value and a group of the parameter values stored in the at least one memory device and corresponding to each predetermined label.” (Datasets collection, pp. 1235; “The real images shown in Figure 6 were collected in the real world on a Baxter robot (Figure 1(B)) with random objects and left arm configurations. There are 11 real distractor objects in total. The ground-truth position of the target blue cuboid was collected by putting the end-effector bottom center on the cuboid top center and recording the left arm configuration (target configuration
q
*
) for forward kinematics, i.e.,
x
*
=
K
(
q
*
)
. The ground-truth position collected in this way is accurate enough for the benchmark task, although some errors might be caused by manually matching the end-effector with the cuboid. This ground truth position collection method was also used in the control performance evaluation in Section 5.” The system uses labeled data to generate movements and train the model. As stated above this uses computing systems to process and store the robotic movements.)
Regarding claim 6, Zhang discloses, “wherein the virtual time series information corresponding to the new internal state is generated by a generative adversarial network (GAN) based on a result of the physical simulation executed through use of the new parameter value.” (Figure 3, pp. 1233; “In ADT, the perception module is divided into two parts: encoder and regressor. The encoder includes all the convolutional layers; the regressor represents all the remaining fully connected layers. We first pre-train a perception module (source encoder + source regressor) with
L
p
S
u
p
using simulated images (
I
S
) and their target object position labels (
x
*
S
). The source encoder is then locked and used as a reference in the ADT to train a target encoder
E
r
with
L
p
A
d
using both simulated (
I
S
) and real (
I
R
) images without labels. In addition to the adversarial loss,
L
p
S
u
p
is also used to train the target encoder and regressor with a small number of labeled real images (
I
R
and
x
*
R
). The target encoder and regressor are initialized with the weights in the source encoder and regressor. The discriminator consists of multiple fully connected layers.” This article discloses the use of a GAN and training robotic movements in a simulated environment. This will use generated data, generated by a generator module and use a discriminator module to evaluate the generated data. This system is designed to train visuomotor policies for robotic systems.)
PNG
media_image2.png
703
710
media_image2.png
Greyscale
Regarding claim 7, Zhang discloses, “wherein the virtual time series information corresponding to the new internal state is generated by a generative adversarial network (GAN) based on a result of the physical simulation executed through use of the new parameter value.” (Figure 3, pp. 1233; “In ADT, the perception module is divided into two parts: encoder and regressor. The encoder includes all the convolutional layers; the regressor represents all the remaining fully connected layers. We first pre-train a perception module (source encoder + source regressor) with
L
p
S
u
p
using simulated images (
I
S
) and their target object position labels (
x
*
S
). The source encoder is then locked and used as a reference in the ADT to train a target encoder
E
r
with
L
p
A
d
using both simulated (
I
S
) and real (
I
R
) images without labels. In addition to the adversarial loss,
L
p
S
u
p
is also used to train the target encoder and regressor with a small number of labeled real images (
I
R
and
x
*
R
). The target encoder and regressor are initialized with the weights in the source encoder and regressor. The discriminator consists of multiple fully connected layers.” This article discloses the use of a GAN and training robotic movements in a simulated environment. This will use generated data, generated by a generator module and use a discriminator module to evaluate the generated data. This system is designed to train visuomotor policies for robotic systems. This system will pretrain the policies and then test the parameters of the policies simulated in the environment. See Figure 3 above.)
Regarding claim 8, Zhang discloses, “wherein the virtual time series information corresponding to the new internal state is generated by a generative adversarial network (GAN) based on a result of the physical simulation executed through use of the new parameter value.” (Figure 3, pp. 1233; “In ADT, the perception module is divided into two parts: encoder and regressor. The encoder includes all the convolutional layers; the regressor represents all the remaining fully connected layers. We first pre-train a perception module (source encoder + source regressor) with
L
p
S
u
p
using simulated images (
I
S
) and their target object position labels (
x
*
S
). The source encoder is then locked and used as a reference in the ADT to train a target encoder
E
r
with
L
p
A
d
using both simulated (
I
S
) and real (
I
R
) images without labels. In addition to the adversarial loss,
L
p
S
u
p
is also used to train the target encoder and regressor with a small number of labeled real images (
I
R
and
x
*
R
). The target encoder and regressor are initialized with the weights in the source encoder and regressor. The discriminator consists of multiple fully connected layers.” This article discloses the use of a GAN and training robotic movements in a simulated environment. This will use generated data, generated by a generator module and use a discriminator module to evaluate the generated data. This system is designed to train visuomotor policies for robotic systems. This system will pretrain the policies and then test the parameters of the policies simulated in the environment. See Figure 3 above.)
Regarding claim 9, Zhang discloses, “wherein the virtual time series information corresponding to the new internal state is generated by a generative adversarial network (GAN) based on a result of the physical simulation executed through use of the new parameter value.” (Figure 3, pp. 1233; “In ADT, the perception module is divided into two parts: encoder and regressor. The encoder includes all the convolutional layers; the regressor represents all the remaining fully connected layers. We first pre-train a perception module (source encoder + source regressor) with
L
p
S
u
p
using simulated images (
I
S
) and their target object position labels (
x
*
S
). The source encoder is then locked and used as a reference in the ADT to train a target encoder
E
r
with
L
p
A
d
using both simulated (
I
S
) and real (
I
R
) images without labels. In addition to the adversarial loss,
L
p
S
u
p
is also used to train the target encoder and regressor with a small number of labeled real images (
I
R
and
x
*
R
). The target encoder and regressor are initialized with the weights in the source encoder and regressor. The discriminator consists of multiple fully connected layers.” This article discloses the use of a GAN and training robotic movements in a simulated environment. This will use generated data, generated by a generator module and use a discriminator module to evaluate the generated data. This system is designed to train visuomotor policies for robotic systems. This system will pretrain the policies and then test the parameters of the policies simulated in the environment. See Figure 3 above.)
Regarding claim 11, Liu discloses, “A machine learning model generation method, comprising:” (Introduction, pp. 2; “(1) We present a new learning paradigm to train control policies for real-world robots with the DRL method. The learning pipeline is divided into a real-to-sim training phase and a sim-to-real inference phase, which trains robot control policy with a higher generalization capability and lower costs. (2) The proposed method automatically constructs a task-relevant simulated environment for policy learning based on semantic information of real-world working scenarios and coordinate transformation, which avoids the challenging problem of manually creating the simulated environments with high fidelity, endowing the policy learning process with high efficiency. (3) The proposed method directly employs the trained policy in real-world scenarios without any real-world training data or fine-tuning.
“acquiring, in association with a predetermined label, actual time series information indicating an operation state of a subject machine being one of a machine or an electric circuit;” (Policy Network, pp. 6; “The policy network takes simulated image
I
s
,
t
captured from the simulated environment at current time step t together with simulated images
I
s
,
t
-
1
and
I
s
,
t
-
2
at the time steps t - 1 and t - 2 as input.” This model will use temporal image data which contains a sequence of event to train a robotic system.) And (Policy Training, pp. 6; “At time step t, the robot agent takes an action at according to current state
I
s
,
t
and policy
π
θ
, receives reward
r
t
, and moves to the next state
I
s
,
t
+
1
.” The model will evaluate the input data as a state of the machine and will evaluate the robotic movement at each state.)
“executing physical simulation of generating a plurality of pieces of virtual time series information by sequentially calculating a virtual state after a unit time based on each of a plurality of parameters representing an internal state of the subject machine;” (Policy Network, pp. 6-7; “At time step t, the robot agent takes an action at according to current state
I
s
,
t
and policy
π
θ
, receives reward
r
t
, and moves to the next state
I
s
,
t
+
1
. Repeating the above procedure, an episode trajectory is obtained
τ
:
{
I
s
,
0
,
a
0
,
r
0
,
…
I
s
,
t
,
a
t
,
r
t
,
…
,
I
S
,
T
}
. The reward function
r
t
is set to be [See Equation (6)] where
d
=
x
-
x
*
is the Euclidean distance between the robot (or gripper) position x and the target object center
x
*
,
δ
is the threshold to determine whether the agent reaches the target position
(
δ
=
1
p
i
x
e
l
)
When encountering obstacles, the agent receives a reward of -10. We adopt the DRL method of proximal policy optimization (PPO) [31] to maximize a surrogate objective
L
c
l
i
p
(
θ
)
: [See Equation (7)] Where [See Equation (8)]
ε
=
0.2
, t specifies the time index in [0, T],
π
θ
o
l
d
denotes the old policy before the update, and
A
^
t
is the estimated advantage function
A
t
, [See Equation (9)] where
V
φ
(
I
s
,
t
)
is the estimated value function for state
I
s
,
t
, and the parameter
φ
is updated by regression on mean-squared error, [See Equation (10)]” This article uses Reinforcement Learning to take the current state of an environment and execute actions in that training space. This will perform these training actions in a simulated space using a simulation-to-real training policy. It will take in timed data, a subset of time stamped frames in a set of frames, and calculate the different states of the machine based on the different images and the previous state of the environment.)
“wherein physical simulation is executed using a physics engine;” (Figure 2, pp. 6; “Illustration of generating a task-relevant simulated environment. The target object is simplified to its geometry center of the corresponding regions in the semantic image. The robot (or gripper) is equivalent to a solid ball, the center and diameter of which are calculated from its semantic pixel region contour. The obstacles are completely preserved, and other irrelevant objects that cannot be objects are ignored.” This figure shows the simulated environment that the machine uses to train different polices. This will focus on giving negative rewards for collisions in this simulated environment.)
PNG
media_image1.png
202
764
media_image1.png
Greyscale
“wherein the plurality of parameters comprise values indicating characteristics of the subject machine;” (Policy Training, pp. 6; “Repeating the above procedure, an episode trajectory is obtained
τ
:
{
I
s
,
0
,
a
0
,
r
0
,
…
I
s
,
t
,
a
t
,
r
t
,
…
,
I
S
,
T
}
. The reward function
r
t
is set to be [See Equation (6)] where
d
=
x
-
x
*
is the Euclidean distance between the robot (or gripper) position x and the target object center
x
*
,
δ
is the threshold to determine whether the agent reaches the target position
(
δ
=
1
p
i
x
e
l
)
When encountering obstacles, the agent receives a reward of -10.” This system uses reinforcement learning concepts to train their model in a simulated environment. This will train a machine the parameters or movements of the policy after training is completed.)
Liu fails to explicitly disclose, “identifying one or more parameter values from the plurality of parameter values based on the plurality of pieces of virtual time series information and the actual time series information, and associating the identified one or more parameter values with the predetermined label;” and “generating a new parameter value and the predetermined label corresponding to the new parameter value based on the one or more parameter values identified in the identifying of the one or more parameter values;”.
However, Zhang discloses, “identifying one or more parameter values from the plurality of parameter values based on the plurality of pieces of virtual time series information and the actual time series information, and associating the identified one or more parameter values with the predetermined label;” (ADT, pp. 1232; “ADT makes use of both adversarial and supervised losses to adapt a perception module with fewer labeled real images. In ADT, the perception module is divided into two parts: encoder and regressor. As shown in Figure 3, the encoder includes all the convolutional layers in a perception module; the regressor represents all the fully connected layers of the perception module. A perception module (source encoder + source regressor) is first pre-trained with simulated images (
I
S
) and their target object position labels (
x
*
S
), using the supervised loss [See Equation (4)] where
y
p
(
I
j
)
is the prediction of
x
*
j
for
I
j
. Here in the pretraining
I
=
I
S
,
x
*
=
x
*
S
The physical meaning of
x
*
guarantees the convenience of collecting labeled training data.” This model is able to take simulated data and real data and use them to train a machine learning system. This system uses a loss system to evaluate the domain label output from the Adversarial Discriminative Transfer module.) and (Figure. 3, pp. 1233; The Adversarial Discriminative Transfer section includes the sources and target encoders. This will take data from real and simulated data and use a discriminator to produce a domain label.)
PNG
media_image2.png
703
710
media_image2.png
Greyscale
“generating a new parameter value and the predetermined label corresponding to the new parameter value based on the one or more parameter values identified in the identifying of the one or more parameter values;” (The Training method, pp. 1232; “Perception The perception module is first pre-trained using labeled simulated data with a supervised loss
L
p
S
u
p
. Then it is adapted using both simulated and real data with a compound loss [See Equation (1)] where
L
p
A
d
is an adversarial loss. Definitions of the loss functions are introduced in Section 3.2. We call this perception training approach ADT. Control The control module is trained using supervised learning with only simulated data [See Equation (2)] where
y
c
(
S
j
)
is the prediction of joint velocity
v
j
for state
S
j
, here
S
=
Θ
(ground-truth); m is the number of samples.” This system is also trained using labeled data. This will learning the different joint angles based on the labeled data. This is seen in the control modules in Fig. 2 Below.)
PNG
media_image3.png
314
1074
media_image3.png
Greyscale
Regarding claim 12, Liu discloses, “A Non-transitory computer-readable information storage medium for storing a program for causing a computer to execute:” (Policy Learning, pp. 9-10; “The task-relevant simulated environment was generated automatically based on our proposed method in the Mujoco physics engine [48] interfaced with OpenAI Gym [49], as is shown in Figure 6. For the manipulation task, the action dimension is 3, which moves the gripper in 3D space. For the navigation task, the action dimension is 2, which controls the TurtleBot’s navigation in 2D space. … For the Transfer-RGB, Transfer-Depth, DR, and DA methods, we built a simulated environment similar to our real-world robot working scenario in Gazebo simulator (https://gazebosim.org). The domain randomization technique refers to [9] (https://github.com/neka-nat/gazebo_domain_ randomization). All of the methods share the same neural networks. We initialized the convolutional layers of the policy networks by the weights pre-trained on ImageNet. The policy networks are trained with TensorFlow (https://www.tensorflow.org) on NVIDIA GTX1080. Table 1 summarizes the parameters used in our experiments.” This article discloses the use of different computer-based physic simulators. This also discloses the use of hardware which contains processors which are connected to memory systems to execute computer instructions.)
“acquiring, in association with a predetermined label, actual time series information indicating an operation state of a subject machine being one of a machine or an electric circuit;” (Policy Network, pp. 6; “The policy network takes simulated image
I
s
,
t
captured from the simulated environment at current time step t together with simulated images
I
s
,
t
-
1
and
I
s
,
t
-
2
at the time steps t - 1 and t - 2 as input.” This model will use temporal image data which contains a sequence of event to train a robotic system.) And (Policy Training, pp. 6; “At time step t, the robot agent takes an action at according to current state
I
s
,
t
and policy
π
θ
, receives reward
r
t
, and moves to the next state
I
s
,
t
+
1
.” The model will evaluate the input data as a state of the machine and will evaluate the robotic movement at each state.)
“executing physical simulation of generating a plurality of pieces of virtual time series information by sequentially calculating a virtual state after a unit time based on each of a plurality of parameters representing an internal state of the subject machine;” (Policy Network, pp. 6-7; “At time step t, the robot agent takes an action at according to current state
I
s
,
t
and policy
π
θ
, receives reward
r
t
, and moves to the next state
I
s
,
t
+
1
. Repeating the above procedure, an episode trajectory is obtained
τ
:
{
I
s
,
0
,
a
0
,
r
0
,
…
I
s
,
t
,
a
t
,
r
t
,
…
,
I
S
,
T
}
. The reward function
r
t
is set to be [See Equation (6)] where
d
=
x
-
x
*
is the Euclidean distance between the robot (or gripper) position x and the target object center
x
*
,
δ
is the threshold to determine whether the agent reaches the target position
(
δ
=
1
p
i
x
e
l
)
When encountering obstacles, the agent receives a reward of -10. We adopt the DRL method of proximal policy optimization (PPO) [31] to maximize a surrogate objective
L
c
l
i
p
(
θ
)
: [See Equation (7)] Where [See Equation (8)]
ε
=
0.2
, t specifies the time index in [0, T],
π
θ
o
l
d
denotes the old policy before the update, and
A
^
t
is the estimated advantage function
A
t
, [See Equation (9)] where
V
φ
(
I
s
,
t
)
is the estimated value function for state
I
s
,
t
, and the parameter
φ
is updated by regression on mean-squared error, [See Equation (10)]” This article uses Reinforcement Learning to take the current state of an environment and execute actions in that training space. This will perform these training actions in a simulated space using a simulation-to-real training policy. It will take in timed data, a subset of time stamped frames in a set of frames, and calculate the different states of the machine based on the different images and the previous state of the environment.)
“wherein physical simulation is executed using a physics engine;” (Figure 2, pp. 6; “Illustration of generating a task-relevant simulated environment. The target object is simplified to its geometry center of the corresponding regions in the semantic image. The robot (or gripper) is equivalent to a solid ball, the center and diameter of which are calculated from its semantic pixel region contour. The obstacles are completely preserved, and other irrelevant objects that cannot be objects are ignored.” This figure shows the simulated environment that the machine uses to train different polices. This will focus on giving negative rewards for collisions in this simulated environment.)
PNG
media_image1.png
202
764
media_image1.png
Greyscale
“wherein the plurality of parameters comprise values indicating characteristics of the subject machine;” (Policy Training, pp. 6; “Repeating the above procedure, an episode trajectory is obtained
τ
:
{
I
s
,
0
,
a
0
,
r
0
,
…
I
s
,
t
,
a
t
,
r
t
,
…
,
I
S
,
T
}
. The reward function
r
t
is set to be [See Equation (6)] where
d
=
x
-
x
*
is the Euclidean distance between the robot (or gripper) position x and the target object center
x
*
,
δ
is the threshold to determine whether the agent reaches the target position
(
δ
=
1
p
i
x
e
l
)
When encountering obstacles, the agent receives a reward of -10.” This system uses reinforcement learning concepts to train their model in a simulated environment. This will train a machine the parameters or movements of the policy after training is completed.)
Liu fails to explicitly disclose, “identifying one or more parameter values from the plurality of parameter values based on the plurality of pieces of virtual time series information and the actual time series information, and associating the identified one or more parameter values with the predetermined label;” and “generating a new parameter value and the predetermined label corresponding to the new parameter value based on the one or more parameter values identified in the identifying of the one or more parameter values;”.
However, Zhang discloses, “identifying one or more parameter values from the plurality of parameter values based on the plurality of pieces of virtual time series information and the actual time series information, and associating the identified one or more parameter values with the predetermined label;” (ADT, pp. 1232; “ADT makes use of both adversarial and supervised losses to adapt a perception module with fewer labeled real images. In ADT, the perception module is divided into two parts: encoder and regressor. As shown in Figure 3, the encoder includes all the convolutional layers in a perception module; the regressor represents all the fully connected layers of the perception module. A perception module (source encoder + source regressor) is first pre-trained with simulated images (
I
S
) and their target object position labels (
x
*
S
), using the supervised loss [See Equation (4)] where
y
p
(
I
j
)
is the prediction of
x
*
j
for
I
j
. Here in the pretraining
I
=
I
S
,
x
*
=
x
*
S
The physical meaning of
x
*
guarantees the convenience of collecting labeled training data.” This model is able to take simulated data and real data and use them to train a machine learning system. This system uses a loss system to evaluate the domain label output from the Adversarial Discriminative Transfer module.) and (Figure. 3, pp. 1233; The Adversarial Discriminative Transfer section includes the sources and target encoders. This will take data from real and simulated data and use a discriminator to produce a domain label.)
PNG
media_image2.png
703
710
media_image2.png
Greyscale
“generating a new parameter value and the predetermined label corresponding to the new parameter value based on the one or more parameter values identified in the identifying of the one or more parameter values;” (The Training method, pp. 1232; “Perception The perception module is first pre-trained using labeled simulated data with a supervised loss
L
p
S
u
p
. Then it is adapted using both simulated and real data with a compound loss [See Equation (1)] where
L
p
A
d
is an adversarial loss. Definitions of the loss functions are introduced in Section 3.2. We call this perception training approach ADT. Control The control module is trained using supervised learning with only simulated data [See Equation (2)] where
y
c
(
S
j
)
is the prediction of joint velocity
v
j
for state
S
j
, here
S
=
Θ
(ground-truth); m is the number of samples.” This system is also trained using labeled data. This will learning the different joint angles based on the labeled data. This is seen in the control modules in Fig. 2.)
PNG
media_image3.png
314
1074
media_image3.png
Greyscale
Regarding claim 13, Liu discloses, “wherein the physics engine is configured to execute a collision determination, a dynamic simulation, a fluid simulation, or a destruction simulation.” (Figure 5, pp. 9; This figure shows an example of the simulated environment. The top is the real world and the bottom images represent the sematic segmentation in the simulated environment. This will evaluate for collision of the systems and the objects.) and (Policy Training, pp. 6-7; “The reward function
r
t
is set to be [See Equation (6)] where
d
=
x
-
x
*
is the Euclidean distance between the robot (or gripper) position x and the target object center
x
*
,
δ
is the threshold to determine whether the agent reaches the target position
(
δ
=
1
p
i
x
e
l
)
When encountering obstacles, the agent receives a reward of -10.” This discloses the penalty for a collision in the simulated environment for training.)
PNG
media_image4.png
256
655
media_image4.png
Greyscale
Regarding claim 14, Liu discloses, “wherein the plurality of parameters comprise a mechanical characteristic including a kinematic state variable or a physical constant.” (Policy Network, pp. 6; “The output of the network is the mean
μ
θ
(
I
s
,
t
)
and variance
σ
θ
(
I
s
,
t
)
of a Gaussian policy
π
θ
∙
I
s
,
t
=
N
(
μ
θ
I
s
,
t
,
σ
θ
I
s
,
t
)
, where θ represents the parameters of the policy neural network.” This network will train a system the different parameters and policies for a robotic system. This article uses reinforcement learning to transfer learned policies to a system) and (Policy Training, pp. 6; “At time step t, the robot agent takes an action at according to current state
I
s
,
t
and policy
π
θ
, receives reward
r
t
, and moves to the next state
I
s
,
t
+
1
. Repeating the above procedure, an episode trajectory is obtained
τ
:
{
I
s
,
0
,
a
0
,
r
0
,
…
I
s
,
t
,
a
t
,
r
t
,
…
,
I
S
,
T
}
.” Reinforcement learning produces an action in a given state and based on that action a reward is given to the system and the state is updated.)
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Liu and Zhang in view of Li et al, (Li et al, “MAD-GAN: Multivariate Anomaly Detection for Time Series Data with Generative Adversarial Networks”, 2019, hereinafter “Li”).
Regarding claim 4, Li discloses, “wherein the predetermined label indicates whether the operation state of the subject machine is normal or abnormal, and” (GAN-Based Anomaly Detection, pp. 706; “Let us now formulate the anomaly detection problem using GAN. Given a training dataset
X
⊆
R
N
×
T
with T streams and M measurements for each stream, and a test dataset
X
t
e
s
t
⊆
R
N
×
T
with T streams and N measurements for each stream, the task is to assign binary (0 for normal and 1 for anomalous) labels to the measurements of test dataset.” The proposed system is designed to detect anomalies in a system. It will label data with different states of the system as normal or abnormal.)
“wherein the new parameter value corresponding to the predetermined label indicating that the operation state is abnormal is selectively generated.” (GAN-Based Anomaly Detection, pp. 706; “Let us now formulate the anomaly detection problem using GAN. Given a training dataset
X
⊆
R
N
×
T
with T streams and M measurements for each stream, and a test dataset
X
t
e
s
t
⊆
R
N
×
T
with T streams and N measurements for each stream, the task is to assign binary (0 for normal and 1 for anomalous) labels to the measurements of test dataset. Note that we assume here that all the points in the training dataset are normal.” This model naturally assumes that all data points in the training set are labeled as normal. This would teach that the system recognizes that initial labels are normal and that the system needs to perform actions and evaluate that data in order to determine if it is abnormal data.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zhang, Liu and Li. Zhang teaches a machine learning system that is able to use simulated motion data and real data to train movements of a robotic system. Liu teaches A machine learning method that uses reinforcement learning and simulated training data to train movements of a robotic system. Li teaches the use of a Conditional GAN which is able to detect anomalies in complex systems. One of ordinary skill would have motivation to combine a machine learning model that is able to use generated adversarial data and a discriminator with art that teaches reinforcement learning methods to train robotic policies using simulated data, with a system that uses similar concepts and is able to detect anomalies in complex systems using GAN concepts, “We evaluated the anomaly detection performance of MAD-GAN on the aforementioned SWaT and WADI. For comparison on the anomaly detection performance, we applied PCA, One-Class SVM (OCSVM), K-Nearest Neighbor Multivariate Anomaly Detection for Time Series Data with GAN 711 (KNN), Feature Bagging (FB), and Auto-Encoder (AE) that are popular unsupervised anomaly detection methods3 on the datasets. To compare with a GANbased method, we also tested both datasets with the Efficient GAN-based (EGAN) method of [8] whose discriminator and generator were implemented as fully connected neural networks.” (Results, pp. 710-711)
Claims 5 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Liu and Zhang in view of Nguyen et al, (Nguyen et al, “Transferring Visuomotor Learning from Simulation to the Real World for Robotics Manipulation Tasks”, 2018, hereinafter “Nguyen”).
Regarding claim 5, Nguyen discloses, “wherein the new parameter value is indicated as a position on a parameter distribution diagram representing a distribution of the one or more parameter values associated with the predetermined label,” (Action-based dataset generation, pp. 6669; “Thus using the iCub simulator [28] has the additional benet that it allows the collection of a larger, more diverse dataset compared to the physical robot. The resulting arm coverage in the simulated workspace is shown in Figure 2, with the arm's sweeping volume approximated as
V
≈
S
c
o
n
v
e
x
∙
d
-
=
0.39
m
3
(where
S
c
o
n
v
e
x
is the area covered by the convex hull spanned in the plane of the y- and z-axes, and
d
-
is the average distance between the robot's end-effector and the robot's shoulder in direction of the x-axis [motion in the x-axis is negligible]).” This system will train a robotic system using trained policies which were generated in a simulated environment. This article will evaluate sets of motion data based on the constraints input into the system which represents a distribution of all possible motions the robot can perform. This is used in training the system)
“wherein the one or more parameter values are depicted in the parameter distribution diagram; and” (Fig. 2, pp. 6669; This discloses the right arm coverage using the random motor commands in the simulation. This is a distribution of all the movements.)
PNG
media_image5.png
282
380
media_image5.png
Greyscale
“wherein the parameter value that is generated as the new parameter value is a parameter value indicated by a position at which a predetermined number of the parameter values exist within a predetermined range on the parameter distribution diagram.” (Action-based dataset generation, pp. 6669; “Covering a sufficiently large working space requires setting large values for A, H and f. While this is feasible in simulation, lower values have to be used on the real robot due to mechanic stress constraints. For this work, we set the parameters as following: A = 5, f = 0.2 and H = 10 in simulation or H = 5 for the physical robot. Thus using the iCub simulator [28] has the additional benefit that it allows the collection of a larger, more diverse dataset compared to the physical robot.” This system will use the distribution of all movements in the given constraints. This is used to train in the simulated environment to train new policies from the generated data.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zhang, Liu and Nguyen. Zhang teaches a machine learning system that is able to use simulated motion data and real data to train movements of a robotic system. Liu teaches A machine learning method that uses reinforcement learning and simulated training data to train movements of a robotic system. Nguyen discloses using machine learning to train a robotic system to perform actions using simulated data. One of ordinary skill would have motivation to combine a machine learning model that is able to use generated adversarial data and a discriminator with art that teaches reinforcement learning methods to train robotic policies using simulated data, with a system that uses machine learning concepts to train a robotic system with simulated data to increase the sample size, “Our next step will be applying the learned visuomotor mapping in vision-based action planning, including reaching with obstacle avoidance and object grasping. For the reaching with avoidance task, we plan to use the proposed method to produce imagined images of the robot's arm, which can be checked against collisions with objects. An advantage of our method is that the planning problem is mapped from vision space to joint space in real-time, where well-established motion planning techniques such as Rapidly exploring Random Trees (RRT*) and Probabilistic RoadMap (PRM*) [34] can be applied. Another benet of our framework is the feasibility of integrating the touch modality using the tactile sensors of the iCub.” (Nguyen, Conclusions, pp. 6673)
Regarding claim 10, Zhang discloses, “wherein the virtual time series information corresponding to the new internal state is generated by a generative adversarial network (GAN) based on a result of the physical simulation executed through use of the new parameter value.” (Figure 3, pp. 1233; “In ADT, the perception module is divided into two parts: encoder and regressor. The encoder includes all the convolutional layers; the regressor represents all the remaining fully connected layers. We first pre-train a perception module (source encoder + source regressor) with
L
p
S
u
p
using simulated images (
I
S
) and their target object position labels (
x
*
S
). The source encoder is then locked and used as a reference in the ADT to train a target encoder
E
r
with
L
p
A
d
using both simulated (
I
S
) and real (
I
R
) images without labels. In addition to the adversarial loss,
L
p
S
u
p
is also used to train the target encoder and regressor with a small number of labeled real images (
I
R
and
x
*
R
). The target encoder and regressor are initialized with the weights in the source encoder and regressor. The discriminator consists of multiple fully connected layers.” This article discloses the use of a GAN and training robotic movements in a simulated environment. This will use generated data, generated by a generator module and use a discriminator module to evaluate the generated data. This system is designed to train visuomotor policies for robotic systems. This system will pretrain the policies and then test the parameters of the policies simulated in the environment. See Figure 3 above.)
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAUL MICHAEL GALVIN-SIEBENALER whose telephone number is (571)272-1257. The examiner can normally be reached Monday - Friday 8AM to 5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PAUL M GALVIN-SIEBENALER/Examiner, Art Unit 2147
/ERIC NILSSON/Primary Examiner, Art Unit 2151
Prosecution Insights