DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment/Status of Claims
Claims 1, 4, 6-9, 11-12, 15-17, and 19-20 were amended.
Claims 2, 10, and 18 were cancelled.
Claims 1, 3-9, 11-17, and 19-20 are pending and examined herein.
Claims 1, 8, 9, 16, 17, and 20 are objected to.
Claims 16 is rejected under 35 U.S.C. 112(b).
Claims 1, 3-9, 11-17, and 19-20 are rejected under 35 U.S.C. 101.
Claims 1, 3-9, 11-17, and 19-20 are rejected under 35 U.S.C. 103.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 03/30/2026 has been entered.
Response to Arguments
Applicant’s arguments, see page 12, filed 10/14/2025, with respect to the 35 U.S.C. 112(b) rejection of claims 6-8, 15, 16, 19, and 20 have been fully considered and are persuasive. The 35 U.S.C. 112(b) rejection of claims 6-8, 15, 16, 19, and 20 has been withdrawn.
Applicant's arguments filed 35 U.S.C. 101 rejection of claims 1-20 have been fully considered but they are not persuasive.
Applicant argues, see pages 11-15 that "Under Prong One of Step 2A, Applicant respectfully submits that claims 1, 9, and 17 do not recite any of the three groupings of subject matter that may be considered as an abstract idea.”
Examiner respectfully disagrees. The following limitations are identified as abstract ideas:
identify a third state from among the plurality of states …, wherein the third state is an intermediate state for switching from the first state to the second state; (Identifying a state of a visual object can practically be performed in the human mind. This is a mental process.)
based at least in part on whether switching from the first state to the second state through the third state is performed within predetermined time, determine a compensation value for the data on the third state; (Determining a compensation value based on whether the state switching is performed within a predetermined time is the mental process of evaluation.)
In regards to the determination of a compensation value, Applicant argues "For example, claim 1 recites a "compensation value," determined "based at least in part on whether switching from the first state to the second state through the third state is performed within a predetermined time and difference between the first state and the third state," which is a specific mechanism designed to solve a computer-centric problem (e.g., rendering latency and responsiveness in virtual environments)."
Firstly, MPEP 2106.05(a) states "Thus, an examiner should evaluate whether a claim contains an improvement to the functioning of a computer or to any other technology or technical field at Step 2A Prong Two and Step 2B, as well as when considering whether the claim has such self-evident eligibility that it qualifies for the streamlined analysis"
Thus, in Prong One of Step 2A, the improvement to technology consideration does not apply to determining whether a claim recites a judicial exception.
Additionally, MPEP 2106.05(a) states "It is important to note, the judicial exception alone cannot provide the improvement. The improvement can be provided by one or more additional elements. See the discussion of Diamond v. Diehr, 450 U.S. 175, 187 and 191-92, 209 USPQ 1, 10 (1981)) in subsection II, below. In addition, the improvement can be provided by the additional element(s) in combination with the recited judicial exception. See MPEP § 2106.04(d) (discussing Finjan, Inc. v. Blue Coat Sys., Inc., 879 F.3d 1299, 1303-04, 125 USPQ2d 1282, 1285-87 (Fed. Cir. 2018)). Thus, it is important for examiners to analyze the claim as a whole when determining whether the claim provides an improvement to the functioning of computers or an improvement to other technology or technical field."
As the determination of the compensation value is an abstract idea of mental process, it alone cannot show an improvement to technology, and thus does not integrate the judicial exception into a practical application in Prong Two of Step 2A nor amounts to significantly more than the judicial exception in Step 2B.
Applicant further argues "Identifying a part of the plurality of states based on the differences between states is not an abstract mental comparison. It is a highly specific data processing step (pruning) that physically restricts the search space within the database. This targeted reduction of data processing directly improves the computational efficiency and training speed of the neural network."
Examiner respectfully disagrees. As written in the claims, the identification of a part of the plurality of states is a mental process, as one could practically perform the identification of a part of a plurality of states of visual objects based on the difference between states. For example, one could look at the different positions of the visual object at the second state and the fourth state and determine states that fall in between the two states that will be used to identify the third state.
Additionally, MPEP 2106.05(a) states "It is important to note, the judicial exception alone cannot provide the improvement. The improvement can be provided by one or more additional elements. See the discussion of Diamond v. Diehr, 450 U.S. 175, 187 and 191-92, 209 USPQ 1, 10 (1981)) in subsection II, below. In addition, the improvement can be provided by the additional element(s) in combination with the recited judicial exception. See MPEP § 2106.04(d) (discussing Finjan, Inc. v. Blue Coat Sys., Inc., 879 F.3d 1299, 1303-04, 125 USPQ2d 1282, 1285-87 (Fed. Cir. 2018)). Thus, it is important for examiners to analyze the claim as a whole when determining whether the claim provides an improvement to the functioning of computers or an improvement to other technology or technical field."
Thus, the abstract idea of "identify, based at least in part on difference between the second state and the fourth state, a part of the plurality of states used for identifying the third state" cannot show an improvement to technology alone. Therefore, the limitation does not integrate the judicial exception into a practical application nor amounts to significantly more than the judicial exception.
Applicant’s arguments, see pages 15-18, filed 3/30/2026, with respect to the 35 U.S.C. 103 rejection of claims 1, 3-9, 11-17, and 19-20 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Kyungho (“Interactive Character Animation by Learning Multi-Objective Control”, 2018), Pardo (“Time Limits in Reinforcement Learning”, 2018) and Schulman (“Proximal Policy Optimization”, 2017). Note that the combination of Kyungho, Pardo, and Schulman does not render Kyungho inoperable. Schulman teaches a differentiable loss that includes the value function in reinforcement learning, which could be used in addition to the teachings of Kyungho. See amended 35 U.S.C. 103 rejection below.
Claim Objections
Claims 1, 8, 9, 16, 17, and 20 are objected to because of the following informalities:
“difference between the first state and the third state” in claims 1, 9, and 17 should be “a difference between the first state and the third state”.
“difference between states of the visual object before the first state” in claim 16 and 20 should be “a difference between states of the visual object before the first state”.
“difference between the first state and a state immediately before the first state” in claims 8, 16, and 20 should be “a difference between the first state and a state immediately before the first state”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 16 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 16 recites the limitation “the operation of identifying a part of the plurality of states”. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 3-9, 11-17, and 19-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
MPEP § 2109(III) sets out steps for evaluating whether a claim is drawn to patent-eligible subject
matter. The analysis of claims 1, 3-9, 11-17, and 19-20, in accordance with these steps, follows.
Step 1 Analysis:
Step 1 is to determine whether the claim is directed to a statutory category (process, machine,
manufacture, or composition of matter. Claims 1 and 3-8 are directed to a manufacture, claims 9 and 11-16 are directed to a process, and claims 17 and 19-20 are directed to a machine. All claims are directed to statutory categories and analysis proceeds.
Step 2A Prong One, Step 2A Prong Two, and Step 2B Analysis:
Step 2A Prong One asks if the claim recites a judicial exception (abstract idea, law of nature, or natural phenomenon). If the claim recites a judicial exception, analysis proceeds to Step 2A Prong Two, which asks if the claim recites additional elements that integrate the abstract idea into a practical application. If the claim does not integrate the judicial exception, analysis proceeds to Step 2B, which asks if the claim amounts to significantly more than the judicial exception. If the claim does not amount to significantly more than the judicial exception, the claim is not eligible subject matter under 35 U.S.C. 101.
None of the claims represent an improvement to technology.
Regarding claim 1, the following claim elements are abstract ideas:
identify a third state from among the plurality of states …, wherein the third state is an intermediate state for switching from the first state to the second state; (Identifying a state of a visual object can practically be performed in the human mind. This is a mental process.)
based at least in part on whether switching from the first state to the second state through the third state is performed within predetermined time and difference between the first state and the third state, determine a compensation value for the data on the third state; (Determining a compensation value based on whether the state switching is performed within a predetermined time and a difference between states is the mental process of evaluation.)
The following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
A non-transitory computer readable storage medium storing one or more programs. the one or more programs comprising instructions which, when executed by at least one processor of an electronic device, cause the electronic device to: (This limitation recites generic computer components and functions. This is mere instructions to apply an exception. See MPEP § 2106.05(f).)
receive, while a visual object is in a first state among a plurality of states, a user input for switching a state of the visual object to a second state among the plurality of states; (Receiving a user input is the known computer process of receiving data. This is mere instructions to apply an exception. See MPEP § 2106.05(f)(2).)
provide data regarding the user input as input data to a neural network for training of the neural network; (Providing data is the known computer process of data transmission. This is mere instructions to apply an exception. See MPEP § 2106.05(f)(2).)
using the neural network in which the data on the user input is input (This describes generic machine learning components/processes. This amounts to mere instructions to apply an exception.)
generate data on the third state as output data from the neural network; (This describes generic machine learning components/processes. This amounts to mere instructions to apply an exception.)
train the neural network based on reinforcement learning such that the compensation value is used to change weights of the neural network. (This limitation recites generic neural network training/reinforcement learning. This is mere instructions to apply an exception.)
Regarding claim 3, the rejection of claim 1 is incorporated herein. The following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
obtain first input data provided to the neural network that is trained based on the compensation value and first output data regarding the first input data obtained from the neural network that is trained based on the compensation value; (Obtaining data is the known process of receiving data. This is mere instructions to apply an exception.)
train another neural network distinct from the neural network, based at least in part on both the first input data and the first output data. (This limitation recites generic teacher/student training. This is mere instructions to apply an exception.)
Regarding claim 4, the rejection of claim 3 is incorporated herein. The following claim element is an abstract idea:
obtain, by performing a time warping with respect to a motion of the visual object within first time interval longer than reference time interval indicated based on the first output data, data on the motion of the visual object within second time interval shorter than the reference time interval as second output data; and (When performing a time warping analysis, the motion is represented as data. Having the data, one could perform the dynamic time warping algorithm practically in the human mind with the aid of pen and paper. This is a mental process.)
The following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
train, based on both the first input data and the second output data, the other neural network. (This limitation recites generic neural network training. This is mere instructions to apply an exception.)
Regarding claim 5, the rejection of claim 4 is incorporated herein. The following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein the neural network is used for training the other neural network, (This recites generic student-teacher training, which is a known process. This is mere instructions to apply an exception.)
wherein the neural network generates output data by processing input data input to the neural network via use of a database storing information on the plurality of states, and (Processing input data is a generic neural network function; this is mere instructions to apply an exception. The use of the database merely specifies a data source, which is an insignificant extra-solution activity. See MPEP § 2106.05(g), “Selecting a particular data source or type of data to be manipulated”.)
wherein the other neural network generates output data by processing input data input to the other neural network without use of the database. (Processing input data is a generic neural network function; this is mere instructions to apply an exception. The limitation of not using the database merely limits the data source. See MPEP § 2106.05(g), “Selecting a particular data source or type of data to be manipulated”.)
Regarding claim 6, the rejection of claim 1 is incorporated herein. The following is an abstract idea:
wherein the data on the user input is used to perform computations with respect to weights of the neural network to be adjusted while the neural network is trained. (Performing computations is equivalent to mathematical calculations, which are a mathematical concept.)
Claim 6 does not recite any additional elements.
Regarding claim 7, the rejection of claim 1 is incorporated herein. The following claim elements are abstract ideas:
identify, in response to receiving the user input, a fourth state of the visual object in which the visual object was scheduled to be switched from the first state among the plurality of states in case that user input is not received; (Identification of a state can be practically performed in the human mind. This is a mental process.)
identify, based at least in part on difference between the second state and the fourth state, a part of the plurality of states used for identifying the third state; and (Identifying a part of the plurality of states can be practically performed in the human mind. This is a mental process.)
identify the third state among the part of the plurality of states (Identifying a state can be practically performed in the human mind. This is a mental process.)
The following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
by using the neural network to input the data on the user input. (Using a neural network to obtain data is a generic function of a generic neural network. This is mere instructions to apply an exception.)
Regarding claim 8, the rejection of claim 7 is incorporated herein. The following claim elements are abstract ideas:
identify the part of the plurality of states, further based on at least one difference between states of the visual object before the first state among the plurality of states and difference between the first state and a state immediately before the first state among the states. (Identifying a part of a plurality of states based on a difference between states can be practically performed in the human mind. This is a mental process.)
Claim 8 does not recite any additional elements.
Regarding claim 9, the following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
A method executed within an electronic device, the method comprising: (This limitation recites generic computer components. This is mere instructions to apply an exception.)
The remainder of claim 9 recites substantially similar subject matter to claim 1 and is rejected with the same rationale, mutatis mutandis.
Claims 11 and 12 recite substantially similar subject matter to claims 3 and 4 respectively and are rejected with the same rationale, mutatis mutandis.
Regarding claim 13, the rejection of claim 12 is incorporated herein. The following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein the neural network is used for training the other neural network. (This recites generic
student-teacher training, which is a known process. This is mere instructions to apply an exception.)
Regarding claim 14, the rejection of claim 12 is incorporated herein. R The following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein the neural network generates output data by processing input data input to the neural network via use of a database storing information on the plurality of states, and (Processing input data is a generic neural network function; this is mere instructions to apply an exception. The use of the database merely specifies a data source, which is an insignificant extra-solution activity. See MPEP § 2106.05(g), “Selecting a particular data source or type of data to be manipulated”.)
wherein the other neural network generates output data by processing input data input to the other neural network without use of the database. (Processing input data is a generic neural network function; this is mere instructions to apply an exception. The limitation of not using the database merely limits the data source. See MPEP § 2106.05(g), “Selecting a particular data source or type of data to be manipulated”.)
Claim 15 recites substantially similar subject matter to claim 7 and is rejected with the same rationale, mutatis mutandis.
Regarding claim 16, the rejection of claim 9 is incorporated herein. Further, the following is an abstract idea:
wherein the operation of identifying a part of the plurality of states includes identifying the part of the plurality of states further based on difference states of the visual object before the first state among the plurality of states and difference between the first state and a state immediately before the first state among the states. (Identifying a part of a plurality of states based on a difference between states can be practically performed in the human mind. This is a mental process.)
Regarding claim 17, the following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
An electronic device comprising: memory comprising one or more storage media storing instructions; and at least one processor comprising processing circuitry, wherein the instructions, when executed by the at least one processor, cause the electronic device to: (This limitation recites generic computer components and functions. This is mere instructions to apply an exception.)
The remainder of claim 17 recites substantially similar subject matter to claim 1 and is rejected with the same rationale, mutatis mutandis.
Claims 19 and 20 recite substantially similar subject matter to claims 7 and 8 respectively and are rejected with the same rationale, mutatis mutandis.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1, 6, 9, and 16-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kyungho (“Interactive Character Animation by Learning Multi-Objective Control”, 2018), Pardo (“Time Limits in Reinforcement Learning”, 2018) and Schulman (“Proximal Policy Optimization”, 2017).
Regarding claim 1, Kyungho teaches
A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, when executed by at least one processor of an electronic device, cause the electronic device to: (Page 7 states "We implemented our algorithm in Python using TensorFlow [TensorFlow 2015] for the learning and evaluation of recurrent neural networks. The computing power of GPUs (NVIDIA GeForce GTX 1080) were utilized to accelerate neural network operations." Python is a programming language used on computers using processors. in order for the method to run, the instructions for the method, stored in one or more programs, must be present on a non-transitory computer readable storage medium. A processor is necessary to run the method on the electronic device. Hereinafter, this is considered to be explanation of how each step is stored in the programs, is executed by the processors, and causes the electronic device to perform the step.)
receive, while a visual object is in a first state among a plurality of states, a user input for switching a state of the visual object to a second state among the plurality of states; (Page 3 states “The character may have a set of actions it can choose from. For example, the basketball player can perform actions such as Dribble, Shoot, Pass, and Catch. Each action is parameterized by a different set of control parameters
f
a
, which is a subset of the full observation. For multi-objective control, we use integrated control parameters
F
as input of the system.
⋃
a
f
a
At each time step
t
, the user provides control parameters
f
a
specific to the action currently performing.” The character is interpreted as the visual object. The state of the character (pose and location) is interpreted as the state of the visual object. As the user provides the control parameters, the program receives the user input. Page 4, “User Control” states "One way of controlling a character is to specify target position
g
p
∈
R
2
and facing direction
g
q
∈
R
2
interactively and command the character to track the target (see Figure 2). Additionally, we can specify which action to perform at the target location and how long it takes to get there." Page 4 states “Let
m
=
(
p
,
q
,
j
,
h
,
c
)
be the full-body configuration of a character, where
p
∈
R
2
is the position of the character,
q
∈
R
2
is its facing direction,
j
is a long vector concatenating joint angles, and
h
is another long vector concatenating joint positions. The current full-body configuration is interpreted as the first state, and the full-body configuration with the target position and facing direction from the user input is interpreted as the second state. The plurality of states is any possible state that can be described using the full-body configuration.)
input data on the user input to a neural network for training of the neural network; (Page 3 states "The control task is to decide a next move at every time step t that best achieves the goal
G
t
=
d
i
a
g
s
F
t
+
d
i
a
g
1
-
S
P
t
". As stated on page 3,
F
includes control parameters
f
a
provided by the user. Page 4 states "We built our RNNs using Long Short-Term Memory units (LSTMs), which preserves gradients well while backpropagating through time/layers and thus can deal with long-term dependencies. Our network model consists of multiple LSTM layers with encoder and decoder units (see Figure 4). The input to the network is the current full-body configuration
m
t
and a goal
G
t
." The RNN is a neural network. As
G
t
includes user input, the user input is provided as input data. Page 5 states "Fortunately, task descriptions can be inferred from training data to build input-output correspondences. This inference step allows RNN to be learned even if we do not know how to create desired outputs given arbitrary inputs." Page 5 further states "We use a hat above a symbol to indicate the measurements in the training data. The task
G
^
t
at every time step t should be inferred from the data." As the task
G
^
t
is inferred before using it as input training data, and the task includes user input, as above, data regarding the user input is used as training data for the neural network.)
identify a third state from among the plurality of states using the neural network in which the data on the user input is input, wherein the third state is an intermediate state for switching the first state to the second state; (Page 4 states “Our network model consists of multiple LSTM layers with encoder and decoder units (see Figure 4). The input to the network is the current full-body configuration
m
t
and a goal
G
t
. The output is the prediction of control parameters
P
^
t
+
1
and the full-body configuration
m
t
+
1
, which recursively feeds back into the network at the next time step." The full-body configuration
m
t
+
1
is interpreted as the third state. As it takes longer than one time step to reach the target (see Fig. 3), the full-body configuration
m
t
+
1
is an intermediate state. In order for the data to be outputted, the third state must be identified.)
generate data on the third state from the neural network (Page 4 states “The output is the prediction of control parameters
P
^
t
+
1
and the full-body configuration
m
t
+
1
, which recursively feeds back into the network at the next time step.")
switching from the first state to the second state through the third state (As stated previously, the target position and facing direction is the second state and the current state is the first state. The output of the network controls the state, predicting a time step ahead with the goal of being in the second state. Therefore, the switch from first to third state will happen through the second state when the model is executed.)
[based on] difference between the first state and the third state, determine a compensation value for the data regarding the third state (Page 5 states that the loss, interpreted as part of the compensation value, is
E
=
E
m
o
t
i
o
n
+
E
c
o
n
t
a
c
t
+
E
r
i
g
i
d
. The equation for the component
E
c
o
n
t
a
c
t
, according to page 5, is
E
c
o
n
t
a
c
t
=
∑
t
∑
k
∈
Feet
c
t
k
c
t
+
1
k
h
t
k
-
T
h
t
+
1
k
2
. Page 5 states
T
h
t
+
1
k
transforms the coordinates of
h
t
+
1
k
to compare with
h
t
k
while taking body translation and rotation at
[
t
,
t
+
1
]
into consideration. The first state is at the current
t
and the third state is at the next timestep
t
+
1
. Therefore, finding the difference between
T
h
t
+
1
k
and
h
t
k
is finding a difference between the first state and the third state. As that occurs within a part of the compensation value calculation, the compensation value is determined based on a difference between the first and third states. Page 5 states "All consecutive pairs
m
^
t
,
m
^
t
+
1
are used to evaluate the loss function, which has three terms:
E
=
E
m
o
t
i
o
n
+
E
c
o
n
t
a
c
t
+
E
r
i
g
i
d
.”
E
is interpreted as part of the compensation value, and
m
^
t
+
1
is interpreted as the third state.)
Kyungho does not appear to explicitly teach
based at least in part on whether [a task] is performed within predetermined time and …, [determine a compensation value]
train the neural network based on reinforcement learning such that the compensation value is used to change weights of the neural network.
However, Pardo—directed to analogous art—teaches
based at least in part on whether [a task] is performed within predetermined time, [determine a compensation value] (Page 6 states "The state-value function of a policy at time
t
can be rewritten in terms of the time-limited return
G
t
:
T
and the value from the last state
v
π
S
T
:
v
π
s
=
E
π
[
G
t
:
T
=
γ
T
-
t
v
π
S
T
|
S
t
=
s
]
(5)”. Page 1 states “However, when the maximum length of an episode is fixed, it is easier to rewrite the expression above by explicitly including the time limit
T
:
G
t
:
T
=
R
t
+
1
+
…
+
γ
T
-
t
-
1
R
T
=
∑
k
=
1
T
-
t
γ
k
-
1
R
t
+
k
(2).” Thus, the state-value, interpreted as part of the compensation value, does change based on the time limit as it is based on
G
t
:
T
, which includes the time-limit.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Kyungho with the teachings of Pardo because, as Pardo states on page 1, "Optimizing for the expectation of the return specified in Equation 2 is suitable for naturally time-limited tasks where the agent has to maximize its expected return only over a fixed episode length."
The combination of Kyungho and Pardo does not appear to explicitly teach
train the neural network based on reinforcement learning such that the compensation value is used to change weights of the neural network.
However, Schulman—directed to analogous art—teaches
train the neural network based on reinforcement learning such that the compensation value is used to change weights of the neural network. (The abstract states "We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a “surrogate” objective function using stochastic gradient ascent." As the loss/objective function is optimized using stochastic gradient ascent, the loss is differentiable. Page 5 states "If using a neural network architecture that shares parameters between the policy and value function, we must use a loss function that combines the policy surrogate and a value function error term." The value function and the loss is interpreted as the compensation value. Page 5 states "A proximal policy optimization (PPO) algorithm that uses fixed-length trajectory segments is shown below. Each iteration, each of N (parallel) actors collect T timesteps of data. Then we construct the surrogate loss on these NT timesteps of data, and optimize it with minibatch SGD (or usually for better performance, Adam [KB14]), for K epochs." As the neural network is optimized based on the loss, which includes a value function error term, the weights of the neural network are changed using the compensation value.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Kyungho and Pardo with the teachings of Schulman because, as Pardo states on page 2, "We evaluate the impact of these considerations on a range of novel and popular benchmark domains using tabular Qlearning and Proximal Policy Optimization (PPO), a modern deep reinforcement learning (Arulkumaran et al., 2017; Henderson et al., 2017) algorithm which has recently been used to achieve state-of-the-art performance in many domains (Schulman et al., 2017; Heess et al., 2017). We use the OpenAI Baselines (Hesse et al., 2017) implementation of PPO with the hyperparameters reported by Schulman et al. (2017), unless stated otherwise." Additionally, as Schulman states on page 1, "This paper seeks to improve the current state of affairs by introducing an algorithm that attains the data efficiency and reliable performance of TRPO, while using only first-order optimization. We propose a novel objective with clipped probability ratios, which forms a pessimistic estimate (i.e., lower bound) of the performance of the policy. To optimize policies, we alternate between sampling data from the policy and performing several epochs of optimization on the sampled data."
Regarding claim 6, the rejection of claim 1 is incorporated herein. Kyungho teaches
wherein the data on the user input is used to perform computations with respect to weights of the neural network to be adjusted while the neural network is trained. (Page 180:4 states "RNN Training is similar to feedforward network training in the sense that network parameters are updated incrementally via backpropagation. Since parameters are shared by all time steps in RNN, gradients at the current time step would affect gradient computation at the previous time steps. This process is called Backpropagation Through Time (BPTT). We built our RNNs using Long Short-Term Memory units (LSTMs), which preserves gradients well while backpropagating through time/layers and thus can deal with long-term dependencies. Our network model consists of multiple LSTM layers with encoder and decoder units (see Figure 4). The input to the network is the current full-body configuration
m
t
and a goal
G
t
." Page 3 states "The control task is to decide a next move at every time step t that best achieves the goal
G
t
=
d
i
a
g
s
F
t
+
d
i
a
g
1
-
S
P
t
". As stated on page 3,
F
includes control parameters
f
a
provided by the user. Therefore, the parameters, interpreted as weights of the neural network, are computed using the data on the user input.)
Regarding claim 9, Kyungho teaches
A method executed within an electronic device, the method comprising: (Page 7 states "We implemented our algorithm in Python using TensorFlow [TensorFlow 2015] for the learning and evaluation of recurrent neural networks. The computing power of GPUs (NVIDIA GeForce GTX 1080) were utilized to accelerate neural network operations." Python is a programming language used on computers using processors. A computer is an electronic device.)
The remainder of claim 9 recites substantially similar subject matter to claim 1 and is rejected with the same rationale, mutatis mutandis.
Regarding claim 16, the rejection of claim 9 is incorporated herein. Kyungho teaches
wherein the operation of identifying a part of the plurality of states includes identifying the part of the plurality on difference between states of the visual object before the first state among the plurality of states and difference between the first state and a state immediately before the first state among the states. (As the output of the model is a state, the model output identifies a part of the plurality of states. Page 8 states "Teacher forcing is a technique that uses the ground truth of the past frame as input rather than the output of the network [Williams and Zipser 1989]. It allows the output to stay in the ground truth, and makes the learning faster and more stable in the training phase. We tested this technique, which helps to reduce the convergence time." The state immediately before the first state is the fourth state in teacher forcing. Therefore, when the fourth state is the input to the network, as happens in teacher forcing, the loss function will find a difference between the state immediately before the first state and the first state.)
Regarding claim 17, Kyungho teaches
An electronic device comprising: at least one memory comprising one or more storage media storing instructions; and the at least one processor comprising processing circuitry, wherein the instructions, when executed by the at least one processor, cause the electronic device to: (Page 7 states "We implemented our algorithm in Python using TensorFlow [TensorFlow 2015] for the learning and evaluation of recurrent neural networks. The computing power of GPUs (NVIDIA GeForce GTX 1080) were utilized to accelerate neural network operations." Python is a programming language used on computers using processors. in order for the method to run, the instructions for the method, stored in one or more programs, must be present on a non-transitory computer readable storage medium. A processor, which inherently comprises processing circuitry, is necessary to run the method on the electronic device.)
The remainder of claim 17 recites substantially similar subject matter to claim 1 and is rejected with the same rationale, mutatis mutandis.
Claim(s) 3-5 and 11-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kyungho (“Interactive Character Animation by Learning Multi-Objective Control”, 2018), Pardo (“Time Limits in Reinforcement Learning”, 2018) and Schulman (“Proximal Policy Optimization”, 2017) as applied to claim 1 above, and further in view of Yao (“Unsupervised Transfer Learning for Spatiotemporal Predictive Networks”, 2020).
Regarding claim 3, the rejection of claim 1 is incorporated herein. Kyungho teaches
the neural network that is trained based on the compensation value (See rejection of claim 1, where the loss function is interpreted as the compensation value.)
The combination of Kyungho, Pardo, and Schulman does not appear to explicitly teach
obtain first input data provided to the neural network that is trained based on the compensation value and first output data on the first input data generated from the neural network trained based on the compensation value;
train another neural network different from the neural network, based at least in part on both the first input data and the first output data.
However, Yao—directed to analogous art—teaches
obtain first input data provided to [a neural network] and first output data on the first input data obtained from [the neural network]; (Page 4 states that “
X
t
is the input state that can be an input frame or the hidden state from the lower layer” in regards to Equation 1 on page 3 that the source model computes. This is interpreted as the input data, and the source model is interpreted as the first neural network. Page 4 further states that “
H
t
m
,
C
t
m
are respectively the hidden state and memory state of the
m
-th pretrained networks, where
m
∈
{
1
,
…
,
M
}
.” The hidden state and memory state is interpreted as the first output data, as it is obtained from pretrained networks.)
train another neural network different from the neural network, based at least in part on both the first input data and the first output data. (Equation 2 on page 4 uses
X
t
,
H
t
m
, and
C
t
m
for the Transferable Memory Unit that is the “basic building block of the target network (see Figure 1)”, as stated on page 4. The intermediate memory state is
C
~
t
, which is calculated by Equation 2, as stated on page 4. The memory distiller, discussed on page 4, Equation 3, uses
C
~
t
to compute features
C
~
t
m
. The loss function that trains the student/target network, interpreted as the “another neural network”, shown by Equation 4, uses the features
C
~
t
m
. Therefore, the first input data and the first output data is used to train another neural network. As can be seen in Figure 1 on page 3, the target model is distinct from the source models.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Kyungho, Pardo, and Schulman with the teachings of Yao because, as Yao states on page 3, "In spatiotemporal predictive learning scenarios, the effectiveness of the memory states has also been explored and validated (Wang et al., 2017). They are important for multi-step future prediction as they convey long-term features of the spatiotemporal data. Besides, training an LSTM-based model in the predictive learning manner, i.e., one of the unsupervised learning paradigms, has been empirically proved to successfully learn concept level representations that can benefit downstream supervised tasks (Wang et al., 2019a). Therefore, we assume that the predictive networks that were pretrained on different unlabeled datasets can provide knowledge of their source domains, and understand the spatiotemporal dynamics of a new task from different perspectives." As the task in the present application is a spaciotemporal predictive learning scenario, one would have been motivated to combine the references.
Regarding claim 4, the rejection of claim 3 is incorporated herein. Kyungho teaches
obtain, by performing a time warping with respect to a motion of the visual object within first time interval longer than reference time interval indicated based on the first output data, data on a motion of the visual object within second time interval shorter than the reference time interval as the second output data; and (Page 6 states "Given a collection of raw motion data, we generate episodes through two steps. The first step is to generate random combinatorial variations by resequencing motion frames. To do so, we constructed a motion graph from the input data [Lee et al. 2002]." Page 6 further states "The length of random walk depends on the complexity of behavioral structures we want to capture from the data set." Page 6 further states " Perturbation should be conducted while preserving the integrity and coherence of human motion. To do so, we employ Laplacian motion editing [Kim et al. 2009], which can deform the spatial trajectory and timing of motion data in an as-rigid as-possible manner subject to user-provided positional, directional, and timing constraints (see Figure 5)." Page 6 further states " Given a motion sequence from random walk, we first divide it into segments of random lengths, then deform the spatial trajectory of each segment randomly in the range of its length from 70% to 100% and its angle from -45°to 45°, and finally connect those segments back together to have the spatial trajectory of the whole sequence perturbed smoothly. We also add timing constraints at random time instances to accelerate or decelerate randomly in the range of 80% to 120% of the original speed.” As the length of the random walk depends on the data set, and the data set is the input data and is used to create the output data from the first neural network, the timing constraint on the time instances of the random walk segment is interpreted as the reference interval. Therefore, when the length of the random walk segment is longer than the reference time interval, will be accelerated to meet the timing constraint, which will mean that the motion is shorter than the reference time interval.)
train, based on both the first input data and the second output data, [the neural network]. (Page 6 states "The performance and accuracy of our model depends mainly on the size and quality of training data. The amount of high-quality motion data available in each individual application domain is insufficient. In our examples, the size (playing time) of raw motion data for each domain ranges from small (only a few seconds) to moderate (20 minutes). We augment the raw data to increase both combinatorial and spatiotemporal variations." Therefore, the raw data (first input data) and accelerated data (second output data) is used for training the neural network.
The combination of Kyungho, Pardo, and Schulman does not appear to explicitly teach
[training] the other neural network
However, Yao—directed to analogous art—teaches
[training] the other neural network (See explanation of rejection of claim 3.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Kyungho, Pardo, and Schulman with the teachings of Yao for the reasons given above in regards to claim 3.
Regarding claim 5, the rejection of claim 4 is incorporated herein. Kyungho teaches
wherein the neural network obtains output data by processing input data provided to the neural network via use of a database storing information regarding the plurality of states, and (Page 8 states "Recent video games provide rich details of characters’ motion using large, high-quality motion databases. Many game developers have designed finite state machines and control mechanisms carefully to create interactively controllable game characters. Our model can be trained exploiting those readily-available motion databases, data structures, and control algorithms, which can generate episodic motion sequences randomly." The training data is interpreted as the input data, as the neural network learns from it and therefore uses it in obtaining output data. As the databases contain motion data, they contain information regarding the plurality of states.
The combination of Kyungho, Pardo, and Schulman does not appear to explicitly teach
wherein the neural network is used for training the other neural network,
wherein the other neural network generates output data by processing input data provided to the other neural network without use of the [source data].
However, Yao—directed to analogous art—teaches
wherein the neural network is used for training the other neural network. (The caption of Figure 1 on page 3 states "An overview of the transferable memory framework, which learns a predictive network on the target dataset from
M
pretrained networks that were collected from different sources.")
wherein the other neural network obtains output data by processing input data provided to the other neural network without use of the [source data]. (Page 3 states "During the training process of the target network on a new dataset, the parameters of the source models are frozen, and they are not taken as the initialization of the target model. In other words, the target model is trained from scratch. It gradually obtains knowledge from the pretrained networks via knowledge distillation." The target network trains on a new dataset and only obtains the knowledge about the source model’s dataset through the source model. Therefore, it does not use the source data.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Kyungho, Pardo, and Schulman with the teachings of Yao for the reasons given above in regards to claim 3.
Claims 11-12 recite substantially similar subject matter to claims 3-4 respectively and are rejected with the same rationale, mutatis mutandis.
Regarding claim 13, the rejection of claim 12 is incorporated herein. The combination of Kyungho, Pardo, and Schulman does not appear to explicitly teach
wherein the neural network is used for training the other neural network.
However, Yao—directed to analogous art—teaches
wherein the neural network is used for training the other neural network. (The caption of Figure 1 on page 3 states "An overview of the transferable memory framework, which learns a predictive network on the target dataset from M pretrained networks that were collected from different sources.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Kyungho, Pardo, and Schulman with the teachings of Yao for the reasons given above in regards to claim 3.
Regarding claim 14, the rejection of claim 12 is incorporated herein. Kyungho teaches
wherein the neural network obtains output data by processing input data provided to the neural network via use of a database storing information regarding the plurality of states, and (Page 8 states "Recent video games provide rich details of characters’ motion using large, high-quality motion databases. Many game developers have designed finite state machines and control mechanisms carefully to create interactively controllable game characters. Our model can be trained exploiting those readily-available motion databases, data structures, and control algorithms, which can generate episodic motion sequences randomly." The training data is interpreted as the input data, as the neural network learns from it and therefore uses it in obtaining output data. As the databases contain motion data, they contain information regarding the plurality of states.
The combination of Kyungho, Pardo, and Schulman does not appear to explicitly teach
wherein the other neural network generates output data by processing input data provided to the other neural network without use of the [source data].
However, Yao—directed to analogous art—teaches
wherein the other neural network obtains output data by processing input data provided to the other neural network without use of the [source data]. (Page 3 states "During the training process of the target network on a new dataset, the parameters of the source models are frozen, and they are not taken as the initialization of the target model. In other words, the target model is trained from scratch. It gradually obtains knowledge from the pretrained networks via knowledge distillation." The target network trains on a new dataset and only obtains the knowledge about the source model’s dataset through the source model. Therefore, it does not use the source data.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Kyungho, Pardo, and Schulman with the teachings of Yao for the reasons given above in regards to claim 3.
Claim(s) 7, 8, 15, 19, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kyungho (“Interactive Character Animation by Learning Multi-Objective Control”, 2018), Pardo (“Time Limits in Reinforcement Learning”, 2018) and Schulman (“Proximal Policy Optimization”, 2017) as applied to claim 1 above, and further in view of Wang (“Design and Implementation of a Voice-Driven Animation System”, 2006).
Regarding claim 7, the rejection of claim 1 is incorporated herein. Kyungho teaches
identify, in response to receiving the user input, a fourth state in which the visual object was scheduled to be switched from the first state among the plurality of states before receiving the user input (Page 8 states "Teacher forcing is a technique that uses the ground truth of the past frame as input rather than the output of the network [Williams and Zipser 1989]. It allows the output to stay in the ground truth, and makes the learning faster and more stable in the training phase. We tested this technique, which helps to reduce the convergence time." The full-body configuration at the past frame is interpreted as the fourth state.)
using the neural network to input the data on the user input. (See rejection of claim 1.)
The combination of Kyungho, Pardo, and Schulman does not appear to explicitly teach
in case that the user input is not received
identify, based at least in part on difference between the second state and the fourth state, a part of the plurality of states used for identifying the third state; and
identify the third state among the part of the plurality of states by …
However, Wang—directed to analogous art—teaches
wherein identifying the fourth state is executed in case that the user input is not received; (See 112(b) rejection for claim interpretation. Page 32 states “Actions such as pick-up, put-down are called discrete actions. These actions cannot be interrupted. Transition among these actions is straight forward, because every action needs to complete before the next one can begin. Transition between continuous actions and discrete actions must go through the rest pose, which serves as a connecting point in the transition.” Page 30 lists the character action commands, which are given by the user during the animation, and are therefore interpreted as the user input. The state of the animation when the discrete action is commenced is interpreted as the fourth state. This state cannot be interrupted by another action, which would be implemented by a user input, and would therefore still activate in case the user input is not received.)
identify, based at least in part on difference between the second state and the fourth state, a part of the plurality of states used for identifying the third state; and (Page 32 states "Transition among these actions is straight forward, because every action needs to complete before the next one can begin. Transition between continuous actions and discrete actions must go through the rest pose, which serves as a connecting point in the transition." The next action would be provided by the user input, and the state of animation for the next action is interpreted as the second state. Therefore, the transition, interpreted as the third state, depends on a difference between the second state and the fourth state, which is the type of action (continuous/discrete). As the third state is identified, identifying the third state is identifying a part of the plurality of states used for identifying the third state.)
identify the third state among the part of the plurality of states by … (As above, identifying the third state is identifying a part of the plurality of states used for identifying the third state.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Kyungho, Pardo, and Schulman with the teachings of Wang because, as Wang states on page 2, “No matter which one of the above methods is used to produce the animation, user interaction is always an important part of an interactive computer animation system. Without a good interface, even an otherwise powerful system cannot enable the creation of good animation.”
Regarding claim 8, the rejection of claim 7 is incorporated herein. Kyungho teaches
identify the part of the plurality of states, further based on at least one difference between states of the visual object before the first state among the plurality of states and difference between the first state and a state immediately before the first state among the states. (Page 8 states "Teacher forcing is a technique that uses the ground truth of the past frame as input rather than the output of the network [Williams and Zipser 1989]. It allows the output to stay in the ground truth, and makes the learning faster and more stable in the training phase. We tested this technique, which helps to reduce the convergence time." The state immediately before the first state is the fourth state in teacher forcing. Therefore, when the fourth state is the input to the network, as happens in teacher forcing, the loss function will find a difference between the state immediately before the first state and the first state. Additionally, in the previous time step, the loss function will have found a difference between the states before the first state.)
Claims 15 recites substantially similar subject matter to claims 7 and is rejected with the same rationale, mutatis mutandis.
Claims 19-20 recite substantially similar subject matter to claims 7-8 respectively and are rejected with the same rationale, mutatis mutandis.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JESSICA THUY PHAM whose telephone number is (571)272-2605. The examiner can normally be reached Monday - Friday, 9 A.M. - 5:00 P.M..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li Zhen can be reached at (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.T.P./Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights