DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This correspondence is in response to amendments filed on February 9, 2026. Claims 11 and 12 are amended. Claims 1-10, 15-16, and 18 are cancelled. Claims 13-14, 17, and 19-20 are filed as previously presented. Applicant filed replacement sheets which address the issues of the drawing objections from the previous action, and as such those objections have been withdrawn. Additionally, Applicant amended the issues presented in the specification objections of the previous action and as such those objections are additionally withdrawn. Amendments to claims 11 and 12 obviate the claim objections and as such those objections are withdrawn. Examiner addresses Applicant’s arguments regarding the prior art rejections below.
Response to Arguments
Applicant argues that Lee does not teach a reward model comprising convolutional layers, pooling layers, and fully connected layers in sequence (see Remarks Page 12). Applicant’s arguments with respect to the specific layering of the reward model have been considered but are moot because the new ground of rejection does not rely on the same combination of references as the prior rejection of record for the specific layering structure of the reward model specifically challenged in the argument.
Applicant further argues that Lee teaches a preference labeling using binary pairwise comparisons between trajectory segments, rather than manually sorting individual state-action pairs and assigning a sequence number after sorting as a label of the corresponding state-action pair (see Remarks Page 13). Examiner respectfully disagrees with this assertion. By instigating a manual preference corresponding to individual state-action pairs σ0 and σ1, the individual state-action pairs are sorted by the input preference. The preference binary assigns the respective sequence with which to consider the individual state action pair as compared to other such state action pairs such that the probability distribution of the preference is associated with the specific preference sequence label when calculating the optimized reward function. As such, Applicant’s argument has been considered but is NOT PERSUASIVE.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are:
“a master-control assembly-strategy model establishing module” in claim 18;
“a master-control assembly-strategy model training module” in claim 18; and
“a robot controlling and executing module” in claim 18.
According to Applicant’s specification, “[T]hose skilled in the art will appreciate that the various modules or steps of the present invention described above may be implemented using general purpose computer means, alternatively they may be implemented using program code executable by computing means such that they may be stored in memory means for execution by computing means, or fabricated separately as individual integrated circuit modules, or multiple of them may be fabricated as a single integrated circuit module” (Pages 9-10). Thus, the modules indicated above are best understood to be software configurations of a computer, and any such computer which executes the desired functions of the modules will be considered pertinent when reviewing the prior art.
Because this/these claim limitations are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have these limitations interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitations to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitations recite sufficient structure to perform the claimed function so as to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 11-14, 17, and 19-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 11, the use of parentheses in the limitation (see lines 39-40) renders the claim indefinite because it is unclear whether the limitation(s) within the parentheses are part of the claimed invention. See MPEP § 2173.05(d). It appears as though the parentheses signify a preference to the intended meaning of the learning process without clearly claiming such interpretation. It is thus not clear whether the limitation in the parentheses should be considered as an alternative, a synonym, or a continuation of the present claim. Examiner will thus merely consider the learning process of the reward function and training of the reward function learning model as synonymous and redundant limitation when reviewing the claim.
Claims 13-14, 17, and 19-20 are rejected as being dependent on claim 11.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 11, 13, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Hou et al. (“Data-efficient hierarchical reinforcement learning for robotic assembly control applications”, 2020; hereinafter “Hou”) in view of Lee et al. (“PEBBLE: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training”, 2021; hereinafter “Lee”) and further in view of Ngyen et al. (“Deep Learning with Experience Ranking Convolutional Neural Network for Robot Manipulator”, 2018; hereinafter “Nguyen”).
Regarding claim 11, Hou teaches a method for a robotic multi-peg-in-hole assembly based on hierarchical reinforcement learning and distributed learning (“From the practical aspect, based on the proposed impedance action space, the proposed HRL algorithm is adopted to solve complicated multiple peg-in-hole assembly tasks” (Page 11566).), comprising:
establishing a master-control assembly-strategy model based on deep reinforcement learning by using data of states and actions of a robot (Control scheme of Fig. 1 shows the hierarchical policy, i.e., deep reinforcement learning, which uses states (s) and actions (a) of a robot. Such control scheme will be the established master-control assembly-strategy model.);
constructing a plurality of sub-process networks based on different assembly interaction environments, updating and training the master-control assembly-strategy model by using interaction data of the robot obtained by the constructed plurality of sub-process networks, and obtaining a trained master-control assembly-strategy model (Training scenarios are divided as “approach”, “search”, and “insertion” tasks (see Fig. 10) which are each trained using different parameter sets (see Table III) for the different assembly interaction environments that result from the varying insertion depths required for each sub-task. Using such parameters as designated in Table III, the master-control assembly model is trained via the HRL algorithm developed by the researchers which results in trained data sets used for the continuous control scheme. Fig. 1, which shows the control scheme (master-control assembly-strategy model), combines tasks as robot and environment responding to actions, which are associated with each sub-task.);
wherein, each of the plurality of sub-process networks comprises a high-level strategy network and a low-level strategy network (Fig. 3 shows breakdown of training policies for networks which are separated into high-level and low-level strategy networks.),
wherein the high-level strategy network obtains a high- level strategy value according to data of a state of the robot at a current time (Control Scheme of Fig. 1 shows the high-level policy, i.e., strategy, receiving the state st of the current time and obtaining option-value ot as a high-level strategy value.), and
the low- level strategy network obtains an action of the robot at a next time according to the high-level strategy value and the data of the state of the robot at the current time (Control scheme of Fig. 1 shows the low-level policy, i.e., strategy, as receiving the option-value ot (high-level strategy value) which is based on the state of the current time (see above limitation regarding derivation of option-value) and obtains the action at at the next time. Although action at is denoted by time step t, such action is contributing to the state for the next time, and as such Examiner best interprets this as the action at the next time as such an interpretation is additionally consistent with Applicant’s specification in which the actor of the low-level network outputs the action at the next time.); and,
controlling and instructing the robot to execute an assembly task of a multi-peg- in-hole assembly by using the trained master-control assembly-strategy model (“The command u¯t to control the robot is calculated as follows: u¯t = clip {at, −b, +b} ◦ ut + ut where at ∈ [−1, 1]6 is the action derived from lower level policies, and b is adopted to control the confidence of action at. Accordingly, compared to adding noise in the original action space (13), the expert command ut takes the relations between pose and force into account, which are the constraints to enable the robots to implement impedance-like behaviors and avoid unpredicted contact force” (Page 11571). Thus, there is a command u¯t for controlling the robot in executing the assembly task which uses results from the trained, i.e., derived, lower level policy as well as the relations of pose and force in deriving an expert command which the command is additionally based on.),
wherein the method further comprises:
storing the state of the robot at the current time, the action corresponding to the state of the robot at the current time, a reward obtained by executing the action corresponding to the state of the robot at the current time, and the action of the robot at the next time in a low-level experience pool (“The collected sample (si, ai, si+1, Ri+1) is stored in the lower level replay buffer DL” (Page 11568). Thus, per the descriptions of Page 6 of Applicant’s disclosure, the prior art stores the same instances of variables in a low-level experience pool which in this case is referred to as “the lower level replay buffer”.), and
updating the low-level strategy network by using the low-level experience pool (“As shown in Fig. 3, at each training step, the lower level policies are trained with the past samples in the lower level replay buffer DL. Mini-batch transitions (si, ai, si+1, Ri+1) ∈ BL are uniformly and randomly sampled from DL” (Page 11568). Thus, for each training step, i.e., each update, the low-level policy uses mini-bath transitions from the lower level replay buffer, i.e., low-level experience pool.);
using the state of the robot at the current time and the action corresponding to the state of the robot at the current time as a state-action pair (States and actions are denoted as state-action pair (s, a) (see Page 11567).)…
However, Hou does not explicitly teach …manually sorting state-action pairs according to experience,
a sequence number after the sorting is regarded as a label of a corresponding state-action pair,
training a reward function learning model by using the state-action pairs and sequence numbers corresponding to the state-action pairs, and
obtaining a reward value of an input state of the robot and an action corresponding to the input state based on the trained reward function model;
wherein, the reward function learning model comprises a first convolution layer, a pooling layer, a second convolution layer, and a fully connected layer connected in sequence,
an input of the reward function learning model is the state action pairs (st , at) with labels, and
an output is a reward value of a current state-action pair; and
using the obtained reward value to iteratively update an initial strategy, comprising:
generating state-action pairs by using the initial strategy to interact with the environment,
training the reward function learning model by manually sorting and learning the state-action pairs, and
then using the reward value output by the reward function learning model to update the initial strategy;
wherein, the initial strategy is a strategy learned at the current time,
a learning process of the learned strategy is carried out alternately with a learning process of the reward function (training of the reward function learning model), and
in the learning process of the reward function, the current strategy serves as the initial strategy.
Lee, pertinent to the problem at hand, teaches …using the state of the robot at the current time and the action corresponding to the state of the robot at the current time as a state-action pair (Algorithm 2 collects state action pairs, as well as resulting next state and corresponding reward to be stored in the replay buffer (see lines 20-23 of pseudocode).),
manually sorting state-action pairs according to experience (Sequences of state-action pairs are manually sorted by comparing experiences and initiating a user preference for each sequence (see lines 9-13 of pseudocode).),
a sequence number after the sorting is regarded as a label of a corresponding state-action pair (Sequence number y, which indicates the preference for each corresponding set and consequently each corresponding state-action pair, is used as a label after the sorting is arranged and user preference is initiated (see Section 3 “Reward learning from preferences.” for definition of “y”).),
training a reward function learning model by using the state-action pairs and sequence numbers corresponding to the state-action pairs (“Concretely, the reward function, modeled as a neural network with parameters, is updated by minimizing the following loss: {Equation 4}” (Section 3, “Reward learning from preferences.”). Thus, the reward function model is trained using the sequence of state-action pairs and the sequence numbers corresponding to the state-action pairs.), and
obtaining a reward value of an input state of the robot and an action corresponding to the input state based on the trained reward function model (“To handle this issue, we relabel all of the agent’s past experience every time we update the reward model” (Section 4.3). Thus, each time the reward model is updated, each state-action pair obtains a relabeling corresponding to the trained reward function model.),
… an input of the reward function learning model is the state action pairs (st , at) with labels (See Lines 14-17 of Algorithm 2 where the input to the reward function is (σ0, σ1, y) which is representative of state action pairs and associated preference label y as additionally described above.), and
an output is a reward value of a current state-action pair (In line 18 of Algorithm 2, the output is the reward value
r
^
ψ
which updates the replay buffer such that past experiences are updated with the new reward model to reflect current state-action pairs which result from the new learning. The reward value
r
^
ψ
is additionally used to update the next series of timestep iterations, i.e., current state-action pairs. Pertinent information can be found in passages of Section 4.); and
using the obtained reward value to iteratively update an initial strategy (See lines 20-23 of Algorithm 2 which uses the obtained reward value
r
^
ψ
to iteratively update the initial strategy for each time step. Additionally, this process iteratively updates the reward model for each iteration which is demonstrated as the initial for loop introduced in Line 6 of Algorithm 2 and terminating in Line 28 of Algorithm 2.), comprising:
generating state-action pairs by using the initial strategy to interact with the environment (State action pairs are generated using the initial strategy to interact with the environment in Lines 21-22 of Algorithm 2. Such state-action pair using this strategy is generated for each desired time step and then stored in the replay buffer.),
training the reward function learning model by manually sorting and learning the state-action pairs (In the following iteration after the generation of updated state-action pairs, the manual sorting and learning of state-action pairs is repeated in Lines 9-17 of Algorithm 2.), and
then using the reward value output by the reward function learning model to update the initial strategy (The reward value
r
^
ψ
which results from the training process of the reward model is used to update all state-action pairs of the replay buffer as well as the state-action pairs for each subsequent time step, i.e., the initial strategy.);
wherein, the initial strategy is a strategy learned at the current time (The updated replay buffer and the newly included state-action pairs resulting for strategies at the current time are each considered as an initial strategy, as such strategies are learned at the current time when the reward model is newly updated.),
a learning process of the learned strategy is carried out alternately with a learning process of the reward function (As illustrated in the flow chart of the method displayed in Fig. 1, the learning process of the learned strategy (left half of the longer box to the right) is carried out alternately with the learning process of the reward function (center of the longer box to the right).), and
in the learning process of the reward function, the current strategy serves as the initial strategy (As seen in Fig. 1, the current strategy depicted as (s,a,s’) in the top left corner of the longer box to the right serves as the initial strategy in the learning process of the reward function, as such strategy is fed into the reward function for learning the updated strategies.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the methods of Hou to include the preference-based reward modeling of Lee with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification because the PEBBLE algorithm (Lee, “Algorithm 2”) learns novel behaviors while avoiding reward exploitation which leads to more desirable behaviors which can be applied to more complex tasks (Lee, Section 6, “Discussion”). Additionally, such a modification is simply a combination of known methods which yield predictable results (see MPEP 2143.I(A)).
However, Hou as modified by Lee does not explicitly teach …wherein, the reward function learning model comprises a first convolutional layer, a pooling layer and a second convolution layer, and a fully connected layer connected in sequence…
Lee instead teaches “For the reward model, we use a three-layer neural network with 256 hidden units each, using leaky ReLUs” (See Appendix B. Experimental Details).
Nguyen, pertinent to the problem at hand, teaches …wherein, the reward function learning model comprises a first convolutional layer, a pooling layer and a second convolution layer, and a fully connected layer connected in sequence (“CNN uses various underlying layers consisting of convolution layers, pooling layers and fully connected layers to predict any sort of recommendation, objects in image or video, or some natural language processing for voice recognition” (Section IV.B). They additionally include their neural network structure which includes multiple convolution layers including ReLU as well as max pooling layers between the convolution layers. The demonstrated network is applied as a reward function learning model.)…
Therefore, it would have been obvious to one of ordinary skill in the art to have modified the three-layer reward model of Lee to instead include the sequential combination of a first convolutional layer, a pooling layer, a second convolutional layer, and a fully connected layer as taught by Nguyen with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification because, as acknowledged by Nguyen, a convolutional reward model would comprise various layers consisting of convolution layers, pooling layers, and fully connected layers (Section IV.B). Such layering of the specified sequence would be motivated by an obvious to try rationale, in which the user chooses from a finite number of identified, predictable solutions, with a reasonable expectation of success (see MPEP 2143.I(E)).
Additionally, known work in the field of endeavor, i.e., reward modeling, could prompt variations for use in the same field based on design incentives if the variations are predictable to one of ordinary skill in the art (see MPEP 2143.I(F)). Any combination of layering for a neural network would be predictable to one of ordinary skill in the art, with specific design choices of such a network based on the specific needs and goals of the user. Applicant has made no specific reference in the disclosure to a feature of the invention which relies specifically on the above-claimed layering order. Thus, the specific layering of the reward model as claimed, or any variation thereof, would be obvious and predictable to one of ordinary skill in the art.
Regarding claim 13, Hou as modified by Lee and Nguyen teaches the method for the robotic multi-peg-in-hole assembly based on hierarchical reinforcement learning and distributed learning according to claim 11,
with Hou further teaching wherein using the data of the state of the robot at the current time as an input of the high-level strategy network to obtain a corresponding high-level strategy value (Fig. 1 indicating the control scheme shows the state of the robot st at the current time as an input to the high-level policy, i.e., strategy, to obtain the option-value as an output, i.e., high-level strategy value.).
Regarding claim 19, Hou as modified by Lee and Nguyen does not explicitly teach a computer device, comprising:
a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, wherein when the machine-readable instructions are executed by the processor, implementing the method for the robotic multi-peg-in-hole assembly based on hierarchical reinforcement learning and distributed learning according to claim 11.
Hou does however recite “A simulation platform, including dual and triple peg-in-hole assembly, is set up on webots” (Page 11566). According to the Webots R2020b User Guide “System Requirements”, a CPU and RAM memory are required to run Webots simulation applications (see https://cyberbotics.com/doc/guide/introduction-to-webots?version=R2020b for full specification of requirements). The R2020b “Introduction to Webots” additionally requires machine-readable instructions which may be executed by the processor (see https://cyberbotics.com/doc/guide/introduction-to-webots?version=R2020b for simulation execution details).
Thus, given that the simulation for the invention was set up on webots, such computer device and associated features would be inherent to the methods disclosed by Hou and as evidenced by the Webots R2020b User Guide.
Regarding claim 20, Hou as modified by Lee and Nguyen does not explicitly teach a non-transitory computer-readable storage medium, having a computer program stored thereon, wherein when the computer program is executed by a processor, implementing the method for the robotic multi-peg-in-hole assembly based on hierarchical reinforcement learning and distributed learning according to claim 11.
Hou does however recite “A simulation platform, including dual and triple peg-in-hole assembly, is set up on webots” (Page 11566). According to the Webots R2020b User Guide “System Requirements”, a CPU and RAM memory are required to run Webots simulation applications (see https://cyberbotics.com/doc/guide/introduction-to-webots?version=R2020b for full specification of requirements). The R2020b “Introduction to Webots” additionally requires machine-readable instructions which may be executed by the processor (see https://cyberbotics.com/doc/guide/introduction-to-webots?version=R2020b for simulation execution details).
Thus, given that the simulation for the invention was set up on webots, such non-transitory computer-readable storage medium and associated features would be inherent to the methods disclosed by Hou and as evidenced by Webots R2020b User Guide.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Hou in view of Lee, further in view of Nguyen, and further in view of Finn et al. (“Deep spatial autoencoders for visuomotor learning”, 2016; hereinafter “Finn”).
Regarding claim 12, Hou as modified by Lee and Nguyen teaches the method for the robotic multi-peg-in-hole assembly based on hierarchical reinforcement learning and distributed learning according to claim 11,
with Hou further teaching wherein the data of the states of the robot comprises a pose of component at an end of the robot, a value of contact force/torque at the end of the robot (“The observed state st representing the assembly environment consists of the pose of pegs and force signals from the F/T sensor” (Page 11571). Thus, the state data is based on the pose and force signals at the end of the robot.)…
However, Hou as modified does not teach …image data of assembly acquired by cameras as part of the data of the states of the robot.
Finn, in the same field of endeavor, teaches …image data of assembly acquired by cameras as part of the data of the states of the robot (“In this paper, we are primarily concerned with the task of learning a state representation for reinforcement learning (RL) from camera images of robotic manipulation skills” (Section III). Additionally, Section V details xt as the combined state space comprising the configuration of the robot and the learned representation (see also Fig. 2 for process of achieving the new state with learned representation). Although force is not additionally included in the state space, Section V further details that the robot state “should be temporally coherent and encode only simple, task-relevant information” which would be inclusive of contact force which is relevant to the assembly task.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the state data of Hou to include image features as taught by Finn with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification because such methods allow high dimensional observations to map to robot state representations which allows ease of learning a control policy (Finn, Section V).
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Hou in view of Lee, further in view of Nguyen, and further in view of Zhenyu et al. (“Three-dimensional path-following control of an autonomous underwater vehicle based on deep reinforcement learning”, 2022; hereinafter “Zhenyu”).
Regarding claim 14, Hou as modified by Lee and Nguyen teaches the method for the robotic multi-peg-in-hole assembly based on hierarchical reinforcement learning and distributed learning according to claim 11,
with Hou further teaching wherein the low-level strategy network comprises an evaluation network and a target network (“Target network is a well-known technique that has been proven to provide a stable target and improve the final performance through reducing the variance of target approximation error (TAE) [33], [34]. Accordingly, we implement the corresponding target networks with parameters(η¯, ω¯ , θ¯) for all networks in Fig. 3” (Page 11569). Additionally, Algorithm 1 initializes both a standard, i.e., evaluation, network and target network for each of the networks shown in Fig. 3.),
the evaluation network and the target network respectively comprise an Actor network and a Critic network (Algorithm 1 initializes an network, i.e., actor evaluation network, and a target actor network. Algorithm 1 additionally initializes a critic network, i.e., critic evaluation network, and a target critic network.), and
the data of the states of the robot and an output of the high-level strategy network are used as an input of the Actor network in the evaluation network to obtain an action of the robot in a current state (Fig. 3 shows the option-value (output of high-level strategy network) and states data as input to the Actor network which determines an action for the robot to take from its current state.);
obtaining a first loss value of the Actor network of the evaluation network by using the data of the states and the actions of the robot as an input of the Critic network in the evaluation network (The critic network in Fig. 3 determines a loss value as a result of state and action inputs.), and
updating the Actor network of the evaluation network according to the loss value (Fig. 3 shows the loss output of the critic network as updating the input for the actor network.);…
Although Hou seemingly follows standard procedures for target networks, especially when considered the algorithm is based off of TD3 theory, Hou does not explicitly teach …using data of a state of the robot at the next time as an input respectively of the Actor network and the Critic network in the target network, an output of the Actor network in the target network is an action corresponding to the next time, an output of the Critic network in the target network is a second loss value of the Critic network in the evaluation network, and updating the Critic network in the evaluation network based on the second loss value.
For further clarity on the structure of a typical TD3 network, Zhenyu, pertinent to the problem at hand, teaches ……using data of a state of the robot at the next time as an input respectively of the Actor network and the Critic network in the target network, an output of the Actor network in the target network is an action corresponding to the next time, an output of the Critic network in the target network is a second loss value of the Critic network in the evaluation network, and updating the Critic network in the evaluation network based on the second loss value (See Fig. 4, wherein the state of the robot at the next time is input to the actor and critic target networks, the output of the actor target network is an action corresponding to said next time, and the output of the critic target network is a critic loss value which is used to update the critic model, i.e., critic evaluation network.).
Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, that Hou shares target network components of those taught by Zhenyu, or would be able to modify the algorithm of Hou to be able to include such components of the TD3 algorithm as taught by Zhenyu. Motivation for such modifications to Hou’s HRL network structure/algorithm (if necessary) could be considered in simple substitution or even combination of known methods and elements to yield predictable results (see MPEP 2143.I(A) and 2143.I(B)).
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Hou in view of Lee, further in view of Nguyen, and further in view of Haarnoja et al. (“Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor”, 2018; hereinafter “Haarnoja”).
Regarding claim 17, Hou as modified by Lee and Nguyen teaches the method for the robotic multi-peg-in-hole assembly based on hierarchical reinforcement learning and distributed learning according to claim 11.
Hou as modified does not explicitly teach …wherein training an Actor network of the low-level strategy network comprising:
calculating a Q value of state-action and an entropy of the action in a strategy network at the current time, obtaining an objective entropy of the strategy network according to the entropy of the action, and
updating parameters of the Actor network in the strategy network by using a gradient descent method combined with the Q value of state-action and the objective entropy; and
training a Critic network of the low-level strategy network comprising:
calculating a target of the Q value of state-action based on empirical data,
updating parameters of the Critic network in the evaluation network by using the gradient descent method combined with the target of the Q value of state-action, and
updating parameters of the Critic network in a target network by using a moving average method and the parameters of the Critic network in the evaluation network.
Such methods are standard updating and training features of the SAC network algorithm (as indicated by Applicant’s specification on Page 6). Such teachings are demonstrated by Haarnoja, and as such is pertinent to the problem at hand. As such, Haarnoja teaches … wherein training an Actor network of the low-level strategy network comprising:
calculating a Q value of state-action and an entropy of the action in a strategy network at the current time, obtaining an objective entropy of the strategy network according to the entropy of the action (See Equation 5 which minimizes the residual error to obtain the equivalent of the objective entropy according to the entropy of the action.), and
updating parameters of the Actor network in the strategy network by using a gradient descent method combined with the Q value of state-action and the objective entropy (The gradient descent of Equation 6 is the updating parameter for the Actor network as indicated by Algorithm 1.); and
training a Critic network of the low-level strategy network comprising:
calculating a target of the Q value of state-action based on empirical data (See Equation 8 which indicates the target Q value based on empirical data.),
updating parameters of the Critic network in the evaluation network by using the gradient descent method combined with the target of the Q value of state-action (See Equation 9 which is the gradient descent method used to update the critic network according to Algorithm 1.), and
updating parameters of the Critic network in a target network by using a moving average method and the parameters of the Critic network in the evaluation network (The parameter ̄ψ of the target value network is updated by an exponentially moving average and such an update is based on the evaluations of the critic evaluation network.).
Although the language in the prior art is slightly varying from that of Applicant’s claim, it is best understood by one of ordinary skill in the art to be performing such calculations as are outlined in the Applicant’s specification on Pages 6 and 7 corresponding to the supported claim above. Therefore, it would be obvious to one of ordinary skill in the art that the network architecture of Hou could be modified to include the training and updating procedures of Haarnoja with a reasonable expectation of success. Motivation for such a modification could be through a combination of known methods to yield predictable results (see MPEP 2143.I(A)).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SIDNEY L MOLNAR whose telephone number is (571)272-2276. The examiner can normally be reached 9 A.M. to 4 P.M. EST Monday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jonathan (Wade) Miles can be reached at (571) 270-7777. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.L.M./Examiner, Art Unit 3656
/WADE MILES/Supervisory Patent Examiner, Art Unit 3656
Prosecution Insights