Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Joint Inventors
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Response to Arguments and Amendments
Applicant’s arguments and amendments, filed May 21st, 2026, have overcome each and every claim objection and 35 U.S.C. 112(b) rejection previously set forth in the Non-Final Office Action sent on February 24th, 2026. However, amended claims 1, 6 and 14 have introduced additional issues, and new claim objections and 35 U.S.C. 112(b) rejections are made below.
Applicant’s arguments and amendments with respect to the rejections of claims 1-3, 5, 7-11, 13 and 15-16 under 35 U.S.C. 102(a)(1) have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further search and consideration, a new ground(s) of rejection is made in view of Bennice et al. (US Patent Pub. No. 2022/0111517 A1). Examiner further notes claims 3-4 and 11-12 have been cancelled.
Claim Objections
Claim 1 is objected to because of the following informalities:
Claim 1 Line 8: “the learning device” should be revised to “the second robot”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 6 and 14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 6 recites the limitation "reaches the target state when each joint of the first robot of the simulated world to change by as much as an amount of movement per unit of time". It is unclear to the Examiner what is being claimed in this limitation. It seems the improperly worded amendments confuse the intention of qualifying when the first robot reaches the target state. The lack of clarity in this limitation renders the claim indefinite.
Claim 14 recites the limitation “reaches the target state when each joint of the first robot of the simulated world each joint by as much as an amount of movement per unit of time”. For similar reasons to Claim 6 above, this claim is rendered indefinite by the lack of clarity in the resulting claim language.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 5, 7-10, 13 and 15-16 are rejected under 35 U.S.C. 103 as being obvious over Sandha et al. (“Sim2Real Transfer for Deep Reinforcement Learning with Stochastic State Transition Delays”), published in 2020, herein “Sandha”, in view of Bennice et al. (US Patent Pub. No. 2022/0111517 A1), herein “Bennice”, published April 14th, 2022.
Regarding Claims 1 and 9, Sandha discloses a method and reinforcement learning apparatus for performing a method of synchronizing actions of a plurality of robots (See Abstract, “Deep Reinforcement Learning (RL) […] for a wide variety of robotics applications.”), the method and reinforcement learning apparatus comprising:
inputting an action command to a first robot of a simulated world and a second robot of a real world, so that the first robot of the simulated world and the second robot of the real world reach a target state (See Abstract, “[…] to train Deep RL policies in a simulator and then deploy to the real world, a process called Sim2Real transfer.” See also Section 1, “RL agents make sequential decisions in a Markov Decision Process (MDP) in discrete time steps, where the input to the agent is the current state st of the environment, where t is the current time step, and output is the action at. The environment transitions to the next state st+1 once the action is executed, and in turn used as the input for the next action […]” Examiner notes reinforcement learning is executed in both simulated and real-world domains, with clear description of parallel training and deployment including input actions driving transitions to new states);
determining whether the first robot of the simulated world and the second robot of the real world reach the target state after one unit of time (See Section 1, “[…] the states st and st+1 were captured at world clock time τ and τ respectively, the timing delay between state transitions is defined as ∆τ =τ−τ. Note that t is a discrete time step in simulation while ∆τ represents the actual passage of time on a robot.” Examiner notes the RL process describes state transitions with a discrete unit of time between sampled states to determine transitions);
when the first robot of the simulated world and the second robot of the real world reach the target state, determining a first delay time, which is a time until the first robot of the simulated world reaches the target state, and a second delay time, which is a time until the second robot of the real world reaches the target state (See Section 1 as referenced above. See also Fig. 1 shown below and Section 2, “[…] the agent takes action at based on state st, and the environment returns with scalar reward rt and next state st+1. We consider episodic MDPs, where the environment is initialized with state s0 and the interaction with the agent continues until the environment reaches the terminal state sT […] changes to the real state transition time ∆τ – as opposed to fixed discrete time steps in t – directly impacts the state transition […]” Examiner notes t represents the first delay time because it denotes a discrete time step in the simulation, and ∆τ represents the second delay time because it is the change in actual time before the physical agent reaches a terminal state. Furthermore, Fig. 1 below shows parallel processing and coordination between the current state and action of both simulation and actual environment); and
PNG
media_image1.png
184
546
media_image1.png
Greyscale
performing a correction between a state of the first robot of the simulated world and a state of the second robot of the real world in reinforcement learning that performs learning in conjunction with the second robot of the real world, based on the first delay time and the second delay time (See Section 6, “[…] deep reinforcement learning approach that incorporates sampling interval and execution latency into its state space. By utilizing domain randomization with time in the state, TSRL’s policies are robust against varying execution latencies and sampling rates for both Sim2Sim and Sim2Real transfer […] staying within the desired reward budget […] evaluation of time in state policies show that the policies are able to maintain higher rewards across a range of timing characteristics and, thus, can be used in presence of deployment uncertainties impacting the timing characteristics at runtime.” See also Appendix A, “To address variations in the state transition delay, we propose augmenting the agent state with execution time ∆τη and sampling interval ∆τσ measurements […] enables the agent to distinguish between state transitions introduced by variations in delays.”),
But does not explicitly disclose when the first robot of the simulated world or the second robot of the real world does not reach the target state, adding time to reach the target state; and repeating the adding of the time until the first robot of the simulated world or the second robot of the real world reaches the target state;
wherein the performing of the correction comprises adding a dummy time by as much as a difference between the first delay time and the second delay time to the robot having a shorter delay time among the first robot of the simulated world and the second robot of the real world.
Bennice, in a similar field of endeavor, teaches when the first robot of the simulated world or the second robot of the real world does not reach the target state, adding time to reach the target state; and repeating the adding of the time until the first robot of the simulated world or the second robot of the real world reaches the target state (See 0013, “When a particular robot avatar lags for some reason (which likely will not be easily recreated), that robot avatar may not be ready to perform its respective operation in synch with the simulated world clock. Accordingly, to ensure that actions by robot avatars can be recreated deterministically, the simulated world clock may be paused until the lagged robot avatar is able to “catch up.”” See also 0021, “[…] operating a first robot avatar of the plurality of robot avatars at a first frequency that is less than the given frequency. In various implementations, the method may further include pausing a simulated world clock until a robot avatar of the plurality of robot avatars advances to a next stage of operation.” See also 0063, “[…] simulation engine 136 may be configured to pause a simulated world clock until a lagging robot avatar of a plurality of robot avatars is able to advance to a next stage of operation.” Examiner notes pausing the simulated clock in response to a detected lag causing a first robot to be unable to reach a next stage (i.e., target state), is the same as adding time for the robot to reach said target state. Furthermore, the repetition of a conventional process well known in the art and as a general concept (i.e., adding time), is not an inventive concept, but a simple design choice);
wherein the performing of the correction comprises adding a dummy time by as much as a difference between the first delay time and the second delay time to the robot having a shorter delay time among the first robot of the simulated world and the second robot of the real world (See 0013, 0021 and 0063 as referenced above. See also 0020, “[…] at least one node representing a simulated sensor of the robot avatar may impose a delay on output of the sensor being passed up the directed acyclic graph, wherein the delay corresponds to a frequency of a real-world sensor corresponding to the simulated sensor.” Examiner notes adjusting state transitions based on a specific calculation of delay, such as the difference between two measured delays to represent a “dummy time”, is simple artificial time augmentation and is a simple design choice).
In view of Bennice’s teachings, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include, with the delay-aware state comparison between simulated and real agents as disclosed by Sandha, a function for adding time so that a robot corresponding with a simulated/real reference frame can reach a target state in the event of a delay, with a reasonable expectation of success, since the combination improves synchronization between simulated and physical robot operations, particularly in situations where lag is introduced to either robot due to communication latencies or errors in robot controller execution.
Regarding Claims 2 and 10, Sandha further discloses the method of claim 1 and reinforcement apparatus of claim 9, wherein the performing of the correction comprises:
receiving a next state of the simulated world according to N number of an amount of movement per unit of time by as much as a difference between the first delay time and the second delay time (See Sections 1-2 as referenced above. See also Abstract, “[…] Time-in State RL (TSRL) approach, which includes delays and sampling rate as additional agent observations at training time […]”); and
synchronizing the state of the first robot of the simulated world with the state of the second robot of the real world based on the next state of the simulated world (See Sections 1-2 as referenced above. See also Section 5, “[…] mechanisms to reduce state transition delays […] by compensating for delays using damping components.”).
Regarding Claims 5 and 13, Sandha further discloses the method of claim 1 and reinforcement apparatus of claim 9, wherein the determining of whether the first robot of the simulated world and the second robot of the real world reach the target state comprises determining whether the first and second robots reach the target state, based on a simulated world state, which is a state after the first robot of the simulated world moves for one unit of time, and a real world state, which is a state after the second robot of the real world moves for one unit of time (See Sections 1-2 as referenced above. Examiner notes explicit definitions for state transition timing at discrete time steps supports comparing states after unit time intervals).
Regarding Claims 7 and 15, Sandha further discloses the method of claim 1 and reinforcement apparatus of claim 9, wherein the second robot of the real world reaches the target state while reducing an error according to a physical state of the real world acting on each joint of the second robot (See Sections 1-2 as referenced above along with, “[…] policies trained using domain randomization (DR) of timing characteristics. Our results demonstrate that the TSRL policies are robust to the varying state transition delays and, as a result, transfer better across simulations and to real-world environments than the DR policies.” See also Appendix A, “[…] To reduce variance, the advantage function A(st, at) is used for the gradient update […] advantage estimates the relative benefit of taking action at compared to other possible actions in state st. The value network is trained with a mean squared error loss function […]”).
Regarding Claims 8 and 16, Sandha further discloses the method of claim 7 and reinforcement apparatus of claim 15, wherein the second robot of the real world reduces the error according to the physical state of the real world, based on a proportional control, a differential control, or an integral control (See Section 5, “[…] model finite-dimensional systems that may require linearization. The goal in these contexts is to develop robust controllers by approximating the worst case time-delays [33] and sampling variation [34], or by compensating for delays using damping components.” Examiner notes controllers requiring linearization are proportional controllers).
Claims 6 and 14 are rejected under 35 U.S.C. 103 as being obvious over Sandha et al. (“Sim2Real Transfer for Deep Reinforcement Learning with Stochastic State Transition Delays”) in view of Bennice et al. (US Patent Pub. No. 2022/0111517 A1) as applied to claims 1 and 9 above, and further in view of Mahmood et al. (US Patent Pub. No. 2020/0074241 A1), herein “Mahmood”.
Regarding Claims 6 and 14, Sandha in view of Bennice does not explicitly disclose the method of claim 1 and reinforcement apparatus of claim 9, wherein the first robot of the simulated world reaches the target state when each joint of the first robot of the simulated world to change by as much as an amount of movement per unit of time,
wherein the amount of movement per unit time is determined based on a physical state of the simulated world acting on each joint of the first robot.
Mahmood, in a similar field of endeavor, teaches the first robot of the simulated world reaches the target state when each joint of the first robot of the simulated world to change by as much as an amount of movement per unit of time (See 0005-0006, “[…] task manager executes on a second processor of the computer system, obtains the state data stored […] stores the joint state vector in a second buffer. The task manager processes the state data and generates a joint state vector based on the state data […] the joint state vector may be generated based on the defined objective to be achieved in the real-world environment. The reinforcement learning agent transitions from a suspended state to an active state and, in the active state, obtains the joint state vector from the second buffer. The reinforcement learning agent generates, based on the joint state vector, a joint action vector indicating actions to be performed by the plurality of devices. The joint action vector may be generated by applying a reinforcement learning policy to information of the joint state vector […] transitions back to the suspended state for a defined period of time. The task manager obtains the joint action vector and parses the joint action vector into a plurality of actuation commands respectively corresponding to actions to be performed by the plurality of devices operating in the real-world environment.”),
wherein the amount of movement per unit time is determined based on a physical state of the simulated world acting on each joint of the first robot (See 0005-0006 as referenced above).
In view of Mahmood’s teachings, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include, with the delay-aware state comparison between simulated and real agents as disclosed by Sandha, changing a joint vector of the physical agent to reach a target state, with a reasonable expectation of success, since merging joint action buffers with delay-aware state comparisons enables more effective action timing in the corrective process of reinforcement learning.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Bryant Tang whose telephone number is (571)270-0145. The examiner can normally be reached M-F 8-5 CST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Thomas Worden can be reached at (571)272-4876. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRYANT TANG/Examiner, Art Unit 3658
/JASON HOLLOWAY/Primary Examiner, Art Unit 3658
Prosecution Insights