DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/22/2025, 10/15/2026 and 09/23/2026 was filed. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Status of the Claims
Claims 1-10 have been examined.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-10 is/are rejected under 35 U.S.C. 102(a)(1) as being unpatentable over Solowjow (US20220331955A1).
Claim.1 Solowjow discloses a control apparatus that performs feedback control on a control target device (see at least fig.1, abstract, Robotics control system (10) and method for training said robotics control system, embodiments make a gracefully blended utilization of Reinforcement Learning (RL) with conventional control by way of a dynamically adaptive interaction between respective control signals (20, 24) generated by a conventional feedback controller (18) and an RL controller (22), p29, a controller 16 is responsive to signals from the suite of sensors 12), the control apparatus comprising a processor configured to: acquire a first output value related to feedback control on the control target device by using Artificial Intelligence (AI) processing (see at east fig.1, p33-34, orthogonality relations indicative of interdependency of conventional feedback controller signal 20 and reinforcement learning controller signal 24 are penalized by reward function 28 so that control conflicts between conventional feedback controller 18 and reinforcement learning controller 22 are avoided, reward function 28 of reinforcement learning controller 22 may be configured to generate a stream of adaptive weights 30 based on respective contributions of conventional feedback control signal 20 and of reinforcement learning control signal 24 towards fulfilling reward function 28); acquire a second output value related to the feedback control by using non-AI processing (see at least fig.1, p30, controller 16 may include a conventional feedback controller 18 configured to generate a conventional feedback control signal 20, and a reinforcement learning controller 22 configured to generate a reinforcement learning control signal 24, p34, reward function 28 of reinforcement learning controller 22 may be configured to generate a stream of adaptive weights 30 based on respective contributions of conventional feedback control signal 20 and of reinforcement learning control signal 24 towards fulfilling reward function 28); correct a selection ratio of the first output value and the second output value based on a degree of reliability of the first output value (see at least fig.1-3, abstract, a gracefully blended utilization of Reinforcement Learning (RL) with conventional control by way of a dynamically adaptive interaction between respective control signals (20, 24) generated by a conventional feedback controller (18) and an RL controller (22), ,p23, a friction coefficient used in the simulation may be adjusted based on real-world measurements rendering virtual experiments more useful because the virtual experiments would become closer to mimicking the physics involved in a real-world task being performed by the robot, such as automated object insertions by the robot, p37-39, the blended control policy for conventional feedback controller 18 and reinforcement learning controller 22 may be learned in machine learning framework 40, claim.1); and output a control signal for performing feedback control on the control target device based on the selection ratio (see at least fig.1, p29, A suite of sensors 12 may be operatively coupled to a robotic system 14 (e.g., robot/s) controlled by robotics control system 10. A controller 16 is responsive to signals from the suite of sensors 12, p35, a signal combiner 32 may be configured to adaptively combine conventional feedback control signal 20 and reinforcement learning control signal 24 based on the stream of adaptive weights 30 generated by reward function 28. Without limitation, signal combiner 32 may be configured to supply an adaptively combined control signal 34 of conventional feedback control signal 20 and reinforcement learning control signal 24. The adaptively combined control signal 34 may be configured to control robot 14, as the robot performs a sequence of tasks).
Claim.2 Solowjow discloses wherein the processor is further configured to: calculate the degree of reliability of the first output value by quantifying reliability of the first output value; and determine whether the first processing unit is reliable based on the degree of reliability of the first output value, wherein the correcting corrects the selection ratio based on a determination result on whether the acquiring of the first output value is reliable (see at least fig.1-3, abstract, a gracefully blended utilization of Reinforcement Learning (RL) with conventional control by way of a dynamically adaptive interaction between respective control signals (20, 24) generated by a conventional feedback controller (18) and an RL controller (22), ,p23, a friction coefficient used in the simulation may be adjusted based on real-world measurements rendering virtual experiments more useful because the virtual experiments would become closer to mimicking the physics involved in a real-world task being performed by the robot, such as automated object insertions by the robot, p37-39, the blended control policy for conventional feedback controller 18 and reinforcement learning controller 22 may be learned in machine learning framework 40, claim.1).
Claim.3 Solowjow discloses wherein the determining determines whether the degree of reliability of the first output value is equal to or larger than a predetermined threshold, determines that the acquiring of the first output value is reliable when the degree of reliability of the first output value is equal to or larger than the predetermined threshold, and determines that the acquiring of the first output value is not reliable when the degree of reliability of the first output value is smaller than the predetermined threshold, and the correcting includes a selector that selects only the first output value as the selection ratio when it is determined that the acquiring of the first output value is reliable, and selects only the second output value as the selection ratio when it is determined that the acquiring of the first output value is not reliable (see at least fig.1, p31-33, a comparator 25 may be configured to compare orthogonality of conventional feedback control signal 20 and reinforcement learning control signal 24. Comparator 25 may be configured to supply a signal 26 indicative of orthogonality relations between conventional feedback control signal 20 and the reinforcement learning control signal 24, orthogonality relations indicative of interdependency of conventional feedback controller signal 20 and reinforcement learning controller signal 24 are penalized by reward function 28 so that control conflicts between conventional feedback controller 18 and reinforcement learning controller 22 are avoided).
Claim.4 Solowjow discloses wherein the determining determines whether an amount of temporal change of the degree of reliability of the first output value is equal to or larger than a predetermined amount of change, determines that the acquiring of the first output value is reliable when the amount of temporal change of the degree of reliability of the first output value is smaller than the predetermined amount of change , and determines that the acquiring of the first output value is not reliable when the amount of temporal change of the degree of reliability of the first output value is equal to or larger than the predetermined amount of change, and the correcting includes a selector that selects only the first output value as the selection ratio when it is determined that the acquiring of the first output value is reliable, and selects only the second output value as the selection ratio when it is determined that the acquiring of the first output value is not reliable (see at least fig.1-3, p2, Conventional feedback control techniques (which may be referred throughout this disclosure as “conventional control”) can solve various types of control problems—such as without limitation, robotics control, autonomous industrial automation, etc, p22, a feedback loop may be used to adjust simulated sensor and actuator statistical properties based on real-world sensor and actuator statistical properties, such as may be obtained from a robot roll-out. It can be shown that appropriate understanding of the statistical properties (e.g., random errors, noise, etc.) of sensors and actuators in connection with a given robotic system may be decisive for appropriately fulfilling the performance of a control policy trained in simulation, when such a control policy is deployed in a real-world implementation, p35, a signal combiner 32 may be configured to adaptively combine conventional feedback control signal 20 and reinforcement learning control signal 24 based on the stream of adaptive weights 30 generated by reward function 28. Without limitation, signal combiner 32 may be configured to supply an adaptively combined control signal 34 of conventional feedback control signal 20 and reinforcement learning control signal 24. The adaptively combined control signal 34 may be configured to control robot 14, as the robot performs a sequence of tasks, p37, for conventional feedback controller 18 and reinforcement learning controller 22 may be learned in machine learning framework 40, where virtual sensor and actuator data 60 acquired in a simulation environment 44, and real-world sensor and actuator data 54 acquired in a physical environment 46 may be iteratively interleaved with one another (as elaborated in greater detail below) to efficiently and reliably learn the blended control policy for conventional feedback controller 18 and reinforcement learning controller 22 in a reduced cycle time).
Claim.5 Solowjow discloses wherein the acquiring of the first output value includes a first sensor that acquires output of the control target device as the first output value by using the AI processing, and the acquiring of the second output value includes a second sensor that acquires output of the control target device as the second output value by using the non-AI processing (see at east fig.1, p29, a suite of sensors 12 may be operatively coupled to a robotic system 14 (e.g., robot/s) controlled by robotics control system 10. A controller 16 is responsive to signals from the suite of sensors 12,p33-34, orthogonality relations indicative of interdependency of conventional feedback controller signal 20 and reinforcement learning controller signal 24 are penalized by reward function 28 so that control conflicts between conventional feedback controller 18 and reinforcement learning controller 22 are avoided, reward function 28 of reinforcement learning controller 22 may be configured to generate a stream of adaptive weights 30 based on respective contributions of conventional feedback control signal 20 and of reinforcement learning control signal 24 towards fulfilling reward function 28).
Claim.6 Solowjow discloses wherein the acquiring of the first output value includes a first sensor that acquires output of the control target device as the first output value by using the AI processing, and the acquiring of the second output value includes a storage that stores therein output of the control target device that is set in advance as the second output value (see at least fig.1, p29, A suite of sensors 12 may be operatively coupled to a robotic system 14 (e.g., robot/s) controlled by robotics control system 10. A controller 16 is responsive to signals from the suite of sensors 12, p35, a signal combiner 32 may be configured to adaptively combine conventional feedback control signal 20 and reinforcement learning control signal 24 based on the stream of adaptive weights 30 generated by reward function 28. Without limitation, signal combiner 32 may be configured to supply an adaptively combined control signal 34 of conventional feedback control signal 20 and reinforcement learning control signal 24. The adaptively combined control signal 34 may be configured to control robot 14, as the robot performs a sequence of tasks).
Claim.7 Solowjow discloses wherein the acquiring of the first output value includes a first controller that acquires, as the first output value, a first control signal for performing feedback control on the control target by using the AI processing (see at east fig.1, p33-34, orthogonality relations indicative of interdependency of conventional feedback controller signal 20 and reinforcement learning controller signal 24 are penalized by reward function 28 so that control conflicts between conventional feedback controller 18 and reinforcement learning controller 22 are avoided, reward function 28 of reinforcement learning controller 22 may be configured to generate a stream of adaptive weights 30 based on respective contributions of conventional feedback control signal 20 and of reinforcement learning control signal 24 towards fulfilling reward function 28), and the acquiring of the second output value includes a second controller that acquires, as the second output value, a second control signal for performing feedback control on the control target by using the non-AI processing (see at least fig.1, p30, controller 16 may include a conventional feedback controller 18 configured to generate a conventional feedback control signal 20, and a reinforcement learning controller 22 configured to generate a reinforcement learning control signal 24, p34, reward function 28 of reinforcement learning controller 22 may be configured to generate a stream of adaptive weights 30 based on respective contributions of conventional feedback control signal 20 and of reinforcement learning control signal 24 towards fulfilling reward function 28).
Claim.8 Solowjow discloses wherein the outputting includes the acquiring of the first output value and the acquiring of the second output value, the acquiring of the first output value includes a first controller that acquires, as the first output value, a first control signal for performing feedback control on the control target by using the AI processing, the acquiring of the second output value includes a second controller that acquires, as the second output value, a second control signal for performing feedback control on the control target by using the non-AI processing, and the outputting outputs, as the control signal, the first control signal and the second control signal based on the selection ratio of the first output value and the second output value (see at least fig.1-3, abstract, a gracefully blended utilization of Reinforcement Learning (RL) with conventional control by way of a dynamically adaptive interaction between respective control signals (20, 24) generated by a conventional feedback controller (18) and an RL controller (22), ,p23, a friction coefficient used in the simulation may be adjusted based on real-world measurements rendering virtual experiments more useful because the virtual experiments would become closer to mimicking the physics involved in a real-world task being performed by the robot, such as automated object insertions by the robot, p37-39, the blended control policy for conventional feedback controller 18 and reinforcement learning controller 22 may be learned in machine learning framework 40, claim.1).
Claim.9 Solowjow discloses a control system (see at least fig.1, abstract, Robotics control system (10) and method for training said robotics control system, embodiments make a gracefully blended utilization of Reinforcement Learning (RL) with conventional control by way of a dynamically adaptive interaction between respective control signals (20, 24) generated by a conventional feedback controller (18) and an RL controller (22), p29, a controller 16 is responsive to signals from the suite of sensors 12)comprising: a control target device (see at least fig.1, element 14 is a robotic system 14 (e.g., robot/s) ); and a control apparatus that performs feedback control on the control target device, wherein the control apparatus includes a processor (see at least fig.1, abstract, Robotics control system (10) and method for training said robotics control system, embodiments make a gracefully blended utilization of Reinforcement Learning (RL) with conventional control by way of a dynamically adaptive interaction between respective control signals (20, 24) generated by a conventional feedback controller (18) and an RL controller (22), p29, a controller 16 is responsive to signals from the suite of sensors 12) configured to: acquire a first output value related to feedback control on the control target device by using Artificial Intelligence (AI) processing (see at east fig.1, p33-34, orthogonality relations indicative of interdependency of conventional feedback controller signal 20 and reinforcement learning controller signal 24 are penalized by reward function 28 so that control conflicts between conventional feedback controller 18 and reinforcement learning controller 22 are avoided, reward function 28 of reinforcement learning controller 22 may be configured to generate a stream of adaptive weights 30 based on respective contributions of conventional feedback control signal 20 and of reinforcement learning control signal 24 towards fulfilling reward function 28); acquire a second output value related to the feedback control by using non-AI processing (see at least fig.1, p30, controller 16 may include a conventional feedback controller 18 configured to generate a conventional feedback control signal 20, and a reinforcement learning controller 22 configured to generate a reinforcement learning control signal 24, p34, reward function 28 of reinforcement learning controller 22 may be configured to generate a stream of adaptive weights 30 based on respective contributions of conventional feedback control signal 20 and of reinforcement learning control signal 24 towards fulfilling reward function 28); correct a selection ratio of the first output value and the second output value based on a degree of reliability of the first output value (see at least fig.1-3, abstract, a gracefully blended utilization of Reinforcement Learning (RL) with conventional control by way of a dynamically adaptive interaction between respective control signals (20, 24) generated by a conventional feedback controller (18) and an RL controller (22), ,p23, a friction coefficient used in the simulation may be adjusted based on real-world measurements rendering virtual experiments more useful because the virtual experiments would become closer to mimicking the physics involved in a real-world task being performed by the robot, such as automated object insertions by the robot, p37-39, the blended control policy for conventional feedback controller 18 and reinforcement learning controller 22 may be learned in machine learning framework 40, claim.1); and output a control signal for performing feedback control on the control target device based on the selection ratio (see at least fig.1, p29, A suite of sensors 12 may be operatively coupled to a robotic system 14 (e.g., robot/s) controlled by robotics control system 10. A controller 16 is responsive to signals from the suite of sensors 12, p35, a signal combiner 32 may be configured to adaptively combine conventional feedback control signal 20 and reinforcement learning control signal 24 based on the stream of adaptive weights 30 generated by reward function 28. Without limitation, signal combiner 32 may be configured to supply an adaptively combined control signal 34 of conventional feedback control signal 20 and reinforcement learning control signal 24. The adaptively combined control signal 34 may be configured to control robot 14, as the robot performs a sequence of tasks).
Claim.10 Solowjow discloses a control method implemented by a control apparatus that performs feedback control on a control target device (see at least fig.1, abstract, Robotics control system (10) and method for training said robotics control system, embodiments make a gracefully blended utilization of Reinforcement Learning (RL) with conventional control by way of a dynamically adaptive interaction between respective control signals (20, 24) generated by a conventional feedback controller (18) and an RL controller (22), p29, a controller 16 is responsive to signals from the suite of sensors 12), the control method comprising: acquiring a first output value related to feedback control on the control target device by using Artificial Intelligence (AI) processing (see at east fig.1, p33-34, orthogonality relations indicative of interdependency of conventional feedback controller signal 20 and reinforcement learning controller signal 24 are penalized by reward function 28 so that control conflicts between conventional feedback controller 18 and reinforcement learning controller 22 are avoided, reward function 28 of reinforcement learning controller 22 may be configured to generate a stream of adaptive weights 30 based on respective contributions of conventional feedback control signal 20 and of reinforcement learning control signal 24 towards fulfilling reward function 28); acquiring a second output value related to the feedback control by using non-AI processing (see at least fig.1, p30, controller 16 may include a conventional feedback controller 18 configured to generate a conventional feedback control signal 20, and a reinforcement learning controller 22 configured to generate a reinforcement learning control signal 24, p34, reward function 28 of reinforcement learning controller 22 may be configured to generate a stream of adaptive weights 30 based on respective contributions of conventional feedback control signal 20 and of reinforcement learning control signal 24 towards fulfilling reward function 28); correcting a selection ratio of the first output value and the second output value based on a degree of reliability of the first output value (see at least fig.1-3, abstract, a gracefully blended utilization of Reinforcement Learning (RL) with conventional control by way of a dynamically adaptive interaction between respective control signals (20, 24) generated by a conventional feedback controller (18) and an RL controller (22), ,p23, a friction coefficient used in the simulation may be adjusted based on real-world measurements rendering virtual experiments more useful because the virtual experiments would become closer to mimicking the physics involved in a real-world task being performed by the robot, such as automated object insertions by the robot, p37-39, the blended control policy for conventional feedback controller 18 and reinforcement learning controller 22 may be learned in machine learning framework 40, claim.1); and outputting a control signal for performing feedback control on the control target device based on the selection ratio (see at least fig.1, p29, A suite of sensors 12 may be operatively coupled to a robotic system 14 (e.g., robot/s) controlled by robotics control system 10. A controller 16 is responsive to signals from the suite of sensors 12, p35, a signal combiner 32 may be configured to adaptively combine conventional feedback control signal 20 and reinforcement learning control signal 24 based on the stream of adaptive weights 30 generated by reward function 28. Without limitation, signal combiner 32 may be configured to supply an adaptively combined control signal 34 of conventional feedback control signal 20 and reinforcement learning control signal 24. The adaptively combined control signal 34 may be configured to control robot 14, as the robot performs a sequence of tasks).
Conclusion
Related References
The relevant art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Takami (US20210333779A1) discloses an acquisition unit that acquires a measurement value measured regarding a control target equipment (20T). A first control unit outputs an operation amount of the of the control target equipment according to the measurement value by a feedback control or feed-forward control. A second control unit outputs an operation amount of the control target equipment according to the measurement value using a model learnt by using learning data. The switching unit performs switching between the first control unit and the second control unit by which the control target equipment is controlled.
Gotou (US20220284158A1) discloses the apparatus has a state data acquisition unit for acquiring state data indicative of a state of an equipment provided with a control target. An operation amount data acquisition section acquires operation amount of the target. A control model generation section generates a control model, which outputs an operation amount corresponding to the state of the equipment. A simulation unit simulates the state by using a simulation model. A determination unit determines whether control of the control target by the control model is possible based on a simulation result. A convergence judgment section judges convergence of a machine learning system.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHARDUL D PATEL whose telephone number is (571)270-7758. The examiner can normally be reached Monday-Friday 8am-5pm (IFP).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, KITO ROBINSON can be reached at (571)270-3921. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHARDUL D PATEL/Primary Examiner, Art Unit 3664
