Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Detailed Action
This Office Action is in response to Amendment filed on 1/13/2026.
Claims 1-20 are pending.
Response to Amendment
This action is in response to the Amendment filled on 1/13/2026. The amendment has been entered. Claims 1-20 have been amended. Claims 1-20 are pending, with claims 1,19 and 20 being independent in the instant application.
Response to Arguments
Applicant's Arguments/Remarks filed on 1/13/2026 on page 10 regarding 35 U.S.C. 112(b) rejections have been fully considered and are found persuasive in view
of the amended claims and presented Arguments/Remarks by the Applicant. Therefore, the previous rejection regarding 35 U.S.C. 112(b) being withdrawn in this current office action.
Applicant's Arguments/Remarks on page 10 regarding 35 U.S.C. 101 rejections have been fully considered and are found persuasive in view of the amended claims and presented Arguments/Remarks by the Applicant. Therefore, the previous rejection regarding 35 U.S.C. 101 being withdrawn in this current office action.
Applicant's Arguments/Remarks filed on pages 10-17 regarding 35 U.S.C. 103
rejections have been fully considered and are moot in view of Applicant's amendments to the claims and amended rejections detailed below. However, a new ground of rejections is necessitated by Applicant's claim amendments. Therefore, the previous rejections regarding 35 U.S.C.103 are being amended in this current office action. (See analysis below Claim Rejections-35 U.S.C. §103).
Examiner Notes
8. Examiner cites particular columns, paragraphs, figures and line numbers in the references as applied to the claims below for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner. The entire reference is considered to provide disclosure relating to the claimed invention. The claims & only the claims form the metes & bounds of the invention. Office personnel are to give the claims their broadest reasonable interpretation in light of the supporting disclosure. Unclaimed limitations appearing in the specification are not read into the claim. Prior art was referenced using terminology familiar to one of ordinary skill in the art. Such an approach is broad in concept and can be either explicit or implicit in meaning. Examiner's Notes are provided with the cited references to assist the applicant to better understand how the examiner interprets the applied prior art. Such comments are entirely consistent with the intent & spirit of compact prosecution.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham, v. John Deere Co., 383 U.S.1.148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
9. Claims 1, 2,8,9,19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Ueda (Pub. No. US20180354125A1) (IDS provided dated 4/6/2022), in view of NAKAGAWA et al. (Pub. No. US20170285584A1) (hereinafter Nakagawa, IDS provided dated 3/17/2023) and further in view of ABE et al. (Pub. No. US20210325838A1) (hereinafter ABE, Patent application originally filed on 2020).
Regarding Claim 1, Ueda teaches a determination apparatus comprising: acquire state data indicative of a state of equipment provided with a control target; (Ueda disclosed in page 1 para [0007]: “A controller according to an embodiment of the present invention determines a compensation amount of a teaching position in control of a robot according to the teaching position included in teaching data … The machine learning device has a state observation section that observes, as state variables expressing a current state of an environment, teaching position compensation amount data indicating the compensation amount of the teaching position in the control of the robot according to the teaching position and motor disturbance value data indicating a disturbance value of each of the motors of the robot in the control of the robot, a determination data acquisition section that acquires determination data indicating an appropriateness determination result of the disturbance value of each of the motors of the robot in the control of the robot, …”).
Ueda teaches acquire operation amount data indicative of an operation amount of the control target; (Ueda disclosed in page 5 para [0057]: “In a case where the robot is controlled according to the teaching position compensated based on a compensation amount of the teaching position determined … the reward R calculated by the reward calculation section 112 may be positive, for example, if an appropriateness determination result of the operation of a robot is determined to be “appropriate” (for example, a case in which a disturbance value of each of the motors of the robot falls within an allowable range, or a case in which a teaching position to which the robot finally moves falls within an allowable range) or may be negative, for example, if the appropriateness determination result of the operation of the robot is determined to be “ inappropriate ” (for example, a case in which the disturbance value of each of the motors of the robot goes beyond the allowable range, or a case in which the teaching position to which the robot finally moves goes beyond the allowable range.”).
Ueda teaches generate a control model, which outputs the operation amount corresponding to the state of the equipment, by machine learning by using the state data and the operation amount data; (Ueda disclosed in page 2-3 para [0034]: “The machine learning device 100 includes software (such as a learning algorithm) … An object to be learned by the machine learning device 100 of the controller 1 corresponds to a model structure expressing the correlation between a disturbance value produced in a motor that drives each joint of a robot and a compensation amount of a teaching position of the robot.” In page 5 para [0052-0053]: “The reinforcement learning is a method in which, while the current state (that is, an input) of an environment in which a learning target exists is observed, a prescribed action (that is, an output) is performed in the current state and the cycle of giving any reward to the action is repeatedly performed by trial and error to learn measures (a compensation amount of a teaching position in the control of a robot according to the teaching position included in teaching data of the robot in the machine learning device of the present application) to maximize the total of the rewards as an optimum solution. In the machine learning device 100 of the controller 1 shown in FIG. 3, the learning section 110 includes a reward calculation section 112 … The learning section 110 learns a compensation amount of a teaching position of a robot with respect to a disturbance value of each of the motors of the robot in the control of the robot according to the teaching position included in teaching data of the robot in such a way that the value function update section 114 repeatedly updates the function Q.”).
However, Ueda doesn’t explicitly teach the limitations “simulate, by using a simulation model, the state of the equipment in a case where the operation amount, which is output by the control model, is given to the control target; determine whether artificial intelligence (AI) control of the control target by the control model is possible, based on a simulation result output by the simulation model;
Nakagawa teaches simulate, by using a simulation model, the state of the equipment in a case where the operation amount, which is output by the control model, is given to the control target; (Nakagawa disclosed in page 2-3 para [0030-0031]: “As illustrated in FIG. 2, the machine learning device 20 includes: a machine learning unit 21 that performs machine learning and outputs a control command; a simulator 22 that executes simulation of the work of the robot 14 based on the control command; a first determination unit 23 that determines the control command based on an execution result of the simulation by the simulator 22; and a second determination unit 24 that determines a work result of the robot 14 by the control command. When there is no problem with the execution result of the simulation by the simulator 22, the first determination unit 23 determines that the control command output from the machine learning unit 21 is good and inputs it to the robot 14. Then, the robot 14 performs the work based on the control command for which the determination result by the first determination unit 23 is good.” In page 4 para [0050-0051]: “For example, a determination may be made in which, with thresholds being provided in two steps, a classification is made into three, i.e., "good", “passing”, and “failing”. This is because, for example, when the robot 14 passes near an obstacle such as the cage 11, depending on accuracy of the simulator 22, a situation occurs in which there is a suspected possibility of interference. … when the determination result by the first determination unit 23 takes three or more states (multiple values) including good and bad, a control can be performed, based on the state, such that the robot 14 is operated by changing the command speed of the robot 14 included in the control command, specifically, by decreasing the command speed when there is a high possibility of interference.).
Nakagawa teaches determine whether artificial intelligence (AI) control of the control target by the control model is possible, based on a simulation result output by the simulation model; (Nakagawa disclosed in page 2-3 para [0029-0031]: “FIG. 2 is a block diagram illustrating a first example of the machine learning device according to the present invention, which may be applied, for example, to the machine system (robot system) described with reference to FIG. 1 … As illustrated in FIG. 2, the machine learning device 20 includes: a machine learning unit 21 that performs machine learning and outputs a control command; a simulator 22 that executes simulation of the work of the robot 14 based on the control command; a first determination unit 23 that determines the control command based on an execution result of the simulation by the simulator 22; and a second determination unit 24 that determines a work result of the robot 14 by the control command. When there is no problem with the execution result of the simulation by the simulator 22, the first determination unit 23 determines that the control command output from the machine learning unit 21 is good and inputs it to the robot 14. Then, the robot 14 performs the work based on the control command for which the determination result by the first determination unit 23 is good.”). Ueda and Nakagawa are analogous art because they are related to have machine learning device that learns a control command for a machine to perform the machine learning to output the control command. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ueda and Nakagawa to modify outputting the operation amount related to equipment state using machine learning of Ueda, to include simulating equipment state to determine control of control target on AI (artificial intelligence) is possible by Nakagawa. The suggestion/motivation for doing so would have been obvious by Nakagawa because “Q-learning is a method for learning a value Q (s, a) for selecting an action a under a certain environmental state s. In other words, under a certain state s, an action a with the highest value Q (s, a) may be selected as optimum action. The agent proceeds to learn selection of a better action, i.e., a correct value Q (s, a). When learning is performed through application of Q-learning, the state quantity s is composed of the first result outputted from the first determination unit and the first state quantity outputted from the simulator 52 or outputted from the robot. (Nakagawa disclosed in page 5 para [0063-0064]).
However, Ueda and Nakagawa do not explicitly teach the limitations “repeatedly switch control of the control target between a feedback control of the control target by a controller and the Al control of the control target by the control model, wherein when the Al control of the control target by the control model is determined to be possible, control of the control target by the operation amount is switched to be performed by the Al control of the control target by the control model, and when the Al control of the control target by the control model is determined to not be possible, control of the control target by the operation amount is switched to be performed by the feedback control of the control target by the controller.”
and ABE teaches repeatedly switch control of the control target between a feedback control of the control target by a controller and the Al control of the control target by the control model, (ABE disclosed in page 2 para [0017]: “The prediction model may be constituted by a regression model such as a self-regression model, or may be constituted by a machine learning model such as a neural network. … Further, in a case where the prediction model is constituted by a machine learning model such as a neural network, a known machine learning method ...” In page 4 para [0051]: “the prediction control development device 1 according to the present embodiment provides the prediction model to the controller to control the operation of the control target device by causing the controller to correct instruction values in accordance with a predicted value of a control amount predicted by the prediction model and giving the corrected instruction values to the control target device. … The PLC 2 may be configured to be able to switch between two modes, that is, a prediction control mode for controlling the operation of a control target device using a prediction model and a normal control mode for determining an instruction value from a measured value of a control amount by a known control method such as PID control …”).
wherein when ABE teaches the Al control of the control target by the control model is determined to be possible, control of the control target by the operation amount is switched to be performed by the Al control of the control target by the control model, (ABE disclosed in page 11 para [0134-0136]: “The control part 21 calculates a correction value R(t) by substituting the predicted value and the basic target value of the acquired control amount into each term of Expression 1 and executing the operation processing of Expression 1. … . Specifically, the control part 21 calculates the correction value R(t) by calculating a product of the calculated difference and the proportional constant r and adding the constant term s to the calculated product. In addition, the control part 21 can calculate the corrected target value 63 by adding the correction value R(t) to the basic target value … When the corrected target value 63 is acquired, the control part 21 causes the processing to proceed to the next step S106. … In step S106, the control part 21 operates as the operation control part 212 and determines an instruction value (operation amount) in accordance with an instruction target value of a control amount. In a situation where prediction control is performed, the corrected target value 63 obtained in step S105 is treated as an instruction target value.”
It has been discussed in para [0051] that the present embodiment provides the prediction model to the controller to control the operation of the control target device by causing the controller to correct instruction values in accordance with a predicted value of a control amount predicted by the prediction model and giving the corrected instruction values to the control target device. The PLC 2 may be configured to be able to switch between two modes, that is, a prediction control mode for controlling the operation of a control target device using a prediction model and a normal control mode. Therefore, the control operation is switched to the prediction model which constituted by a machine learning model or artificial intelligence, because the control part 21 operates as the operation control part 212 and determines an instruction value (operation amount) in accordance with an instruction target value of a control amount (as discussed above)).
and when ABE teaches the Al control of the control target by the control model is determined to not be possible, control of the control target by the operation amount is switched to be performed by the feedback control of the control target by the controller. (ABE disclosed in page 11 para [0134-0136]: “The control part 21 calculates a correction value R(t) by substituting the predicted value and the basic target value of the acquired control amount into each term of Expression 1 and executing the operation processing of Expression 1. … . Specifically, the control part 21 calculates the correction value R(t) by calculating a product of the calculated difference and the proportional constant r and adding the constant term s to the calculated product. In addition, the control part 21 can calculate the corrected target value 63 by adding the correction value R(t) to the basic target value … When the corrected target value 63 is acquired, the control part 21 causes the processing to proceed to the next step S106. … In step S106, the control part 21 operates as the operation control part 212 and determines an instruction value (operation amount) in accordance with an instruction target value of a control amount. … On the other hand, in a situation where prediction control is not performed, that is, a situation where normal control is performed, the basic target value of the control amount obtained by step S101 is treated as an instruction target value as it is.”
It has been discussed in para [0051] that the present embodiment provides the prediction model to the controller to control the operation of the control target device by causing the controller to correct instruction values in accordance with a predicted value of a control amount predicted by the prediction model and giving the corrected instruction values to the control target device. The PLC 2 may be configured to be able to switch between two modes, that is, a prediction control mode for controlling the operation of a control target device using a prediction model and a normal control mode. The “normal control mode” corresponds to claim element “feedback control of the control target” is performed, therefore, the control operation is switched to be performed by the feedback control of the control target by the controller. Since, the prediction control (or AI control) is not performed, that means, a situation where normal control is performed, the basic target value of the control amount obtained by step S101 is treated as an instruction target value as it is (as discussed above)).
Ueda, Nakagawa and ABE are analogous art because they are related to have machine learning device that learns a control command for a machine to perform the machine learning to output the control command. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ueda, Nakagawa and ABE before him or her, to modify generating control model or device using machine learning technique of Ueda and Nakagawa, to include switching control between AI control and feedback control in ABE’s teaching. The suggestion/motivation for doing so would have been obvious by ABE because “The present invention is contrived in view of such circumstances in one aspect, and an objective thereof is to provide a technique for generating a prediction model making it possible to predict a control amount with high accuracy when not only a large - scale production device but also a relatively small - scale production device is used as a control target device.” (ABE disclosed in page 1 para [0009]).
Regarding claim 2, Ueda, Nakagawa and ABE teach the determination apparatus according to Claim 1, however, Ueda doesn’t explicitly teach the limitations “the AI control of the control target by the control model is determined to be possible when the simulation result indicates that a period during which the equipment can be operated normally by the control model exceeds a predetermined threshold”.
wherein Nakagawa teaches the AI control of the control target by the control model is determined to be possible when the simulation result indicates that a period during which the equipment can be operated normally by the control model exceeds a predetermined threshold. (Nakagawa disclosed in page 2-3 para [0029-0031]: “FIG. 2 is a block diagram illustrating a first example of the machine learning device according to the present invention, which may be applied, for example, to the machine system (robot system) described with reference to FIG. 1 … As illustrated in FIG. 2, the machine learning device 20 includes: a machine learning unit 21 that performs machine learning and outputs a control command; a simulator 22 that executes simulation of the work of the robot 14 based on the control command; a first determination unit 23 that determines the control command based on an execution result of the simulation by the simulator 22; and a second determination unit 24 that determines a work result of the robot 14 by the control command. When there is no problem with the execution result of the simulation by the simulator 22, the first determination unit 23 determines that the control command output from the machine learning unit 21 is good and inputs it to the robot 14. Then, the robot 14 performs the work based on the control command for which the determination result by the first determination unit 23 is good.”
In page 4 para [0050-0051]: “Although the first determination unit 23 and the second determination unit 24 perform the good/bad determination, the result output from each determination unit does not necessarily need to be binary (“0” or “1”). For example, a determination may be made in which, with thresholds being provided in two steps, a classification is made into three, i.e., "good", "passing”, and “failing". This is because, for example, when the robot 14 passes near an obstacle such as the cage 11, depending on accuracy of the simulator 22, a situation occurs in which there is a suspected possibility of interference. … it is possible to more minutely control the control command by configuring such that the determination by the first determination unit 23 is a determination in a ternary form including additionally “passing (intermediate)" or in a more multiple value form … when the determination result by the first determination unit 23 takes three or more states (multiple values) including good and bad, a control can be performed, based on the state, such that the robot 14 is operated by changing the command speed of the robot 14 included in the control command, …”. It has been disclosed in page 4 para [0047]: “the first result label 41 obtained from the simulator 22 is calculated by an error calculation unit 28, and the error is back-propagated so that learning by the neural network 25 can be performed. Specifically, when it is arranged such that the extraction success probability 26 takes on a numerical value in a range from “0” representing failure to “1” representing success, the first result label 41 takes on the value of “0” or “1” depending on the result of extraction success/failure, so that the error can be calculated by taking difference between the two. When a determination of being good (right) is made by the first determination unit 23, it means that it is assured that the robot 14 (actual machine) operates (works) without problems, regardless of a control signal being input ted thereto. As a result, a control command is inputted to cause the actual machine (robot 14) to operate actually.”
The disclosure “the first determination unit 23 takes three or more states (multiple values) including good and bad, a control can be performed, based on the state, such that the robot 14 is operated by changing the command speed of the robot 14 included in the control command” corresponds to the claim limitation “the equipment can normally operate exceeds a predetermined threshold”).
Ueda and Nakagawa are analogous art because they are related to have machine learning device that learns a control command for a machine to perform the machine learning to output the control command. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ueda and Nakagawa to modify outputting the operation amount related to equipment state using machine learning of Ueda, to include simulating equipment state to determine control of control target on AI (artificial intelligence) is possible by Nakagawa. The suggestion/motivation for doing so would have been obvious by Nakagawa because “Q-learning is a method for learning a value Q (s, a) for selecting an action a under a certain environmental state s. In other words, under a certain state s, an action a with the highest value Q (s, a) may be selected as optimum action. The agent proceeds to learn selection of a better action, i.e., a correct value Q (s, a). When learning is performed through application of Q-learning, the state quantity s is composed of the first result outputted from the first determination unit and the first state quantity outputted from the simulator 52 or outputted from the robot. (Nakagawa disclosed in page 5 para [0063-0064]).
Regarding Claim 8, Ueda, Nakagawa and ABE teach the determination apparatus according to Claim 1, however, Ueda and Nakagawa do not explicitly teach the limitation “instruct the control target to switch to the Al control of the control target by the control model when it is determined that the Al control of the control target by the control model is possible”.
further ABE teaches instruct the control target to switch to the Al control of the control target by the control model when it is determined that the Al control of the control target by the control model is possible. ((ABE disclosed in page 11 para [0134-0136]: “The control part 21 calculates a correction value R(t) by substituting the predicted value and the basic target value of the acquired control amount into each term of Expression 1 and executing the operation processing of Expression 1. … . Specifically, the control part 21 calculates the correction value R(t) by calculating a product of the calculated difference and the proportional constant r and adding the constant term s to the calculated product. In addition, the control part 21 can calculate the corrected target value 63 by adding the correction value R(t) to the basic target value … When the corrected target value 63 is acquired, the control part 21 causes the processing to proceed to the next step S106. … In step S106, the control part 21 operates as the operation control part 212 and determines an instruction value (operation amount) in accordance with an instruction target value of a control amount. In a situation where prediction control is performed, the corrected target value 63 obtained in step S105 is treated as an instruction target value.”
It has been discussed in para [0051] that the present embodiment provides the prediction model to the controller to control the operation of the control target device by causing the controller to correct instruction values in accordance with a predicted value of a control amount predicted by the prediction model and giving the corrected instruction values to the control target device. The PLC 2 may be configured to be able to switch between two modes, that is, a prediction control mode for controlling the operation of a control target device using a prediction model and a normal control mode. Therefore, the control operation is switched to the prediction model which constituted by a machine learning model or artificial intelligence, because the control part 21 operates as the operation control part 212 and determines an instruction value (operation amount) in accordance with an instruction target value of a control amount (as discussed above)).
Ueda, Nakagawa and ABE are analogous art because they are related to have machine learning device that learns a control command for a machine to perform the machine learning to output the control command. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ueda, Nakagawa and ABE before him or her, to modify generating control model or device using machine learning technique of Ueda and Nakagawa, to include switching control between AI control and feedback control in ABE’s teaching. The suggestion/motivation for doing so would have been obvious by ABE because “The present invention is contrived in view of such circumstances in one aspect, and an objective thereof is to provide a technique for generating a prediction model making it possible to predict a control amount with high accuracy when not only a large - scale production device but also a relatively small - scale production device is used as a control target device.” (ABE disclosed in page 1 para [0009]).
Regarding claim 9, Ueda, Nakagawa and ABE teach the determination apparatus according to Claim 2, is incorporating the rejections of claim 8, because claim 9 has substantially similar claim language as claim 8, therefore claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Ueda, Nakagawa and ABE as discussed above for substantially similar rationale.
Regarding claim 19, the same ground of rejection is made as discussed in claim 1 for substantially similar rationale, therefore claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Ueda, Nakagawa and ABE as discussed above for substantially similar rationale. In addition, claim 19 recites following limitation:
Ueda teaches a determination method (Ueda disclosed in page 7 para [0071]: “controller 1 may be described as a machine learning method (or software) per formed by the processor 101. The machine learning method is a method for learning a compensation amount of a teaching position in the control of a robot according to the teaching position included in teaching data of the robot.” In page 8 para [0090]: “Workers engaging in the systems 170 and 170' may perform a determination as to whether the achievement degree (the reliability of the compensation amount of the teaching position in the control of the robot …) of learning a compensation amount of a teaching position in the control of a robot according to the teaching position included in teaching data of the robot with the machine learning device 120 (or 100) has reached a required level at an appropriate timing after the start of learning by the machine learning device 120 (or 100)”).
However, Ueda doesn’t explicitly teach the limitation “determine whether artificial intelligence (AI) control of the control target by the control model is possible, based on a simulation result output by the simulation model;”
Nakagawa teaches determine whether artificial intelligence (AI) control of the control target by the control model is possible, based on a simulation result output by the simulation model; (Nakagawa disclosed in page 2-3 para [0029-0031]: “FIG. 2 is a block diagram illustrating a first example of the machine learning device according to the present invention, which may be applied, for example, to the machine system (robot system) described with reference to FIG. 1 … As illustrated in FIG. 2, the machine learning device 20 includes: a machine learning unit 21 that performs machine learning and outputs a control command; a simulator 22 that executes simulation of the work of the robot 14 based on the control command; a first determination unit 23 that determines the control command based on an execution result of the simulation by the simulator 22; and a second determination unit 24 that determines a work result of the robot 14 by the control command. When there is no problem with the execution result of the simulation by the simulator 22, the first determination unit 23 determines that the control command output from the machine learning unit 21 is good and inputs it to the robot 14. Then, the robot 14 performs the work based on the control command for which the determination result by the first determination unit 23 is good.”). Ueda and Nakagawa are analogous art because they are related to have machine learning device that learns a control command for a machine to perform the machine learning to output the control command. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ueda and Nakagawa to modify outputting the operation amount related to equipment state using machine learning of Ueda, to include simulating equipment state to determine control of control target on AI (artificial intelligence) is possible by Nakagawa. The suggestion/motivation for doing so would have been obvious by Nakagawa because “Q-learning is a method for learning a value Q (s, a) for selecting an action a under a certain environmental state s. In other words, under a certain state s, an action a with the highest value Q (s, a) may be selected as optimum action. The agent proceeds to learn selection of a better action, i.e., a correct value Q (s, a). When learning is performed through application of Q-learning, the state quantity s is composed of the first result outputted from the first determination unit and the first state quantity outputted from the simulator 52 or outputted from the robot. (Nakagawa disclosed in page 5 para [0063-0064]).
However, Ueda and Nakagawa do not explicitly teach the limitations “repeatedly switch control of the control target between a feedback control of the control target by a controller and the Al control of the control target by the control model, wherein when the Al control of the control target by the control model is determined to be possible, control of the control target by the operation amount is switched to be performed by the Al control of the control target by the control model, and when the Al control of the control target by the control model is determined to not be possible, control of the control target by the operation amount is switched to be performed by the feedback control of the control target by the controller.”
and ABE teaches repeatedly switch control of the control target between a feedback control of the control target by a controller and the Al control of the control target by the control model, (ABE disclosed in page 2 para [0017]: “The prediction model may be constituted by a regression model such as a self-regression model, or may be constituted by a machine learning model such as a neural network. … Further, in a case where the prediction model is constituted by a machine learning model such as a neural network, a known machine learning method ...” In page 4 para [0051]: “the prediction control development device 1 according to the present embodiment provides the prediction model to the controller to control the operation of the control target device by causing the controller to correct instruction values in accordance with a predicted value of a control amount predicted by the prediction model and giving the corrected instruction values to the control target device. … The PLC 2 may be configured to be able to switch between two modes, that is, a prediction control mode for controlling the operation of a control target device using a prediction model and a normal control mode for determining an instruction value from a measured value of a control amount by a known control method such as PID control …”).
wherein when ABE teaches the Al control of the control target by the control model is determined to be possible, control of the control target by the operation amount is switched to be performed by the Al control of the control target by the control model, (ABE disclosed in page 11 para [0134-0136]: “The control part 21 calculates a correction value R(t) by substituting the predicted value and the basic target value of the acquired control amount into each term of Expression 1 and executing the operation processing of Expression 1. … . Specifically, the control part 21 calculates the correction value R(t) by calculating a product of the calculated difference and the proportional constant r and adding the constant term s to the calculated product. In addition, the control part 21 can calculate the corrected target value 63 by adding the correction value R(t) to the basic target value … When the corrected target value 63 is acquired, the control part 21 causes the processing to proceed to the next step S106. … In step S106, the control part 21 operates as the operation control part 212 and determines an instruction value (operation amount) in accordance with an instruction target value of a control amount. In a situation where prediction control is performed, the corrected target value 63 obtained in step S105 is treated as an instruction target value.”
It has been discussed in para [0051] that the present embodiment provides the prediction model to the controller to control the operation of the control target device by causing the controller to correct instruction values in accordance with a predicted value of a control amount predicted by the prediction model and giving the corrected instruction values to the control target device. The PLC 2 may be configured to be able to switch between two modes, that is, a prediction control mode for controlling the operation of a control target device using a prediction model and a normal control mode. Therefore, the control operation is switched to the prediction model which constituted by a machine learning model or artificial intelligence, because the control part 21 operates as the operation control part 212 and determines an instruction value (operation amount) in accordance with an instruction target value of a control amount (as discussed above)).
and when ABE teaches the Al control of the control target by the control model is determined to not be possible, control of the control target by the operation amount is switched to be performed by the feedback control of the control target by the controller. (ABE disclosed in page 11 para [0134-0136]: “The control part 21 calculates a correction value R(t) by substituting the predicted value and the basic target value of the acquired control amount into each term of Expression 1 and executing the operation processing of Expression 1. … . Specifically, the control part 21 calculates the correction value R(t) by calculating a product of the calculated difference and the proportional constant r and adding the constant term s to the calculated product. In addition, the control part 21 can calculate the corrected target value 63 by adding the correction value R(t) to the basic target value … When the corrected target value 63 is acquired, the control part 21 causes the processing to proceed to the next step S106. … In step S106, the control part 21 operates as the operation control part 212 and determines an instruction value (operation amount) in accordance with an instruction target value of a control amount. … On the other hand, in a situation where prediction control is not performed, that is, a situation where normal control is performed, the basic target value of the control amount obtained by step S101 is treated as an instruction target value as it is.”
It has been discussed in para [0051] that the present embodiment provides the prediction model to the controller to control the operation of the control target device by causing the controller to correct instruction values in accordance with a predicted value of a control amount predicted by the prediction model and giving the corrected instruction values to the control target device. The PLC 2 may be configured to be able to switch between two modes, that is, a prediction control mode for controlling the operation of a control target device using a prediction model and a normal control mode. The “normal control mode” corresponds to claim element “feedback control of the control target” is performed, therefore, the control operation is switched to be performed by the feedback control of the control target by the controller. Since, the prediction control (or AI control) is not performed, that means, a situation where normal control is performed, the basic target value of the control amount obtained by step S101 is treated as an instruction target value as it is (as discussed above)).
Ueda, Nakagawa and ABE are analogous art because they are related to have machine learning device that learns a control command for a machine to perform the machine learning to output the control command. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ueda, Nakagawa and ABE before him or her, to modify generating control model or device using machine learning technique of Ueda and Nakagawa, to include switching control between AI control and feedback control in ABE’s teaching. The suggestion/motivation for doing so would have been obvious by ABE because “The present invention is contrived in view of such circumstances in one aspect, and an objective thereof is to provide a technique for generating a prediction model making it possible to predict a control amount with high accuracy when not only a large - scale production device but also a relatively small - scale production device is used as a control target device.” (ABE disclosed in page 1 para [0009]).
Regarding claim 20, the same ground of rejection is made as discussed in claim 1 for substantially similar rationale, therefore claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Ueda, Nakagawa and ABE as discussed above for substantially similar rationale. In addition, claim 20 recites following limitation:
Ueda teaches a non-transitory recording medium having recorded thereon a determination program that, when executed by a computer, causes the computer to: (Ueda disclosed in page 2 para [0029]: “The non-volatile memory 14 stores teaching data input from the teach pendant 60 via an interface 19, a robot-controlling program input via an interface … Further, the ROM 12 stores in advance various system programs (including a system program for controlling communication with a machine learning device 100 …) for running processing for the control of a robot …”. In page 7 para [0078]: “A decision- making section 122 may be configured …, one of the functions of the processor 101 or software stored in the ROM 102 for functioning the processor 101. The decision-making section 122 generates and outputs a command value C including a command for determining a compensation amount of a teaching position of a robot with respect to a disturbance value …”).
However, Ueda doesn’t explicitly teach the limitation “determine whether artificial intelligence (AI) control of the control target by the control model is possible, based on an output of the simulation model;”
Nakagawa teaches determine whether artificial intelligence (AI) control of the control target by the control model is possible, based on an output of the simulation model; (Nakagawa disclosed in page 2-3 para [0029-0031]: “FIG. 2 is a block diagram illustrating a first example of the machine learning device according to the present invention, which may be applied, for example, to the machine system (robot system) described with reference to FIG. 1 … As illustrated in FIG. 2, the machine learning device 20 includes: a machine learning unit 21 that performs machine learning and outputs a control command; a simulator 22 that executes simulation of the work of the robot 14 based on the control command; a first determination unit 23 that determines the control command based on an execution result of the simulation by the simulator 22; and a second determination unit 24 that determines a work result of the robot 14 by the control command. When there is no problem with the execution result of the simulation by the simulator 22, the first determination unit 23 determines that the control command output from the machine learning unit 21 is good and inputs it to the robot 14. Then, the robot 14 performs the work based on the control command for which the determination result by the first determination unit 23 is good.”). Ueda and Nakagawa are analogous art because they are related to have machine learning device that learns a control command for a machine to perform the machine learning to output the control command. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ueda and Nakagawa to modify outputting the operation amount related to equipment state using machine learning of Ueda, to include simulating equipment state to determine control of control target on AI (artificial intelligence) is possible by Nakagawa. The suggestion/motivation for doing so would have been obvious by Nakagawa because “Q-learning is a method for learning a value Q (s, a) for selecting an action a under a certain environmental state s. In other words, under a certain state s, an action a with the highest value Q (s, a) may be selected as optimum action. The agent proceeds to learn selection of a better action, i.e., a correct value Q (s, a). When learning is performed through application of Q-learning, the state quantity s is composed of the first result outputted from the first determination unit and the first state quantity outputted from the simulator 52 or outputted from the robot. (Nakagawa disclosed in page 5 para [0063-0064]).
However, Ueda and Nakagawa do not explicitly teach the limitations “repeatedly switch control of the control target between a feedback control of the control target by a controller and the Al control of the control target by the control model, wherein when the Al control of the control target by the control model is determined to be possible, control of the control target by the operation amount is switched to be performed by the Al control of the control target by the control model, and when the Al control of the control target by the control model is determined to not be possible, control of the control target by the operation amount is switched to be performed by the feedback control of the control target by the controller.”
and ABE teaches repeatedly switch control of the control target between a feedback control of the control target by a controller and the Al control of the control target by the control model, (ABE disclosed in page 2 para [0017]: “The prediction model may be constituted by a regression model such as a self-regression model, or may be constituted by a machine learning model such as a neural network. … Further, in a case where the prediction model is constituted by a machine learning model such as a neural network, a known machine learning method ...” In page 4 para [0051]: “the prediction control development device 1 according to the present embodiment provides the prediction model to the controller to control the operation of the control target device by causing the controller to correct instruction values in accordance with a predicted value of a control amount predicted by the prediction model and giving the corrected instruction values to the control target device. … The PLC 2 may be configured to be able to switch between two modes, that is, a prediction control mode for controlling the operation of a control target device using a prediction model and a normal control mode for determining an instruction value from a measured value of a control amount by a known control method such as PID control …”).
wherein when ABE teaches the Al control of the control target by the control model is determined to be possible, control of the control target by the operation amount is switched to be performed by the Al control of the control target by the control model, (ABE disclosed in page 11 para [0134-0136]: “The control part 21 calculates a correction value R(t) by substituting the predicted value and the basic target value of the acquired control amount into each term of Expression 1 and executing the operation processing of Expression 1. … . Specifically, the control part 21 calculates the correction value R(t) by calculating a product of the calculated difference and the proportional constant r and adding the constant term s to the calculated product. In addition, the control part 21 can calculate the corrected target value 63 by adding the correction value R(t) to the basic target value … When the corrected target value 63 is acquired, the control part 21 causes the processing to proceed to the next step S106. … In step S106, the control part 21 operates as the operation control part 212 and determines an instruction value (operation amount) in accordance with an instruction target value of a control amount. In a situation where prediction control is performed, the corrected target value 63 obtained in step S105 is treated as an instruction target value.”
It has been discussed in para [0051] that the present embodiment provides the prediction model to the controller to control the operation of the control target device by causing the controller to correct instruction values in accordance with a predicted value of a control amount predicted by the prediction model and giving the corrected instruction values to the control target device. The PLC 2 may be configured to be able to switch between two modes, that is, a prediction control mode for controlling the operation of a control target device using a prediction model and a normal control mode. Therefore, the control operation is switched to the prediction model which constituted by a machine learning model or artificial intelligence, because the control part 21 operates as the operation control part 212 and determines an instruction value (operation amount) in accordance with an instruction target value of a control amount (as discussed above)).
and when ABE teaches the Al control of the control target by the control model is determined to not be possible, control of the control target by the operation amount is switched to be performed by the feedback control of the control target by the controller. (ABE disclosed in page 11 para [0134-0136]: “The control part 21 calculates a correction value R(t) by substituting the predicted value and the basic target value of the acquired control amount into each term of Expression 1 and executing the operation processing of Expression 1. … . Specifically, the control part 21 calculates the correction value R(t) by calculating a product of the calculated difference and the proportional constant r and adding the constant term s to the calculated product. In addition, the control part 21 can calculate the corrected target value 63 by adding the correction value R(t) to the basic target value … When the corrected target value 63 is acquired, the control part 21 causes the processing to proceed to the next step S106. … In step S106, the control part 21 operates as the operation control part 212 and determines an instruction value (operation amount) in accordance with an instruction target value of a control amount. … On the other hand, in a situation where prediction control is not performed, that is, a situation where normal control is performed, the basic target value of the control amount obtained by step S101 is treated as an instruction target value as it is.”
It has been discussed in para [0051] that the present embodiment provides the prediction model to the controller to control the operation of the control target device by causing the controller to correct instruction values in accordance with a predicted value of a control amount predicted by the prediction model and giving the corrected instruction values to the control target device. The PLC 2 may be configured to be able to switch between two modes, that is, a prediction control mode for controlling the operation of a control target device using a prediction model and a normal control mode. The “normal control mode” corresponds to claim element “feedback control of the control target” is performed, therefore, the control operation is switched to be performed by the feedback control of the control target by the controller. Since, the prediction control (or AI control) is not performed, that means, a situation where normal control is performed, the basic target value of the control amount obtained by step S101 is treated as an instruction target value as it is (as discussed above)).
Ueda, Nakagawa and ABE are analogous art because they are related to have machine learning device that learns a control command for a machine to perform the machine learning to output the control command. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ueda, Nakagawa and ABE before him or her, to modify generating control model or device using machine learning technique of Ueda and Nakagawa, to include switching control between AI control and feedback control in ABE’s teaching. The suggestion/motivation for doing so would have been obvious by ABE because “The present invention is contrived in view of such circumstances in one aspect, and an objective thereof is to provide a technique for generating a prediction model making it possible to predict a control amount with high accuracy when not only a large - scale production device but also a relatively small - scale production device is used as a control target device.” (ABE disclosed in page 1 para [0009]).
Claims 3-7,10 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Ueda, Nakagawa and ABE and further in view of Handa et al. (Pub. No. US2020/0306960A1) (IDS provided dated 6/15/2023).
Regarding claim 3, Ueda, Nakagawa and ABE teach the determination apparatus according to Claim 1, wherein Nakagawa teaches the AI control of the control target by the control model is determined to be possible (Nakagawa disclosed in page 2-3 para [0029-0031]: “FIG. 2 is a block diagram illustrating a first example of the machine learning device according to the present invention, which may be applied, for example, to the machine system (robot system) described with reference to FIG. 1 … As illustrated in FIG. 2, the machine learning device 20 includes: a machine learning unit 21 that performs machine learning and outputs a control command; a simulator 22 that executes simulation of the work of the robot 14 based on the control command; a first determination unit 23 that determines the control command based on an execution result of the simulation by the simulator 22; and a second determination unit 24 that determines a work result of the robot 14 by the control command. When there is no problem with the execution result of the simulation by the simulator 22, the first determination unit 23 determines that the control command output from the machine learning unit 21 is good and inputs it to the robot 14. Then, the robot 14 performs the work based on the control command for which the determination result by the first determination unit 23 is good.”).
However, Ueda, Nakagawa and ABE do not explicitly teach the claim limitation “the simulation result indicates that a number of times that it is judged that the equipment can be operated normally by the control model exceeds a predetermined threshold.”
when Handa teaches the simulation result indicates that a number of times that it is judged that the equipment can be operated normally by the control model exceeds a predetermined threshold. (Handa disclosed in page 8 para [0070-0071]: “In an embodiment, the system performing the process 900 determines 912 an accuracy score for the iterated set of simulation parameters … In an embodiment, the accuracy score corresponds to the degree of accuracy between the simulation and the real-world results. In an embodiment, a higher accuracy score can reflect a higher degree of accuracy, … the system performing the process 900 identifies 916 the most accurate simulation based on the scores. In an embodiment, the system can utilize an error signal generated from the scores in a least mean square optimization algorithm, although other optimization algorithms can be used, to determine the score that corresponds to a minimal error and maximum accuracy. In an embodiment, the determined score corresponds to a set of parameters that corresponds to the most accurate simulation. In an embodiment, the system can utilize the parameters to generate an accurate simulation that can be utilized to train a control system to determine controls to perform the task in the real world”.
The disclosures above “an accuracy score for the iterated set of simulation parameters; the score that corresponds to a minimal error and maximum accuracy; the system can utilize the parameters to generate an accurate simulation that can be utilized to train a control system to determine controls to perform the task” corresponds to the claim limitation “when it is judged based on the simulation result that a number of times that it is judged that the equipment can be operated normally by the control model exceeds a predetermined threshold”).
Ueda, Nakagawa, ABE and Handa are analogous art because they are related to have machine learning device that learns a control command for a machine to perform the machine learning to output the control command. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ueda, Nakagawa, ABE and Handa to modify simulating equipment state to determine control of control target is possible by Nakagawa and ABE, to include simulation occurrences (or number of times) an equipment can perform based on the simulation result by Handa. The suggestion/motivation for doing so would have been obvious by Handa because “In an embodiment, if the attempt is successful, the simulation is accurate and control system is trained. An accurate simulation can refer to a simulation in which the simulation accurately re-creates and generates a representation of an environment, such as the environment comprising the performance of the task. In an embodiment, the accurate simulation is utilized to train the control system, the trained control system can accurately produce controls to perform the task in the real world.” (Handa disclosed in page 7 para [0064]).
Regarding Claim 5, Ueda, Nakagawa and ABE teach the determination apparatus according to Claim 1, however, Ueda doesn’t explicitly teach the limitation “the AI control of the control target by the control model is determined to be possible”.
wherein Nakagawa teaches the AI control of the control target by the control model is determined to be possible (Nakagawa disclosed in page 2-3 para [0029-0031]: “FIG. 2 is a block diagram illustrating a first example of the machine learning device according to the present invention, which may be applied, for example, to the machine system (robot system) described with reference to FIG. 1 … As illustrated in FIG. 2, the machine learning device 20 includes: a machine learning unit 21 that performs machine learning and outputs a control command; a simulator 22 that executes simulation of the work of the robot 14 based on the control command; a first determination unit 23 that determines the control command based on an execution result of the simulation by the simulator 22; and a second determination unit 24 that determines a work result of the robot 14 by the control command. When there is no problem with the execution result of the simulation by the simulator 22, the first determination unit 23 determines that the control command output from the machine learning unit 21 is good and inputs it to the robot 14. Then, the robot 14 performs the work based on the control command for which the determination result by the first determination unit 23 is good.”).
Ueda and Nakagawa are analogous art because they are related to have machine learning device that learns a control command for a machine to perform the machine learning to output the control command. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ueda and Nakagawa to modify outputting the operation amount related to equipment state using machine learning of Ueda, to include simulating equipment state to determine control of control target on AI (artificial intelligence) is possible by Nakagawa. The suggestion/motivation for doing so would have been obvious by Nakagawa because “Q-learning is a method for learning a value Q (s, a) for selecting an action a under a certain environmental state s. In other words, under a certain state s, an action a with the highest value Q (s, a) may be selected as optimum action. The agent proceeds to learn selection of a better action, i.e., a correct value Q (s, a). When learning is performed through application of Q-learning, the state quantity s is composed of the first result outputted from the first determination unit and the first state quantity outputted from the simulator 52 or outputted from the robot. (Nakagawa disclosed in page 5 para [0063-0064]).
However, Ueda, Nakagawa and ABE do not explicitly teach the claim limitation “an instruction to permit control is acquired in response to an output of the simulation result”.
when Handa teaches an instruction to permit control is acquired in response to an output of the simulation result. (Handa disclosed in page 3 para [0030]: “In an embodiment, initial values for the parameters are determined by the control computer 122 to generate a simulation of the computer-controlled robot 110 performing the bag placing task. In an embodiment, the computer-controlled robot 110 utilizes the determined controls to attempt to perform the bag placing task. In an embodiment, data relating to the inputs, outputs, and results of the attempted performance of the bag placing task is gathered. In an embodiment, the control computer 122 utilizes the gathered data from the attempted performance of the bag placing task to determine new values for the parameters of the simulation.”).
Ueda, Nakagawa, ABE and Handa are analogous art because they are related to have machine learning device that learns a control command for a machine to perform the machine learning to output the control command. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ueda, Nakagawa, ABE and Handa to modify simulating equipment state to determine control of control target is possible by Nakagawa and ABE, to include simulation occurrences (or number of times) an equipment can perform based on the simulation result by Handa. The suggestion/motivation for doing so would have been obvious by Handa because “In an embodiment, if the attempt is successful, the simulation is accurate and control system is trained. An accurate simulation can refer to a simulation in which the simulation accurately re-creates and generates a representation of an environment, such as the environment comprising the performance of the task. In an embodiment, the accurate simulation is utilized to train the control system, the trained control system can accurately produce controls to perform the task in the real world.” (Handa disclosed in page 7 para [0064]).
Regarding Claim 4, Ueda, Nakagawa and ABE teach the determination apparatus according to claim 2, is incorporating the rejections of claim 3, because claim 4 has substantially similar claim language as claim 3, therefore claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Ueda, Nakagawa, ABE and Handa as discussed above for substantially similar rationale.
Regarding Claim 6, Ueda, Nakagawa and ABE teach the determination apparatus according to claim 2, is incorporating the rejections of claim 5, because claim 6 has substantially similar claim language as claim 5, therefore claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Ueda, Nakagawa, ABE and Handa as discussed above for substantially similar rationale.
Regarding Claim 7, Ueda, Nakagawa and ABE teach the determination apparatus according to claim 3, is incorporating the rejections of claim 5, because claim 7 has substantially similar claim language as claim 5, therefore claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Ueda, Nakagawa, ABE and Handa as discussed above for substantially similar rationale.
Regarding Claim 10, Ueda, Nakagawa and ABE teach the determination apparatus according to Claim 1, however, Ueda, Nakagawa and ABE do not explicitly teach the claim limitation “re-generate the control model by the machine learning, when it is determined that the AI control of the control target by the control model is not possible”.
wherein Handa teaches the one or more processors are further configured to: re-generate the control model by the machine learning, when it is determined that the AI control of the control target by the control model is not possible. (Handa disclosed in page 2 para [0025]: “In an embodiment, a control system is a system that regulates, manages, and controls a system, such as the computer-controlled robot 110, utilizing control loops, feedback, and various other mechanisms. In an embodiment, the machine learning control system 124 can comprise various control schemes such as proportional-integral - derivative (“PID”) control, feedback control, logic control, linear control, and/or variations thereof. Furthermore, in an embodiment, the machine learning control system 124 can utilize various structures such as a neural network, structured prediction system, anomaly detection system, supervised learning system, artificial intelligence system …”.
In page 3 para [0030-0031]: “In an embodiment, the generated simulation is utilized to train the machine learning control system 124 to determine controls for the computer-controlled robot 110 to attempt to perform the bag placing task. … In an embodiment, a simulation utilizing the new values is generated to train the machine learning control system 124 to re-determine controls for the computer-controlled robot 110 to re-attempt to perform the bag placing task. In an embodiment, following the attempt, data relating to the inputs, outputs, and results of the re-attempted performance of the bag placing task is gathered and utilized by the control computer 122 and machine learning control system 124 to determine new values of the parameters for an updated simulation … In an embodiment, the cycle of attempting to perform the task with controls derived from determined parameters, and analyzing produced data and re-determining parameters to derive controls for the next attempt is continuously repeated until the desired results are achieved.”).
Ueda, Nakagawa, ABE and Handa are analogous art because they are related to have machine learning device that learns a control command for a machine to perform the machine learning to output the control command. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ueda, Nakagawa, ABE and Handa to modify simulating equipment state to determine control of control target is possible by Nakagawa and ABE, to include simulation occurrences (or number of times) an equipment can perform based on the simulation result by Handa. The suggestion/motivation for doing so would have been obvious by Handa because “In an embodiment, if the attempt is successful, the simulation is accurate and control system is trained. An accurate simulation can refer to a simulation in which the simulation accurately re-creates and generates a representation of an environment, such as the environment comprising the performance of the task. In an embodiment, the accurate simulation is utilized to train the control system, the trained control system can accurately produce controls to perform the task in the real world.” (Handa disclosed in page 7 para [0064]).
Regarding Claim 11, Ueda, Nakagawa and ABE teach the determination apparatus according to Claim 2, is incorporating the rejections of claim 10, because claim 11 has substantially similar claim language as claim 10, therefore claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Ueda, Nakagawa, ABE and Handa as discussed above for substantially similar rationale.
Claims 12-18 are rejected under 35 U.S.C. 103 as being unpatentable over Ueda, Nakagawa and ABE and further in view of Matsubara et al. (Pub. No. US2020/0057416A1) (IDS provided dated 3/17/2023).
Regarding Claim 12, Ueda, Nakagawa and ABE teach the determination apparatus according to claim 1, however Ueda, Nakagawa and ABE do not explicitly teach the limitation “judge convergence of the machine learning, wherein the simulation model is configured to simulate the state of the equipment when it is judged that the machine learning has converged.”
further Matsubara teaches a convergence judgment unit configured to judge convergence of the machine learning, wherein the simulation unit is configured to simulate the state of the equipment when it is judged that the machine learning has converged. (Matsubara disclosed in page 4 para [0043]: “processing of the model 415 is performed by each agent 41 using the learning data that includes the state data of the facility 2 and the control condition data of the target device 20 (T), … Accordingly , the learning of the model 415 can be made to converge, and the control condition data recommended for each target device 20 (T) can be acquired by inputting the state data to the obtained model 415.”
In page 6 para [0063]: “the target device 20 (T) is controlled according to the recommended control condition data and the learning process is repeated, to optimize the operational state of the facility when the processes of steps S1 to S11 are repeated, the period of step 1 may be set according to a time constant of the facility 2, … Furthermore, when the processes of steps S1 to S11 are repeated, the number of devices 20 and sensors 21 in the facility 2 may be increased or decreased …”. Further, in page 6-7 para [0069]: “According to the apparatus 4A described above, it is possible to narrow down the state parameters in each agent 41 and perform the learning process, thereby making it possible to converge the learning in a short time. Furthermore, the learning can be caused to converge in a manner to be globally optimal across a plurality of devices 20 having defined correlation probabilities.”).
Ueda, Nakagawa, ABE and Matsubara are analogous art because they are related to have machine learning device that learns a control command for a machine to perform the machine learning to output the control command. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ueda, Nakagawa, ABE and Matsubara, to modify simulating equipment state to determine control of control target is possible by Nakagawa, to include simulating the state of the equipment to judge convergence of the machine learning by Matsubara. The suggestion/motivation for doing so would have been obvious by Matsubara because “Each of the plurality of agents may include a state acquiring section that acquires state data indicating a state of the facility in an apparatus. Each of the plurality of agents may include a control condition acquiring section that acquires control condition data indicating a control condition of each target device. Each of the plurality of agents may include a learning processing section that uses learning data including the state data and the control condition data to perform learning processing of a model that outputs recommended control condition data indicating a control condition recommended for each target device in response to input of the state data.” (Matsubara disclosed in page 1 para [0006]).
Regarding Claim 14, Ueda, Nakagawa, ABE and Matsubara teach the determination apparatus according to Claim 12, however Ueda, Nakagawa and ABE do not explicitly teach the limitation “the machine learning is judged to have converged based on an elapsed time since the machine learning is started”.
wherein Matsubara teaches the machine learning is judged to have converged based on an elapsed time since the machine learning is started. (Matsubara disclosed in page 4 para [0049]: “Here, the plurality of future timings (t+1), (t+2), etc. may be timings at every unit time (e.g. 30 seconds) within a reference time period (e.g. 10 minutes) from the current timing. The predicted state data may display the predicted state of the facility 2 in a case where any of the controls indicated by the control conditions (C1), (C2), etc. are performed on a target device 20 (T) at each timing t+1), (t+2), etc. As an example, the predicted state data may comprehensively include the predicted state data (D(C1_t+1)), (D(C2_t+2)), etc. (where the suffixes “t+1” and “t+2” indicate the timings at which the control is to be performed) …”).
Ueda, Nakagawa, ABE and Matsubara are analogous art because they are related to have machine learning device that learns a control command for a machine to perform the machine learning to output the control command. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ueda, Nakagawa, ABE and Matsubara, to modify simulating equipment state to determine control of control target is possible by Nakagawa, to include simulating the state of the equipment to judge convergence of the machine learning by Matsubara. The suggestion/motivation for doing so would have been obvious by Matsubara because “Each of the plurality of agents may include a state acquiring section that acquires state data indicating a state of the facility in an apparatus. Each of the plurality of agents may include a control condition acquiring section that acquires control condition data indicating a control condition of each target device. Each of the plurality of agents may include a learning processing section that uses learning data including the state data and the control condition data to perform learning processing of a model that outputs recommended control condition data indicating a control condition recommended for each target device in response to input of the state data.” (Matsubara disclosed in page 1 para [0006]).
Regarding Claim 15, Ueda, Nakagawa, ABE and Matsubara teach the determination apparatus according to Claim 12, however Ueda, Nakagawa and ABE do not explicitly teach the limitation “the machine learning is judged to have converged based on a value of an evaluation function of the machine learning”.
wherein Matsubara teaches the machine learning is judged to have converged based on a value of an evaluation function of the machine learning. (Matsubara disclosed in page 3 para [0034]: “The reward value acquiring section 40 acquires reward values used for reinforcement learning by the agents 41, and acquires reward values for evaluating the operational state of the facility 2. The reward values may be values determined by a preset reward function … The reward function may output a reward value obtained by evaluating a state, in response to the input of state data indicating this state.” In page 3 para [0040]: “The model 415 outputs recommended control condition data indicating a recommended control condition for each target device 20 (T), in response to the input of state data. ... The reference reward value may be a reward value corresponding to a work state of the facility 2 at a prescribed timing (e.g. the current time), such as a reward value obtained by inputting the state data at this timing into the reward function, or may be a fixed value (e.g. a value obtained by subtracting a tolerance value from the maximum value of the reward value).”).
Ueda, Nakagawa, ABE and Matsubara are analogous art because they are related to have machine learning device that learns a control command for a machine to perform the machine learning to output the control command. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ueda, Nakagawa, ABE and Matsubara, to modify simulating equipment state to determine control of control target is possible by Nakagawa, to include simulating the state of the equipment to judge convergence of the machine learning by Matsubara. The suggestion/motivation for doing so would have been obvious by Matsubara because “Each of the plurality of agents may include a state acquiring section that acquires state data indicating a state of the facility in an apparatus. Each of the plurality of agents may include a control condition acquiring section that acquires control condition data indicating a control condition of each target device. Each of the plurality of agents may include a learning processing section that uses learning data including the state data and the control condition data to perform learning processing of a model that outputs recommended control condition data indicating a control condition recommended for each target device in response to input of the state data.” (Matsubara disclosed in page 1 para [0006]).
Regarding Claim 17, Ueda, Nakagawa and ABE teach the determination apparatus according to claim 1, however Ueda, Nakagawa and ABE do not explicitly teach the limitation “the machine learning by the control model is performed using reinforcement learning so that the operation amount has a reward value determined by a predetermined reward function is output as a recommended operation amount, in response to the acquired state data”.
wherein Matsubara teaches the machine learning by the control model is performed using reinforcement learning so that the operation amount has a reward value determined by a predetermined reward function is output as a recommended operation amount, in response to the acquired state data. (Matsubara disclosed in page 3 para [0034]: “The reward value acquiring section 40 acquires reward values used for reinforcement learning by the agents 41, and acquires reward values for evaluating the operational state of the facility 2. The reward values may be values determined by a preset reward function.” In page 4 para [0050]: “The model 415 may output the reward value predicted in a case where the state of the facility 2 is the state indicated by the predicted state data, … the model 415 may output the reward value in response to the selection of a piece of predicted state data that is predicted in response to the input of the current state data, or may output the reward value in response to the input of the predicted state data.” In page 5 para [0054]: “the control condition series specifying section 4162 generates a plurality of control condition sequences (control condition series) by selecting and linking together one control condition (Cy) at every timing (t+1), (t+2), etc. Furthermore, the control condition series specifying section 4162 specifies a control condition series having the highest total of predicted reward values corresponding to the control conditions, as an example of the control condition series that is most highly recommended among the plurality of control conditions series.”).
Ueda, Nakagawa, ABE and Matsubara are analogous art because they are related to have machine learning device that learns a control command for a machine to perform the machine learning to output the control command. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ueda, Nakagawa, ABE and Matsubara, to modify simulating equipment state to determine control of control target is possible by Nakagawa, to include simulating the state of the equipment to judge convergence of the machine learning by Matsubara. The suggestion/motivation for doing so would have been obvious by Matsubara because “Each of the plurality of agents may include a state acquiring section that acquires state data indicating a state of the facility in an apparatus. Each of the plurality of agents may include a control condition acquiring section that acquires control condition data indicating a control condition of each target device. Each of the plurality of agents may include a learning processing section that uses learning data including the state data and the control condition data to perform learning processing of a model that outputs recommended control condition data indicating a control condition recommended for each target device in response to input of the state data.” (Matsubara disclosed in page 1 para [0006]).
Regarding Claim 13, Ueda, Nakagawa and ABE teach the determination apparatus according to claims 2, is incorporating the rejections of claim 12, because claims 13 has substantially similar claim language as claim 12, therefore claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Ueda, Nakagawa, ABE and Matsubara as discussed above for substantially similar rationale.
Regarding Claim 16, Ueda, Nakagawa and ABE teach the determination apparatus according to claim 14, is incorporating the rejections of claim 15, because claim 16 has substantially similar claim language as claim 15, therefore claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Ueda, Nakagawa, ABE and Matsubara as discussed above for substantially similar rationale.
Regarding Claim 18, Ueda, Nakagawa and ABE teach the determination apparatus according to claim 2, is incorporating the rejections of claim 17, because claim 18 has substantially similar claim language as claim 17, therefore claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Ueda, Nakagawa, ABE and Matsubara as discussed above for substantially similar rationale.
Conclusion
10. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior arts made of record and not relied upon is considered pertinent to applicant's disclosure. An article “A Disturbance Rejection Control Method Based on Deep Reinforcement Learning for a Biped Robot” by Chuzhao Liu et al. disclosed the disturbance rejection performance of a biped robot when walking has long been a focus of roboticists in their attempts to improve robots. There are many traditional stabilizing control methods, such as modifying foot placements and the target zero moment point (ZMP), e.g., in model ZMP control. The disturbance rejection control method in the forward direction of the biped robot is an important technology, whether it comes from the inertia generated by walking or from external forces. The control method based on the model ZMP control is among the main methods of disturbance rejection for biped robots. The state of-the-art deep-reinforcement-learning algorithm combined with model ZMP control in simulating the balance experiment of the cart–table model and the disturbance rejection experiment. To solve disturbance rejection from a practical point of view, the present paper develops a biped robot x-direction disturbance rejection training environment. Different algorithms can be used to test the disturbance rejection ability in the simulation, so that actual problems encountered in the research and development of a simulation robot can be solved quickly and conveniently.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NUPUR DEBNATH whose telephone number is (571)272-8161. The examiner can normally be reached M-F 8:00 am -4:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Renee D Chavez can be reached on (571)270-1104. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NUPUR DEBNATH/Examiner, Art Unit 2186
/RENEE D CHAVEZ/Supervisory Patent Examiner, Art Unit 2186
Prosecution Insights