DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 3-9 have been amended. Claims 1-10 have been examined.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
Claim 1
a state information acquisition unit that acquires state information
an action information output unit that outputs action information
a reward output unit that determines a reward
a value function updating unit that updates a value function
Claim 8
an optimization action information output unit that outputs adjustment information
Claim 9
a control unit configured to set a feedback gain
a frequency response calculation device that calculates an output/input gain
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5, 8 and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication 20180335758 by Shinoda et al. ("Shinoda") in view of “Deep Deterministic Policy Gradient-based Parameter Selection Method of Notch Filters for Suppressing Mechanical Resonance in Industrial Servo Systems” by Oh et al. (“Oh”) and “Determining Stability using the Nyquist Plot” by Cheever (“Cheever”).
In regard to claim 1, Shinoda discloses:
1. A machine learning device which performs machine learning to optimize at least one selected from a coefficient of at least one filter and a feedback gain which are provided in a servo controller for controlling a motor, the machine learning device comprising: Shinoda, Fig. 4 element 100 “Machine Learning Device.”
a state information acquisition unit that acquires state information including at least one selected from the coefficient of the … [transfer function] and the feedback gain, and including an output/input gain and an output/input phase delay of the servo controller; Shinoda, Fig. 4 element 11 “State Information Acquiring Part.” Also Abstract, “state information including a deviation between an actual operation of the target device and a command input to the controller, a phase of the motor, and the coefficients of the transfer function of the controller gain.” Also ¶ 0049, “In order to perform the machine learning, the phase calculating part 21 outputs the phase of the present position to the machine learning device 100. Moreover, in order to realize the machine learning, the coefficients of the transfer function of the present controller gain are input from the position controller 23, the speed controller 25, and the current controller 27 to the machine learning device 100.” Also ¶ 0073, “The state information acquiring part 11 acquires, from the servo control apparatus 200, the state information s …”
Shinoda does not expressly disclose: the filter. However, this is taught by Oh. See Oh, p. 320, bottom right column, “In this paper, the elements of the reinforcement learning for the selection problem of the notch filter parameters are defined and utilized to select these notch filter parameters.” Also section III-A on p. 321, “First, the Bode plot data of the plant are obtained from the servo driver.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Oh’s filter with Shinoda’s servo controller in order to provide resonance suppression as suggested by Oh (see p. 320, section I, first paragraph).
Shinoda also discloses:
an action information output unit that outputs action information including adjustment information of at least one selected from the coefficient and the feedback gain included in the state information; Shinoda, Fig. 4 element 13 “Action Information Output Part.” Also ¶ 0013, “an action information output step of outputting action information including adjustment information of coefficients of a transfer function of a controller gain to a controller (for example, a position controller 23, a speed controller 25, and a current controller 27 to be described later) included in the servo control apparatus;” Also ¶ 0090, “The action information output part 13 is a part that transmits the action information a output from the learning part 12 to a learning target controller (the current controller 27) of the servo control apparatus 200. As described above, the learning target controller finely adjusts the present state s (that is, each of the presently set coefficients of the transfer function of the controller gain of the current controller 27) on the basis of the action information to thereby transition to the next state s′ (that is, each of the corrected coefficients of the transfer function of the controller gain).”
a reward output unit that determines a reward depending on … margin, and outputs the reward; and Shinoda, Fig. 4 element 121 “Reward Output Part.” Also ¶ 0076, “The reward output part 121 is a part that calculates a reward when the action a is selected under a certain state s.” Also ¶ 0080-0082, e.g. “The reward output part 121 sets the value of a reward to a negative value when the value f(PD(s′)) … is larger than the value f(PD(s)) …” Also note Oh, section III-A, “A stability margin is defined as a reward. In this paper, the gain margin from the open-loop Bode plot of the system is given as the reward. However, the reward can be defined differently such as phase margin …”
Shinoda does not expressly disclose determining whether a Nyquist path calculated from the output/input gain and the output/input phase delay passes through an inside of a closed curve which contains therein (−1, 0) on a complex plane and passes through a predetermined gain margin and phase margin. Oh teaches use of Bode diagrams for filter parameter selection (e.g. see Fig. 1 on p. 321), but does not expressly teach the use of Nyquist diagrams. However, this is taught by Cheever. See p. 10, depicting Nyquist diagrams, along with text: “If we zoom in, we can see that the plot in "L(s)" does not encircle the -1+j0, so the system is stable.” Also p. 12, “As you can see, the plot crosses the real axis at about -2/3, or -0.67. This tells us that if we multiply L(s) by a number greater than 3/2that the path would encircle the -1+j0 and the systems would be unstable. So the "gain margin" is 3/2 or 20log10(3/2)=3.5dB. The greater the gain margin, the more stable the system. If the gain margin is zero, the system is marginally stable. … A higher phase margin yields a more stable system. A phase margin of 0° indicates a marginally stable system.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Cheever’s Nyquist diagram with Shinoda’s state information in order to determine system stability as suggested by Cheever (e.g. see top of p. 1).
Shinoda also discloses:
a value function updating unit that updates a value function, based on a value of the reward output by the reward output unit, the state information, and the action information. Shinoda, Fig. 4 element 122 “Value Function Updating Part.” Also ¶ 0084, “The value function updating part 122 updates the value function Q stored in the value function storing part 14 by performing Q-learning with respect to the coefficients of the transfer function of the controller gain of the current controller 27 on the basis of the state s, the action a, the state s′ when the action a was applied to the state s, and the value of the reward calculated in the above-described manner.”
In regard to claim 2, Shinoda does not expressly disclose:
2. The machine learning device according to claim 1, wherein the reward output unit determines the reward based on a distance between the closed curve and the Nyquist path, and outputs the reward. This is taught by Oh and Cheever. See Oh, section III-A, “A stability margin is defined as a reward. In this paper, the gain margin from the open-loop Bode plot of the system is given as the reward. However, the reward can be defined differently such as phase margin …” As noted above, Oh does not expressly teach utilization of a Nyquist diagram. However, Cheever teaches this. See Cheever, p. 12, depicting Nyquist diagrams along with text: “This tells us that if we multiply L(s) by a number greater than 3/2that the path would encircle the -1+j0 and the systems would be unstable. So the "gain margin" is 3/2 or 20log10(3/2)=3.5dB. The greater the gain margin, the more stable the system.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Oh’s rewards and Cheever’s Nyquist diagram in order to maximize stability as suggested by Oh (see section I on p. 320).
In regard to claim 3, Shinoda, Oh and Cheever also teach:
3. The machine learning device according to claim 1, wherein the closed curve is a circle. See Cheever, at least p. 11, “circle.”
In regard to claim 5, Shinoda also discloses:
5. The machine learning device according to claim 1, wherein the reward output unit outputs a total reward obtained by adding a reward calculated based on a closed loop characteristic to the reward. See Shinoda, Fig. 2, element 100 along with Fig. 4 element 11, depicting state/feedback acquisition, broadly interpreted as closed loop characteristics. Also see ¶ 0135, “In this case, the reward output part 121 may add the reward based on the deviation PD(s) and the reward based on the current command value CC(s) …”
In regard to claim 8, Shinoda also discloses:
8. The machine learning device according to claim 1, further comprising an optimization action information output unit that outputs adjustment information of at least one selected from the coefficient and the feedback gain based on a value function updated by the value function updating unit. Shinoda, Fig. 4 element 15. Also ¶ 0092, “As described above, the value function Q is updated by the value function updating part 122 performing the Q-learning with respect to the coefficients of the transfer function of the controller gain of the current controller 27.”
In regard to claim 10, Shinoda discloses:
10. A machine learning method for a machine learning device which performs machine learning to optimize at least one selected from a coefficient of at least one filter and a feedback gain which are provided in a servo controller for controlling a motor, Shinoda, Fig. 5, depicting a method.
All further limitations of claim 10 have been addressed in the above rejection of claim 1.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shinoda in view of Oh and Cheever as applied above, and further in view of U.S. Patent Application Publication 20130057191 by Yoshiura et al. ("Yoshiura").
In regard to claim 4, Shinoda also teaches:
4. The machine learning device according to claim 1, wherein the reward output unit outputs a total reward obtained by adding a reward … to the reward. See Shinoda ¶ 0135, “In this case, the reward output part 121 may add the reward based on the deviation PD(s) and the reward based on the current command value CC(s) …”
Shinoda does not expressly disclose: calculated based on a cutoff frequency. This is taught by Yoshiura. See ¶ 0140, “The cutoff frequencies ω1 and ω2 are set to satisfy the relationship: ω2<ω1. This helps correct phase delay.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Yoshiura’s cutoff frequency with Shinoda’s reward addition in order to help correct phase delay as suggested by Yoshiura.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shinoda in view of Oh and Cheever as applied above, and further in view of U.S. Patent Application Publication 20170270434 by Takigawa et al. ("Takigawa").
In regard to claim 6, Shinoda also discloses:
6. The machine learning device according to claim 1, wherein the reward output unit outputs a total reward obtained by adding a reward calculated by comparing the output/input gain with … [state information] … to the reward. Shinoda, Abstract “outputting a value of a reward in the reinforcement learning based on the deviation included in the state information.” Also see ¶ 0135, “In this case, the reward output part 121 may add the reward based on the deviation PD(s) and the reward based on the current command value CC(s) …”
Shinoda does not expressly disclose: a pre-calculated normative gain. This is taught by Takigawa. See ¶ 0085, “The reward calculation unit 20 provides, when a difference between a laser machined result including a machining speed or time expended for predetermined machining obtained by an operation result acquisition unit 12 and a substantially ideal machined result or a target machined result including a machining speed or time expended for predetermined machining set for each laser machining content is small, a plus reward according to a small size thereof, and when the difference is large, provides a minus reward according to a large size thereof.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Takigawa’s comparison with an ideal along with Shinoda’s gain in order to obtain a substantially ideal results as suggested by Takigawa (see ¶ 0072).
Claim(s) 7 and 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shinoda in view of Oh and Cheever as applied above, and further in view of U.S. Patent Application Publication 20070268797 by Abrishamchian et al. ("Abrishamchian").
In regard to claim 7, Shinoda also discloses:
7. The machine learning device according to claim 1, wherein the output/input gain and the output/input phase delay are calculated … Shinoda, Fig. 4 element 11 “State Information Acquiring Part.” Also Abstract, “state information including a deviation between an actual operation of the target device and a command input to the controller, a phase of the motor, and the coefficients of the transfer function of the controller gain.” Also ¶ 0049, “In order to perform the machine learning, the phase calculating part 21 outputs the phase of the present position to the machine learning device 100. Moreover, in order to realize the machine learning, the coefficients of the transfer function of the present controller gain are input from the position controller 23, the speed controller 25, and the current controller 27 to the machine learning device 100.” Also ¶ 0073, “The state information acquiring part 11 acquires, from the servo control apparatus 200, the state information s …”
Shinoda does not expressly disclose: … by a frequency response calculation device, and the frequency response calculation device calculates the output/input gain and the output/input phase delay by using an input signal of a frequency-changing sinusoidal wave and speed feedback information of the servo controller. This is taught by Abrishamchian. See ¶ 0049, “The open loop transfer function may be obtained in a conventional manner by applying sinusoidal inputs to the control loop over a frequency range of interest and measuring the resulting PES.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Abrishamchian’s frequency response measurement and adjustment with Shinoda’s gain in order to adjust loop gain as suggested by Abrishamchian (see ¶ 0049).
In regard to claim 9, Shinoda discloses:
9. A controller comprising: See Fig. 1, depicting a controller system.
the machine learning device according to claim 1; See citations with respect to claim 1 above.
a servo controller that controls a motor and includes … and a control unit configured to set a feedback gain; and Shinoda Fig. 1 element 200, “Servo Control Apparatus.” Also ¶ 0025, “The servo control apparatus 200 is a device that controls driving of the control target device 300 by performing feedback control.” ¶ 0026, “Shinoda, ¶ 0026, “The control target device 300 is a device having a servo motor that is driven under the control by the servo control apparatus 200.” ¶ 0035, “The speed controller 25 generates a current command from the input speed deviation according to a transfer function K2(s,Θ) of the controller gain represented by “K2P(Θ)+K2I(Θ)/s+K2D(Θ)s” where K2P(Θ) is a proportional gain, K2I(Θ) is an integral gain, and K2D(Θ) is a differential gain, and outputs the generated current command to the current controller 27.”
Shinoda does not expressly disclose: at least one filter. However, this is taught by Oh. See Oh, p. 320, bottom right column, “In this paper, the elements of the reinforcement learning for the selection problem of the notch filter parameters are defined and utilized to select these notch filter parameters.” Also section III-A on p. 321, “First, the Bode plot data of the plant are obtained from the servo driver.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Oh’s filter with Shinoda’s servo controller in order to provide resonance suppression as suggested by Oh (see p. 320, section I, first paragraph).
Shinoda does not expressly disclose:
a frequency response calculation device that calculates an output/input gain and an output/input phase delay of the servo controller, in the servo controller. This is taught by Abrishamchian. See ¶ 0049, “The open loop transfer function may be obtained in a conventional manner by applying sinusoidal inputs to the control loop over a frequency range of interest and measuring the resulting PES.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Abrishamchian’s frequency response measurement and adjustment with Shinoda’s gain in order to adjust loop gain as suggested by Abrishamchian (see ¶ 0049).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
U.S. Patent Application Publication 20160252896 by Nakamura et al. Teaches servo motor vibration suppression. See Abstract.
U.S. Patent Application Publication 20170032282 by Senoo. Teaches the use of machine learning for gain optimization. See Abstract.
“Deep RL Based Notch Filter Design Method for Complex Industrial Servo Systems” by Oh et al. Teaches Filter design for servo systems. See Abstract.
“Approximate Optimal Stabilization Control of Servo Mechanisms based on Reinforcement Learning Scheme” by Lv et al. Teaches servo system stabilization using reinforcement learning. See Abstract.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to James D Rutten whose telephone number is (571)272-3703. The examiner can normally be reached M-F 9:00-5:30 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at (571)272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/James D. Rutten/Primary Examiner, Art Unit 2121