DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 2025/12/08 & 2024/01/23. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1–5 are directed to patent-eligible subject matter under 35 U.S.C. 101. No rejection under 35 U.S.C. 101 is made. The analysis under the 2019 PEG, as incorporated into MPEP § 2106 and updated by the July 2024 Guidance Update on Patent Subject Matter Eligibility Including on Artificial Intelligence, is set forth below.
Step 1 -- whether the claim falls within any statutory category. See MPEP 2106.03.
Claims 1–4 are drawn to a reinforcement learning method and therefore recite a series of steps, falling within the "process" category. Claim 5 is drawn to a motor control unit comprising processing circuitry and a storage and therefore falls within the "machine" category. Each of claims 1–5 falls within one of the four statutory categories. (Step 1: YES.)
Step 2A Prong One -- whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Independent claim 1 recites, among other limitations, "calculate a first reward …," "calculate a second reward as the reward such that a return decreases as a time from start of cranking to completion of starting of the engine increases," "selecting an action that maximizes a Q value," and "update the Q table such that the action is selected with which the return that is the sum of the rewards in the trials becomes larger."
These limitations set forth the calculation of rewards, the computation of a return as a sum of rewards, the selection of an action that maximizes a Q value, and the iterative updating of the Q value — i.e., mathematical calculations and mathematical relationships used to optimize the action value function. Such limitations fall within the "mathematical concepts" grouping of abstract ideas. See MPEP § 2106.04(a)(2), subsection I. This treatment is consistent with the July 2024 Guidance Update, Example 47, in which training operations performed by mathematical optimization algorithms (e.g., gradient descent/backpropagation) were found to recite mathematical concepts.
Independent claim 5 recites corresponding limitations in apparatus form and is found to recite the same abstract idea for the same reasons. (Step 2A Prong One: YES, claims 1–5 recite an abstract idea.)
Step 2A Prong Two -- whether the claim as a whole integrates the recited judicial exception into a practical application, or whether the claim is "directed to" the judicial exception. See MPEP 2106.04(d).
This evaluation (1) identifies the additional elements recited beyond the judicial exception and (2) evaluates those additional elements individually and in combination to determine whether the claim as a whole integrates the exception into a practical application. One manner of integration is an improvement to the functioning of a computer or to another technology or technical field, where the improvement is described in the specification and reflected in the claim. See MPEP § 2106.04(d)(1) and § 2106.05(a).
Regarding independent claim 1, the claim recites the following additional elements beyond the abstract idea:
"cranking an engine to start the engine by controlling a motor using the Q table";
"the Q table is used in a motor control unit that determines a torque command value for the motor";
"a sound pressure detected by a noise meter that detects noise emitted from a vehicle"; and
"an acceleration detected by an acceleration sensor that detects vibration of the vehicle."
Taken as a whole, these additional elements do more than generally link the abstract idea to a technological environment or recite a generic computer as a tool. In each trial, the recited method physically cranks the engine to start it by controlling a motor, obtains the first-reward quantity from physical measurements of vehicle noise (a noise meter) and vehicle vibration (an acceleration sensor), and applies the optimized Q table in a motor control unit that determines the torque command value for the motor. The claim thereby uses the result of the recited mathematical operations to control a particular machine — the motor that cranks the engine — and reflects an improvement in the technical field of engine starting, namely starting the engine quickly while suppressing the noise and vibration generated during cranking. The specification describes this improvement (the objective of completing engine starting promptly while suppressing vibration and noise emitted during cranking), and claim 1 reflects that improvement through the first reward (noise/vibration), the second reward (cranking-to-start time), and the application of the Q table to determine the motor torque command. This is analogous to the eligible claim of Example 47 (Claim 3) of the July 2024 Guidance Update, in which additional elements that used a model's output to take real-world action improving a technical field integrated the exception into a practical application. See MPEP § 2106.04(d)(1), § 2106.05(a).
Regarding independent claim 5, the claim recites the additional elements of "processing circuitry," "a storage storing a Q table," and, in particular, "the processing circuitry is configured to cause the motor to crank the engine by driving the motor based on the determined torque command value." The last element applies the optimized Q table to physically drive the motor to crank and start the engine, controlling a particular machine and reflecting the same improvement to engine-starting technology described in the specification. The additional elements, individually and in combination, integrate the abstract idea into a practical application for the same reasons set forth above with respect to claim 1.
Because the additional elements of claims 1 and 5, considered as a whole, integrate the recited abstract idea into a practical application by improving the technical field of engine starting and by applying the exception in the control of a particular machine, claims 1 and 5 are not "directed to" the judicial exception. (Step 2A Prong Two: YES, integrated into a practical application.)
Regarding dependent claims 2–4, these claims depend from claim 1 and further specify the manner of calculating the first reward from the measured sound pressure (claim 2), the manner of calculating the first reward from the measured acceleration (claim 3), and the action as an amount of change of the torque command value together with a third reward keyed to the torque command value range (claim 4). These limitations further refine the recited rewards and action used to control the motor and therefore likewise do not cause the claims to be directed to the abstract idea, for the same reasons set forth above with respect to claim 1.
Step 2B -- whether the claim amounts to significantly more than the judicial exception. See MPEP § 2106.05.
Because the analysis concludes at Step 2A Prong Two that claims 1–5 are not directed to the judicial exception, Step 2B is not reached. See MPEP § 2106.04(d).
Conclusion
Claims 1–5 recite an abstract idea (mathematical concepts) but integrate that exception into a practical application at Step 2A Prong Two and are therefore eligible under 35 U.S.C. 101. No § 101 rejection is made.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 3, and 5 are rejected under 35 U.S.C. 103 as being unpatentable over Hashimoto et al. (Hashimoto '984), U.S. Patent No. 11,230,984 B2, in view of Hashimoto (Hashimoto '580), U.S. Patent No. 8,666,580 B2, and Kanayama et al. (Kanayama), U.S. Patent No. 9,233,685 B2, and further in view of Katsuki (Katsuki), U.S. Patent No. 10,353,351 B2.
Regarding claim 1, (Hashimoto '984) teaches a reinforcement learning method, comprising:
"A reinforcement learning method of optimizing a Q table through reinforcement learning in which a computer is caused to repeatedly execute a trial … using the Q table, wherein"
"by selecting an action that maximizes a Q value,"
"the Q table defines a correspondence relationship between a state variable that includes an engine rotation speed … the action, and the Q value,"
"the reinforcement learning method comprises causing the computer to repeatedly, in each trial:"
"update the Q table such that the action is selected with which the return that is the sum of the rewards in the trials becomes larger."
As to "A reinforcement learning method of optimizing a Q table through reinforcement learning in which a computer is caused to repeatedly execute a trial … using the Q table, wherein," (Hashimoto '984) discloses that the relationship defining data set is used to define an action value function Q, and that this action value function Q is "a table-type function" representing an expected return for a state s and an action a, which is updated through the reinforcement learning process ((Hashimoto '984), p. 14; p. 10). (Hashimoto '984) further discloses that the CPU repeatedly executes the series of processes of FIG. 6 (steps S510–S590) at each execution of the operation process ((Hashimoto '984), p. 14), i.e., repeatedly executes a trial using the Q table.
As to "by selecting an action that maximizes a Q value," (Hashimoto '984) discloses a policy that maximizes the probability of selecting the action that maximizes the action value function Q(s[t], a), i.e., a greedy action, implemented by an ε-greedy selection method ((Hashimoto '984), p. 14).
As to "the Q table defines a correspondence relationship between a state variable that includes an engine rotation speed … the action, and the Q value," (Hashimoto '984) discloses that the state s is determined on the basis of eight variables including "the engine rotation speed NE," that the action a is determined on the basis of operated-amount variables, and that "the action value function Q(s, a)" is a table-type function — i.e., a correspondence among the state variable (including engine rotation speed), the action, and the Q value ((Hashimoto '984), p. 14).
As to "the reinforcement learning method comprises causing the computer to repeatedly, in each trial:," (Hashimoto '984) discloses the CPU repeatedly executing, at each trial, the steps of acquiring the state, selecting the action, calculating a reward, and updating the data set ((Hashimoto '984), p. 14, FIG. 6).
As to "update the Q table such that the action is selected with which the return that is the sum of the rewards in the trials becomes larger," (Hashimoto '984) discloses calculating an error (temporal-difference) used to compute an update amount that updates the value of the action value function Q(s[t], a[t]), the reward being calculated as a sum of multiple rewards, and updating the data set so as to increase the expected return of the reward ((Hashimoto '984), p. 15; p. 10).
(Hashimoto '984) teaches subject matter related to determining, through reinforcement learning, an operated amount for controlling an internal combustion engine. However, (Hashimoto '984) does not teach "of cranking an engine to start the engine by controlling a motor using the Q table"; "the Q table is used in a motor control unit that determines a torque command value for the motor"; and "the most recent torque command value to the motor."
In the same field of endeavor, (Hashimoto '580) teaches "of cranking an engine to start the engine by controlling a motor using the Q table"; "the Q table is used in a motor control unit that determines a torque command value for the motor"; and "the most recent torque command value to the motor." Specifically, (Hashimoto '580) discloses that "the engine is started by being motored by the first motor generator (MG1)" ((Hashimoto '580), p. 15), i.e., the engine is cranked to start it by controlling a motor; that the control device (ECU) sets the torque command value for the motor, e.g., "the ECU sets torque command value Tacom1" and sets torque command value Tacom2 by adding a vibration-reduction compensation torque ((Hashimoto '580), p. 22; claim, p. 27); and that the motor torque command value is controlled in accordance with, and smoothed relative to, its preceding value to prevent abrupt variation of "the torque command value" ((Hashimoto '580), p. 15), i.e., the control uses the most recent torque command value to the motor.
(Hashimoto '984) and (Hashimoto '580) are analogous to the claimed invention as both are from the same field of endeavor of controlling a motor/engine of a vehicle to determine a torque-related command. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the table-type Q-value reinforcement learning of (Hashimoto '984) with the motor cranking-torque control of (Hashimoto '580) so that the action whose value is represented in the Q table is the motor torque command value used to crank the engine, and so that the state variable further includes the most recent torque command value to the motor. The motivation to combine (Hashimoto '984) and (Hashimoto '580) is to reduce the man-hours required for calibration by determining the cranking-torque command through reinforcement learning rather than by manual adaptation ((Hashimoto '984), p. 10, noting that the reinforcement-learning controller reduces the number of man-hours of skilled workers), which would have predictably yielded a reinforcement-learning-optimized motor cranking-torque policy (combining prior-art elements according to known methods to yield predictable results; MPEP § 2143(A)).
The combination of (Hashimoto '984) and (Hashimoto '580), however, does not teach "calculate a first reward as a reward based on a sound pressure detected by a noise meter that detects noise emitted from a vehicle or an acceleration detected by an acceleration sensor that detects vibration of the vehicle."
In the same field of endeavor, (Kanayama) teaches the subject matter underlying "calculate a first reward as a reward based on a sound pressure detected by a noise meter that detects noise emitted from a vehicle or an acceleration detected by an acceleration sensor that detects vibration of the vehicle." Specifically, (Kanayama) discloses that, at the start of the engine, the electric motor cranks the engine to increase the engine rotation speed, and that as the engine rotation speed passes through the speed corresponding to the resonant frequency, "shock and noise due to vibration" are produced ((Kanayama), p. 11; p. 1), thereby identifying the noise (sound pressure) and the vibration (acceleration) generated during cranking as the quantities to be suppressed. (Hashimoto '984) further teaches calculating the reward as a sum of multiple reward factors, including a reward related to driver comfort ((Hashimoto '984), p. 15), thereby providing the framework for calculating such a quantity as a reward.
(Hashimoto '984), (Hashimoto '580), and (Kanayama) are analogous to the claimed invention as all are from the same field of endeavor of controlling a motor to start/crank an engine of a hybrid vehicle. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to calculate, as one of the rewards in the reinforcement learning of (Hashimoto '984) as applied to the motor cranking control of (Hashimoto '580), a first reward based on the sound pressure (noise) or the acceleration (vibration) generated during cranking as recognized by (Kanayama). The motivation to combine (Hashimoto '984), (Hashimoto '580), and (Kanayama) is that (Kanayama) teaches the cranking-induced vibration and noise degrade drivability and are to be suppressed ((Kanayama), p. 11), so that incorporating the sound pressure or acceleration into the reward would predictably cause the reinforcement learning to optimize the cranking-torque command so as to suppress noise and vibration.
The combination of (Hashimoto '984), (Hashimoto '580), and (Kanayama), however, does not teach "calculate a second reward as the reward such that a return decreases as a time from start of cranking to completion of starting of the engine increases."
In the same field of endeavor, (Katsuki) teaches the subject matter underlying "calculate a second reward as the reward such that a return decreases as a time from start of cranking to completion of starting of the engine increases." Specifically, (Katsuki) discloses a machine-learning system for a motor control system that observes the rotation number and torque of the motor as state variables, calculates a reward based on the motor output, and updates an action value table based on the reward ((Katsuki), p. 1; p. 10), i.e., a reward-driven reinforcement-learning loop applied to motor control. (Hashimoto '984) further discloses calculating the reward as a sum of reward factors and weighting the value of later returns by a discount factor, whereby the contribution of a reward to the return diminishes as time advances ((Hashimoto '984), p. 15), and (Kanayama) teaches that completing the engine start quickly, by rapidly passing through the resonant rotation-speed region, is desirable ((Kanayama), p. 11). One of ordinary skill in the art would interpret these teachings to derive a reward, applied in the reinforcement-learning loop of (Hashimoto '984) and (Katsuki), that assigns a negative value at each step of a trial such that the return that is the sum of the rewards decreases as the time from start of cranking to completion of starting of the engine increases, thereby causing the reinforcement learning to favor cranking-torque commands that shorten the time required to start the engine.
(Hashimoto '984), (Hashimoto '580), (Kanayama), and (Katsuki) are analogous to the claimed invention as all are from the same field of endeavor of reinforcement-learning-based control of a motor for a vehicle. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to calculate a second reward, in the reinforcement learning of (Hashimoto '984) as applied to the motor cranking control of (Hashimoto '580), such that the return decreases as the time from the start of cranking to completion of starting of the engine increases. The motivation to combine (Hashimoto '984), (Hashimoto '580), (Kanayama), and (Katsuki) is that completing the engine start quickly is a recognized objective in cranking control ((Kanayama), p. 11, describing the desirability of rapidly passing through the resonant rotation-speed region at the start of the engine), so that assigning a reward whose return decreases with increasing cranking-to-start time — a known reinforcement-learning technique evidenced by the reward-driven motor learning of (Katsuki) and the discounting of (Hashimoto '984) — would predictably cause the reinforcement learning to optimize the cranking-torque command so as to shorten the time required to start the engine.
Claim 2 depends from claim 1. All limitations of claim 1 that are incorporated into claim 2 are rejected under the same rationale set forth above with respect to claim 1.
Claim 2 further recites:
"wherein the computer calculates the first reward such that the return is smaller when a logical disjunction is satisfied than when the logical disjunction is not satisfied, the logical disjunction being the sound pressure being greater than or equal to a threshold value and an amount of change of the sound pressure per unit time being greater than or equal to a prescribed amount."
Regarding claim 2, (Hashimoto '984) teaches:
"the computer calculates the first reward such that the return is smaller when a logical disjunction is satisfied than when the logical disjunction is not satisfied"
As to "the computer calculates the first reward such that the return is smaller when a logical disjunction is satisfied than when the logical disjunction is not satisfied," (Hashimoto '984) discloses that the computer (CPU) calculates the reward as a sum of signed reward factors, and that a reward factor is assigned a smaller value when a corresponding condition is satisfied than when the condition is not satisfied ((Hashimoto '984), p. 15). (Hashimoto '984) therefore teaches calculating the first reward such that the return is smaller when a condition is satisfied than when the condition is not satisfied.
(Hashimoto '984) teaches subject matter related to a reward whose return is smaller upon satisfaction of a condition. However, (Hashimoto '984) does not teach "the logical disjunction being the sound pressure being greater than or equal to a threshold value" and "an amount of change of the sound pressure per unit time being greater than or equal to a prescribed amount."
In the same field of endeavor, (Kanayama) teaches:
"the logical disjunction being the sound pressure being greater than or equal to a threshold value"
"an amount of change of the sound pressure per unit time being greater than or equal to a prescribed amount"
As to "the logical disjunction being the sound pressure being greater than or equal to a threshold value," (Kanayama) discloses that, at the start of the engine, the electric motor cranks the engine to increase the engine rotation speed, and that when the engine rotation speed reaches the rotation speed corresponding to the resonant frequency, "shock and noise due to vibration" are produced ((Kanayama), p. 11; p. 1). (Kanayama) therefore teaches that the cranking sound pressure becomes significant — i.e., reaches or exceeds a threshold level — when the engine rotation speed reaches the resonant region, such that one of ordinary skill in the art would interpret this teaching to derive the condition that the sound pressure is greater than or equal to a threshold value.
As to "an amount of change of the sound pressure per unit time being greater than or equal to a prescribed amount," (Kanayama) discloses that the cranking sound pressure rises as the engine rotation speed increases toward and through the resonant region during the engine start ((Kanayama), p. 11), i.e., the sound pressure increases over time as the cranking proceeds. (Kanayama) therefore teaches that the sound pressure changes per unit time during cranking, such that one of ordinary skill in the art would interpret this teaching to derive the condition that an amount of change of the sound pressure per unit time is greater than or equal to a prescribed amount.
(Hashimoto '984) and (Kanayama) are analogous to the claimed invention as both are from the same field of endeavor of controlling a motor to start/crank an engine of a hybrid vehicle. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the conditional reward calculation of (Hashimoto '984) with the cranking-sound-pressure teaching of (Kanayama) so that the computer calculates the first reward such that the return is smaller when the sound pressure is greater than or equal to a threshold value and when an amount of change of the sound pressure per unit time is greater than or equal to a prescribed amount, than when those conditions are not satisfied. The motivation to combine (Hashimoto '984) and (Kanayama) is that (Kanayama) teaches the cranking-induced noise degrades drivability and is to be suppressed ((Kanayama), p. 11), so that calculating the first reward such that the return is smaller upon satisfaction of the sound-pressure-magnitude condition or the sound-pressure-rate-of-change condition would predictably cause the reinforcement learning to favor cranking-torque commands that avoid both a high level of, and a rapid rise in, the cranking noise.
Regarding Claim 3, Claim 3 depends from claim 1. All limitations of claim 1 that are incorporated into claim 3 are rejected under the same rationale set forth above with respect to claim 1.
Claim 3 further recites:
"wherein the computer calculates the first reward such that the return decreases as the acceleration increases."
Regarding claim 3, (Hashimoto '984) teaches:
"the computer calculates the first reward"
As to "the computer calculates the first reward," (Hashimoto '984) discloses that the computer (CPU) calculates the reward as a sum of signed reward factors, the value of each factor being set in accordance with a corresponding state quantity ((Hashimoto '984), p. 15). (Hashimoto '984) therefore teaches that the computer calculates a reward, including the first reward, as part of the reinforcement-learning loop.
(Hashimoto '984) teaches subject matter related to calculating a reward whose value is set in accordance with a state quantity. However, (Hashimoto '984) does not teach "such that the return decreases as the acceleration increases."
In the same field of endeavor, (Hashimoto '580) teaches "such that the return decreases as the acceleration increases." Specifically, (Hashimoto '580) discloses that the vibration reduction torque applied through the motor is set, based on the crank angle, as a periodic torque that suppresses the engine torque pulsation, such that the magnitude of the vibration reduction torque corresponds to the magnitude of the torque pulsation (vibration) to be suppressed ((Hashimoto '580), p. 1; p. 15). (Hashimoto '580) therefore teaches that a larger vehicle vibration is treated as a more adverse condition warranting a correspondingly larger suppressing response, such that one of ordinary skill in the art would interpret this teaching to derive a first reward whose return decreases as the magnitude of the vibration increases.
(Hashimoto '984) and (Hashimoto '580) are analogous to the claimed invention as both are from the same field of endeavor of controlling a motor/engine of a vehicle to determine a torque-related command. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the reward calculation of (Hashimoto '984) with the magnitude-proportional vibration treatment of (Hashimoto '580) so that the computer calculates the first reward such that the return decreases as the magnitude of the vibration increases. The motivation to combine (Hashimoto '984) and (Hashimoto '580) is that (Hashimoto '580) teaches that a larger vehicle vibration is a more adverse condition to be suppressed by a correspondingly larger response ((Hashimoto '580), p. 1), so that calculating the first reward such that the return decreases as the vibration increases would predictably cause the reinforcement learning to favor cranking-torque commands that minimize the vehicle vibration.
The combination of (Hashimoto '984) and (Hashimoto '580), however, does not teach "based on … the acceleration," i.e., that the vibration on which the first reward is based is an acceleration detected by an acceleration sensor that detects vibration of the vehicle.
In the same field of endeavor, (Kanayama) teaches "based on … the acceleration." Specifically, (Kanayama) discloses that, at the start of the engine, the electric motor cranks the engine to increase the engine rotation speed, and that as the engine rotation speed passes through the speed corresponding to the resonant frequency, "shock and noise due to vibration" are produced ((Kanayama), p. 11; p. 1). (Kanayama) therefore teaches that the vibration of the vehicle generated during cranking — i.e., the acceleration — is the quantity on which the first reward is based.
(Hashimoto '984), (Hashimoto '580), and (Kanayama) are analogous to the claimed invention as all are from the same field of endeavor of controlling a motor to start/crank an engine of a hybrid vehicle. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to calculate the first reward such that the return decreases as the acceleration increases. The motivation to combine (Hashimoto '984), (Hashimoto '580), and (Kanayama) is that (Kanayama) teaches the cranking-induced vibration degrades drivability and is to be suppressed ((Kanayama), p. 11), so that calculating the first reward, based on the acceleration, such that the return decreases as the acceleration increases would predictably cause the reinforcement learning to favor cranking-torque commands that minimize the vehicle vibration.
Regarding Claim 5, Claim 5 is the apparatus counterpart of claim 1. The following limitations of claim 5 recite, in apparatus form, subject matter corresponding to limitations of claim 1, and are rejected under the same rationale set forth above with respect to claim 1: "the Q table is used in the motor control unit"; "the motor control unit determines a torque command value for a motor by selecting an action that maximizes a Q value"; "the Q table defines a correspondence relationship between a state variable that includes an engine rotation speed and the most recent torque command value to the motor, the action, and the Q value"; "the reinforcement learning method optimizes the Q table through reinforcement learning in which a computer is caused to repeatedly execute a trial of cranking an engine to start the engine by controlling the motor using the Q table"; "the reinforcement learning method includes causing the computer to repeatedly, in each trial:"; "calculate a first reward as a reward based on a sound pressure detected by a noise meter that detects noise emitted from a vehicle or an acceleration detected by an acceleration sensor that detects vibration of the vehicle"; "calculate a second reward as the reward such that a return decreases as a time from start of cranking to completion of starting of the engine increases"; and "update the Q table such that the action is selected with which the return that is the sum of the rewards in the trials becomes larger."
Claim 5 additionally recites, in pertinent part:
"A motor control unit, comprising: processing circuitry; and a storage storing a Q table updated by a reinforcement learning method, wherein"
"the processing circuitry is configured to refer to the Q table stored in the storage to determine the torque command value by selecting the action that maximizes the Q value based on a state variable including an engine rotation speed and the most recent torque command value for the motor, and"
"the processing circuitry is configured to cause the motor to crank the engine by driving the motor based on the determined torque command value."
Regarding claim 5, (Hashimoto '984) teaches a motor control unit, comprising:
"processing circuitry; and"
"a storage storing a Q table updated by a reinforcement learning method, wherein"
"refer to the Q table stored in the storage to determine the torque command value by selecting the action that maximizes the Q value based on a state variable including an engine rotation speed"
As to "processing circuitry," (Hashimoto '984) discloses a controller implemented by a CPU (e.g., "the CPU 72") that executes the operation and reinforcement-learning processes ((Hashimoto '984), p. 12).
As to "a storage storing a Q table updated by a reinforcement learning method," (Hashimoto '984) discloses that a nonvolatile memory stores the relationship defining data set, which is used to define the table-type action value function Q and is updated through the reinforcement learning process ((Hashimoto '984), p. 12; p. 14).
As to "refer to the Q table stored in the storage to determine the torque command value by selecting the action that maximizes the Q value based on a state variable including an engine rotation speed," (Hashimoto '984) discloses that the policy refers to the table-type action value function Q to select the action that maximizes the action value function Q(s[t], a), i.e., the greedy action, based on the state s that includes "the engine rotation speed NE" ((Hashimoto '984), p. 14).
(Hashimoto '984) teaches subject matter related to determining, through reinforcement learning, an operated amount for controlling an internal combustion engine. However, (Hashimoto '984) does not teach "determine the torque command value … and the most recent torque command value for the motor"; and "the processing circuitry is configured to cause the motor to crank the engine by driving the motor based on the determined torque command value."
In the same field of endeavor, (Hashimoto '580) teaches "determine the torque command value … and the most recent torque command value for the motor," and "the processing circuitry is configured to cause the motor to crank the engine by driving the motor based on the determined torque command value." Specifically, (Hashimoto '580) discloses that the control device (ECU) sets the torque command value for the motor, e.g., "the ECU sets torque command value Tacom1" and sets torque command value Tacom2 by adding a vibration-reduction compensation torque ((Hashimoto '580), p. 22), with the torque command value being controlled relative to its preceding value to prevent abrupt variation ((Hashimoto '580), p. 15), i.e., based on the most recent torque command value for the motor; and that "the engine is started by being motored by the first motor generator (MG1)" ((Hashimoto '580), p. 15), i.e., the motor is driven, based on the set torque command value, to crank the engine and start it.
(Hashimoto '984) and (Hashimoto '580) are analogous to the claimed invention as both are from the same field of endeavor of controlling a motor/engine of a vehicle to determine a torque-related command. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the processing circuitry and the storage storing the table-type Q of (Hashimoto '984) with the motor cranking-torque control of (Hashimoto '580) so that the processing circuitry refers to the stored Q table to determine the motor torque command value based on a state variable including an engine rotation speed and the most recent torque command value for the motor, and causes the motor to crank the engine by driving the motor based on the determined torque command value. The motivation to combine (Hashimoto '984) and (Hashimoto '580) is to reduce the man-hours required for calibration by determining the cranking-torque command through reinforcement learning rather than by manual adaptation ((Hashimoto '984), p. 10), which would have predictably yielded a motor control unit that determines and applies a reinforcement-learning-optimized cranking-torque command.
The combination of (Hashimoto '984) and (Hashimoto '580), however, does not teach "calculate a first reward as a reward based on a sound pressure detected by a noise meter that detects noise emitted from a vehicle or an acceleration detected by an acceleration sensor that detects vibration of the vehicle." In the same field of endeavor, (Kanayama) teaches this subject matter, disclosing that, at the start of the engine, the electric motor cranks the engine to increase the engine rotation speed, and that as the engine rotation speed passes through the speed corresponding to the resonant frequency, "shock and noise due to vibration" are produced ((Kanayama), p. 11; p. 1), thereby identifying the noise (sound pressure) and the vibration (acceleration) generated during cranking as the quantities to be calculated as a reward in the framework of (Hashimoto '984) ((Hashimoto '984), p. 15). (Hashimoto '984), (Hashimoto '580), and (Kanayama) are analogous to the claimed invention as all are from the same field of endeavor of controlling a motor to start/crank an engine of a hybrid vehicle. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to calculate the first reward based on the sound pressure or the acceleration generated during cranking, for the same reasons and under the same motivation set forth above with respect to claim 1.
The combination of (Hashimoto '984), (Hashimoto '580), and (Kanayama), however, does not teach "calculate a second reward as the reward such that a return decreases as a time from start of cranking to completion of starting of the engine increases." In the same field of endeavor, (Katsuki) teaches a machine-learning system for a motor control system that calculates a reward based on the motor output and updates an action value table based on the reward ((Katsuki), p. 1; p. 10), i.e., a reward-driven reinforcement-learning loop applied to motor control, and (Katsuki) likewise discloses storing and updating the action value table ((Katsuki), p. 1; p. 10), corroborating "a storage storing a Q table updated by a reinforcement learning method." (Hashimoto '984) additionally discloses weighting later returns by a discount factor ((Hashimoto '984), p. 15). One of ordinary skill in the art would interpret these teachings to derive a reward, applied in the reinforcement-learning loop, that assigns a negative value at each step of a trial such that the return that is the sum of the rewards decreases as the time from start of cranking to completion of starting of the engine increase.
(Hashimoto '984), (Hashimoto '580), (Kanayama), and (Katsuki) are analogous to the claimed invention as all are from the same field of endeavor of reinforcement-learning-based control of a motor for a vehicle. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to calculate the second reward such that the return decreases as the time from start of cranking to completion of starting of the engine increases, for the same reasons and under the same motivation set forth above with respect to claim 1.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Hashimoto et al. (Hashimoto '984), U.S. Patent No. 11,230,984 B2, in view of Hashimoto (Hashimoto '580), U.S. Patent No. 8,666,580 B2, and Kanayama et al. (Kanayama), U.S. Patent No. 9,233,685 B2, in view of Katsuki (Katsuki), U.S. Patent No. 10,353,351 B2, and further in view of Hashimoto et al. (Hashimoto '086), U.S. Patent Application Publication No. 2021/0253086 A1.
Claim 4 depends from claim 1. All limitations of claim 1 that are incorporated into claim 4 are rejected under the same rationale set forth above with respect to claim 1.
Claim 4 further recites:
"the method optimizing the Q table used in the motor control unit, wherein"
"the motor control unit refers to the Q table so as to select, as the action to be selected to determine the torque command value, one of multiple options for the amount of change of the torque command value, the options being set with different amounts of change, and"
"the reinforcement learning method further comprises causing the computer to calculate a third reward as the reward such that the return is smaller when the torque command value exceeds a prescribed range than when the torque command value does not exceed the prescribed range."
Regarding claim 4, (Hashimoto '984) teaches:
"the method optimizing the Q table"
"the motor control unit refers to the Q table so as to select, as the action to be selected … one of multiple options"
"the reinforcement learning method further comprises causing the computer to calculate a third reward as the reward"
As to "the method optimizing the Q table," (Hashimoto '984) discloses that the relationship defining data set, which defines the table-type action value function Q, is updated through the reinforcement learning process so as to increase the expected return ((Hashimoto '984), p. 14; p. 10).
As to "the motor control unit refers to the Q table so as to select, as the action to be selected … one of multiple options," (Hashimoto '984) discloses that the policy refers to the table-type action value function Q and selects the action that maximizes Q(s[t], a) from among the finite set of action values defined in the table, i.e., selects one of multiple options ((Hashimoto '984), p. 14).
As to "the reinforcement learning method further comprises causing the computer to calculate a third reward as the reward," (Hashimoto '984) discloses that the computer calculates the reward as a sum of multiple signed reward factors ((Hashimoto '984), p. 15), thereby providing for the calculation of an additional reward factor among the rewards.
(Hashimoto '984) teaches subject matter related to selecting an action from a table-type Q to control an internal combustion engine. However, (Hashimoto '984) does not teach "used in the motor control unit," nor that the selected action is "to determine the torque command value."
In the same field of endeavor, (Hashimoto '580) teaches "used in the motor control unit" and that the selected action is "to determine the torque command value." Specifically, (Hashimoto '580) discloses that the control device (ECU) sets the torque command value for the motor, e.g., "the ECU sets torque command value Tacom1" and sets torque command value Tacom2 by adding a vibration-reduction compensation torque ((Hashimoto '580), p. 22; claim, p. 27), i.e., a motor control unit that determines the torque command value for the motor.
(Hashimoto '984) and (Hashimoto '580) are analogous to the claimed invention as both are from the same field of endeavor of controlling a motor/engine of a vehicle to determine a torque-related command. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the table-type Q action selection of (Hashimoto '984) with the motor control unit of (Hashimoto '580) so that the motor control unit refers to the Q table to select the action that determines the torque command value for the motor. The motivation to combine (Hashimoto '984) and (Hashimoto '580) is to reduce the man-hours required for calibration by determining the cranking-torque command through reinforcement learning rather than by manual adaptation ((Hashimoto '984), p. 10), which would have predictably yielded a reinforcement-learning-optimized motor cranking-torque policy (MPEP § 2143(A)).
The combination of (Hashimoto '984) and (Hashimoto '580), however, does not teach "for the amount of change of the torque command value, the options being set with different amounts of change"; and "such that the return is smaller when the torque command value exceeds a prescribed range than when the torque command value does not exceed the prescribed range."
In the same field of endeavor, (Hashimoto '086) teaches "for the amount of change of the torque command value, the options being set with different amounts of change," and "such that the return is smaller when the torque command value exceeds a prescribed range than when the torque command value does not exceed the prescribed range."
As to "for the amount of change of the torque command value, the options being set with different amounts of change," (Hashimoto '086) discloses that, in the reinforcement learning of an operation/command variable of the vehicle, the action handled by the reinforcement learning is expressed in terms of a change amount of the operation variable ((Hashimoto '086), p. 16), the action variable being the command value operated upon ((Hashimoto '086), p. 15). One of ordinary skill in the art would interpret this teaching of (Hashimoto '086) to derive an action defined as one of multiple options for the amount of change of the torque command value, the options being set with different amounts of change.
As to "such that the return is smaller when the torque command value exceeds a prescribed range than when the torque command value does not exceed the prescribed range," (Hashimoto '086) discloses a reward-calculating process that provides a greater reward when a characteristic of the vehicle meets a standard than when the characteristic does not meet the standard ((Hashimoto '086), p. 1; p. 12–13), i.e., a reward whose return is smaller when a value fails to satisfy a prescribed condition. One of ordinary skill in the art would interpret this teaching of (Hashimoto '086) to derive a third reward whose return is smaller when the torque command value exceeds a prescribed range than when the torque command value does not exceed the prescribed range.
(Hashimoto '984), (Hashimoto '580), and (Hashimoto '086) are analogous to the claimed invention as all are from the same field of endeavor of reinforcement-learning-based control of a vehicle to determine a command value. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to configure the motor control unit of the combination so as to select, as the action that determines the torque command value, one of multiple options for the amount of change of the torque command value, the options being set with different amounts of change, and to calculate a third reward such that the return is smaller when the torque command value exceeds a prescribed range than when the torque command value does not exceed the prescribed range. The motivation to combine (Hashimoto '984), (Hashimoto '580), and (Hashimoto '086) is that (Hashimoto '086) teaches expressing the action as a change amount of the command and penalizing the reward when a characteristic of the vehicle fails to satisfy a prescribed standard ((Hashimoto '086), p. 1; p. 16), so that selecting the torque command from a finite set of bounded change amounts and penalizing the return when the torque command value exceeds a prescribed range would predictably cause the reinforcement learning to confine the cranking-torque command within an acceptable range.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HUNG VAN LE whose telephone number is (571)270-0164. The examiner can normally be reached 8 a.m. - 5 p.m..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571) 272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HUNG VAN LE/Examiner, Art Unit 2145
/CHAU T NGUYEN/Primary Examiner, Art Unit 2145
Prosecution Insights