Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5, 7-8, and 11-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fuji et al. (US Pub. No. 2018/0181089, published June 2018, hereinafter “Fuji”) in view of Zhu et al. (NPL dated Feb. 2021, Self-correcting Q-Learning, hereinafter “Zhu”), and further in view of Mousavi et al. (NPL dated Dec. 2014, Context Transfer in Reinforcement Learning Using Action-Value Functions, hereinafter “Mousavi”).
Regarding claim 1, Fuji teaches an information processing device comprising:
one or more hardware processors configured to function as (Fuji, [0035] – “The control device 4 can be configured on, for example, a general-purpose computer, and a hardware configuration (not illustrated) of the control device 4 includes an arithmetic unit configured by a central processing unit (CPU), a random access memory (RAM), and the like, a storage unit configured by a read only memory (ROM), a hard disk drive (HDD), a solid state drive (SSD) using a flash memory or the like, and the like, a connection device of a parallel interface format or a serial interface format, and the like.”):
an acquisition unit that acquires a current state of a device (Fuji, [0027] – “the control device 4 according to the present embodiment includes a state acquisition unit 51 that processes input values from at least one sensor 2 or the like mounted inside the machine and determines state values that are output to control units 11 to 1n.sub.2 and 21 to 2n.sub.2 and a learning unit 71,” – teaches an acquisition unit that acquires a current state of a device (state acquisition unit 51 acquires state values from control device 4));
a first action value function specifying unit that has a function of learning a first inference model by reinforcement learning, and specifies a first action value function of the device on a basis of the current state and the first inference model (Fuji, [0028] – “The control device 4 according to the present embodiment operates the control units 11 to 1n.sub.2 identifying the control models 31 to 3n.sub.1 by learning and the control units 21 to 2n.sub.2 having one or more existing control models 41 to 4n.sub.2, which are illustrated in FIG. 1 in parallel to output the action value and the action of each of the control units 11 to 1n.sub.2 and 21 to 2n.sub.2 to the action value selection unit 61,” – teaches a first action value function specifying unit that has a function of learning a first inference model, and specifies a first action value function of the device on a basis of the current state and the first inference model (device operates control units identifying control models 31, models are identified by learning and output an action value based on a basis of current state, as state is input to control units in [0027], and the first inference model), and in [0050] – “In the present embodiment, an example using Q learning in reinforcement learning is illustrated as a learning method for acquiring a control model.” – teaches a function of learning a first inference model by reinforcement learning (uses Q-learning in reinforcement learning as learning method for acquiring a control model));
a second action value function specifying unit that specifies a second action value function of the device on a basis of the current state and a second inference model that is not a parameter update target (Fuji, [0028] – “The control device 4 according to the present embodiment operates the control units 11 to 1n.sub.2 identifying the control models 31 to 3n.sub.1 by learning and the control units 21 to 2n.sub.2 having one or more existing control models 41 to 4n.sub.2, which are illustrated in FIG. 1 in parallel to output the action value and the action of each of the control units 11 to 1n.sub.2 and 21 to 2n.sub.2 to the action value selection unit 61,” – teaches a second action value specifying unit (control models 41) that specifies a second action value function of the device (outputs action value of control unit 21) on a basis of current state (state is input to control unit at [0027]), and a second inference model, and in [0058] – “One control unit 11a that updates a parameter of the control model 31a and a control unit 21a having one existing control model 41a are operated in parallel.” – teaches that the second inference model is not a parameter update target (updates first inference model and existing second inference model is not a parameter update target)); and
an action determination unit that determines a first action of the device on a basis of the first action value function and the second action value function (Fuji, [0028] – “The control device 4 according to the present embodiment operates the control units 11 to 1n.sub.2 identifying the control models 31 to 3n.sub.1 by learning and the control units 21 to 2n.sub.2 having one or more existing control models 41 to 4n.sub.2, which are illustrated in FIG. 1 in parallel to output the action value and the action of each of the control units 11 to 1n.sub.2 and 21 to 2n.sub.2 to the action value selection unit 61, outputs a control output value (action) selected by the action value selection unit 61 to at least one actuator 3 or the like mounted inside a machine, and updates the parameters of the control models 31 to 3n.sub.1 of the learning destination control units 11 to 1n.sub.1, based on observation data output from the sensor 2 and the selected action value.” – teaches an action determination unit (action selection unit 61) that determines a first action of the device on a basis of the first action value function and the second action value function (action value selection unit 61 selects one of the action values output by control models 31 and 41)).
wherein the action determination unit includes:
a third action value function specifying unit that specifies a third action value function obtained by synthesizing the first action value function and the second action value function; an action selection unit that selects the first action on a basis of the third action value function (Fuji, Eq. 2 and [0053] – “The existing control model Q.sub.A is synthesized (learned) to the control model Q.sub.Z of a synthesis target by the following method. For example, Q.sub.A can be synthesized with Q.sub.Z by establishing the following updating formula.”, and [0055] – “in Formula (2) and Formula (3), an action is selected by comparing the maximum action values of the synthesis destination control model Q.sub.Z and the existing control model Q.sub.A. At least one of the respective control models is required.” – teaches wherein the action determination unit includes a third action value function specifying unit that specifies a third action value function obtained by synthesizing the first action value function and the second action value function, and an action selection unit that selects the first action on a basis of the third action value function (Q.sub.A can be synthesized with Q.sub.Z, action is selected by a synthesis of the first and second action value functions, thus the third action value function is a synthesis of the first and second action value functions and the selection unit selects the first action on a basis of the third action value function));
wherein the third action value function specifying unit specifies, as the third action value function, a maximum function of [[a]] fourth value function and the second action value function (Fuji, [0030] – “The action value selection unit 61 outputs the selected action to the actuator 3 and the selected action and action value to the learning unit 71. For example, an action having the maximum action value may be selected by using a Max function as action value selection means output from a plurality of the control units 11 to 1n.sub.1 and 21 to 2n.sub.2 by the action value selection unit 61, and stochastic selection means such as ε-greedy selection or Boltzmann selection may be taken.” and [0055] – “In general Q learning, the Q learning is updated by selecting the action with the highest action value in a certain state, but in Formula (2) and Formula (3), an action is selected by comparing the maximum action values of the synthesis destination control model Q.sub.Z and the existing control model Q.sub.A. At least one of the respective control models is required.” – teaches specifying the third action value function as a maximum of a fourth value function and the second action value function (Q learning is updated by selecting action with highest action value in certain state, action is selected by comparing maximum action values of action value functions, thus teaching specifying a third action value function as a maximum function of a fourth and second action value function));
moving the device based on the first action (Fuji, [0040] – “In step S3, the action value selection unit 61 selects an action (a control value which is output to the actuator), based on the action value output from each control model, outputs the selected action and action value to the learning unit 71, and outputs the control value (selected action) to the actuator 3.” and in [0041] – “In step S4, the actuator 3 performs an operation according to the control value (operation command).” – teaches moving the device based on the first action (action value selection unit selects action based on action values output from each control model, outputs selected action to actuator, actuator performs operation according to selected action. Action and device may be moving a travel robot as in Fuji at [0047])).
Fuji fails to explicitly teach an action value function correction unit that specifies a fourth action value function obtained by deriving a correction value from at least one statistical value of the second action value function in the current state and adding the derived correction value to the first action value function.
However, analogous to the field of the claimed invention, Zhu teaches:
an action value function correction unit that specifies a fourth action value function obtained by deriving a correction value from the second action value function in the current state and adding the derived correction value to the first action value function (Zhu, Section 5 Paragraph 3 – “Specifically, our proposed Self-correcting Deep Q Network algorithm (ScDQN) equates the current and previous estimates of the action value function Qn(s 0 , a) and Qn−1(s 0 , a) in Eqn. (4) to the target network Q(s, a; θ −) and the online network Q(s, a; θ), respectively. In other words, we compute Eq. (5)” – teaches an action value function correction unit that specifies a fourth action value function (the result of Eq. (5)) obtained by deriving a correction value from the second action value function in the current state and adding the derived correction value to the first action value function (corrects target network based on online network, Eq (5) shows correction value (second term of Eq (5)) being added to the first action value function Qn(s,a))).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the action value function correction of Zhu to the action value functions and device of Fuji. Doing so would combine correlated estimators of the optimal action value into a corrected estimator, which would remove maximization bias and attain faster convergence speed (Zhu, Introduction).
The combination of Fuji and Zhu fails to explicitly teach deriving a correction value from at least one statistical value of the second action value function in the current state.
However, analogous to the field of the claimed invention, Mousavi teaches:
deriving a correction value from at least one statistical value of the second action value function in the current state and adding the derived correction value to the first action value function (Mousavi, Section 5 Paragraphs 3-4 – “Eq. (12) This is the set of possible Q-values for the pair of (sl, al) using the knowledge of the source tasks. These definitions are used to initialize the Q-values of the target task. We can use a statistical average operator to estimate a single value from the set-value CT(sl, al) as an initial value of Ql(sl, al). For example, we can use mean, median, or midrange operators. In this paper, we use the midrange operator, defined as follows: Eq. (13) where mathematical equation is an initial estimation of Ql(sl, al) and Eq. (14)” – teaches deriving a correction value from at least one statistical value (uses a statistical average operator (such as a midrange operator, midrange is used within the Specification of the claimed invention at [0251-0252] to derive a correction value from a statistical value) of the second action value function CT(sl, al) in current sate (sl)) and adds the derived correction value to the first action value function (derived statistical value is used as an initial value of Ql(sl, al), thus adding the derived statistical value to the first action value function)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the statistical values of an action value function of Mousavi to the action value functions, correction values, and system of Fuji and Zhu. Doing so would provide statistical measures for Q-values, where the statistical measures are used as a tool for reasoning about knowledge when multiple agents or Q-value functions are present (Mousavi, Section 6).
Claims 15 and 16 incorporate substantively all the limitations of claim 1 in a method and non-transitory computer-readable storage medium and are rejected on similar grounds as above.
Regarding claim 2, the combination of Fuji, Zhu, and Mousavi teaches the information processing device according to claim 1,
wherein the action determination unit selects, as a third action value function, any one of the first action value function and the second action value function, and determines the first action based on the selected third action value function (Fuji, Eq. 3 and [0055] – “In general Q learning, the Q learning is updated by selecting the action with the highest action value in a certain state, but in Formula (2) and Formula (3), an action is selected by comparing the maximum action values of the synthesis destination control model Q.sub.Z and the existing control model Q.sub.A. At least one of the respective control models is required.” – a third action value function which is any one of the first action value function and second action value function, and determines the first action on a basis of the selected third action value function (the determined first action is based on the third action value function which is the maximum of the first and second action value functions, action is selected based on the comparison)).
Regarding claim 3, the combination of Fuji, Zhu, and Mousavi teaches the information processing device according to claim 2,
wherein the action determination unit changes a first selection probability of selecting the first action value function as the third action value function and a second selection probability of selecting the second action value function as the third action value function according to a learning time of the first inference model (Fuji, Eq.4 and [0056] – “Furthermore, in order to reduce a probability that an existing model is selected even in a state where learning is sufficiently progressed, for example, an oblivion factor f may be defined as in Formula (4), and a factor f multiplied by the action value according to the progress of learning may be provided.” – teaches wherein the action determination unit changes a first selection probability of selecting the first action value function as the third action value function and a second selection probability of selecting the second action value function as the third action value function according to a learning time of the first inference model (oblivion factor f changes probabilities associated with selecting the first and second action value functions according to the progress of learning)),
decreases the first selection probability and increases the second selection probability as the learning time is shortened (Fuji, Eq. 4 and [0057] – “As for the factor f, a constant value may be subtracted from the oblivion factor for each trial, and a method of gradually making a selection probability of the existing control model approach zero may be adopted” – teaches decreasing the first selection probability and increasing the second selection probability as the learning time is shortened (f at shorter learning progress increases probability of existing (or second) control model being selected and thus decreases the first selection probability)), and
increases the first selection probability and decreases the second selection probability as the learning time is lengthened (Fuji, Eq. 4 and [0057] – “As for the factor f, a constant value may be subtracted from the oblivion factor for each trial, and a method of gradually making a selection probability of the existing control model approach zero may be adopted” – teaches increasing the first selection probability and decreasing the second selection probability as the learning time is lengthened (method of updating f after each trial such that selection probability of existing (second) control model may approach 0, thus increasing the first selection probability as learning time is lengthened)).
Regarding claim 5, the combination of Fuji, Zhu, and Mousavi teaches the information processing device according to claim [[4]] 1,
wherein the third action value function specifying unit specifies, as the third action value function, a maximum function of the first action value function and the second action value function (Fuji, Eq. 3 and [0055] – “In general Q learning, the Q learning is updated by selecting the action with the highest action value in a certain state, but in Formula (2) and Formula (3), an action is selected by comparing the maximum action values of the synthesis destination control model Q.sub.Z and the existing control model Q.sub.A. At least one of the respective control models is required.” – teaches wherein the third action value function specifying unit specifies a maximum function of the first action value function and the second action value function (action selected by comparing maximum action values of first and second action value functions)).
Regarding claim 7, the combination of Fuji, Zhu, and Mousavi teaches the information processing device according to claim [[6]] 1,
wherein the action value function correction unit specifies the fourth action value function obtained by correcting the first action value function (Zhu, Section 5 Paragraph 3 – “Specifically, our proposed Self-correcting Deep Q Network algorithm (ScDQN) equates the current and previous estimates of the action value function Qn(s 0 , a) and Qn−1(s 0 , a) in Eqn. (4) to the target network Q(s, a; θ −) and the online network Q(s, a; θ), respectively. In other words, we compute Eq. (5)” – teaches an action value function correction unit that specifies a fourth action value function (the result of Eq. (5)) obtained by correcting the first action value function on a basis of the second action value function (corrects target network based on online network))
The combination of Fuji and Zhu fails to explicitly teach an action value for an action represented by the first action value function becomes a value between a maximum value and a minimum value of action values for actions represented by the second action value function.
However, analogous to the field of the claimed invention, Mousavi teaches:
such that an action value for an action represented by the first action value function becomes a value between a maximum value and a minimum value of action values for actions represented by the second action value function (Mousavi, Section 5 Paragraphs 3-4 – “Eq. (12) This is the set of possible Q-values for the pair of (sl, al) using the knowledge of the source tasks. These definitions are used to initialize the Q-values of the target task. We can use a statistical average operator to estimate a single value from the set-value CT(sl, al) as an initial value of Ql(sl, al). For example, we can use mean, median, or midrange operators. In this paper, we use the midrange operator, defined as follows: Eq. (13) where mathematical equation is an initial estimation of Ql(sl, al) and Eq. (14)” – teaches an action value for an action represented by the first action value function becoming a value between a maximum and minimum value of action values for actions represented by the second action value function (takes midrange, which is the average of the minimum and maximum values for the action value function, to initialize Q-values of the next target task)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the action values between maximum and minimum action values of an action value function of Mousavi to further modify the action value functions, correction values, and system of Fuji, Zhu, and Mousavi. Doing so would combine the knowledge of different source tasks to be used by the target task (Mousavi, Section 3) and increase average reward while decreasing regret (Mousavi, Section 7.2).
Regarding claim 8, the combination of Fuji, Zhu, and Mousavi teaches the information processing device according to claim [[6]] 1,
such that a second selection probability of selecting the second action value function as the third action value function at start of learning of the first inference model becomes a predetermined selection probability (Fuji, Eq.4 and [0056] – “Furthermore, in order to reduce a probability that an existing model is selected even in a state where learning is sufficiently progressed, for example, an oblivion factor f may be defined as in Formula (4), and a factor f multiplied by the action value according to the progress of learning may be provided.” – teaches a predetermined selection probability (oblivion factor f) that represents a probability of selecting the second action value function as the third action value function at the start of learning (predetermined probability selected at any state of learning, even when learning has progressed or has just started)).
Fuji fails to explicitly teach wherein the action value function correction unit specifies the fourth action value function obtained by correcting the first action value function.
However, analogous to the field of the claimed invention, Zhu teaches:
wherein the action value function correction unit specifies the fourth action value function obtained by correcting the first action value function (Zhu, Section 5 Paragraph 3 – “Specifically, our proposed Self-correcting Deep Q Network algorithm (ScDQN) equates the current and previous estimates of the action value function Qn(s 0 , a) and Qn−1(s 0 , a) in Eqn. (4) to the target network Q(s, a; θ −) and the online network Q(s, a; θ), respectively. In other words, we compute Eq. (5)” – teaches an action value function correction unit that specifies a fourth action value function (the result of Eq. (5)) obtained by correcting the first action value function on a basis of the second action value function (corrects target network based on online network))
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the action value function correction of Zhu to further modify the action value functions and predetermined selection probability of Fuji, Zhu, and Mousavi. Doing so would combine correlated estimators of the optimal action value into a corrected estimator, which would remove maximization bias and attain faster convergence speed (Zhu, Introduction).
Regarding claim 11, the combination of Fuji, Zhu, and Mousavi teaches the information processing device according to claim [[6]] 1,
wherein the first action value function specifying unit learns the first inference model by reinforcement learning by using the current state, a reward in the current state, and the first action, and learns the first inference model by reinforcement learning by using the first action when the fourth action value function is used to specify the third action value function (Fuji, [0050] – “ In the present embodiment, an example using Q learning in reinforcement learning is illustrated as a learning method for acquiring a control model. The Q learning is a method of learning a value (action value) Q(s,a) for selecting an action a under a certain state value s obtained by processing the observation data from the sensor 2 by using the state acquisition unit 51.”, [0051] – “ A Q table according to the present embodiment holds the grid square of each maze, and a coordinate value represented by symbols 1 to 10 and A to P in the vertical and horizontal directions is set as the state value s. In addition, scores are allocated for each grid square (predefined by a designer), and this is searched as a reward value r. The control model 330 in eight directions is handled one by one in the vertical, horizontal, and diagonal directions as the action a. For the Q learning, state transition calculation is performed by using the following updating formula.”, and in [0055] – “In general Q learning, the Q learning is updated by selecting the action with the highest action value in a certain state, but in Formula (2) and Formula (3), an action is selected by comparing the maximum action values of the synthesis destination control model Q.sub.Z and the existing control model Q.sub.A. At least one of the respective control models is required.” – teaches wherein the first action value function specifying unit learns the first inference model by reinforcement learning using the current state, a reward in the current state, and the first action (uses Q learning in reinforcement learning to learn a control model using current state, action, and reward), and learns the first inference model by reinforcement learning by using the first action when the fourth action value function is used to specify the third action value function (selects action with highest action value in certain state between action value functions, thus learning by reinforcement learning using the action defined by the fourth action value function or the control models)).
Regarding claim 12, the combination of Fuji, Zhu, and Mousavi teaches the information processing device according to claim 1,
wherein the second inference model performs learning in advance based on the current state and data of an action of the device based on a first rule (Fuji, [0053] – “In the present embodiment, the existing control model is specifically set as a Q table (Q.sub.A) in which a convergence condition is obtained when continuously reaching the goal 10 times on the shortest path in a shortest path search problem movable in the vertical and horizontal four directions. In addition, the control model of a synthesis destination (the control model for updating parameters) is specifically set as a Q table Q.sub.Z in which a convergence condition is obtained when continuously reaching the goal 10 times on the shortest path in a condition movable in eight directions to which the diagonal four directions are added.” – teaches wherein the second inference model (existing control model) performs learning in advance on a basis of the current state and data of an action of the device based on a first rule (performs Q-learning in advance, Q-learning utilizes state and action to perform learning as in [0050] and is based on a first rule, which is the convergence condition that is obtained by shortest path search problem)).
Regarding claim 13, the combination of Fuji, Zhu, and Mousavi teaches the information processing device according to claim 1,
wherein the one or more hardware processors are configured to further function as a plurality of second action value function specifying units (Fuji, [0028] – “The control device 4 according to the present embodiment operates the control units 11 to 1n.sub.2 identifying the control models 31 to 3n.sub.1 by learning and the control units 21 to 2n.sub.2 having one or more existing control models 41 to 4n.sub.2,” – teaches wherein the one or more hardware processors (processors as in [0035]) are configured to further function as plurality of second action value function specifying units (one or more existing control models 41 to 4n.sub.2)).
Regarding claim 14, the combination of Fuji, Zhu, and Mousavi teaches the information processing device according to claim 1, wherein the one or more hardware processors are configured to further function as:
a display control unit that displays, on a display unit, information indicating at least one of a progress of learning of the first inference model, a selection probability of the action determination unit selecting at least one of the first action value function and the second action value function, a number of times of selection of the action determination unit selecting at least one of the first action value function and the second action value function, whether the first action is an action that maximizes an action value represented by one of the first action value function and the second action value function, a selection probability that an action that maximizes an action value represented by the second action value function is selected as the first action, and a transition of the selection probability (Fuji, [0046] – “The selection monitoring unit 91 monitors a situation of learning by displaying the action value and the action selected by the action value selection unit 61 and the number of times of each of the selected control units 11 to 1n.sub.1 and 21 to 2n.sub.2, on, for example, a visualization tool such as a display connected to the outside of the control device 4, or by taking a log and describing in text. For example, it can be used as information for changing a connection relationship with the learning units 71 of the control models 31 to 3n.sub.1 of a learning destination and the existing control models 41 to 4n.sub.2, based on the monitoring results.” – teaches a display control unit that displays information indicating at least one of a progress of learning of the first inference model (monitors situation of learning), a number of times of selection of the action determination unit selecting at least one of the first action value function and the second action value function (number of times of each of the selected control units), and whether the first action is an action that maximizes an action value represented by one of the first action value function and the second action value function (selected action would be an action that maximizes an action value of either action value functions)).
Claim(s) 9-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fuji, Zhu, and Mousavi as applied to claims 1 and 15-16 above, and further in view of Chen et al. (US Pub. No. 2017/0286860, published Oct. 2017, hereinafter “Chen”).
Regarding claim 9, the combination of Fuji, Zhu, and Mousavi teaches the information processing device according to claim 8,
wherein the action value function correction unit specifies the fourth action value function obtained by correcting the first action value function (Zhu, Section 5 Paragraph 3 – “Specifically, our proposed Self-correcting Deep Q Network algorithm (ScDQN) equates the current and previous estimates of the action value function Qn(s 0 , a) and Qn−1(s 0 , a) in Eqn. (4) to the target network Q(s, a; θ −) and the online network Q(s, a; θ), respectively. In other words, we compute Eq. (5)” – teaches an action value function correction unit that specifies a fourth action value function (the result of Eq. (5)) obtained by correcting the first action value function on a basis of the second action value function (corrects target network based on online network))
The combination of Fuji, Zhu, and Mousavi fails to explicitly teach a selection probability input by a user.
However, analogous to the field of the claimed invention, Chen teaches:
so as to become a selection probability input by a user (Chen, [0046] – “In some examples, the control program 142 can be configured to receive inputs, e.g., via a keyboard, transmit corresponding queries to a computing device 102, receive responses from computing device 102, and present the responses, e.g., via a display. In some examples, training and operation of computational models are carried out on computing device(s) 102. In some examples, training and operation are carried out on a computing device 104. In some of these examples, the control program 142 can be configured to receive inputs, train and/or operate computational model(s) 128 using instructions of representation engine 120 and action engine 122 based at least in part on those inputs to determine an action, and implement the determined action.” and in [0049] – “In some examples, computing device 104 can include a user interface 146. For example, computing device 104(3) can provide user interface 146 to control and/or otherwise interact with cluster 106 and/or computing devices 102 therein. For example, processing unit(s) 136 can receive inputs of user actions via user interface 146 and transmit corresponding data via communications interface 144 to computing device(s) 102.” – teaches a computing device receiving inputs provided by a user to determine an action based at least in part on the user inputs (user interface enables user to control and/or otherwise interact with computing device)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the user interface of Chen to the selection probabilities and action value functions of Fuji, Zhu, and Mousavi in order to utilize a user input selection probability. Doing so would permit users to control and interact with the computing devices (Chen, [0049]) and adapt the system according to user interactions to obtain effective results (Chen, [0017]).
Regarding claim 10, the combination of Fuji, Zhu, and Mousavi teaches the information processing device according to claim 7,
wherein the action value function correction unit specifies the fourth action value function obtained by correcting the first action value function (Zhu, Section 5 Paragraph 3 – “Specifically, our proposed Self-correcting Deep Q Network algorithm (ScDQN) equates the current and previous estimates of the action value function Qn(s 0 , a) and Qn−1(s 0 , a) in Eqn. (4) to the target network Q(s, a; θ −) and the online network Q(s, a; θ), respectively. In other words, we compute Eq. (5)” – teaches an action value function correction unit that specifies a fourth action value function (the result of Eq. (5)) obtained by correcting the first action value function on a basis of the second action value function (corrects target network based on online network))
such that an action value for an action represented by the first action value function becomes a value that is between a maximum value and a minimum value of action values for actions represented by the second action value function (Mousavi, Section 5 Paragraphs 3-4 – “Eq. (12) This is the set of possible Q-values for the pair of (sl, al) using the knowledge of the source tasks. These definitions are used to initialize the Q-values of the target task. We can use a statistical average operator to estimate a single value from the set-value CT(sl, al) as an initial value of Ql(sl, al). For example, we can use mean, median, or midrange operators. In this paper, we use the midrange operator, defined as follows: Eq. (13) where mathematical equation is an initial estimation of Ql(sl, al) and Eq. (14)” – teaches an action value for an action represented by the first action value function becoming a value between a maximum and minimum value of action values for actions represented by the second action value function (takes midrange, which is the average of the minimum and maximum values for the action value function, to initialize Q-values of the next target task)).
The combination of Fuji, Zhu, and Mousavi fails to explicitly teach a value that is input by a user.
However, analogous to the field of the claimed invention, Chen teaches:
a value that is input by a user (Chen, [0046] – “In some examples, the control program 142 can be configured to receive inputs, e.g., via a keyboard, transmit corresponding queries to a computing device 102, receive responses from computing device 102, and present the responses, e.g., via a display. In some examples, training and operation of computational models are carried out on computing device(s) 102. In some examples, training and operation are carried out on a computing device 104. In some of these examples, the control program 142 can be configured to receive inputs, train and/or operate computational model(s) 128 using instructions of representation engine 120 and action engine 122 based at least in part on those inputs to determine an action, and implement the determined action.” and in [0049] – “In some examples, computing device 104 can include a user interface 146. For example, computing device 104(3) can provide user interface 146 to control and/or otherwise interact with cluster 106 and/or computing devices 102 therein. For example, processing unit(s) 136 can receive inputs of user actions via user interface 146 and transmit corresponding data via communications interface 144 to computing device(s) 102.” – teaches a computing device receiving inputs provided by a user to determine an action based at least in part on the user inputs (user interface enables user to control and/or otherwise interact with computing device)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the user interface of Chen to the action value function correction and values between a minimum and maximum of Fuji, Zhu, and Mousavi in order to enable users to input values between a minimum and maximum based on the second action value function. Doing so would permit users to control and interact with the computing devices (Chen, [0049]) and adapt the system according to user interactions to obtain effective results (Chen, [0017]).
Response to Arguments
Applicant’s arguments, see pp. 1-3 of Remarks, filed 22 April 2026, with respect to claims 1 and 15-16 have been fully considered and are persuasive. The 35 U.S.C. 101 rejection of claims 1-3, 5, and 7-16 has been withdrawn. The claims, as amended, now recite technical steps for correcting, or adjusting, a first action value function based on at least one statistical value derived from a second action value function, where the second action value function is based on the current state and a second inference model that is not a parameter update target. As noted in Remarks at pp. 2, paragraph 1 – “This sequence defines a specific, structured processing pipeline in which outputs of a non-updating inference model are used to dynamically calibrate a reinforcement-learned model prior to action selection” and in pp. 2, paragraph 2 – “By deriving a correction value from statistical properties of the second action value function in the current state and applying that correction to the first action value function, the claimed subject matter ensures that the two value functions are aligned and comparable. This enables stable and safe control of the device, particularly during early stages of learning when the reinforcement-learned model is not yet reliable”. Additionally, the amended claims reflect improvements described in the Specification at [0173] and [0243-0246] which regards utilizing the second action value function of the second inference model at the initial learning stage in which the reinforcement learning of the first inference model is insufficient, and adjusting reward settings without manual adjustment and/or trial and error. Accordingly, the claims as amended provide a technical improvement in the functioning of the information processing device and in the control of the device itself.
Applicant’s arguments, see pp. 3-4 of Remarks, filed 22 April 2026, with respect to the rejection(s) of claim(s) 1-5 and 12-16 under 35 U.S.C. 102(a)(1) have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made under 35 U.S.C. 103 over Fuji in view of Zhu, and further in view of Mousavi. Fuji teaches the amended limitations of claim 1 regarding “wherein the action determination unit includes: a third action value function specifying unit…”, “an action selection unit…”, “wherein the third action value function specifying unit specifies… a maximum function…”, and “wherein the device moves based on the first action”. Zhu teaches the amended limitations of claim 1 regarding “an action value function correction unit that specifies a fourth action value function obtained by deriving a correction value from the second action value function in the current state and…”. Mousavi teaches the amended limitation of claim 1 regarding “deriving a correction value from at least one statistical value of the second action value function in the current state and…”.
Applicant states on pp. 4 of Remarks that an understanding was reached during the applicant-initiated interview on 13 April 2026. Examiner points to Examiner Interview Summary Record, which states that the amendment may overcome prior rejection over Fuji in view of Zhu and that “No agreement was reached”. Discussion during the interview was limited to specific references, and discussion did not conclude that all the cited references do not appear to disclose or suggest the amended independent claims. Upon further consideration, Examiner determines that Mousavi teaches the amended limitations of independent claims 1 and 15-16 regarding “deriving a correction value from at least one statistical value of the second action value function in the current state and…”.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Lan et al. (NPL: Maxmin Q-Learning: Controlling the Estimation Bias of Q-Learning, published Aug. 2021) teaches methods for Q-Learning that include a parameter to flexibly control bias. Teaches deriving an update value, or correction value, for a first action value function based on a statistical value, such as maximum or minimum, of a second action value function. Teaches that expected estimation bias is based on the number of action-value functions in Maxmin Q-Learning.
Kostrikov et al. (NPL: Offline Reinforcement Learning with Implicit Q-Learning, published Oct. 2021) teaches methods for offline reinforcement learning by approximating a policy improvement step implicitly by treating the state value function as a random variable. Teaches deriving a statistical value of the state value function to modify policy evaluation objectives.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LOUIS C NYE whose telephone number is 571-272-0636. The examiner can normally be reached Monday - Friday 9:00AM - 5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MATT ELL can be reached at 571-270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LOUIS CHRISTOPHER NYE/Examiner, Art Unit 2141
/DANIEL T PELLETT/Primary Examiner, Art Unit 2121
Prosecution Insights