Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1 - 21 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract ideas without significantly more.
Regarding claim 1: Step 2A Prong 1: Generating, by the computer system, an action including information for modifying an aggregation function employed by the parameter server during the training run; (Mental process, simply generating an action can be done in the mind. It doesn’t actually modify the function.)
Step 2A Prong 2:
Receiving, by a computer system implementing a reinforcement learning (RL) agent, a state from a distributed learning or federated learning (DL/FL) system, the DL/FL system including a parameter server and a plurality of clients, the state including information regarding one or more runtime properties of the DL/FL system with respect to a training run being executed by the DL/FL system for training an artificial neural network (ANN); (Adding insignificant extra-solution activity to the judicial exception MPEP 2106.05(g) - Examiner’s note: merely receiving data of a state)
Receiving, by the computer system, a reward from the DL/FL system, the reward including one or more values that are proportional to one or more metrics of the training run that the RL agent is designed to optimize; (Adding insignificant extra-solution activity to the judicial exception MPEP 2106.05(g) - Examiner’s note: merely receiving data of a reward)
And transmitting, by the computer system, the action to the parameter server (Adding insignificant extra-solution activity to the judicial exception MPEP 2106.05(g) - Examiner’s note: merely transmitting data of an action)
Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Receiving, by a computer system implementing a reinforcement learning (RL) agent, a state from a distributed learning or federated learning (DL/FL) system, the DL/FL system including a parameter server and a plurality of clients, the state including information regarding one or more runtime properties of the DL/FL system with respect to a training run being executed by the DL/FL system for training an artificial neural network (ANN); (Receiving state data is well understood and routine conventional activity that adds nothing meaningful to the exception, see MPEP 2106.05(d). Reinforcement learning requires state data.)
Receiving, by the computer system, a reward from the DL/FL system, the reward including one or more values that are proportional to one or more metrics of the training run that the RL agent is designed to optimize; (Receiving reward data is well understood and routine conventional activity that adds nothing meaningful to the exception, see MPEP 2106.05(d). Reinforcement learning requires reward data.)
And transmitting, by the computer system, the action to the parameter server (Transmitting data is well understood and routine conventional activity, see MPEP 2106.05(g))
The additional elements as disclosed above alone or in combination do not integrate the judicial exception into a practical application as they are mere insignificant extra solution activity in combinations.
The claim is ineligible.
Regarding claim 2:
The method of claim 1 wherein the information included in the action modifies the aggregation function in a manner that maximizes future rewards that are expected to be received from the DL/FL system in view of the state.
This provides a further description of the method within the abstract ideas, as discussed with regards to claim 1. The limitations of claim 2, under broadest reasonable interpretation, does not include any new abstract ideas.
The claim recites additional elements of adding insignificant extra-solution activity to the judicial exception MPEP 2106.05(g) – (Examiner’s note: merely mentions what the action information is)
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions when taken alone or in combination with previous claims. The claimed additional element is just insignificant extra-solution activity. Mentioning what the information includes, when taken individually or in combination with previous additional elements, is well understood and routine conventional activity that adds nothing meaningful to the exceptions. Therefore, the judicial exceptions
are not integrated into a practical application.
Claim 2 is ineligible
Regarding claim 3:
The method of claim 1 wherein upon receiving the action, the parameter server modifies the aggregation function in accordance with the information in the action.
This provides a further description of the method within the abstract ideas, as discussed with regards to claim 1. The limitations of claim 3, under broadest reasonable interpretation, does not include any new abstract ideas.
Claim 3 recites additional elements of adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – (Examiner’s note: high level recitation of applying the exception Generating, by the computer system, an action including information for modifying an aggregation function employed by the parameter server during the training run. It applies the system’s action to modify an aggregation function.)
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions when taken alone or in combination with previous claims. The claimed additional element is just an application of the judicial exceptions.
Claim 3 is ineligible.
Regarding claim 4:
The method of claim 1 wherein the state comprises information pertaining to gradients computed by a subset of the plurality of clients for one or more rounds of the training run.
This provides a further description of the method within the abstract ideas, as discussed with regards to claim 1. The limitations of claim 4, under broadest reasonable interpretation, does not include any new abstract ideas.
The claim recites additional elements of adding insignificant extra-solution activity to the judicial exception MPEP 2106.05(g) – (Examiner’s note: merely mentions what the state data is)
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions when taken alone or in combination with previous claims. The claimed additional element is just insignificant extra-solution activity. Mentioning what the data is, when taken individually or in combination with previous additional elements, is well understood and routine conventional activity that adds nothing meaningful to the exceptions. Therefore, the judicial exceptions
are not integrated into a practical application.
Claim 4 is ineligible
Regarding claim 5:
The method of claim 1 wherein the state comprises aggregated statistics for one or more rounds of the training run that are collected by the parameter server.
This provides a further description of the method within the abstract ideas, as discussed with regards to claim 1. The limitations of claim 5, under broadest reasonable interpretation, does not include any new abstract ideas.
The claim recites additional elements of adding insignificant extra-solution activity to the judicial exception MPEP 2106.05(g) – (Examiner’s note: merely mentions what the state data is)
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions when taken alone or in combination with previous claims. The claimed additional element is just insignificant extra-solution activity. Mentioning what the data is, when taken individually or in combination with previous additional elements, is well understood and routine conventional activity that adds nothing meaningful to the exceptions. Therefore, the judicial exceptions
are not integrated into a practical application.
Claim 5 is ineligible
Regarding claim 6:
The method of claim 1 wherein the reward comprises a loss improvement value that is proportional to an improvement in loss for the ANN from a previous round of the training run to a current round of the training run.
This provides a further description of the method within the abstract ideas, as discussed with regards to claim 1. The limitations of claim 6, under broadest reasonable interpretation, does not include any new abstract ideas.
The claim recites additional elements of adding insignificant extra-solution activity to the judicial exception MPEP 2106.05(g) – (Examiner’s note: merely mentions what the reward data is)
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions when taken alone or in combination with previous claims. The claimed additional element is just insignificant extra-solution activity. Mentioning what the data is, when taken individually or in combination with previous additional elements, is well understood and routine conventional activity that adds nothing meaningful to the exceptions. Therefore, the judicial exceptions
are not integrated into a practical application.
Claim 6 is ineligible
Regarding claim 7:
The method of claim 1 wherein the information included in the action comprises modified values for one or more coefficients or weights of the aggregation function.
This provides a further description of the method within the abstract ideas, as discussed with regards to claim 1. The limitations of claim 7, under broadest reasonable interpretation, does not include any new abstract ideas.
The claim recites additional elements of adding insignificant extra-solution activity to the judicial exception MPEP 2106.05(g) – (Examiner’s note: merely mentions what the action data is)
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions when taken alone or in combination with previous claims. The claimed additional element is just insignificant extra-solution activity. Mentioning what the data is, when taken individually or in combination with previous additional elements, is well understood and routine conventional activity that adds nothing meaningful to the exceptions. Therefore, the judicial exceptions
are not integrated into a practical application.
Claim 7 is ineligible
Regarding claims 8 – 14:
Claims 8 – 14 refer to a non-transitory computer readable medium which does the same activities as listed in the limitations of claims 1 - 7. Therefore, they receive the same judicial exceptions and additional elements as claims 1 – 7 where the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
Claims 8 – 14 are ineligible.
Regarding claims 15 – 21:
Claims 15 – 21 refer to a computer system which does the same activities as listed in the limitations of claims 1 - 7. Therefore, they receive the same judicial exceptions and additional elements as claims 1 – 7 where the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
Claims 15 – 21 are ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 - 21 are rejected under 35 U.S.C. 103 as being unpatentable over by Han et al (US PG PUB 20210166158 A1) in view of Wu et al (US 20230016827 A1).
Claim 1 recites:
A method comprising:
receiving, by a computer system implementing a reinforcement learning (RL) agent, a state from a distributed learning or federated learning (DL/FL) system, the DL/FL system including a parameter server and a plurality of clients, the state including information regarding one or more runtime properties of the DL/FL system with respect to a training run being executed by the DL/FL system for training an artificial neural network (ANN);
Han Fig 1 discloses a full FL/DL system consisting of a plurality of devices (300), device controllers (100) as RL agents for each device, and one federated reinforcement learning managing server (200). The system's state data, as disclosed by Han Fig 2 reinforcement learning data, includes current device state data, device control information, predicted device's next state data, and current rewards. [0095] and Fig 4 disclose a reinforcement learning data construction unit (140) that receives next state information from other state data coming from the DL/FL system's devices. Fig 2 also discloses reinforcement learning done by a learning model. [0115] discloses the learning model can be an artificial neural network (ANN). [0020] discloses the reinforcement learning reports a gradient and a learning parameter which are runtime properties of the DL/FL system used for eventually training the ANN. These also fall within the system's state data. Furthermore, Fig 3 discloses the managing server has a gradient sharing process where an average gradient is computed in an aggregation function using gradients as input. This also falls within the system's state data.
receiving, by the computer system, a reward from the DL/FL system, the reward including one or more values that are proportional to one or more metrics of the training run that the RL agent is designed to optimize;
Han [0099] and [0050] disclose the system's reward is a compensation value that's proportional to whether the device's next state information falls within a preset threshold. A positive compensation value if within the threshold and a negative compensation value if outside the threshold. [0093] discloses configuring data based on receiving state info and a generated reward made by the reward generation unit (130) in Fig 4.
and transmitting, by the computer system, the action to the parameter server.
Han teaches the system's action is sharing the learning parameter if a training run is completed. Fig. 5 discloses a learning parameter receiving unit (230) within the managing server. [0121] discloses the managing server containing a unit 230 that receives a learning parameter from the device controllers. [0105] also discloses the device controller's federated reinforcement learning unit (170) contains a learning parameter reporting unit (173) that reports the parameter to the managing server.
Han does not teach “generating, by the computer system, an action for modifying an aggregation function employed by the parameter server during the training run;”
Wu does teach “generating, by the computer system, an action for modifying an aggregation function employed by the parameter server during the training run;” (Wu [0037] teaches generating an action by making an offloading decision which specifies what layers of a Neural Network (NN) partition to use in training. This changes each instance of a neural network. [0024] and [0021] teach an aggregation function used by one server controller that uses neural network instances as input. The action provides information meant to change a neural network's composition which is modifying the aggregation function.)
It would have been obvious for a person of ordinary skill in the art to combine Wu’s teaching of generating an action for making a global model of a neural network on the parameter server that could be copied to its individual devices, which would’ve led to a less computationally expensive training run than Han’s action.
Claim 2 recites:
The method of claim 1 wherein the information included in the action modifies the aggregation function in a manner that maximizes future rewards that are expected to be received from the DL/FL system in view of the state.
Claim 2 invokes the same rejection as claim 1. Using the rationale from claim 1, the action refers to transferring a learning parameter to each device controller. Han [0114] discloses the learning parameter of a model with a completed training run is used to update models with unfinished training runs. This will result in new gradients, which changes the inputs of the managing server’s aggregation function as disclosed by [0090]. Which in turn maximizes future rewards by using learning parameters that result in the device's next state being within the predicted threshold.
Claim 3 recites:
The method of claim 1 wherein upon receiving the action, the parameter server modifies the aggregation function in accordance with the information in the action.
Claim 3 invokes the same rejection as claim 1. Using the same rationale as claim 2, the learning parameter modifies the aggregation function by changing its gradient inputs. Han Fig 5 and [0121] discloses the parameter server's learning parameter providing unit (240) is what shares the learning parameters back to the device controllers.
Claim 4 recites:
The method of claim 1 wherein the state comprises information pertaining to gradients computed by a subset of the plurality of clients for one or more rounds of the training run.
Claim 4 invokes the same rejection as claim 1. Using the same rationale as claim 1, Han Fig 3 discloses a gradient sharing process that uses gradients from device controllers and computes an average gradient in the managing server. These values are within the system's state information.
Claim 5 recites:
The method of claim 1 wherein the state comprises aggregated statistics for one or more rounds of the training run that are collected by the parameter server.
Claim 5 invokes the same rejection as claim 1. Using the same rationale as claim 1, the system's state includes gradients acquired from one round of the training run. Han [0020] discloses the parameter server collects the gradients and aggregates them by finding their average.
Claim 6 recites:
The method of claim 1 wherein the reward comprises a loss improvement value that is proportional to an improvement in loss for the ANN from a previous round of the training run to a current round of the training run.
Claim 6 invokes the same rejection as claim 1. Using the rationale from claim 1, the reward is a compensation value. Han [0049] discloses the compensation value is applied to the device's next state data. During the next round of the training data, the next state data becomes current state data. The system uses that state data from a previous training round to create a new reward for the current round. [0050] discloses that the associated reward can be ratioed to the threshold range where next state further outside the preset threshold would result in a greater negative reward. This compensation value acts as the loss improvement value that is proportional to the device's next state data falling within a threshold. [0066], [0070], and Fig 2 discloses the device's next state data is a result of the ANN or learning model generating information to control a device. Therefore, the loss improvement value is proportional to the ANN's improvement in loss via the device’s next state data.
Claim 7 recites:
The method of claim 1 wherein the information included in the action comprises modified values for one or more coefficients or weights of the aggregation function.
Claim 7 invokes the same rejection as claim 1. Han also [0090] discloses the aggregation formula that uses gradients from devices. [0091] discloses the gradient formulas updated with the learning parameters. The aggregation function changes over training runs as the input gradients change due to the learning parameter.
Claims 8 – 14 are machine claims which are rejected under 35 U.S.C. 103 as being unpatentable over Han et al (US PG PUB 20210166158 A1) as applied to the limitations of claims 1 – 7 above, and further in view of Wu et al (US 20230016827 A1).
Claims 15 – 21 are system claims which are rejected under 35 U.S.C. 103 as being unpatentable over Han et al (US PG PUB 20210166158 A1) as applied to the limitations of claims 1 - 7 above, and further in view of Wu et al (US 20230016827 A1).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AJAY J KANJOOR whose telephone number is (571)270-0965. The examiner can normally be reached Monday-Friday 8am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AJAY J KANJOOR/Examiner, Art Unit 2142
/Mariela Reyes/Supervisory Patent Examiner, Art Unit 2142
Prosecution Insights