DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 06/21/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of
matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the
conditions and requirements of this title.
Claims 1-8 and 19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without reciting additional elements that amount to significantly more.
Regarding Claim 1, the claims recite a method for adjusting one or more operational parameters for a first cell of a communication network based on reinforcement learning, comprising: “determining a plurality of reward metric values based on measurements representative of conditions in the first cell and in one or more neighbor cells of the first cell at a corresponding plurality of time instances; determining a plurality of reward values based on differences between reward metric values at successive time instances; associating each of the reward values with a corresponding previous action that changed the one or more operational parameters; and selecting the previous action associated with a highest reward value as an action to change the one or more operational parameters for the first cell.” Therefore, claim 1 is a process (or method) claim. (see MPEP §2106, III. Summary of Analysis and Flowchart, in “Subject Matter Eligibility Test for Products and Processes” Flowchart, Step 1 [Yes] and so on as following).
Step 1: Statutory Category
The claim is directed to a method, which is one of the four statutory categories (process, machine, manufacture, or composition of matter) under 35 U.S.C. § 101.
Step 2A – Prong One: Is the claim directed to a judicial exception (law of nature, natural phenomenon, or abstract idea)?
➔ Yes. The claimed method recites an abstract idea- specifically, a mathematical concept and data analysis. The limitations, under their broadest reasonable interpretation, describe an algorithmic process for adjusting parameters using reinforcement learning (a mathematical concept and data analysis), which is an abstract idea.
Step 2A – Prong Two: Does the claim recite additional elements that integrate the exception into a practical application?
➔ No. While the claim mentions that adjusted operational parameters for a first cell of a communication network are derived based on “reinforcement learning, RL,” the claims do not recite any specific hardware, apparatus, or improvement to computer technology. The phrase “computer-implemented” is not enough, and there is no specified controller or memory within the claim for implementing the limitations of method Claim 1. Additionally, Claim 1 does not specify how the method is tied to a particular machine that is integral to the claim or how the method improves the functioning of a computer or network in a technical field.
Step 2B: Do the additional elements amount to significantly more than the judicial exception?
➔ No. Under Step2B, the claim is analyzed to determine whether it includes an inventive concept that transforms the abstract idea into patent-eligible subject matter. There is no unconventional machine and no new arrangement of components that would be considered “significantly more”.
Without reciting a specific hardware component or a particular machine, the method claims are more likely to be viewed as an attempt to patent an abstract idea. Therefore, Step 2B is not met.
In summary, Claim 1 is rejected under 35 U.S.C. § 101 because the claimed invention is directed to a judicial exception (i.e., an abstract idea in the form of mathematical concept, data analysis) and does not recite significantly more than the exception. Examiner recommends amending the claim to explicitly recite that the steps are performed “by a processor” or “by a computing device”, for example, communicatively coupled to non-transitory memory, or otherwise tie the method to a specific machine or technical environment.
The dependents Claims 2-8 are also rejected under 35 U.S.C. 101 as explained in Claim 1 above.
Regarding Claim 19, the body of the Claim 19 recites similar limitations as in Claim 1 and is also rejected under 35 U.S.C. 101 as explained in Claim 1 above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-15, 17, 19, 22 are rejected under 35 U.S.C. 103 as being unpatentable over Tan et al., US 2019 0014488 A1 (hereinafter “Tan”) in view of Chen et al., US 2021 0241090 A1 (hereinafter “Chen”).
Regarding Claim 1, Tan teaches a computer-implemented method for adjusting one or more operational parameters for a first cell of a communication network based on reinforcement learning, RL, the method comprising:
determining a plurality of reward metric values based on measurements representative of conditions in the first cell and in one or more neighbor cells of the first cell at a corresponding plurality of time instances (Page 11, Paragraph 86, Fig. 12, adjusting cell parameters of a plurality of cells in a wireless network using a DRL technique; Paragraph 87, generation of experience tuples for training the DRL-NN, time instances t; Paragraph 88, initial state of cell, including information of the cell of interest, relationship information with neighboring cells, e.g. RSRP, RSRQ; Tan discloses a cost function in terms of a performance or optimization goal, wherein the cost function may be defined for coverage and interference, coefficients to reflect weight of coverage to and interference to cost function, for example, Page 9, Paragraph 70); determining a plurality of reward values based on differences between reward metric values at successive time instances (Page 12, Paragraph 89, calculated reward value); associating each of the reward values with a corresponding previous action that changed the one or more operational parameters (Pages 11-12, Paragraphs 87 and 89, previous action, action that moves cell from first state to second state, operational parameter adjustment); and selecting the previous action associated with a highest reward value as an action to change the one or more operational parameters for the first cell (Pages 8-10, Paragraphs 67-71, selection of reward according to highest reward; Paragraph 76, determining whether action is acceptable at global level based on global reward associated with action, according to various criteria; Page 12, Paragraph 89, selection of action, reward indicates whether action for a cell is on the right track for target improvement).
Tan fails to fully teach the limitation:
determining a plurality of reward metric values based on measurements representative of conditions in the first cell and in one or more neighbor cells of the first cell at a corresponding plurality of time instances (Tan fails to teach determining a plurality of reward metric values at a corresponding plurality of time instances, further defined in Claims 8 and 9, which are in part determined according to an average of the good traffic states in the neighbor cells at time instance t, as well as the average of the congestion rates in the neighbor cells at time instance t. Both of the aforementioned values are weighted averages, with each neighbor cell’s good traffic rate and congestion rate being weighted by a degree of overlap between the first cell and the neighbor cell (Tan fails to explicitly teach weighting by a degree of overlap between a first cell and the neighbor cell).
However, Chen further teaches the limitation: determining a plurality of reward metric values based on measurements representative of conditions in the first cell and in one or more neighbor cells of the first cell at a corresponding plurality of time instances (Chen, Page 2, Paragraphs 16-17, weighted average, RLA may select setting based on weighted average; Pages 7-8, Paragraph 52, weighted majority arbitration may be applied among setting selections from different sub-agents for a parameter having a discrete parameter space).
Although Tan addresses the remaining limitations of claim 1, Chen teaches a computer-implemented method for adjusting one or more operational parameters for a first cell of a communication network based on reinforcement learning, RL, the method comprising: determining a plurality of reward metric values based on measurements representative of conditions in the first cell and in one or more neighbor cells of the first cell at a corresponding plurality of time instances (Chen, Page 8, Paragraph 54, Fig. 3, example use cases that may be associated with different sub-agents of a reinforcement learning [RL] agent, respective state information, reward information, and actions that may relate to different use cases; Pages 8-10, Paragraphs 55-67, Fig. 4, method for determining settings for RAN parameters via reinforcement learning agent); determining a plurality of reward values based on differences between reward metric values at successive time instances (Chen, Pages 2-3, Paragraphs 17-23, determination of reward value at given time, Q-learning model); associating each of the reward values with a corresponding previous action that changed the one or more operational parameters (Chen, Pages 2-3, Paragraphs 17-23); and selecting the previous action associated with a highest reward value as an action to change the one or more operational parameters for the first cell (Chen, Page 5, Paragraphs 38-40, selection of different actions, with associated rewards, with respect to a given state).
Chen and Tan are considered to be analogous to the claimed invention because they are in the same field of reinforcement learning models as well as supervisory, monitoring, and testing arrangements for optimizing operational conditions. It would have been obvious before the effective filing date of the claimed invention a person having ordinary skill in the art to which the claimed invention pertains to have modified Tan which clearly comprises adjusting cell parameters of a plurality of cells in a wireless network using a DRL technique, wherein a performance or optimization goal is defined with consideration to coverage and interference, wherein coefficients reflecting weight of coverage and weight of interference are used as variables in a cost function used for assisting in determination of the performance or optimization goal (Tan, Page 9, Paragraph 70), to utilize the weighted averages distinguishing discrete parameter spaces, as taught by Chen (Chen, Page 2, Paragraphs 16-17, weighted average, RLA may select setting based on weighted average; Pages 7-8, Paragraph 52, weighted majority arbitration may be applied among setting selections from different sub-agents for a parameter having a discrete parameter space) because determining a plurality of reward metric values based on measurements of conditions in the first cell and in one or more neighbor cells at a corresponding plurality of time instances is a routine optimization in adjustment of cell parameters of a plurality of cells in a wireless network using a DRL technique. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tan to incorporate the teachings Chen for determining a plurality of reward metric values based on measurements representative of conditions in the first cell and in one or more neighbor cells of the first cell at a corresponding plurality of time instances.
Regarding Claim 2, Tan in view of Chen teaches the method of claim 1, wherein the method is performed by an RL agent associated with the first cell (Tan, Page 4, Paragraph 41, Fig. 3, reinforcement learning [RL] system, RL agent selects an action to perform on the environment, action may refer to an adjustment of a cell parameter of a cell associated with a base station; Page 10, Paragraph 76, DRL agent, selects an action for applying to the cell; Chen, Page 4, Paragraph 35, reinforcement learning agent [RLA], plurality of sub-agents for determining settings of parameters of a RAN, obtaining performance indicators, state information, reward information from cell site(s) in access network to process state information, reward information in order to determine settings).
Regarding Claim 3, Tan in view of Chen teaches the method of claim 1, wherein the conditions include the following: downlink, DL, coverage; DL quality; and congestion (Tan, Pages 3-4, Paragraph 36, key performance indicators [KPIs], Paragraph 39, optimization or adjustment of KPI[s] of a wireless network; Page 8, Paragraph 65, state of cell, KPIs (e.g. throughput and cell load); Page 12, Paragraph 89, RSRP value should be greater than or equal to a RSRP threshold, RSRQ greater than or equal to RSRQ threshold; Chen, Page 4, Paragraph 35, cell parameters, performance indicators [e.g. RSRP, RSRQ]).
Regarding Claim 4, Tan in view of Chen teaches the method of claim 1, wherein associating each of the reward values with a corresponding previous action comprises, for each of the previous actions: determining a pre-action state of the one or more operational parameters and a post-action state of the one or more operational parameters (Tan, Pages 11-12, Paragraphs 86-90, Fig. 12, adjusting parameters of a plurality of cells in a wireless network using a DRL technique, determining reward based on previous action, action performed based on reward value; Chen, Pages 2-3, Paragraphs 17-23, determination of reward value at given time, consideration of action consequences to a state); determining a loss function based on the corresponding reward value and an estimated reward value for performing the previous action on the pre-action state to obtain the post-action state, and minimizing the loss function to associate the previous action with the reward value (Tan, Page 12, Paragraph 90, calculation of temporal difference [TD] error corresponding to each action of the experience tuples in the mini batch using a loss function; TD error may be calculated for minimizing mean squared error (MSE) loss; Chen does not disclose determining a loss function, but discloses deriving performance indicators [e.g. KPIs] based on raw operational data; configurations may be applied before and/or after KPI aggregations, configurable KPI calculation intervals, Page 7, Paragraphs 49-50).
Regarding Claim 5, Tan in view of Chen teaches the method of claim 4, wherein the loss function is based on a mean square residual between the reward value and the estimated reward value (Tan, Page 12, Paragraph 90, MSE loss).
Regarding Claim 6, Tan in view of Chen teaches the method of claim 1, wherein selecting the previous action comprises: determining a current state of the one or more operational parameters; determining respective estimated reward values for performing the previous actions on the current state; and selecting the previous action associated with the highest estimated reward value as the action to change the current state (Tan, Pages 11-12, Paragraph 87-89, cell states, such as a first state, a first reward associated with a previous action, a second state, an action that moves the cell from the first state to the second state, and a second reward associated with the action; Paragraph 88, cell states; Paragraph 89, action selected to adjust parameters of cell, updating respective state; Chen, Page 5, Paragraphs 38-40, selection of different actions, with associated rewards, with respect to a given state).
Regarding Claim 7, Tan in view of Chen teaches the method of claim 6, wherein: a random one of the previous actions is selected with probability 0
≤
ε
≤
1; and the previous action associated with the highest estimated reward value is selected with probability 1
–
ε
(Tan, Page 6, Paragraph 52, determine whether or not the expert experience is used to explore, i.e., to generate experience tuples, according to various criteria, such as probability-based criteria, similarity-based criteria, or threshold-based criteria; method may make determination using e.g. e-greedy technique or any other probabilistic techniques/approaches; Page 8, Paragraph 67, when probability of an action is higher than any of the other actions, select action that has highest probability among the actions).
Regarding Claim 8, Tan in view of Chen teaches the method of claim 1, wherein the reward metric value at time instance t , RMt , is determined according to: RMt = GTt + GTNt + ( 1 – CRt ) + ( 1 - CRNt ) and wherein: GTt is the good traffic rate in the first cell at time instance t; CRt is the congestion rate in the first cell at time instance t; GTNt is an average of the good traffic rates in the neighbor cells at time instance t; and CRNt is an average of the congestion rates in the neighbor cells at time instance t (Tan, Pages 3-4, Paragraph 36, key performance indicators [KPIs], Paragraph 39, optimization or adjustment of KPI[s] of a wireless network; Page 8, Paragraph 65, state of cell, KPIs (e.g. throughput and cell load); Pages 11-12, Paragraph 86, Fig. 12, adjusting cell parameters of a plurality of cells in a wireless network using a DRL technique; Paragraph 87, generation of experience tuples for training the DRL-NN, time instances t; Paragraph 88, initial state of cell, including information of the cell of interest, relationship information with neighboring cells, e.g. RSRP and RSRQ; Paragraph 89, RSRP value should be greater than or equal to a RSRP threshold, RSRQ greater than or equal to RSRQ threshold; Chen, Pages 4-6, Paragraph 35, cell parameters, performance indicators; Paragraphs 38-43, sub-agent[s] select a parameter setting at a given state at a given time).
Regarding Claim 9, Tan in view of Chen teaches the method of claim 8, wherein GTNt and CRNt are weighted averages, with each neighbor cell’s good traffic rate and congestion rate being weighted by a degree of overlap between the first cell and the neighbor cell (Tan fails to disclose weighted averages, with each neighbor cell’s good traffic and congestion rate being weighted by a degree of overlap between the first cell and the neighbor cell. Tan discloses a cost function in terms of a performance or optimization goal, wherein the cost function may be defined for coverage and interference, coefficients to reflect weight of coverage to and interference to cost function, for example, Page 9, Paragraph 70. Chen, Page 2, Paragraphs 16-17, weighted average, RLA may select setting based on weighted average; Pages 7-8, Paragraph 52, weighted majority arbitration may be applied among setting selections from different sub-agents for a parameter having a discrete parameter space).
Regarding Claim 10, Tan in view of Chen teaches the method of claim 9, wherein the respective degrees of overlap between the first cell and the neighbor cells are based on the portion of the total DL traffic in the first cell for which UEs also receive DL reference signals, RS, from the respective neighbor cells (Tan discloses a cost function in terms of a performance or optimization goal, wherein the cost function may be defined for coverage and interference, coefficients to reflect weight of coverage to and interference to cost function, for example, Page 9, Paragraph 70; Chen, Pages 5-6, Paragraph 38-43, state information, reward information performance indicators; Page 8, Paragraph 54, different use cases may have sub-agents/agents and/or neural networks that expect different sets of state information; coordination among actions of different sub-agents by weighted average, for example; Paragraph 56, operational data may be particular to one or more cells of the RAN of interest).
Regarding Claim 11, Tan in view of Chen teaches the method of claim 8, wherein the good traffic rate at time instance t, for each particular cell of the first cell and one or more neighbor cells, is the portion of total downlink, DL, traffic in the particular cell that is delivered with good coverage and good quality during a period including or immediately preceding time instance t (Tan, Pages 11-12, Paragraph 87, generation of experience tuples for training the DRL-NN, time instances t; Paragraph 88, initial state of cell, including information of the cell of interest, relationship information with neighboring cells, e.g. RSRP and RSRQ); Chen, Page 2, Paragraph 15, real-time exploration of different configurations, updating policies based upon the outcomes of the changes to determine the optimal configurations [e.g., parameter settings] of the network).
Regarding Claim 12, Tan in view of Chen teaches the method of claim 11, wherein determining the reward metric value at time instance t comprises: obtaining user equipment, UE, measurements of DL reference signal received power, RSRP, and DL signal-to-interface-plus-noise ratio, SINR, for the particular cell during the period including or immediately preceding time instance t (Tan, Page 11, Paragraph 88, initial state of cell, including information of the cell of interest, relationship information with neighboring cells; Chen, Page 8, Paragraph 54); and determining the good traffic rate for each particular cell as the portion of total DL traffic, during the period including or immediately preceding time instance t, that is associated with DL RSRP measurements above a first threshold and with DL SINR measurements above a second threshold (Tan, Page 12, Paragraph 89, comparison of RSRP and RSRQ against a respective threshold; Chen, Page 4, Paragraph 35, cell parameters, performance indicators, RSRP, RSRQ; Pages 5-6, Paragraphs 38-43, selection of parameter setting at given state, given time).
Regarding Claim 13, Tan in view of Chen teaches the method of claim 8, wherein the congestion rate at time instance t, for each particular cell of the first cell and one or more neighbor cells, is the congestion rate for radio resource control, RRC, signaling in the particular cell during a period including or immediately preceding time instance t (Tan, Page 8, Paragraph 65, state of cell, KPIs (e.g. throughput and cell load); Chen, Pages 4-6, Paragraphs 40-43).
Regarding Claim 14, Tan in view of Chen teaches the method of claim 8, wherein the reward value at time instance t + 1, Rt+1 , is determined according to: Rt+1 = 1000
∙
R
M
t
+
1
-
R
M
t
R
M
t
where RMt and RMt+1 are reward metric values at time instances t and t+1, respectively (Tan, Page 12, Paragraph 89, calculated reward value; Chen, Pages 2-3, Paragraphs 17-23, determination of reward value).
Regarding Claim 15, Tan in view of Chen teaches the method of claim 1, wherein the one or more operational parameters include remote electrical tilt, RET, of one or more antennas associated with the first cell (Tan, Page 4, Paragraph 41, Fig. 3, example of cell parameters, including antenna tilt; Chen, Page 8, Paragraph 54, Fig. 3, ‘Action’ column, tilt, settings of parameters of a RAN network).
Regarding Claim 17, it differs from Claim 1 only in that it is a reinforcement learning, RL, agent configured to adjust one or more operational parameters for a first cell of a communication network, wherein:
the RL agent is implemented by communication interface circuitry and processing circuitry that are operably coupled and configured to communicate with at least a network node that provides the first cell (Tan, Pages 3-4, Paragraphs 34-35, Fig. 1, Paragraph 41, Fig. 3, RL Agent, environment, wireless network including a plurality of cells; Chen, Pages 4-5, Paragraphs 35-37, Fig. 1, element AS 145); and the processing circuitry and interface circuitry are configured to perform the method of Claim 1. It recites similar limitations as in Claim 1 and Tan in view of Chen discloses them (Tan, Pages 8-10, Paragraphs 67-71 and Pages 11-12, Paragraph 86-89, Fig. 12, adjusting cell parameters of a plurality of cells in a wireless network using a DRL technique; Chen, Pages 2-3, Paragraphs 17-23, RL techniques for adjust cell parameters, determination of reward values; Page 5, Paragraphs 38-40, action selection with associated rewards with respect to given state; Pages 8-10, Paragraphs 54-67, Fig. 4, method for determining RAN parameters via reinforcement learning agent). Therefore, claim 17 is rejected for the same reasons of anticipation (obviousness) used above.
Regarding Claim 19, it differs from Claim 1 only in that it is a reinforcement learning, RL, agent configured to adjust one or more operational parameters for a first cell of a communication network (Tan, Page 4, Paragraph 41, Fig. 3, reinforcement learning [RL] system, RL agent selects an action to perform on the environment, action may refer to an adjustment of a cell parameter of a cell associated with a base station; Page 10, Paragraph 76, DRL agent, selects an action for applying to the cell; Chen, Page 4, Paragraph 35, reinforcement learning agent [RLA], plurality of sub-agents for determining settings of parameters of a RAN, obtaining performance indicators, state information, reward information from cell site[s] in access network to process state information, reward information in order to determine settings), wherein the RL agent is further configured to perform the method of Claim 1. It recites similar limitations as in Claim 1 and Tan in view of Chen discloses them (Tan, Pages 8-10, Paragraphs 67-71 and Pages 11-12, Paragraph 86-89, Fig. 12, adjusting cell parameters of a plurality of cells in a wireless network using a DRL technique; Chen, Pages 2-3, Paragraphs 17-23, RL techniques for adjust cell parameters, determination of reward values; Page 5, Paragraphs 38-40, action selection with associated rewards with respect to given state; Pages 8-10, Paragraphs 54-67, Fig. 4, method for determining RAN parameters via reinforcement learning agent). Therefore, claim 19 is rejected for the same reasons of anticipation (obviousness) used above.
Regarding Claim 22, Tan in view of Chen teaches a non-transitory, computer-readable medium storing computer-executable instructions that, when executed by processing circuitry associated with a reinforcement learning, RL, agent configured to adjust one or more operational parameters for a first cell of a communication network (Tan, Pages 15-16, Paragraphs 112-115, Fig. 17, processing system for performing method, programming instructions executed by processor; Chen, Page 7, Paragraph 50, Fig. 2, RLA 240, sub-agents 241-243; Pages 10-11, Paragraphs 68-72, Fig. 5, computing system programmed to perform the described functions), configure the RL agent to perform the method of claim 1. Claim 22 corresponds to the method of claim 1. Therefore, the recited operations performed by the non-transitory, computer-readable medium storing computer-executable instructions are mapped to the proposed combination of Tan in view of Chen in the same manner as the corresponding steps/elements in the corresponding method claim. Claim 22 is rejected for the same reasons of anticipation (obviousness) used above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GABRIELLE N DAI whose telephone number is (571)272-6693. The examiner can normally be reached Mon - Thu. 8:30am - 5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, AKWASI SARPONG can be reached at (571) 270-3438. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GABRIELLE N DAI/Examiner, Art Unit 2681
/AKWASI M SARPONG/SPE, Art Unit 2681 6/17/2026
Prosecution Insights