DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This Office Action is in response to claims filed on 8/14/2024.
Claims 1-14 remain pending in the application.
Claim Objections
Claim 1 is objected to because of the following informalities: DRL should be expanded as Deep Reinforcement Learning (DRL) in line 18. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 1-14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 1; claim 1 recites with a second probability selects: either a random action selected randomly among all the possible actions, with a third probability, or with a fourth probability, a random action among a set of actions that satisfy a predetermined action constraint. This renders claim indefinite because it is unclear whether/how a second probability is used in selecting a random action.
Regarding claims 2-14; claims 2-14 are dependent claims of claim 1, and therefore inherit the 35 U.S.C 112(b) issues of the independent claim.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-5, 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Shafin et al. (Self-tuning sectorization: Deep Reinformcement Learning Meets Broadcast Beam Optimization; IEEE Transactions on Wireless Communications, Vol. 19, No. 6, June 2020); in view of Outes et al. (WO 2021/244765 A1).
Regarding claim 1; Shalfin discloses the method comprising:
- providing an Environment configured for simulating said mobile communications network, wherein the Environment is configured to simulate said mobile communications network based upon (an offline training mechanism using ray-tracing data is used to train the DRL network; ray-tracing can well-capture the cellular environment so that the learned beam in the offline training platform could be the same as the online deployment case; see paragraph 1 of right column of page 4):
- radio measurement data, comprising radio measurements with associated geolocalization and time stamp (the location data includes UEs’ location and the corresponding time stamp; see paragraph 1 of right column of page 4), performed by user equipment connected to the mobile communications network and received from the deployed mobile communications network (UE report measurement results, time stamp, and location information; see paragraph 2 of right column of page 4);
- network performance data provided by the deployed mobile communications network (ray-tracing data includes network performance data (e.g. path loss value of each path for each location in a cell); see paragraph 1 of right column of page 4), and
- simulation data obtained by an electromagnetic field propagation simulator and corresponding to different possible configurations of the values of the modifiable parameters of the configurable cells (together with ray-tracing data, which contains the information about signal propagation environment, UEs’ location history data are used to train the DRL network; see paragraph 1 of right column of page 4);
- providing a DRL Agent configured for interacting with the Environment by acting on the Environment to cause the Environment to simulate effects, in terms of network performance, of modifications of the values of the modifiable parameters of the configurable cells (a DRL agent interacts with environment by observing the state of the network, and taking action that maximizes the reward or network performance metric; see paragraph 2 of left column of page 5 and Fig. 2 ), the Environment being configured for calculating and returning to the DRL Agent a Reward indicative of the goodness of the actions selected by the DRL Agent and undertaken on the Environment (the Environment provides observation/state reward to the DRL agent; see paragraph 2 of left column of page 5 and Fig. 2), the Reward being exploited by the DRL Agent for training and estimating Q-values (a reward refers to any network performance metric; the Q-value is defined as the expected cumulative discounted reward for taking action; see paragraph 1 of right column of page 5 and paragraph 2 of left column of page 6), wherein:
- the DRL Agent, during training time, in selecting an action to be undertaken on the Environment among all the possible actions (the DRL agent dynamically interacts with an environment, by taking sequential actions in discrete time steps; see paragraph 3 of left column of page 5), adopts an action selection policy that (the DRL agent selects an action based on Ɛ-greedy policy; see paragraph 3 of left column of page 6):
- with a certain first probability selects a greedy action based on the estimated Q-values (with probability (1- Ɛ), the agent selects the based know action so far; the optimum action in a particular state is selected based on maximum Q-values; paragraph 3 of left column of page 6),
- with a second probability selects (with probability Ɛ; see paragraph 3 of left column of page 6):
either a random action selected randomly among all the possible actions, with a third probability (the DRL agent tries a random action; see paragraph 3 of left column of page 6), or
with a fourth probability, a random action among a set of actions that satisfy a predetermined action constraint (no patentable weight is given due to a claim language or).
Shalfin discloses an offline training mechanism using ray-tracing data to train a DRL network.
Shalfin does not explicitly disclose the environment returns a penalizing reward if an action violates a predetermined constraint.
Outes discloses a method, implemented by a data processing system, of adjusting modifiable parameters of network cells of a deployed self- organizing cellular mobile communications network comprising network cells covering a geographic area of interest, said network cells comprising configurable cells having modifiable parameters, and non-configurable cells in the neighborhood of the configurable cells (the method is performed by a system that includes a number of DRL agent that are each associated with a cell group in a communication network; each DRL agent has a policy that the DRL uses to determine adjustments to the cell parameter in the cell group; each cell has its own set of configurable parameters; some of these parameters only affect the cell on which they are applied; see paragraphs 1-2 of page 1, paragraph 3 of page 5, Fig. 5 and Fig. 6)
wherein - in case the DRL Agent selects, and causes the Environment to simulate the effects of, an action that violates said predetermined constraint, the Environment is configured to return to the DRL Agent a penalizing Reward (an action selected by the DRL agent may cause a negative reward if the performance degraded; see paragraph 2 of page 27).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Shalfin and Outes to return a penalizing reward to reflect the performance degradation that the action is generating in the environment (see paragraph 5 of page 23).
Regarding claim 2; Shafin discloses the method of claim 1, wherein said radio measurement data comprising radio measurements with associated geolocalization and time stamp are Minimization of Drive Test, MDT, data (a UE report MDT measurement data, time stamp, and location history; see paragraph 1 of right column of page 4).
Regarding claim 3; Shalfin discloses an offline training mechanism using ray-tracing data to train a DRL network.
Shalfin does not explicitly disclose the predetermined action constraint is not modifying a parameter.
Outes discloses the method of claim 1, wherein said predetermined action constraint is a constraint for not attempting to modify a modifiable parameter of a configurable cell already subjected, in past actions selected by the DRL Agent, to a modification of its modifiable parameters (the possible actions that can be performed with respect to a cell parameter in a cell of a cell group includes do nothing, i.e. do not modify the cell parameter and maintain the current value of the cell parameter; see paragraph 2 of page 23 and Fig. 3).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Shalfin and Outes to not to modify a parameter to optimize the performance (see paragraph 2 of page 23 of Outes).
Regarding claim 4; Shalfin discloses the method of claim 1, wherein:
- the sum of said first and second probabilities is 1 (in Ɛ-greedy policy, on probability is Ɛ and the other probability is (1- Ɛ); see paragraph 3 of left column of page 6), and
- the sum of said third and fourth probabilities is 1 (in Ɛ-greedy policy, on probability is Ɛ and the other probability is (1- Ɛ); see paragraph 3 of left column of page 6).
Regarding claim 5; Shalfin discloses an offline training mechanism using ray-tracing data to train a DRL network.
Shalfin does not explicitly disclose an action is to modify the value of a parameter of a cell.
Outes discloses the method of claim 1, wherein:
- each action selected by the DRL Agent is an action that attempts to modify the value of one single configurable parameter of one single configurable cell of the configurable cells (possible action that can be performed with respect to a cell parameter in a cell of a cell group are increase/decrease the value of the cell parameter; see paragraph 2 of page 23 and Fig. 3).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Shalfin and Outes to modify the value of a parameter of a cell to optimize the performance (see paragraph 2 of page 23).
Regarding claim 7; Shafin discloses the method of claim 1, wherein: wherein the Environment is configured for analyzing and aggregating said radio measurement data comprising radio measurements with associated geolocalization and time stamp, based on geolocation information included in the radio measurement data, in territory pixels corresponding to the territory pixels of the simulation data (a UE reports measurement results, time stamp and location information; select a batch of UE location at different timestamps; incorporate UE location distribution to ray-tracing data to create scenario-specific ray-tracing data; see paragraph 2 of right column of page 4 and Algorithm 1 offline training of page 5).
Regarding claim 14; Shafin discloses a data processing system configured for automatically adjusting modifiable parameters of network cells of a self-organizing cellular mobile communications network (self-optimization within self-organizing network is a process of self-tunning of network parameters for achieving optimum performance in terms of any predefined metric of interest; see paragraph 1 of right column of page 1), the system comprising:
- a self-organizing network module comprising a capacity and coverage optimization module, wherein the capacity and coverage optimization module is configured to execute the method of claim 1 (the DRL can enhance the capacity of femtocells while ensuring a minimum quality of service to macrocells; the DRL-based framework for MIMO broadcast beam optimization for optimal cell-sectorization to maximize the coverage of cellular network; see paragraph 3 of left column of page 2 and paragraph 2 of right column of page 2).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Shafin; in view of Outes; and in further view of Zadorojniy et al. (US 2022/0101177 A1).
Regarding claim 6; the combination of Shafin and Outes discloses an offline training mechanism using ray-tracing data to train a DRL network.
The combination of Shafin and Outes does not explicitly disclose stopping an ongoing sequence of actions after a condition is met.
Zadorojniy discloses the method of claim 1, wherein:
- the Environment is configured to command the DRL Agent to stop an ongoing sequence of actions after a predetermined number of actions selected by the DRL Agent that violates said predetermined constraint (when the stopping condition is met, the execution of the DRL algorithm stops; see paragraph [0061]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Shalfin, Outes and Zadorojniy to stop an ongoing sequence of actions after a condition is met to control the machine for solving a predefined optimization problem (see ABSTRACT of Zadorojniy).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Shafin; in view of Outes; and in further view of Yousefi'zadeh et al. (US 2017/0359752 A1).
Regarding claim 8; the combination of Shafin and Outes discloses a UE reports MDT measurement results, time stamp and location information.
The combination of Shafin and Outes does not explicitly disclose calculating RSRP, SINR and number of active UEs in the region.
Yousefi'zadeh discloses the method of claim 7, wherein the Environment is configured for calculating, for each territory pixel and based on the MDT data:
- a pixel RSRP being an average of the RSRPs included in the MDT data corresponding to such pixel (calculate average RSRP; see paragraph [0081]);
- a pixel SINR (calculating SINR; see paragraph [0081]), and
- a pixel weight providing an indication of an average number of active UEs or RRC connected UEs, in such pixel (calculating average active UEs; average connected UEs; see paragraph [0081]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Shalfin, Outes and Yousefi'zadeh to calculate RSRP, SINR and number of active UEs in the region to predict PRB utilization of cells (see paragraph [0081] of Yousefi'zadeh).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Shafin; in view of Outes; in view of Yousefi'zadeh; and in further view of Su et al. (CN 105992263 A).
Regarding claim 9; the combination of Shalfin, Outes and Yousefi'zadeh disclose calculating RSRP and SINR based on MDT data.
The combination of Shalfin, Outes and Yousefi'zadeh do not explicitly disclose calculating RSRP and SINR difference values.
Su discloses the method of claim 8, wherein the Environment is configured to calculate, for every pixel:
- a RSRP difference between the calculated pixel RSRP, calculated based on the MDT data, and the RSRP resulting from the simulation data (calculating RSRP values of all grids in the calculation region; obtaining difference between the RSRP value of each grid by covering simulation according to a magnitude of RSRP value defining the grid; see (2) – (4) of page 8), and
- a SINR difference between the calculated pixel SINR, calculated based on the MDT data, and the SINR resulting from the simulation data (calculating SINR values for all the grids in the calculation region; obtaining difference between the SINR value of each grid by covering simulation according to a magnitude of SINR value defining the grid; see (2) – (4) of page 8).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Shalfin, Outes, Yousefi'zadeh and Su to calculate RSRP and SINR difference values to address offset of the evaluation scheme (see paragraph 3 of page 2 of Su).
Claims 10 is rejected under 35 U.S.C. 103 as being unpatentable over Shafin; in view of Outes; in view of Yousefi'zadeh; in view of Su; and in further view of Tan et al. (US 2019/0014488 A1).
Regarding claim 10; the combination of Shafin, Outes, Yousefi'zadeh and Su disclose calculating RSRP and SINR difference values.
The combination of Shafin, Outes, Yousefi'zadeh and Su do not disclose obtaining estimated RSRP and SINR using RSRP difference and SINR difference.
Tan discloses the method of claim 9, wherein the Environment, when the DRL Agent undertakes an action on it, is configured to, for every pixel:
- taking the RSRP and the SINR from the simulation data that correspond to the new configuration of the configurable cells indicated in the actions requested by the DRL Agent (generate a state tensor for each of the plurality of cells, each state tensor indicating a state of the respective cell; and a state tensor includes RSRP and SINR; see paragraph [0103] and Fig. 15), and
- applying to the RSRP and SINR taken from the simulation data said RSRP difference and said SINR difference, respectively, to obtain estimated RSRP and estimated SINR for the new configuration of the configurable cells (the DRL agent performs the following steps: select an action for each of the plurality of cells, the action moving the respective cell from one state to another state; apply respective actions selected for the plurality of cells to the respective cells to adjust one or more cell parameters and generate an experience tuple for each of the plurality of cells based on the respective action applied; see paragraph [0103] and Fig. 15).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Shalfin, Outes, Yousefi'zadeh, Su and Tan to obtain estimated RSRP and SINR using RSRP difference and SINR difference to improve wireless network performance (see paragraph [0003] of Tan).
Claims 11-12 are rejected under 35 U.S.C. 103 as being unpatentable over Shafin; in view of Outes; in view of Yousefi'zadeh; in view of Su; in view of Tan; and in further view of Acharya et al. (WO 2014/140667 A1).
Regarding claim 11; the combination of Shafin, Outes, Yousefi'zadeh, Su and Tan disclose estimating a RSRP value for a cell.
The combination of Shafin, Outes, Yousefi'zadeh, Su and Tan do not explicitly disclose re-assign regions to best-server network cells based on estimated RSRP value.
Acharya discloses the method of claim 10, wherein the Environment is configured to:
- re-assigning territory pixels to respective best-server network cells based on the estimated RSRP for the new configuration of the configurable cells (UEs may be reassigned to the cells to which they had the next strongest RSRP value; see paragraph [0056]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Shalfin, Outes, Yousefi'zadeh, Su, Tan and Acharya to re-assign regions to best-server network cells based on estimated RSRP value in order to reduce interference (see paragraph [0003] of Acharya).
Regarding claim 12; the combination of Shafin, Outes, Yousefi'zadeh, Su and Tan disclose estimate RSRP value for a cell.
The combination of Shafin, Outes, Yousefi'zadeh, Su and Tan do not explicitly disclose
redistributing UEs to the network cells based on the re-assignment of regions.
Acharya discloses the method of claim 11, wherein the Environment is configured to:
- redistribute UEs to the network cells based on the re-assignment of the territory pixels to the respective best-server network cells (UEs may be reassigned to the cells to which they had the next strongest RSRP value; see paragraph [0056]), and
- calculate numbers of UEs per network cell for the new configuration (keeps track of the changes in the number of UEs in the network; see paragraphs [0003] and [0056]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Shalfin, Outes, Yousefi'zadeh, Su, Tan and Acharya to redistribute UEs to the network cells based on re-assignment of regions to adapt to dynamic changes in UE traffic (see paragraph [0004] of Acharya).
Allowable Subject Matter
Claim 13 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The closest prior art on the record Outes discloses a method is performed by a system that includes a number of DRL agent that are each associated with a cell group in a communication network. Each DRL agent has a policy that the DRL uses to determine adjustments to the cell parameter in the cell group. An action selected by the DRL agent may cause a negative reward if the performance degraded or a positive reward if the performance improved (paragraphs 1-2 of page 1, paragraph 3 of page 5, paragraph 2 of page 27, Fig. 5 and Fig. 6)
However, Outes does not teach wherein said Reward indicative of the goodness of the actions selected by the DRL Agent is indicative of an estimated overall performance of a network configuration resulting from a simulation of modifications of the values of the modifiable parameters of the configurable network cells by the Environment, wherein the Environment is configured to calculate said Reward by calculating an estimation of an overall throughput as a weighted average of estimated average user throughputs per network cell, where the weights in the weighted average are based on said calculated numbers of UEs per network cell.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NING LI whose telephone number is (571)270-0624. The examiner can normally be reached Monday, Tuesday, Thursday 8:30am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jeffrey Rutkowski can be reached at (571) 270-1215. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MANSOUR OVEISSI/Primary Examiner, Art Unit 2415
/N.L/Examiner, Art Unit 2415
Prosecution Insights