DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d) based on an application filed in PEOPLE'S REPUBLIC OF CHINA on 08/10/2022. The certified copy has been filed in parent Application No. 18/125,327, filed on 03/23/2023. Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-2, 8-9, and 15-16, is/are rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Zhang et al., (“Intelligent Electric Vehicle Charging Recommendation Based on Multi-Agent Reinforcement Learning”).
Regarding claim 1 and analogous claim 8, 15:
Zhang teaches:
A method for training an information adjustment model of a charging station, comprising: (Section 3, “we present the MARL formulation for the EV charging [charging station] recommendation task and detail our Master framework with centralized training decentralized execution (CTDE). Moreover, we elaborate on the generalized multi-critic architecture for multiple objectives optimization.”)
acquiring a battery charging request, and (Section 2, Definition 1, “Charging request. A charging request qt = ⟨lt , Tt ,T c t ⟩ ∈ Q is defined as the t-th request (i.e., step t) of a day. Specifically, lt is the location of qt [a battery charging request] (i.e., wherein under the broadest reasonable interpretation (BRI) a charging request is interpreted as receiving a request), Tt is the real-world time of the step t, and T c t is the real-world time qt finishes the charging request.”)
determining an environment state information corresponding to each charging station in a charging station set; (Section 3.1, “Observation o i t . Given a charging request qt , we define the observation o i t [environment state information] of agent c i [each charging station] as a combination of the index of c i , the real-world time Tt , the number of current available charging spots of c i (supply), the number of charging requests around c i in the near future (future demand), the charging power of c i , the estimated time of arrival (ETA) from location lt to c i , (i.e., wherein the environment state information is interpreted as supply, future demand, and ETA.) and the CP of c i at the next ETA. We further define st = {o 1 t , o 2 t , . . . , o N t } [chagrining station set] as the state of all agents at step t.”)
determining, through an initial policy network, (Section 3.21, paragraph 1, “we devise the multi-agent actor-critic framework with a centralized attentive critic for deterministic policies learning (i.e., wherein the policy network under the broadest reasonable interpretation (BRI) is interpreted as an actor of the multi-agent actor critic framework, hence (SPEC [0044], In this embodiment, an actor (policy network)-critic (value network) architecture is used. Here, the policy network is used to determine action))”)
target operational information of each charging station in the charging station set for the battery charging request, (“Action a i t . Given an observation o i t , an intuitional design for the action of agent ci is a binary decision, i.e., recommending qt to itself for charging or not. However, because one qt can only choose one station for charging, multiple agents’ actions may be tied together and are difficult to coordinate. Inspired by the bidding mechanism [47], we design each agent c i offers a scalar value to "bid" for qt as its action a i t [target operational information] (i.e., wherein the bidding under the broadest reasonable interpretation is interpreted as an action to be taken, ‘bid’). By defining ut = {a 1 t , a 2 t , . . . , a N t } as the joint action, qt will be recommended to the agent with the highest "bid" value, i.e., rct = c i , where i = arg max(ut ).”)
according to the environment state information corresponding to each charging station in the charging station set; (Section 3.1, “Observation o i t . Given a charging request qt , we define the observation o i t [environment state information] of agent c i [each charging station] as a combination of the index of c i , the real-world time Tt , the number of current available charging spots of c i (supply), the number of charging requests around c i in the near future (future demand), the charging power of c i , the estimated time of arrival (ETA) from location lt to c i , (i.e., wherein the environment state information is interpreted as supply, future demand, and ETA.) and the CP of c i at the next ETA. We further define st = {o 1 t , o 2 t , . . . , o N t } [chagrining station set] as the state of all agents at step t.”)
determining, through an initial value network, a cumulative reward expectation corresponding to the battery charging request according to the environment state information and the target operational information corresponding to each charging station in the charging station set; (Section 3.2.1, “To motivate the agents to make recommendations cooperatively, we devise the multi-agent actor-critic framework with a centralized attentive critic for deterministic policies learning. A similar MARL algorithm with CTDE architecture is proposed in [24], which incorporates the full state st and joint action ut of all agents into the critic [initial value network] to motivate agents to learn coordinated and cooperative policies.”…Section 3.3, “Particularly, in our task, we learn two centralized attentive critics, Q cw t b [environment state information] (i.e., wherein cwt is interpreted as charging wait time (Zhang, see definition 2)) and Q cp b [target operational information] (i.e., wherein cp is interpreted as charging price (Zhang, see definition 3)), which correspond to the expected returns of reward r cw t and r cp , respectively where E denotes the environment, and R cw t t:t+j is the cumulative discounted reward [cumulative reward expectation] (defined in Eq. (3)) with respect to r cw t .”)
training the initial policy network and the initial value network by using a deep deterministic policy gradient algorithm, to obtain a trained policy network and a trained value network, (Section 3.2.1, “To motivate the agents to make recommendations cooperatively, we devise the multi-agent actor-critic framework (i.e., wherein the initial policy network and the initial value network are interpreted as the actor and critic) with a centralized attentive critic for deterministic policies learning [deep deterministic policy gradient algorithm]. A similar MARL algorithm with CTDE architecture is proposed in [24], which incorporates the full state st and joint action ut of all agents into the critic to motivate agents to learn coordinated and cooperative policies (i.e., wherein the result of training is interpreted as the trained policy network and a trained value network)”
wherein, during the training, the initial value network is updated through a temporal difference method, and the initial policy network is updated with a goal of maximizing the cumulative reward expectation corresponding to the battery charging request; and (Section 3.2.1, paragraph 3, “The centralized attentive critic Qb is updated by minimizing the following loss: L(θQ ) = Es a t ,u a t ,pt ,s a t+j ,pt+j ,Rt :t+j∼D h. (i.e., wherein a temporal difference method under the broadest reasonable interpretation (BRI) is interpreted as the loss over time ‘t:t+j’)”…Section 3.3, paragraph 2, “A simple way to optimize multiple objectives is to average the reward of multiple objectives with a set of prior weights and maximize [maximizing] the combined reward [cumulative reward expectation] as a single objective ”)
determining the trained policy network as an information adjustment model corresponding to each charging station in the charging station set (Section 3.2.3, “The execution process is fully decentralized, by only invoking the learned [trained policy network] actor policy with its own observation. Specifically, for a charging request qt , the agent c i ∈ C a t takes action a i t based on its o i t by a i t = b i (o i t ) (i.e., wherein the information adjustment model under the broadest reasonable interpretation (BRI) is interpreted as the decision maker based on the data). (11) And qt will be recommended to the active agent with the highest a i t among all the actions of C a t .”)
Regarding claim 2 and analogous claim 9, 16:
Zhang further teaches:
wherein determining, through the initial value network, the cumulative reward expectation corresponding to the battery charging request according to the environment state information and the target operational information corresponding to each charging station in the charging station set comprises: (Section 3.2.1, “To motivate the agents to make recommendations cooperatively, we devise the multi-agent actor-critic framework with a centralized attentive critic for deterministic policies learning. A similar MARL algorithm with CTDE architecture is proposed in [24], which incorporates the full state st and joint action ut of all agents into the critic [initial value network] to motivate agents to learn coordinated and cooperative policies.”…Section 3.3, “Particularly, in our task, we learn two centralized attentive critics, Q cw t b [environment state information] (i.e., wherein cwt is interpreted as charging wait time (Zhang, see definition 2)) and Q cp b [target operational information] (i.e., wherein cp is interpreted as charging price (Zhang, see definition 3)), which correspond to the expected returns of reward r cw t and r cp , respectively where E denotes the environment, and R cw t t:t+j is the cumulative discounted reward [cumulative reward expectation] (defined in Eq. (3)) with respect to r cw t .”)
determining, through an agent pooling module, integrated representation information representing all charging stations in the charging station set according to the environment state information and the target operational information corresponding to each charging station in the charging station set; and (Section 3.2.1, paragraph 2, “we propose to use the attention mechanism which is permutation-invariant [agent pooling module] (i.e., wherein permutation invariant under the broadest reasonable interpretation (BRI) is interpreted as pooling) to integrate information of the active agents”…Section 3.2.2, “Once the influence weight α i t of each active agent c i ∈ C a t is obtained, we can derive the attentive representation [integrated representation information] of all active agents by xt = ReLU © - « Wc Õ i ∈C a t α i t o i t ⊕ a i t ⊕ p i t ª ® ¬ , (6) where Wc are learnable parameters”…“Section 3.3, “Particularly, in our task, we learn two centralized attentive critics, Q cw t b [environment state information] (i.e., wherein cwt is interpreted as charging wait time (Zhang, see definition 2)) and Q cp b [target operational information] (i.e., wherein cp is interpreted as charging price (Zhang, see definition 3))”)
determining, through the initial value network, the cumulative reward expectation corresponding to the battery charging request according to the integrated representation information (Section 3.2.1, “To motivate the agents to make recommendations cooperatively, we devise the multi-agent actor-critic framework with a centralized attentive critic for deterministic policies learning. A similar MARL algorithm with CTDE architecture is proposed in [24], which incorporates the full state st and joint action ut of all agents into the critic [initial value network] to motivate agents to learn coordinated and cooperative policies”... Section 3.2.2, “Once the influence weight α i t of each active agent c i ∈ C a t is obtained, we can derive the attentive representation [integrated representation information] of all active agents by xt = ReLU © - « Wc Õ i ∈C a t α i t o i t ⊕ a i t ⊕ p i t ª ® ¬ , (6) where Wc are learnable parameters”)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 3, 10, and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al., in view of Su et al., Non-Patent Literature (“Operating Status Prediction Model at EV Charging Stations With Fusing Spatiotemporal Graph Convolutional Network”).
Regarding claim 3 and analogous claim 10, 17:
Zhang teaches the method of claim 2.
Zhang further teaches:
wherein determining, through the agent pooling module, integrated representation information representing all charging stations in the charging station set according to the environment state information and the target operational information corresponding to each charging station in the charging station set comprises: (Section 3.2.1, paragraph 2, “we propose to use the attention mechanism which is permutation-invariant [agent pooling module] to integrate information of the active agents”…“ Once the influence weight α i t of each active agent c i ∈ C a t is obtained, we can derive the attentive representation [integrated representation information] of all active agents by xt = ReLU © - « Wc Õ i ∈C a t α i t o i t ⊕ a i t ⊕ p i t ª ® ¬ , (6) where Wc are learnable parameters”…“Section 3.3, “Particularly, in our task, we learn two centralized attentive critics, Q cw t b [environment state information] (i.e., wherein cwt is interpreted as charging wait time (Zhang, see definition 2)) and Q cp b [target operational information] (i.e., wherein cp is interpreted as charging price (Zhang, see definition 3))”)
mapping, through a mapping vector, the environment state information and the target operational information corresponding to each charging station in the charging station set to a score feature representing an importance of each charging station; (Section 4.6, paragraph 1, “the attention weights and some input features [environment state information and target operational information] of two centralized attentive critics (i.e., Q cw t b and Q cp b ), to qualitatively analysis the effectiveness of Master. Two cases are depicted in Figure 9. We can observe these two critics pay more attention to the charging stations with high action values (i.e., wherein the high action values are interpreted as a score feature representing an importance of the charging station). This makes sense, since these charging stations are highly competitive bidding participants. The charging request will be recommended to the charging station with the highest action, then environment returns rewards depending on this recommended station (i.e., wherein the mapping is interpreted as taking the attention weights to map the input features)”)
determining a preset number of charging stations from the charging station set according to the score feature, and (Section 3.2.1, paragraph 2, “given a charging request qt , we only activate the agents nearby qt [score feature] (e.g., top-k nearest qt ) [preset number] to take actions, denoted as C a t . We set other agents who are far away from qt inactive and don’t participate in the recommendation for qt (i.e., wherein under the broadest reasonable interpretation (BRI), the agents nearby are interpreted as assigning a score feature (i.e., ranking etc.,) to the closet agent)”)
determining the environment state information, the target operational information and the score feature corresponding to each charging station of the preset number of charging stations; (Section 4.1.1, “All real-time availability (supplies) records, charging prices and charging powers of charging stations [target operational information] (i.e., wherein the target operational information is interpreted as the price and charging power of the charging stations) are crawled from a publicly accessible app [46], in which all charging spot occupancy information is collected by real-time sensors [environment state information] (i.e., wherein the environment state information is interpreted as the availability of the charging station). The charging request data is collected through Baidu Maps API [22, 45]. We split each city as 1 × 1km2 grids, and aggregate the number of future 15 minutes charging requests in the surrounding area (i.e., the grid the station locates in and eight neighboring grids) as the future demands of the corresponding charging stations”…Section 3.2.1, paragraph 2, “given a charging request qt , we only activate the agents nearby qt [score feature] (e.g., top-k nearest qt ) [preset number] to take actions, denoted as C a t . We set other agents who are far away from qt inactive and don’t participate in the recommendation for qt (i.e., wherein under the broadest reasonable interpretation (BRI), the agents nearby are interpreted as assigning a score feature (i.e., ranking etc.,) to the closet agent)”)
normalizing score features corresponding to the preset number of charging stations to obtain a gate control vector; (Section 3.2.1, paragraph 2, “given a charging request qt , we only activate the agents nearby qt (e.g., top-k nearest qt ) [preset number] to take actions, denoted as C a t . We set other agents who are far away from qt inactive and don’t participate in the recommendation for qt”…Page 1861, Col 1, “we derive dynamic update weights to adaptively adjust the step size of the two objectives, which is learned by the Boltzmann softmax function [normalizing score features]”…Page 1859, Col 1, “the attention mechanism [gate control vector] (i.e., wherein the attention mechanism is interpreted as a gate control) automatically quantifies the influence of each active agents where v and Wa are learnable parameters, ⊕ is the concatenation operation, and p i t are future information”)
Zhang does not explicitly teach:
determining a gate control feature according to the environment state information, the target operational information, and the gate control vector corresponding to the preset number of charging stations; and
determining the integrated representation information of all the charging stations in the charging station set according to the gate control feature.
Su teaches:
determining a gate control feature according to the environment state information, the target operational information, and the gate control vector corresponding to the preset number of charging stations; and (Section 3, Part B, paragraph 2, “Here, ht−1 represents the hidden state at t − 1; xt [environment state information] (i.e., wherein the xt flow information is interpreted as the environment state information of the charging station) represents the flow information at the charging station at the time of t; rt is the reset gate used to control the degree of ignoring the status information at the previous time step; ut is an update gate for controlling the degree to which the state at the previous time is brought to the current state [gate control feature]; ct is the information stored at the time of t; and ht is the output state at the time of t (i.e., wherein reset and update gate is interpreted as a gate control feature)”)
determining the integrated representation information of all the charging stations in the charging station set according to the gate control feature. (Page 116, Col 1, “feature of GCN is the representation of each node in a graph network under the influence of neighboring nodes and other nodes using the Laplace matrix, describing the characteristics of nodes in a graph network and the flow and propagation of information (i.e., wherein the integrated representation information of a charging station is interpreted as the representation of each node) in the topology. Inspired by the application characteristics of GCN, we propose the spatiotemporal graph convolutional network (SGCN) model to predict the operating status at charging stations. The contributions of this article are curated as follows.”)
Su and Zhang are both related to the same field of endeavor (i.e., EV charging). In view of the teachings of Su it would have been obvious for a person of ordinary skill in the art to apply the teachings of Su to Zhang before the effective filing date of the claimed invention in order to improve the efficiency of managing charging demands at a EV charging station (Su, Page 115, Col 1, “By utilizing machine learning or statistical methods, we can predict the charging load demand for a day at charging stations. In [16] and [17], the hourly charging station status was predicted using gated recurrent unit (GRU) and LSTM, respectively. The historical charging data of charging stations were used to forecast the future charging load.”)
Claim(s) 4-6, 11-13, and 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al., in view of Khosla et al., Non-Patent Literature (“Supervised Contrastive Learning”).
Regarding claim 4 and analogous claim 11, 18:
Zhang teaches the method of claim 2.
Zhang further teaches:
wherein training the initial policy network and the initial value network by using the deep deterministic policy gradient algorithm comprises: ((Section 3.2.1, “To motivate the agents to make recommendations cooperatively, we devise the multi-agent actor-critic framework with a centralized attentive critic for deterministic policies learning. A similar MARL algorithm with CTDE architecture is proposed in [24], which incorporates the full state st and joint action ut of all agents into the critic [initial value network] to motivate agents to learn coordinated and cooperative policies.”… Section 3.21, paragraph 1, “we devise the multi-agent actor-critic framework with a centralized attentive critic for deterministic policies learning [deep deterministic policy gradient algorithm] (i.e., wherein the policy network under the broadest reasonable interpretation (BRI) is interpreted as an actor of the multi-agent actor critic framework, hence (SPEC [0044], In this embodiment, an actor (policy network)-critic (value network) architecture is used. Here, the policy network is used to determine action))”)
determining a first loss corresponding to the initial value network through the temporal difference method; (Section 3.2.1, “To motivate the agents to make recommendations cooperatively, we devise the multi-agent actor-critic framework with a centralized attentive critic for deterministic policies learning. A similar MARL algorithm with CTDE architecture is proposed in [24], which incorporates the full state st and joint action ut of all agents into the critic [initial value network] to motivate agents to learn coordinated and cooperative policies”…Section 3.2.1, paragraph 3, “The centralized attentive critic Qb is updated by minimizing the following loss: L(θQ ) = Es a t ,u a t ,pt ,s a t+j ,pt+j ,Rt :t+j∼D h. (i.e., wherein a temporal difference method under the broadest reasonable interpretation (BRI) is interpreted as the loss over time ‘t:t+j’)”)
updating the initial value network and the agent pooling module according to the first loss and the second loss; and ((Section 3.2.1, “To motivate the agents to make recommendations cooperatively, we devise the multi-agent actor-critic framework with a centralized attentive critic for deterministic policies learning. A similar MARL algorithm with CTDE architecture is proposed in [24], which incorporates the full state st and joint action ut of all agents into the critic [initial value network] to motivate agents to learn coordinated and cooperative policies”…Section 3.2.2, paragraph 2, “The entire process is shown in Figure 3(b). As the centralized attentive critic perceived more complete information of all active agents, it can motivate the agents to learn policies in a coordinated and cooperative way. The centralized attentive critic Qb is updated [updating] by minimizing the following loss where θQ are the learnable parameters of critic Qb , b ′ i and Q ′ b ′ are the target actor policy of c i and target critic function with delayed parameters θ ′ i b and θ ′ Q (i.e., wherein the updating the loss is interpreted as first and second loss)”)
updating the initial policy network with the goal of maximizing the cumulative reward expectation corresponding to the battery charging request. (Section 3.3, paragraph 2, “A simple way to optimize multiple objectives is to average the reward of multiple objectives with a set of prior weights and maximize [maximizing] the combined reward [cumulative reward expectation] as a single objective ”)
Zhang does not explicitly teach:
determining a second loss corresponding to the agent pooling module through a self- supervised contrastive learning method;
Khosla teaches:
determining a second loss corresponding to the agent pooling module through a self- supervised contrastive learning method; (Section 3.2, “we now look at the family of contrastive losses [self-supervised contrastive learning], starting from the self-supervised domain and analyzing the options for adapting it to the supervised domain, showing that one formulation is superior. For a set of N randomly sampled sample/label pairs, {xk, yk}k=1...N , the corresponding batch used for training consists of 2N pairs, {x˜`, y˜`}`=1...2N , where x˜2k and x˜2k−1 are two random augmentations (a.k.a., “views”) of xk (k = 1...N) and y˜2k−1 = y˜2k = yk . For the remainder of this paper, we will refer to a set of N samples as a “batch” and the set of 2N augmented samples as a “multiviewed batch””)
Khosla and Zhang are both related to the same field of endeavor (i.e., self-supervised learning). In view of the teachings of Khosla it would have been obvious for a person of ordinary skill in the art to apply the teachings of Khosla to Zhang before the effective filing date of the claimed invention in order to improve the efficiency of multi-agent systems at a EV charging station (Khosla, Section 3.2.2, paragraph 3, “The supervised losses encourage the encoder to give closely aligned representations to all entries from the same class, resulting in a more robust clustering of the representation space”)
Regarding claim 5 and analogous claim 12, 19:
Zhang, as modified by Khosla, teaches the method of claim 4.
Zhang further teaches:
determining, for a first subset in a joint feature, first integrated representation information through the agent pooling module, (Section 3.2.1, “To motivate the agents to make recommendations cooperatively, we devise the multi-agent actor-critic framework with a centralized attentive critic for deterministic policies learning. A similar MARL algorithm with CTDE architecture is proposed in [24], which incorporates the full state st and joint action ut [joint feature] of all agents into the critic to motivate agents to learn coordinated and cooperative policies. However, such an approach suffers from the large state and action space problem in our task. In practice, the EVs tend to go to nearby stations for charging. Based on this fact, given a charging request qt , we only activate the agents nearby qt (e.g., top-k nearest qt ) to take actions, denoted as C a t . We set other agents who are far away from qt inactive and don’t participate in the recommendation for qt (i.e., wherein the first subset is interpreted as a portion of top-k agents from all agents)”…Section 3.2.1, paragraph 2, “we propose to use the attention mechanism which is permutation-invariant [agent pooling module] to integrate information of the active agents”…“ Once the influence weight α i t of each active agent c i ∈ C a t is obtained, we can derive the attentive representation [integrated representation information] of all active agents by xt = ReLU © - « Wc Õ i ∈C a t α i t o i t ⊕ a i t ⊕ p i t ª ® ¬ , (6) where Wc are learnable parameters”)
wherein the joint feature comprises the environment state information and the target operational information corresponding to each charging station in the charging station set; (Section 3.2.1, “To motivate the agents to make recommendations cooperatively, we devise the multi-agent actor-critic framework with a centralized attentive critic for deterministic policies learning. A similar MARL algorithm with CTDE architecture is proposed in [24], which incorporates the full state st and joint action ut [joint feature] of all agents into the critic to motivate agents to learn coordinated and cooperative policies”… Section 3.3, “Particularly, in our task, we learn two centralized attentive critics, Q cw t b [environment state information] (i.e., wherein cwt is interpreted as charging wait time (Zhang, see definition 2)) and Q cp b [target operational information] (i.e., wherein cp is interpreted as charging price (Zhang, see definition 3))”)
determining, for a second subset in the joint feature, second integrated representation information through the agent pooling module; (Section 3.2.1, “To motivate the agents to make recommendations cooperatively, we devise the multi-agent actor-critic framework with a centralized attentive critic for deterministic policies learning. A similar MARL algorithm with CTDE architecture is proposed in [24], which incorporates the full state st and joint action ut [joint feature] of all agents into the critic to motivate agents to learn coordinated and cooperative policies. However, such an approach suffers from the large state and action space problem in our task. In practice, the EVs tend to go to nearby stations for charging. Based on this fact, given a charging request qt , we only activate the agents nearby qt (e.g., top-k nearest qt ) to take actions, denoted as C a t . We set other agents who are far away from qt inactive and don’t participate in the recommendation for qt (i.e., wherein a second subset is interpreted as a another portion of top-k agents from all agents)”…Section 3.2.1, paragraph 2, “we propose to use the attention mechanism which is permutation-invariant [agent pooling module] to integrate information of the active agents”…Section 3.2.2, “Once the influence weight α i t of each active agent c i ∈ C a t is obtained, we can derive the attentive representation [integrated representation information] of all active agents by xt = ReLU © - « Wc Õ i ∈C a t α i t o i t ⊕ a i t ⊕ p i t ª ® ¬ , (6) where Wc are learnable parameters”)
determining, for a third subset in the joint feature corresponding to another battery charging request different from the battery charging request, third integrated representation information through the agent pooling module; and (Section 3.2.1, “To motivate the agents to make recommendations cooperatively, we devise the multi-agent actor-critic framework with a centralized attentive critic for deterministic policies learning. A similar MARL algorithm with CTDE architecture is proposed in [24], which incorporates the full state st and joint action ut [joint feature] of all agents into the critic to motivate agents to learn coordinated and cooperative policies. However, such an approach suffers from the large state and action space problem in our task. In practice, the EVs tend to go to nearby stations for charging. Based on this fact, given a charging request qt , we only activate the agents nearby qt (e.g., top-k nearest qt ) to take actions, denoted as C a t . We set other agents who are far away from qt inactive and don’t participate in the recommendation for qt (i.e., wherein a third subset is interpreted as a another portion of top-k agents from all agents)”…Section 3.2.1, paragraph 2, “we propose to use the attention mechanism which is permutation-invariant [agent pooling module] to integrate information of the active agents”…Section 3.2.2, “Once the influence weight α i t of each active agent c i ∈ C a t is obtained, we can derive the attentive representation [integrated representation information] of all active agents by xt = ReLU © - « Wc Õ i ∈C a t α i t o i t ⊕ a i t ⊕ p i t ª ® ¬ , (6) where Wc are learnable parameters”)
being determined according to the first integrated representation information, the second integrated representation information, and the third integrated representation information (Section 3.2.1, “To motivate the agents to make recommendations cooperatively, we devise the multi-agent actor-critic framework with a centralized attentive critic for deterministic policies learning. A similar MARL algorithm with CTDE architecture is proposed in [24], which incorporates the full state st and joint action ut of all agents into the critic to motivate agents to learn coordinated and cooperative policies. However, such an approach suffers from the large state and action space problem in our task. In practice, the EVs tend to go to nearby stations for charging. Based on this fact, given a charging request qt , we only activate the agents nearby qt (e.g., top-k nearest qt ) to take actions, denoted as C a t . We set other agents who are far away from qt inactive and don’t participate in the recommendation for qt (i.e., wherein a first, second, third subset are interpreted as a another portion of top-k agents from all agents based on the charging request received)”…Section 3.2.2, “Once the influence weight α i t of each active agent c i ∈ C a t is obtained, we can derive the attentive representation [integrated representation information] of all active agents by xt = ReLU © - « Wc Õ i ∈C a t α i t o i t ⊕ a i t ⊕ p i t ª ® ¬ , (6) where Wc are learnable parameters”)
Zhang does not explicitly teach:
wherein determining the second loss corresponding to the agent pooling module through the self-supervised contrastive learning method comprises: using a self-supervised contrastive learning loss as the second loss, the self- supervised contrastive learning loss
Khosla further teaches:
wherein determining the second loss corresponding to the agent pooling module through the self-supervised contrastive learning method comprises: (Section 3.2, “we now look at the family of contrastive losses [self-supervised contrastive learning], starting from the self-supervised domain and analyzing the options for adapting it to the supervised domain, showing that one formulation is superior. For a set of N randomly sampled sample/label pairs, {xk, yk}k=1...N , the corresponding batch used for training consists of 2N pairs, {x˜`, y˜`}`=1...2N , where x˜2k and x˜2k−1 are two random augmentations (a.k.a., “views”) of xk (k = 1...N) and y˜2k−1 = y˜2k = yk . For the remainder of this paper, we will refer to a set of N samples as a “batch” and the set of 2N augmented samples as a “multiviewed batch”)
using a self-supervised contrastive learning loss as the second loss, the self- supervised contrastive learning loss (Section 3.2, “we now look at the family of contrastive losses [self-supervised contrastive learning], starting from the self-supervised domain and analyzing the options for adapting it to the supervised domain, showing that one formulation is superior. For a set of N randomly sampled sample/label pairs, {xk, yk}k=1...N , the corresponding batch used for training consists of 2N pairs, {x˜`, y˜`}`=1...2N , where x˜2k and x˜2k−1 are two random augmentations (a.k.a., “views”) of xk (k = 1...N) and y˜2k−1 = y˜2k = yk . For the remainder of this paper, we will refer to a set of N samples as a “batch” and the set of 2N augmented samples as a “multiviewed batch””)
The motivation for claim 5 is the same motivation for claim 4.
Regarding claim 6 and analogous claim 13, 20:
Zhang, as modified by Khosla, teaches the method of claim 4.
Zhang further teaches:
wherein determining the first loss corresponding to the initial value network through the temporal difference method comprises: (Section 3.2.1, paragraph 3, “The centralized attentive critic Qb [initial value network] is updated by minimizing the following loss: L(θQ ) = Es a t ,u a t ,pt ,s a t+j ,pt+j ,Rt :t+j∼D h. (i.e., wherein a temporal difference method under the broadest reasonable interpretation (BRI) is interpreted as the loss over time ‘t:t+j’)”
determining, through a preset reward function, reward information according to a battery charging behavior of a charging object corresponding to the battery charging request, (Section 2, Definition 1, “Charging request. A charging request qt = ⟨lt , Tt ,T c t ⟩ ∈ Q is defined as the t-th request (i.e., step t) of a day. Specifically, lt is the location of qt [a battery charging request] (i.e., wherein under the broadest reasonable interpretation (BRI) a charging request is interpreted as receiving a request), Tt is the real-world time of the step t, and T c t is the real-world time qt finishes the charging request”…Section 3.1, “we propose a lazy reward [reward information] settlement scheme (i.e., return rewards when a charging request is finished), and integrate three goals into two natural reward functions. Specifically, if a charging request qt succeeds in charging, then the environment will return the negative of CWT and negative of CP as the part of reward r cw t (st ,ut ) and reward r cp (st ,ut ) respectively. For the case that the CWT of qt exceeds a threshold (i.e., wherein a preset reward), the recommendation will be regarded as failure and environment will return much smaller rewards as penalty to stimulate agents reducing the CFR”)
wherein each charging station in the charging station set shares the reward information, and the preset reward function provides a different reward for a different battery charging behavior; and (Section 3.1, Col 2, “For the case that the CWT of qt exceeds a threshold (i.e., wherein a preset reward), the recommendation will be regarded as failure and environment will return much smaller rewards as penalty to stimulate agents reducing the CFR”…“All agents in our model share [shares the reward information] the unified rewards, means agents make the recommendation decisions cooperatively”)
determining, through the temporal difference method, the first loss corresponding to the initial value network according to the cumulative reward expectation corresponding to the battery charging request, (Section 3.2.1, paragraph 3, “The centralized attentive critic Qb [initial value network] is updated by minimizing the following loss: L(θQ ) = Es a t ,u a t ,pt ,s a t+j ,pt+j ,Rt :t+j∼D h. (i.e., wherein a temporal difference method under the broadest reasonable interpretation (BRI) is interpreted as the loss over time ‘t:t+j’)”…Section 3.3, “which correspond to the expected returns of reward r cw t and r cp , respectively where E denotes the environment, and R cw t t:t+j is the cumulative discounted reward [cumulative reward expectation] (defined in Eq. (3)) with respect to r cw t”)
a reward corresponding to the battery charging request, and a cumulative reward expectation corresponding to a second battery charging request next to the battery charging request (Section 3.1, “we propose a lazy reward [reward information] settlement scheme (i.e., return rewards when a charging request is finished), and integrate three goals into two natural reward functions”…Section 3.3, “which correspond to the expected returns of reward r cw t and r cp , respectively where E denotes the environment, and R cw t t:t+j is the cumulative discounted reward [cumulative reward expectation] (defined in Eq. (3)) with respect to r cw t”)
The motivation for claim 6 is the same motivation for claim 4.
Claim(s) 7 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al., in view of Moura et al., (US20220188946A1)
Regarding claim 7 and analogous claim 14:
Zhang teaches the method of claim 1.
Zhang further teaches:
acquiring a new battery charging request; (Section 2, Definition 1, “Charging request. A charging request qt = ⟨lt , Tt ,T c t ⟩ ∈ Q is defined as the t-th request (i.e., step t) of a day. Specifically, lt is the location of qt [battery charging request] (i.e., wherein under the broadest reasonable interpretation (BRI) a charging request is interpreted as receiving a request, therefore, new requests are received), Tt is the real-world time of the step t, and T c t is the real-world time qt finishes the charging request.”)
determining the environment state information corresponding to each charging station in the charging station set; for each charging station in the charging station set, (Section 3.1, “Observation o i t . Given a charging request qt , we define the observation o i t [environment state information] of agent c i [each charging station] as a combination of the index of c i , the real-world time Tt , the number of current available charging spots of c i (supply), the number of charging requests around c i in the near future (future demand), the charging power of c i , the estimated time of arrival (ETA) from location lt to c i , (i.e., wherein the environment state information is interpreted as supply, future demand, and ETA.) and the CP of c i at the next ETA. We further define st = {o 1 t , o 2 t , . . . , o N t } [chagrining station set] as the state of all agents at step t.”)
determining, through the information adjustment model corresponding to each charging station, (Section 3.2.3, “The execution process is fully decentralized, by only invoking the learned actor policy with its own observation. Specifically, for a charging request qt , the agent c i ∈ C a t takes action a i t based on its o i t by a i t = b i (o i t ) (i.e., wherein the information adjustment model under the broadest reasonable interpretation (BRI) is interpreted as the decision maker based on the data). (11) And qt will be recommended to the active agent with the highest a i t among all the actions of C a t .”)
target operational information of each charging station for the new battery charging request according to the environment state information of the charging station, (“Action a i t . Given an observation o i t , an intuitional design for the action of agent ci is a binary decision, i.e., recommending qt to itself for charging or not. However, because one qt can only choose one station for charging, multiple agents’ actions may be tied together and are difficult to coordinate. Inspired by the bidding mechanism [47], we design each agent c i offers a scalar value to "bid" for qt as its action a i t [operational information] (i.e., wherein the bidding under the broadest reasonable interpretation is interpreted as an action to be taken, ‘bid’). By defining ut = {a 1 t , a 2 t , . . . , a N t } as the joint action, qt will be recommended to the agent with the highest "bid" value, i.e., rct = c i , where i = arg max(ut )”…Section 3.1, “Observation o i t . Given a charging request qt , we define the observation o i t [environment state information] of agent c i [each charging station] as a combination of the index of c i , the real-world time Tt , the number of current available charging spots of c i (supply), the number of charging requests around c i in the near future (future demand), the charging power of c i , the estimated time of arrival (ETA) from location lt to c i , (i.e., wherein the environment state information is interpreted as supply, future demand, and ETA.) and the CP of c i at the next ETA. We further define st = {o 1 t , o 2 t , . . . , o N t } [chagrining station set] as the state of all agents at step t.”)
wherein each charging station in the charging station set is configured to perceive the environment state information of each other, and (Section 3.2.2, “The entire process is shown in Figure 3(b). As the centralized attentive critic perceived more complete information (i.e., wherein the complete information is interpreted to include environment state information) of all active agents, it can motivate the agents to learn policies in a coordinated and cooperative way”)
Zhang does not explicitly teach:
displaying the target operational information of each charging station in the charging station set for the new battery charging request; and receiving a selection instruction and determining a target charging station from the charging station set according to the selection instruction.
Moura teaches:
displaying the target operational information of each charging station in the charging station set for the new battery charging request; and receiving a selection instruction and determining a target charging station from the charging station set according to the selection instruction ([0051], “each charging terminal 104 i may have a display screen 118 (i.e., wherein displaying)i, which can show information, such as charging time [target operational information] (i.e., wherein the target operational information is interpreted to display charging time etc.), ON, OFF, or the like, to a driver of a vehicle. The driver may interact with the charging system controller 150 through a downloadable native application or by accessing a website through his/her user device, e.g., a smartphone, tablet, personal computer connected to a hotspot, or the like (i.e., wherein under the broadest reasonable interpretation a driver may make a selection by ‘interacting’))
receiving a selection instruction and determining a target charging station from the charging station set according to the selection instruction ([0116], “u is a charging power for a given pricing option selected by an incoming user (i.e., wherein selected by is interpreted as a selection is received), Pr(M=flex) is a probability that the incoming user will select the charging-FLEX pricing option, fflex(zflex, yflex, uflex, v) is a function of a charging-FLEX profit of the charging-FLEX pricing option, zri is a per-unit price of the charging-FLEX pricing option, (i.e., wherein a user charges at a charging station with selected pricing option)”)
Moura and Zhang are both related to the same field of endeavor (i.e., EV charging). In view of the teachings of Moura it would have been obvious for a person of ordinary skill in the art to apply the teachings of Moura to Zhang before the effective filing date of the claimed invention in order to improve the customer experience at a EV charging station (Moura, [0038], “Pricing and/or carbon intensity of each option is updated in real time based on the time-varying cost of energy for both the site host and the electricity provider, maximum power constraints and/or demand charges, greenhouse gas emissions associated with electricity production, and charge point demand, with the objective of maximizing financial value for the charge point operator while meeting customer expectations for quality of service.”)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMINA BENOURAIDA whose telephone number is (571)272-4340. The examiner can normally be reached Monday-Friday 8:30am-5pm ET..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J. Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AMINA MORENO BENOURAIDA/ Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129