Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The office action is responsive to the amendment filed on 03/18/2026. As directed by the amendments claims 1, 3, 5-8, 10, 12-14, 16 and 19-20 are amended. Claims 1-20 are pending for examination.
Response to Arguments
Regarding objection to claim 5:
Applicant’s arguments, see pg. 11, filed 03/18/2026, with respect to claim 5 being objected to because of informalities have been fully considered and are persuasive. The objection of claim 5 has been withdrawn.
Regarding Rejections Under 35 US.C. § 112(a):
Applicant’s arguments, see pg. 11-12, filed 03/18/2026, with respect to claims 1-6, 8-12, and 14-19 under 35 US.C. § 112 (a) have been fully considered and are persuasive. The rejection of claims 1-6, 8-12, and 14-19 under 35 US.C. § 112 (a) has been withdrawn. However, applicant's arguments with respect to claims 7, 13, and 20 under 35 US.C. § 112 (a) have been fully considered but they are not persuasive. The rejection of claims 7, 13, and 20 under 35 US.C. § 112 (a) has not been withdrawn.
Regarding Rejections Under 35 US.C. § 112(b):
Applicant’s arguments, see pg. 12, filed 03/18/2026, with respect to claims 1-20 under 35 US.C. § 112 (b) have been fully considered and are persuasive. The rejection of claims 1-20 under 35 US.C. § 112 (b) has been withdrawn.
Regarding Rejections Under 35 US.C. § 101:
Applicant’s arguments, see pg. 12-15 filed 03/18/2026, with respect to claims 1-20 under 35 US.C. § 101 have been fully considered and are persuasive. The rejection of claims 1-20 under 35 US.C. § 101 has been withdrawn.
Regarding Rejections Under 35 US.C. § 103:
Applicant’s arguments with respect to claims 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
EXAMINER’S NOTE
Regarding Claim Interpretation Under 35 US.C. § 112(f), No arguments were presented. Thus, claims 1 and 3 remain interpreted under 35 US.C. § 112(f).
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“the path planning module is configured to” in claim 1, line 20.
“a reward module is configured to” in claim 3, line 1.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 7, 13, and 20 rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claims contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 7 recites “a motion sensing device configured to detect joint positions and movements”, however, the specification do not teach or suggest detecting joint position and movements by a motion sensing device. Rather, paragraph [0047] of the instant application teaches a “sensor module 102 receives spatio-temporal historical observations associated at least one element in an environment [where] ...In a skeletal scenario the elements may include limbs, joints”.
Claims 13 and 20 are similar to claim 7, therefore the rejection of claim 7 applies.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the following limitations:
PNG
media_image1.png
609
813
media_image1.png
Greyscale
However, it’s not clear how the components of the reinforced hybrid attention for motion forecasting are linked to each other. Specifically, it is unclear how the ranking information is being utilized to drive the motion prediction. The claim states that ranking information is being obtained by the “soft attention module” which is used by the “motion module” to generate “motion prediction” by machine learning model that includes two Long Short-Term Memory (LSTM) networks, however, the claims does not disclose any value being utilized from the ranking information by the LSTM networks which are the networks making the motion predictions. Further, it is unclear what is the relation or connection between agent state information and the environment information which was utilized to generate the ranking information and how the generated ranking is applicable of utilized with agent state. Also it is unclear how the motion module utilizes the ranking information for path planning as none of the information from ranking is being utilized in the motion module or has any relation with agent or any other elements of the motion module during the processing of state of the future. Therefore, it’s unclear what is the purpose/functionality of the ranking information, and how is utilized in the in the machine learning to generate the motion predictions. Furthermore, claim 1 also recites “...the second Long Short-Term Memory network takes in a complete node attribute”. It’s not clear what constitute a “complete node attribute” and it’s not clear how the “complete node attribute” is being obtained. Paragraph [0074] of the instant application teaches the two LSTM networks include a soft graph attention in between them and [0071-72] teaches the soft graph attention generates the “complete node attribute” by concatenating the self-attribute and neighbor attribute obtained from the first LSTM which is then by utilized by the second LSTM to predict change in state. Therefore, examiner interprets the “complete node attribute” as being generated by the soft attention module/graph attention mechanism with the first LSTM outputs.
Independent claims 8 and 14 recites similar limitation as claim 1, therefore are rejected for reasons set forth in the rejection of claim 1.
Claims 2-7,9-13 and 15-20 are dependent on claims 1, 8, and 14, and thus are rejected for reasons set forth in the rejection of claims 1, 8, and 14.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6, 8-12 and 14-19 are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al. “Attention-based Hierarchical Deep Reinforcement Learning for Lane Change Behaviors in Autonomous Driving” (hereinafter Chen) as cited in the PTO-892 dated 08/21/2024 in view of Sorokin et al. Deep Attention Recurrent Q-Network (hereinafter Sorokin) as cited in the PTO-892 dated 08/07/2025 in further view of Huang et al. STGAT: Modeling Spatial-Temporal Interactions for Human Trajectory Prediction as cited in Information Disclosure Statement (IDS) dated 04/16/2021 and Naghshvar et al. US 2020/0150672 A1 (hereinafter Naghshvar).
Regarding claim 1:
A system for reinforced hybrid attention for motion forecasting, the system comprising: ( Chen pg. 1 , Abstract, lines 1-16, teaches predicting motions for autonomous driving “ we design a hierarchical Deep Reinforcement Learning (DRL) algorithm to learn lane change behaviors in dense traffic. By breaking down overall behavior to sub policies, faster and safer lane change actions can be learned. We also apply temporal and spatial attention to the DRL architecture, which helps the vehicle focus more on surrounding vehicles and leads to smoother lane change behavior”).
a processor and a memory, the processor being configured to execute a machine learning model including a attention module, a soft attention module, and a motion module, wherein the machine learning model is implemented using one or more trained neural networks; and ( Chen pg. 5 sec. 4. Experiments, teaches evaluating the algorithm in the TORCS simulator. Someone ordinary skilled in the art will appreciate that in order to perform this experiments a computer with a processor and a memory will be required. Further, Chen pg. 3: sec 3.2 Actor Critic Based DRL Architecture, para. 1 teaches performing training with two neural networks. Furthermore, Chen pg. 4, Fig. 4 teaches an “Attention Mechanism (i.e., attention module), and pg. 5, sec. 3.3.3 Spatial Attention, paras. 1-6, lines 8-25, teaches a soft version of attention (i.e., soft attention module) and pg. 5, sec. 3.3.3 Spatial Attention, para. 1, lines 1-15, teaches adding spatial attention to a model to better predict motions based on the images (i.e., motion module)).
a path planning module, wherein ( Chen pg. 2-3,sec. 3.1. Hierarchical Action Space for Lane Change and Fig. 2, teaches an agent (i.e., path planning module)).
the attention module is configured to select information from the spatio-temporal historical observations associated with the at least one element based on a reinforcement learning model, and wherein the at least one element and its relationships with other elements in the environment are represented as nodes and edges in an inferred graph, and the attention module selects relevant edges as the selected information in the inferred graph based on the reinforcement learning model, (Chen pg. 4, Fig. 4 teaches an “Attention Mechanism” and in pg. 3-4, para. 6, lines 1-21, Chen introduces an “Attention Mechanism” in their deep reinforcement learning algorithm shaped after humans drives and the way they consider a “series of historical observation to make driving decision”. The attention mechanism is further modeled to “weight importance according to the time and location of the observations”. In addition, Chen pg. 2-3, sec. 3.1. Hierarchical Action Space for Lane Change and Fig. 2 also teaches a graph with hierarchical actions placed for lane change. Where each “actions” can be seen as nodes/elements and each “lines” can be view as edge/relationship in the environment).
the soft attention module is configured to generate ranked information by applying attention weights to the selected edges, (Chen pg.5, sec. 3.3.3 Spatial Attention, paras. 1-6, lines 8-25, teaches “a soft version of attention” and pg. 4, para. 1, lines 4-6 implies the ability to generate the ranked information “ The Spatial Attention detects the most important and relevant regions in the image for driving”. Further, Chen pg. 4, para. 5, lines 1-27, and Fig. 4 teaches temporal attention mechanism depended on information with weights, “The temporal attention mechanism learns scalar weights for LSTM outputs at different time steps. The weight of each LSTM output
w
i
is defined as an inner product of the feature vector and LSTM hidden vector,
v
i
followed by a softmax function to normalize the sum of weights to 1. By this definition, each learned weight is dependent on the previous time step’s information and current state information”. Further, it would be obvious to one ordinary skill in the art to deduce that a “temporal attention mechanism” inherently uses attention weights as they are key part of the attention mechanism).
the motion module is configured to generate motion predictions based on the ranked information such that the motion predictions are generated by machine learning... ( Chen pg. 5, sec. 3.3.3 Spatial Attention, para. 1, lines 1-15 teaches adding spatial attention to a model (motion module) to better predict motions based on the images “Human drivers pay special attention to certain objects when performing specific tasks. For example, drivers care more about the direction of the road when traversing curvy roads and pay more attention to surrounding vehicles when making lane changes. This intuition can be enforced in the algorithm by adding importance to different locations in the image, which we call spatial attention”, this suggest the motion prediction are generated by the machine learning model that was used. Further, Chen pg. 7, para. 4, lines 1-12, Chen teaches how “The attention model gives large weight to the region where the neighboring vehicle appears, which helps the agent figure out when and how to launch the proper lane change behavior...By learning a mask over the input image...the agent can extract the relevant context to the task and learn the proper behavior more efficiently”).
the path planning module is configured to plan a path for a primary element based on the motion predictions ( Chen pg. 2-3, sec. 3.1. Hierarchical Action Space for Lane Change and Fig. 2 teaches an agent [path planning module] that for each of the steps in the “Illustration of the hierarchical action space for lane change behavior...must choose one of the three high-level actions to execute”. To add Chen, pg. 2, right column, para. 2-3, teaches the proposed attention mechanism automatically focus on feasible paths or surrounding on-road vehicles that may influence the driving behavior and teaches the proposed method “can be easily extended to learn multiple driving policies in one model ...This helps improve compositionality of network: learn better performance with fewer examples”).
Chen does not explicitly teaches a “hard attention module” and does not teach a sensor module, having at least one sensor, configured to receive real-time spatio- temporal historical observations associated with at least one element in an environment; the motion predictions being generated by machine learning that comprises a first Long Short-Term Memory network and a second Long Short-Term Memory network, the first Long Short-Term Memory network takes in agent state information and outputs a self-attribute, and the second Long Short-Term Memory network takes in a complete node attribute and outputs a predicted change in state, which is used to calculate a state at a future time.
Nonetheless Sorokin teaches the following:
a processor and a memory, the processor being configured to execute a machine learning model including a hard attention module, a soft attention module, wherein the machine learning model is implemented using one or more trained neural networks; and (Sorokin pg. 2, sec: Deep Attention Recurrent Q-Network, para. 1, teaches a Deep Attention Recurrent Q-Network (DARQN) architecture that comprises three types of networks including attention networks (i.e., hard attention module, a soft attention module).
the hard attention module is configured to select information from the observations associated with the at least one element based on a reinforcement learning model,... (Sorokin pg. 3, sec: Hard Attention, para. 1, teaches the “hard attention mechanism requires sampling only one attention location from available at
L
each time step in accordance with some stochastic attention policy... this policy is represented by the neural network whose output (1) consists of location selection probabilities g and whose weights are the policy parameters”, this implies the hard attention mechanism (i.e., hard attention module) selects information from the observed environment based on neural network (i.e., reinforcing learning model)).
the soft attention module is configured to generate ranked information by applying attention weights
Sorokin is also in the same field of endeavor as Chen (machine learning). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the functionality of a soft and hard attention mechanism and a policy function that is trained using reinforcement learning and is iteratively updated as being disclosed and taught by Sorokin, in the system taught by Chen to yield the predictable results of providing an attention-based algorithm that enables to gain insights into the logic of agent’s behavior ( Sorokin pg. 6, sec: 4 Conclusion and Future Work, para. 1).
Neither Chen and Sorokin teach a sensor module, having at least one sensor, configured to receive real-time spatio- temporal historical observations associated with at least one element in an environment; the motion predictions being generated by machine learning that comprises a first Long Short-Term Memory network and a second Long Short-Term Memory network, the first Long Short-Term Memory network takes in agent state information and outputs a self-attribute, and the second Long Short-Term Memory network takes in a complete node attribute and outputs a predicted change in state, which is used to calculate a state at a future time.
Nonetheless, Huang teaches the following:
the soft attention module is configured to generate ranked information by applying attention weights to the selected edges, (Huang pg. 6273, left colm, para. 1 teaches a graph attention network (GAT) ( i.e., soft attention module) that model complex interactions and pg. 6278, left colm, para. 3, teaches GAT that assigns attention weights to “neighbors” (i.e., edges) by their motion status).
the motion module is configured to generate motion predictions based on the ranked information such that the motion predictions are generated by machine learning that comprises a first Long Short-Term Memory network and a second Long Short-Term Memory network, the first Long Short-Term Memory network takes in agent state information and outputs a self-attribute, and the second Long Short-Term Memory network takes in a complete node attribute and outputs a predicted change in state, which is used to calculate a state at a future time, and (Huang Abstract teaches a Spatial-Temporal Graph Attention network (STGAT), based on a sequence-to-sequence architecture to predict future trajectories of pedestrians and Fig. 2 teaches the proposed architecture is based on machine learning that consist of an encoder module that include three components, that is 2 types Long-Short Term Memory networks (LSTM) and a Graph Attention Network (GAT) in between them (see section of Fig. 2 below).
PNG
media_image2.png
342
1062
media_image2.png
Greyscale
In addition, pg. 6273, left col. sec: 3.2, para. 1 & left colm. para. 1-3 teaches using “one LSTM for each pedestrian to get the motion state” which is denoted by M-LSTM (LSTM for motion encoding) and it takes in a pedestrian relative position (i.e., agent state information) denoted by function (2) and outputs the hidden state of the M-LSTM at time-step t (i.e., a self attribute) denoted by function 3. Further, pg. 6274, right colm., para. 1-3, teaches using “another LSTM to model the temporal correlations between interactions explicitly. We term this LSTM as G-LSTMTM” takes in a complete node attribute denotes as
m
^
i
t
and outputs a predicted change in state, which is used to calculate a state at a future time denoted as
g
i
T
o
b
s
. Furthermore, Huang pg. 6274, right colm., sec: 3.5 Future Trajectory Prediction, para. 3-4 teaches a decoder LSTM termed as D-LSTM that is implemented to calculate a state at a future time).
Huang is also in the same field of endeavor as Chen and Sorokin (machine learning). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the functionality of a soft graph attention module, two LSTM and a path planning module as being disclosed and taught by Huang, in the system taught by Chen and Sorokin to yield the predictable results of significantly improves the prediction accuracy compared to other baseline methods as shown in pg. 6276 Table 1.
Chen, Sorokin and Huang do not teach or suggest a sensor module, having at least one sensor, configured to receive real-time spatio- temporal historical observations associated with at least one element in an environment;
Nevertheless, Naghshvar teaches the following:
a sensor module, having at least one sensor, configured to receive real-time spatio- temporal historical observations associated with at least one element in an environment; (Naghshvar [0023] teaches “One or more sensors, such as a light detection and ranging (LIDAR) sensor, a radio detection and ranging (RADAR) sensor, a camera, and/or another type of sensor, may observe the surrounding environment” & [0080] teaches “The autonomous driving system 700 may include a mapping and localization module. The mapping and localization module may receive location information from a location sensor, such as a global positioning system (GPS) sensor”. Furthermore to clarify, [0081] teaches the “autonomous driving system 700 may also include a sensor fusion and road world model (RWM)... the sensor fusion and road world model module receive information obtained from the vehicle's sensors, such as a vision sensor (e.g., camera) and a RADAR sensor”, as would be familiar to one skilled in the art, vision sensor such as camera are used to receive and process observation/information in real-time).
Naghshvar is also in the same field of endeavor as Chen, Sorokin and Huang (machine learning). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the functionality of a sensor module, as being disclosed and taught by Naghshvar, in the system taught by Chen, Sorokin and Huang to yield the predictable results of “improve reinforcement learning systems”(Naghshvar [0005]).
Regarding claim 2:
Chen, Sorokin, Huang and Naghshvar teaches The system of claim 1. Naghshvar specifically teaches wherein a number of elements of the at least one element is unbounded (Naghshvar [0080-0081] does not pose a limit to the amount of elements collected from the sensor “The autonomous driving system 700 may include a mapping and localization module. The mapping and localization module may receive location information from a location sensor, such as a global positioning system (GPS) sensor and/or an inertial measurement unit (IMU) sensor”).
Regarding claim 3:
Chen, Sorokin, Huang and Naghshvar teach The system of claim 1. Naghshvar teaches further comprising a reward module configured to generate rewards for the motion predictions based on performance metrics of an action of an element (Naghshvar, Abstract, teaches “using a model-based solution based on a reward” and [0089] teaches “Rewards may be determined for actions. A positive reward may be provided for keeping a safe distance, approaching a maximum desired speed, and successfully executing high-level options, such as a lane change or merge. A negative reward may be provided for increasing risk (relative distance to lead vehicle < safe distance) or passenger discomfort (high frequency of lane changes, high jerk, high lateral acceleration, etc.). The negative reward may also be provided if a motion planner cannot execute high-level options”).
Regarding claim 4:
Chen, Sorokin, Huang and Naghshvar teach The system of claim 3. Naghshvar specifically teaches wherein the reinforcement learning model determines relevancy of the information of the spatio-temporal historical observations based on the rewards (Naghshvar [0024] teaches “Reinforcement learning may be used to train an agent to perform tasks in an environment. The agent may transition between different scenarios of the environment, referred to as states, by performing actions. Actions, in return, yield rewards, which may be positive, negative, or zero. The agent may be trained to maximize a total reward obtained between an initial state and a terminal state. That is, the agent is reinforced to perform certain actions by providing positive rewards, and to stray away from others by providing negative rewards. The positive and negative rewards”).
Regarding claim 5:
Chen, Sorokin, Huang and Naghshvar teach The system of claim 1. Chen teaches wherein the system is trained with an alternating training strategy using both the reinforcement learning model and gradient based back propagation (Chen pg. 3, 3rd- 4th paragraph, lines 1-43 teaches “a deep reinforcement learning algorithm for continuous control” with a “back-propagation stage”. Further, Fig. 3, show the “The Actor-Critic architecture used in deep reinforcement learning” and “on the right is the gradient flow during training (back-propagation)”).
Regarding claim 6:
Chen, Sorokin, Huang and Naghshvar teach The system of claim 1. Naghshvar teaches wherein the environment is a roadway, and wherein the at least one element includes a plurality of agents traveling the roadway, (Naghshvar teaches [0023] “The surrounding environment may include dynamic objects, such as autonomous agents and non-autonomous agents. The surrounding environment may also include static objects, such as roads and buildings”).
Regarding claim 8:
Chen teaches the following A computer-implemented method for reinforced hybrid attention for motion forecasting, the computer-implemented method comprising: ( Chen pg. 1 , Abstract, lines 1-16 teaches predicting motions for autonomous driving “we design a hierarchical Deep Reinforcement Learning (DRL) algorithm to learn lane change behaviors in dense traffic. By breaking down overall behavior to sub policies, faster and safer lane change actions can be learned. We also apply temporal and spatial attention to the DRL architecture, which helps the vehicle focus more on surrounding vehicles and leads to smoother lane change behavior”).
selecting, by way of a processor and a memory wherein the processor is configured to execute a machine learning model implemented using one or more trained neural networks, information from the spatio-temporal historical observations associated with the at least one element based on a reinforcement learning model, wherein the at least one element and its relationships with other elements in the environment are represented as nodes and edges in an inferred graph, and a attention module of the processor selects relevant edges as the selected information in the inferred graph based on the reinforcement learning model; (Chen pg. 5, sec. 4. Experiments, para. 1 teaches evaluating the algorithm in the TORCS simulator. Someone ordinary skilled in the art will appreciate that in order to perform this experiments a computer with a processor and a memory will be required. Further, Chen pg. 3, sec. 3.2 Actor Critic Based DRL Architecture, para. 1 teaches performing training with two neural networks. Furthermore, Chen pg. 4, Fig. 4 teaches an “Attention Mechanism” and in (pg. 3-4, para. 6, lines 1-21) Chen introduces an “Attention Mechanism” in their deep reinforcement learning algorithm shaped after humans drives and the way they consider a “series of historical observation to make driving decision”. The mechanism is further modeled to “weight importance according to the time and location of the observations”. In addition, to further clarify, Chen pg. 2-3, sec. 3.1. Hierarchical Action Space for Lane Change and Fig. 2 also teaches a graph with hierarchical actions placed for lane change. Where each “actions” can be seen as nodes/elements and each “lines” can be view as edge/relationship in the environment).
generating, by way of the processor, ranked information by applying attention weights to the selected edges; (Chen pg. 5, sec. 4. Experiments teaches evaluating the algorithm in the TORCS simulator. This implies a computer with a processor will be required. Further, Chen pg. 4, para. 1, lines 4-6 implies the ability to generate the ranked information “The Spatial Attention detects the most important and relevant regions in the image for driving”. Further, Chen pg. 4, para. 5, lines 1-27, Fig. 4 teaches temporal attention mechanism depended on information with weights, “The temporal attention mechanism learns scalar weights for LSTM outputs at different time steps. The weight of each LSTM output
w
i
is defined as an inner product of the feature vector and LSTM hidden vector,
v
i
followed by a softmax function to normalize the sum of weights to 1. By this definition, each learned weight is dependent on the previous time step’s information and current state information”. Further, it would be obvious to one ordinary skill in the art to deduce that a “temporal attention mechanism” inherently uses attention weights as they are key part of the attention mechanism).
generating, by way of the processor, motion predictions based on the ranked information such that the motion predictions are generated by machine learning that comprises; ( Chen pg. 5, sec. 4. Experiments teaches evaluating the algorithm in the TORCS simulator. This implies a computer with a processor will be required. Further, Chen pg. 5, para. 1, lines 1-15, teaches adding spatial attention to a model to better predict motions based on the images “Human drivers pay special attention to certain objects when performing specific tasks. For example, drivers care more about the direction of the road when traversing curvy roads and pay more attention to surrounding vehicles when making lane changes. This intuition can be enforced in the algorithm by adding importance to different locations in the image, which we call spatial attention” this suggest the motion prediction are generated by the machine learning model that was used. Further, Chen pg. 7, para. 4, lines 1-12 teaches how “The attention model gives large weight to the region where the neighboring vehicle appears, which helps the agent figure out when and how to launch the proper lane change behavior... By learning a mask over the input image...the agent can extract the relevant context to the task and learn the proper behavior more efficiently”).
planning a path for a primary element based on the motion predictions ( Chen pg. 2-3, sec. 3.1. Hierarchical Action Space for Lane Change and Fig. 2 teaches an agent [path planning module] that for each of the steps in the “Illustration of the hierarchical action space for lane change behavior...must choose one of the three high-level actions to execute”. To add Chen, pg. 2, right column, para. 2-3 teaches the proposed attention mechanism automatically focus on feasible paths or surrounding on-road vehicles that may influence the driving behavior and teaches the proposed method “can be easily extended to learn multiple driving policies in one model ...This helps improve compositionality of network: learn better performance with fewer examples”).
Chen does not explicitly teaches receiving, through at least one sensor, real-time spatio-temporal historical observations associated with at least one element in an environment; a “hard attention module” configure to make edge selections and motion predictions being generated by machine learning that comprises; determining by a first Long Short-Term Memory network a self-attribute based on agent state information; determining by a second Long Short-Term Memory network a predicted change in state based on a complete node attribute; determining a predicted state at a future time based on the predicted change in state;
Nonetheless, Sorokin teaches the following:
...a hard attention module of the processor selects relevant edges as the selected information in the inferred graph based on the reinforcement learning model; (Sorokin pg. 3, sec: Hard Attention, para. 1, teaches the “hard attention mechanism requires sampling only one attention location from available at
L
each time step in accordance with some stochastic attention policy... this policy is represented by the neural network whose output (1) consists of location selection probabilities g and whose weights are the policy parameters”, this implies the hard attention mechanism (i.e., hard attention module) selects information from the observed environment based on neural network (i.e., reinforcing learning model)).
Sorokin is also in the same field of endeavor as Chen (machine learning). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the functionality of a soft and hard attention mechanism as being disclosed and taught by Sorokin, in the system taught by Chen to yield the predictable results of providing an attention-based algorithm that enables to gain insights into the logic of agent’s behavior ( Sorokin pg. 6, sec: 4 Conclusion and Future Work, para. 1).
Neither Chen or Sorokin teach receiving, through at least one sensor, real-time spatio-temporal historical observations associated with at least one element in an environment; and motion predictions being generated by machine learning that comprises; determining by a first Long Short-Term Memory network a self-attribute based on agent state information; determining by a second Long Short-Term Memory network a predicted change in state based on a complete node attribute; determining a predicted state at a future time based on the predicted change in state;
Nonetheless, Huang teaches the following:
generating, by way of the processor, motion predictions based on the ranked information such that the motion predictions are generated by machine learning that comprises; (Huang Abstract teaches a Spatial-Temporal Graph Attention network (STGAT), based on a sequence-to-sequence architecture to predict future trajectories of pedestrians and Fig. 2 teaches the proposed architecture is based on machine learning that consist of an encoder module that include three components, that is 2 types Long-Short Term Memory networks (LSTM) and a Graph Attention Network (GAT) in between them).
determining by a first Long Short-Term Memory network a self-attribute based on agent state information; (Huang pg. 6273, left col. sec: 3.2, para. 1 & left colm. para. 1-3 teaches using “one LSTM for each pedestrian to get the motion state” which is denoted by M-LSTM (LSTM for motion encoding) and it takes in a pedestrian relative position (i.e., agent state information) denoted by function (2) and outputs the hidden state of the M-LSTM at time-step t (i.e., a self-attribute) denoted by function 3).
determining by a second Long Short-Term Memory network a predicted change in state based on a complete node attribute; (Huang pg. 6274, right colm., para. 1-3, teaches using “another LSTM to model the temporal correlations between interactions explicitly. We term this LSTM as G-LSTMTM” takes in a complete node attribute denotes as
m
^
i
t
and outputs a predicted change in state, which is used to calculate a state at a future time denoted as
g
i
T
o
b
s
).
determining a predicted state at a future time based on the predicted change in state; (Huang pg. 6274, right colm., sec: 3.5 Future Trajectory Prediction, para. 3-4 teaches a decoder LSTM termed as D-LSTM that is implemented to predicted state at a future time based on the predicted change in state).
Chen, Sorokin and Huang do not teach or suggest a sensor module, having at least one sensor, configured to receive real-time spatio- temporal historical observations associated with at least one element in an environment;
Nevertheless, Naghshvar teaches the following:
a sensor module, having at least one sensor, configured to receive real-time spatio- temporal historical observations associated with at least one element in an environment; (Naghshvar [0023] teaches “One or more sensors, such as a light detection and ranging (LIDAR) sensor, a radio detection and ranging (RADAR) sensor, a camera, and/or another type of sensor, may observe the surrounding environment” & [0080] teaches “The autonomous driving system 700 may include a mapping and localization module. The mapping and localization module may receive location information from a location sensor, such as a global positioning system (GPS) sensor”. Furthermore to clarify, [0081] teaches the “autonomous driving system 700 may also include a sensor fusion and road world model (RWM)... the sensor fusion and road world model module receive information obtained from the vehicle's sensors, such as a vision sensor (e.g., camera) and a RADAR sensor”, as would be familiar to one skilled in the art, vision sensor such as camera are used to receive and process observation/information in real-time).
Naghshvar is also in the same field of endeavor as Chen, Sorokin and Huang (machine learning). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the functionality of a sensor module, as being disclosed and taught by Naghshvar, in the system taught by Chen, Sorokin and Huang to yield the predictable results of “improve reinforcement learning systems”(Naghshvar [0005]).
Regarding claim 9: is rejected under the same rational of claim 2.
Regarding claim 10: is rejected under the same rational of claim 3.
Regarding claim 11: is rejected under the same rational of claim 4.
Regarding claim 12: is rejected under the same rational of claim 6.
Regarding claim 14: is rejected under the same rational of claim 8. Claim 14 only recites the additional element of A non-transitory computer readable storage medium storing instructions that when executed by a computer having a processor perform a method for reinforced hybrid attention for motion forecasting, the method comprising... for which Naghshvar [0116] teaches a computer-readable medium for hybrid reinforcement learning in accordance with certain
aspects of the disclosure.
Regarding claim 15: is rejected under the same rational of claim 2.
Regarding claim 16: is rejected under the same rational of claim 3.
Regarding claim 17: is rejected under the same rational of claim 4.
Regarding claim 18:
Chen, Sorokin, Huang and Naghshvar teach The non-transitory computer readable storage medium of claim 14. Naghshvar specifically teaches wherein selecting the edges from the spatio-temporal historical observations includes identifying relevant observations from the spatio-temporal historical observations and discarding remaining spatio-temporal historical observations ( Naghshvar [0057] teaches “A model-free reinforcement learning solution does not make an assumption on the dynamics. That is, the model-free reinforcement learning solution is trained using available data and makes decisions without explicitly determining a transition model...The model-free reinforcement learning solution may be referred to as a model-free solution or a model-free neural network” and [0061] teaches “At any state, the model-free neural network is used to identify the top N actions. The best action may be selected by comparing the N actions via a model-based solution using the known dynamics...The action refers to the next action of an autonomous device. For example, an action may refer to a specific movement of an autonomous vehicle (e.g., turn right)”).
Regarding claim 19: is rejected under the same rational of claim 6.
Claim 7, 13 and 20 is rejected under 35 U.S.C. 103 as being unpatentable over Chen, Sorokin, Huang and Naghshvar in further view of Ionescu et al. “Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments” (hereinafter Ionescu).
Regarding claim 7:
Chen, Sorokin, Huang and Naghshvar teach The system of claim 1.
Neither Chen, Sorokin, Huang and Naghshvar teach wherein the spatio-temporal historical observations are based on human skeleton motions of a human captured by a motion sensing device configured to detect joint positions and movements, and wherein the at least one element includes at least one joint of the human, and wherein the motion predictions are used for recognizing human activities or generating synthetic human motion sequences.
Nevertheless, Ionescu teaches the following:
wherein the spatio-temporal historical observations are based on human skeleton motions of a human captured by a motion sensing device configured to detect joint positions and movements, and wherein the at least one element includes at least one joint of the human ( Ionescu, Abstract teaches Human3.6M dataset used for pose estimation and motion prediction. Further, pg. 1327, sec: 2.1 Experimental Setting teaches how the data was captures using 15 sensors (4 digital video cameras, 1 time-of-flight sensor, 10 motion cameras), for which one skilled in the art will recognize any of the sensor described are motion sensing devices. In addition, pg. 1328, left col., sec: Joint Positions and Joint Angle Skeleton Representations, para. 1, teaches how the dataset includes joints of the humans which can also been seen in Fig. 8 Illustration of additional annotations in our dataset. Lastly, pg. 1330, right col., lines 26-32 and Fig. 6 teaches how Human3.6M can be used for motion synthesis).
Ionescu is also in the same field of endeavor as Chen, Sorokin, Huang and Naghshvar (Machine Learning). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the functionality of human based skeleton motion collected by motion sensing devices as being disclosed and taught by Ionescu, in the system taught by Chen, Sorokin, Huang and Naghshvar to yield the predictable results of to capture human skeleton data for motion prediction and to “stimulate further research in computer vision, machine learning,” to help improve “development of improved 3D human sensing systems that can operate robustly in the real world” (Ionescu pg. 1337, left col., para. 1).
Regarding claim 13: is rejected under the same rational of claim 7.
Regarding claim 20: is rejected under the same rational of claim 7.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Wang el at. A Soft Graph Attention Reinforcement Learning for Multi-Agent Cooperation (2020).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GISEL G FACCENDA whose telephone number is (703)756-1919. The examiner can normally be reached Monday - Friday 8:00 am - 4:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Al Kawsar can be reached at (571) 270-3169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/G.G.F./Examiner, Art Unit 2127
/ABDULLAH AL KAWSAR/Supervisory Patent Examiner, Art Unit 2127
Prosecution Insights