DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1-20 of US Application No. 19/084,329, filed on 03/19/2025, are currently pending and have been examined.
Information Disclosure Statement
The information Disclosure Statement filed on 04/08/2025 has been considered. An initialed copy of form 1449 is enclosed herewith.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 3-11, and 13-15 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1
Claim 1 is directed towards a system for causal trajectory prediction. Claim 11 is directed towards a computer-implemented method for causal trajectory prediction.
Step 2A, Prong 1
A claim that recites an abstract idea, a law of nature, or a natural phenomenon is directed to a judicial exception. Abstract ideas include the following groupings of subject matter, when recited as such in a claim limitation: (a) Mathematical concepts – mathematical relationships, mathematical formulas or equations, mathematical calculations; (b) Certain methods of organizing human activity – fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions); and (c) Mental processes – concepts performed in the human mind (including an observation, evaluation, judgment, opinion). See the 2019 Revised Patent Subject Matter Eligibility Guidance.
In the instant application, independent claim 1 recites:
“…generating a sparsified causal graph including two or more nodes and two or more edges, wherein a node of the two or more nodes represents an agent of one or more agents within an environment and wherein an edge of the two or more edges between a first node and a second node represents a causal relationship between the first node and the second node…
…generating one or more agent future features based on the sparsified causal graph and an encoder…
…generating a trajectory prediction for a target agent based on the one or more agent future features and a decoder.”
Independent claim 11 recites substantially similar limitations.
These claim limitations, when given their broadest reasonable interpretation, may be performed in the human mind. Therefore these limitations are abstract ideas and claims 1 and 11 are directed to a judicial exception.
Step 2A, Prong 2
Even when a judicial element is recited in the claim, an additional claim element(s) that integrates the judicial exception into a practical application of that exception renders the claim eligible under §101. A claim that integrates a judicial exception into a practical application will apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception. The following examples are indicative that an additional element or combination of elements may integrate the judicial exception into a practical application:
the additional element(s) reflects an improvement in the functioning of a computer, or an improvement to other technology or technical field;
the additional element(s) that applies or uses a judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition;
the additional element(s) implements a judicial exception with, or uses a judicial exception in conjunction with, a particular machine or manufacture that is integral to the claim;
the additional element(s) effects a transformation or reduction of a particular article to a different state or thing; and
the additional element(s) applies or uses the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception.
Examples in which the judicial exception has not been integrated into a practical application include:
the additional element(s) merely recites the words ‘‘apply it' ' (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea;
the additional element(s) adds insignificant extra-solution activity to the judicial exception; and
the additional element does no more than generally link the use of a judicial exception to a particular technological environment or field of use.
See the 2019 Revised Patent Subject Matter Eligibility Guidance.
In the instant application, claims 1 and 11 do not recite additional elements that integrate the judicial exception into a practical application of that exception. Claim 1 recites “a processor” at a high level. The specification identifies the processor generically as a general processor, i.e., “‘processor’, as used herein, processes signals and performs general computing and arithmetic functions. (See instant Specification ¶ [0015]). The processor(s) is merely a computer used as a tool to perform the abstract idea. Claim 1 further recites a “memory storing one or more instructions” generically, i.e., “‘memory’, as used herein, may include volatile memory and/or non-volatile memory. Non-volatile memory may include, for example, ROM (read only memory), PROM (programmable read only memory), EPROM (erasable PROM), and EEPROM (electrically erasable PROM). Volatile memory may include, for example, RAM (random access memory), synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), and direct RAM bus RAM (DRRAM).” (See instant Specification ¶ [0016]) These elements are not meaningful limitations on the judicial exception. The processor and memory are recited so generically (no details whatsoever are provided other than that they are a processor and memory) that they represent no more than mere instructions to apply the judicial exception on a computer. These limitations can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of a computer. It should be noted that because the courts have made it clear that mere physicality or tangibility of an additional element or elements is not a relevant consideration in the eligibility analysis, the physical nature of these computer components does not affect this analysis. See MPEP 2106.05(I) for more information on this point, including explanations from judicial decisions including Alice Corp. Pty. Ltd. v. CLS Bank Int'l, 573 U.S. 208, 224-26 (2014). Therefore, claims 1 and 11 do not recite additional elements that integrate the judicial exception into a practical application of that exception.
Step 2B
Finally, even when a judicial element is recited in the claim, an additional claim element(s) that amounts to significantly more than the judicial exception renders the claim eligible under §101. Examples that are not enough to amount to significantly more than the abstract idea include 1) mere instructions to implement the abstract idea on a computer, 2) simply appending well-understood, routine and conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, e.g., a claim to an abstract idea requiring no more than a generic computer to perform generic computer functions that are well understood, routine and conventional activities previously known to the industry, 3) adding insignificant extra-solution activity to the judicial exception, and 4) generally linking the use of the judicial exception to a particular technological environment or field of use are not enough to amount to significantly more than the abstract idea. Examples of generic computing functions that are not enough to amount to significantly more than the abstract idea include 1) performing repetitive calculations, 2) receiving, processing, and storing data, 3) electronically scanning or extracting data from a physical document, 4) electronic recordkeeping, 5) automating mental tasks, and 6) receiving or transmitting data over a network, e.g., using the Internet to gather data.
In the instant application, claims 1 and 11 do not include additional elements that are sufficient to amount to significantly more than the judicial exception. In this particular application, the same analysis above in determining whether the recited additional elements integrate the judicial exception into a practical application of that exception is applicable to determine if the additional elements amount to significantly more than the judicial exception.
Based on the above analysis, claims 1 and 11 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claims 3 and 13 recite: “…wherein the processor generates the sparsified causal graph based on regularized Bernoulli distribution.” Which further defines an abstract idea identified above. Which further defines an abstract idea identified above. In the instant application claim 3 recites “the processor”. However, as in claim 1 the processor is disclosed at a high level of generality. Therefore, “the processor” is no more than a generic computing element that is performing a generic computing activity. Thus, claim 3 does not recite elements that integrate the judicial exception into a practical application of that exception or amount to significantly more than the judicial exception.
Claims 4 and 14 recite: “…wherein the processor generates the sparsified causal graph based on an entmax function or a softmax function.” Which further defines an abstract idea identified above. In the instant application claim 4 recites “the processor”. However, as in claim 1 the processor is disclosed at a high level of generality. Therefore, “the processor” is no more than a generic computing element that is performing a generic computing activity. Thus, claim 4 does not recite elements that integrate the judicial exception into a practical application of that exception or amount to significantly more than the judicial exception.
Claim 5 recites additional abstract ideas that may be performed mentally, i.e., “…wherein the processor generates a coarse trajectory prediction for one or more of the agents within the environment based on one or more of the agent future features.” In the instant application claim 5 recites “the processor”. However, as in claim 1 the processor is disclosed at a high level of generality. Therefore, “the processor” is no more than a generic computing element that is performing a generic computing activity. Thus, claim 5 does not recite elements that integrate the judicial exception into a practical application of that exception or amount to significantly more than the judicial exception.
Claim 6 recites additional abstract ideas that may be performed mentally, i.e., wherein the processor generates the trajectory prediction for the target agent based on the coarse trajectory prediction for one or more of the agents.” In the instant application claim 6 recites “the processor”. However, as in claim 1 the processor is disclosed at a high level of generality. Therefore, “the processor” is no more than a generic computing element that is performing a generic computing activity. Thus, claim 6 does not recite elements that integrate the judicial exception into a practical application of that exception or amount to significantly more than the judicial exception.
Claim 7 recites: “…wherein the processor generates the sparsified causal graph based on an adjacency matric and sparse self-attention.” Which further defines an abstract idea identified above. In the instant application claim 7 recites “the processor”. However, as in claim 1 the processor is disclosed at a high level of generality. Therefore, “the processor” is no more than a generic computing element that is performing a generic computing activity. Thus, claim 7 does not recite elements that integrate the judicial exception into a practical application of that exception or amount to significantly more than the judicial exception.
Claim 8 recites: “…wherein the system for causal trajectory prediction is equipped on an autonomous vehicle.” Which further defines an abstract idea identified above. However, the claim does not recite any additional elements and, therefore, does not recite any additional elements that integrate the judicial exception into a practical application of that exception or amount to significantly more than the judicial exception.
Claim 9 recites: “…wherein the encoder includes one or more encoder layers and in each encoder layer, a message is only passed from each agent's parents to each agent itself.” Which further defines an abstract idea identified above. However, the claim does not recite any additional elements and, therefore, does not recite any additional elements that integrate the judicial exception into a practical application of that exception or amount to significantly more than the judicial exception.
Claim 10 recites: “…wherein the decoder includes one or more decoder layers and in each decoder layer, a message is only passed from each agent's parents to each agent itself.” Which further defines an abstract idea identified above. However, the claim does not recite any additional elements and, therefore, does not recite any additional elements that integrate the judicial exception into a practical application of that exception or amount to significantly more than the judicial exception.
Claim 15 recites additional abstract ideas that may be performed mentally, i.e., “…generating a coarse trajectory prediction for one or more of the agents within the environment based on one or more of the agent future features; and generating the trajectory prediction for the target agent based on the coarse trajectory prediction for one or more of the agents.” However, the claim does not recite any additional elements and, therefore, does not recite any additional elements that integrate the judicial exception into a practical application of that exception or amount to significantly more than the judicial exception.
To overcome the rejections under 35 USC § 101 the Examiner suggests amended claims 1 and 11 to include the limitations of claims 2 and 12, respectively. Performing the driving maneuvers based on the trajectory prediction, similar to claim 16, would integrate the abstract idea into a practical application.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 4, 8, 11, 12, 14, 16, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (US 2021/0261148 A1, “Li”) in view of Yu (US 2024/0028878 A1, “Yu”).
Regarding claims 1 and 11, Li discloses driver-centric risk assessment: risk object identification via causal inference with intent-aware driving models and teaches:
A system for causal trajectory prediction, comprising: (FIG. 1 is a schematic view of an exemplary system 100 for predicting driving actions based on intent-aware driving models according to an exemplary embodiment of the present disclosure – See at least ¶ [0027]; Examiner notes that the system determines predicted positions of objects over a time period, i.e., the trajectory of the moving objects in the environment – See at least ¶ [0033] and ¶ [0075])
a memory storing one or more instructions; and (According to another aspect, a system for predicting driving actions based on intent-aware driving models that includes a memory storing instructions – See at least ¶ [0004])
a processor executing one or more of the instructions stored on the memory to perform: (A “processor”, as used herein, processes signals and performs general computing and arithmetic functions... The processor may include various modules to execute various functions – See at least ¶ [0024]; A “module”, as used herein, includes, but is not limited to, non-transitory computer readable medium that stores instructions, instructions in execution on a machine, hardware, firmware, software in execution on a machine, and/or combinations of each to perform a function(s) or an action(s), and/or to cause a function or action from another module, method, and/or system – See at least ¶ [0022])
generating a [] causal graph (In one embodiment, the neural network 108 may thereby output a predicted causality score of two alternative driving behaviors (e.g., stop/go) and may analyze the driving scene 400 to determine a level of change pertaining to the two driving behaviors with the presence of the dynamic object and without the presence of dynamic object as it's electronically removed from the driving scene 400 included in the image frame. In other words, the dynamic object detection module 120 may analyze a level of change associated with a driving behavior with respect to the removal of each of the dynamic objects and may thereby assign a causality score that is associated with a causal relationship with the driving behavior based on the level of change. The causality score may be associated with the causal relationship between the presence of each dynamic object and the particular driving behavior (e.g., stop vs. go) based on the level of change associated with the electronical removal of each of the dynamic objects (e.g., if not for the presence of the particular dynamic object, the ego vehicle 102 would not stop and would go) – See at least ¶ [0058]-[0059]) including two or more nodes and two or more edges, (As Shown in Fig. 2, the system contains Ego-Stuff Graph 220 and Ego-Thing Graph 218. Fig. 2 further illustrates the relationships between nodes, e.g., Ego Node and Stuff Nodes and Ego Node and Thing Nodes, using causal relationships – See at least ¶ [0052]-[0063] and Fig. 2) wherein a node of the two or more nodes represents an agent of one or more agents within an environment (The nodes of the Ego-Thing Graph 218 in Figure 2 are an Ego Node, i.e., the ego vehicle, and dynamic objects within the environment Thing nodes, i.e., other moving objects – See at least ¶ [0035]) and wherein an edge of the two or more edges between a first node and a second node represents a causal relationship between the first node and the second node; (The nodes are connected with edges that represent a causal relationship – See at least ¶ [0062]-[0063])
generating one or more agent future features based on the [] causal graph and an encoder; and (In an exemplary embodiment, ego-thing features extracted from the ego-thing graph 218 and ego-stuff features extracted from the ego-stuff graph 220 may and fused to accomplish spatial-temporal driving scene modeling of the driving scene of the ego vehicle 102. Upon fusion of the ego-features extracted from the graphs 218, 220, fused data is processed via the encoder LSTM 222 to make spatial temporal determinations based on the fused data to deter mine the interaction representation 224. In one configuration, ego features from the ego-thing graph 218 and the ego-stuff graph 220 are aggregated by element-wise summation and fed into the encoder LSTM 222 to obtain a 1xD feature vector as the interaction representation 224 – See at least ¶ [0074])
generating a trajectory prediction for a target agent (The system determines predicted positions of objects over a time period, i.e., the trajectory of the moving objects in the environment – See at least ¶ [0033] and ¶ [0075]) based on the one or more agent future features and a decoder. (As shown in FIG. 6, a schematic overview of the structure of the temporal decoder 228, the intention representation 226 and the interaction representation 224 are inputted to the temporal decoder 228. The intention representation 226 serves as the initial hidden state to a decoder LSTM 602. A future gate 604 and spatiotemporal accumulator (STA) 606 aggregate futures from historical, current, and predicted future information to estimate the driver stimulus action 232, shown as stop and go as an illustrative example – See at least ¶ [0077]-[0078] and Fig. 2 and 6)
Li does not explicitly disclose that the causal graph is a sparsified causal graph. However, Yu discloses organizing neural network graph information and teaches:
generating a sparsified causal graph [] (In at least one embodiment, processor 102 is a processor comprising one or more circuits to cause neural network graph data to be organized based, at least in part, on one or more sparsity constraints. In at least one embodiment, processor 102 is a processor comprising one or more circuits to cause a set of data of one or more graph neural networks to be ordered to cause the set of data to have a sparsity property. In at least one embodiment, neural network graph data (or data of one or more neural networks) is data that corresponds to one or more graph neural networks such as those described herein – See at least ¶ [0072] [0077])
In summary, Li discloses driver-centric risk assessment: risk object identification via causal inference with intent-aware driving models and teaches generating causal graphs to model the surrounding environment. Li further discloses the use of a softmax function to normalize the influence on one dynamic object from another dynamic object in the environment. While claims 3 and 13 of the instant application indicate that a sparsified causal graph is based on a softmax function, Li is not explicitly disclosing a sparsified causal graph based on the softmax function. However, Yu discloses organizing neural network graph information and teaches an autonomous vehicle platform that implements sparsity constraints to reduce the number of irrelevant nodes within the graph it creates.
Therefore it would have been obvious to a person having ordinary skill in the art before the effective filing date of the instant application to have modified the driver-centric risk assessment: risk object identification via causal inference with intent-aware driving models of Li to provide for organizing neural network graph information, as taught in Yu, to provide one or more deep learning accelerator ("DLA") which may include, without limitation, one or more Tensor processing units ("TPUs") that may be configured to provide an additional ten trillion operations per second for deep learning applications and inferencing. (At Yu ¶ [0179])
Regarding claims 2 and 12, Li does not explicitly teach, but Yu further teaches:
an actuator, wherein the processor controls the actuator to cause the system for causal trajectory prediction to perform a driving maneuver based on the trajectory prediction for the target agent. (In at least one embodiment, a steering system 1354, which may include, without limitation, a steering wheel, is used to steer vehicle 1300 (e.g., along a path or route) when propulsion system 1350 is operating (e.g., when vehicle 1300 is in motion). In at least one embodiment, steering system 1354 may receive signals from steering actuator(s) 1356. In at least one embodiment, a steering wheel may be optional for full automation (Level 5) functionality. In at least one embodiment, a brake sensor system 1346 may be used to operate vehicle brakes in response to receiving signals from brake actuator(s) 1348 and/or brake sensors – See at least ¶ [0148])
Therefore it would have been obvious to a person having ordinary skill in the art before the effective filing date of the instant application to have modified the driver-centric risk assessment: risk object identification via causal inference with intent-aware driving models of Li to provide for organizing neural network graph information, as taught in Yu, to provide one or more deep learning accelerator ("DLA") which may include, without limitation, one or more Tensor processing units ("TPUs") that may be configured to provide an additional ten trillion operations per second for deep learning applications and inferencing. (At Yu ¶ [0179])
Regarding claims 4, 14, and 18, Li further teaches:
wherein the processor generates the [] causal graph based on an entmax function or a softmax function. (A softmax function is used to normalize the influence on dynamic object I from other dynamic objects located within the driving scene – See at least ¶ [0063])
Li does not explicitly disclose, but Yu further teaches:
generates the sparsified causal graph [] (In at least one embodiment, processor 102 is a processor comprising one or more circuits to cause neural network graph data to be organized based, at least in part, on one or more sparsity constraints. In at least one embodiment, processor 102 is a processor comprising one or more circuits to cause a set of data of one or more graph neural networks to be ordered to cause the set of data to have a sparsity property. In at least one embodiment, neural network graph data (or data of one or more neural networks) is data that corresponds to one or more graph neural networks such as those described herein – See at least ¶ [0072] [0077])
Therefore it would have been obvious to a person having ordinary skill in the art before the effective filing date of the instant application to have modified the driver-centric risk assessment: risk object identification via causal inference with intent-aware driving models of Li to provide for organizing neural network graph information, as taught in Yu, to provide one or more deep learning accelerator ("DLA") which may include, without limitation, one or more Tensor processing units ("TPUs") that may be configured to provide an additional ten trillion operations per second for deep learning applications and inferencing. (At Yu ¶ [0179])
Regarding claim 8, Li further teaches:
wherein the system for causal trajectory prediction is equipped on an autonomous vehicle. (As shown in Figure 1, the system 100 is equipped onto ego vehicle 102. In some embodiments, the vehicle systems/control units 114 may be configured to include an engine control unit, a braking control unit, a transmission control unit, a steering control unit, and the like to control the ego vehicle 102 to be autonomously driven based on autonomous commands that are communicated by the action pre diction application 106 – See at least ¶ [0048])
Regarding claim 16, Li discloses driver-centric risk assessment: risk object identification via causal inference with intent-aware driving models and teaches:
A system for causal trajectory prediction, comprising: (FIG. 1 is a schematic view of an exemplary system 100 for predicting driving actions based on intent-aware driving models according to an exemplary embodiment of the present disclosure – See at least ¶ [0027])
a memory storing one or more instructions; and (According to another aspect, a system for predicting driving actions based on intent-aware driving models that includes a memory storing instructions – See at least ¶ [0004])
a processor executing one or more of the instructions stored on the memory to perform: (A “processor”, as used herein, processes signals and performs general computing and arithmetic functions... The processor may include various modules to execute various functions – See at least ¶ [0024]; A “module”, as used herein, includes, but is not limited to, non-transitory computer readable medium that stores instructions, instructions in execution on a machine, hardware, firmware, software in execution on a machine, and/or combinations of each to perform a function(s) or an action(s), and/or to cause a function or action from another module, method, and/or system – See at least ¶ [0022])
generating a [] causal graph (In one embodiment, the neural network 108 may thereby output a predicted causality score of two alternative driving behaviors (e.g., stop/go) and may analyze the driving scene 400 to determine a level of change pertaining to the two driving behaviors with the presence of the dynamic object and without the presence of dynamic object as it's electronically removed from the driving scene 400 included in the image frame. In other words, the dynamic object detection module 120 may analyze a level of change associated with a driving behavior with respect to the removal of each of the dynamic objects and may thereby assign a causality score that is associated with a causal relationship with the driving behavior based on the level of change. The causality score may be associated with the causal relationship between the presence of each dynamic object and the particular driving behavior (e.g., stop vs. go) based on the level of change associated with the electronical removal of each of the dynamic objects (e.g., if not for the presence of the particular dynamic object, the ego vehicle 102 would not stop and would go) – See at least ¶ [0058]-[0059]) including two or more nodes and two or more edges, (As Shown in Fig. 2, the system contains Ego-Stuff Graph 220 and Ego-Thing Graph 218. Fig. 2 further illustrates the relationships between nodes, e.g., Ego Node and Stuff Nodes and Ego Node and Thing Nodes, using causal relationships – See at least ¶ [0052]-[0063] and Fig. 2) wherein a node of the two or more nodes represents an agent of one or more agents within an environment, (The nodes of the Ego-Thing Graph 218 in Figure 2 are an Ego Node, i.e., the ego vehicle, and dynamic objects within the environment Thing Nodes, i.e., other moving objects – See at least ¶ [0035]) wherein an edge of the two or more edges between a first node and a second node represents a causal relationship between the first node and the second node, [] (The nodes are connected with edges that represent a causal relationship – See at least ¶ [0062]-[0063])
generating one or more agent future features based on the sparsified causal graph and an encoder; (In an exemplary embodiment, ego-thing features extracted from the ego-thing graph 218 and ego-stuff features extracted from the ego-stuff graph 220 may and fused to accomplish spatial-temporal driving scene modeling of the driving scene of the ego vehicle 102. Upon fusion of the ego-features extracted from the graphs 218, 220, fused data is processed via the encoder LSTM 222 to make spatial temporal determinations based on the fused data to deter mine the interaction representation 224. In one configuration, ego features from the ego-thing graph 218 and the ego-stuff graph 220 are aggregated by element-wise summation and fed into the encoder LSTM 222 to obtain a 1xD feature vector as the interaction representation 224 – See at least ¶ [0074])
generating a trajectory prediction for a target agent based on the one or more agent future features and a decoder; and (As shown in FIG. 6, a schematic overview of the structure of the temporal decoder 228, the intention representation 226 and the interaction representation 224 are inputted to the temporal decoder 228. The intention representation 226 serves as the initial hidden state to a decoder LSTM 602. A future gate 604 and spatiotemporal accumulator (STA) 606 aggregate futures from historical, current, and predicted future information to estimate the driver stimulus action 232, shown as stop and go as an illustrative example – See at least ¶ [0077]-[0078] and Fig. 2 and 6)
controlling the [system] to cause the system for causal trajectory prediction to perform a driving maneuver based on the trajectory prediction for the target agent. (the driving action prediction module 124 may be configured to communicate with the ECU 104 of the ego vehicle 102 to autonomously or semi-autonomously control operation of the ego vehicle 102 based on the predictions of the driver stimulus action 232 and/or the driver intention action 234 to avoid any potential overlap with dynamic objects and/or static objects within the driving scene. Accordingly, the ECU 104 may communicate with the one or more vehicle systems/control units 114 to thereby control the ego vehicle 102 to perform one or more maneuvers to travel within the driving scene at a respective speed, braking rate, steering rate, acceleration rate, and the like that avoids overlap with static objects and/or dynamic objects in accordance with the driving scene characteristics of the driving scene (e.g., number of lanes, navigable pathways based on lane markings, traffic light status) – See at least ¶ [0081])
Li does not explicitly teach the use of an actuator to perform its autonomous vehicle control. Further, Li does not explicitly disclose that the casual graph is a sparsified causal graph. However, Yu discloses organizing neural network graph information and teaches:
an actuator; (In at least one embodiment, a steering system 1354, which may include, without limitation, a steering wheel, is used to steer vehicle 1300 (e.g., along a path or route) when propulsion system 1350 is operating (e.g., when vehicle 1300 is in motion). In at least one embodiment, steering system 1354 may receive signals from steering actuator(s) 1356. In at least one embodiment, a steering wheel may be optional for full automation (Level 5) functionality. In at least one embodiment, a brake sensor system 1346 may be used to operate vehicle brakes in response to receiving signals from brake actuator(s) 1348 and/or brake sensors – See at least ¶ [0148])
generating a sparsified causal graph [] (In at least one embodiment, processor 102 is a processor comprising one or more circuits to cause neural network graph data to be organized based, at least in part, on one or more sparsity constraints. In at least one embodiment, processor 102 is a processor comprising one or more circuits to cause a set of data of one or more graph neural networks to be ordered to cause the set of data to have a sparsity property. In at least one embodiment, neural network graph data (or data of one or more neural networks) is data that corresponds to one or more graph neural networks such as those described herein – See at least ¶ [0072] and [0077]) and wherein the sparsified causal graph includes less edges than a full causal graph associated with the same one or more [nodes]; (In at least one embodiment, for example, neural network 104 may include several hundred nodes and corresponding edges, which may be more than graphics acceleration module 112 can process efficiently. In such an embodiment, processor 102 may execute instructions to subdivide neural network 104 into smaller subnetworks before, during, and/or after executing instructions to permute 106 neural network 104 to satisfy one or more sparsity conditions 108 so that graphics acceleration module 112 may perform permuted neural network processing 114 on said small subnetworks – See at least ¶ [0070])
In summary, Li discloses driver-centric risk assessment: risk object identification via causal inference with intent-aware driving models and teaches generating causal graphs to model the surrounding environment. Li further discloses the use of a softmax function to normalize the influence on one dynamic object from another dynamic object in the environment. While claims 3 and 13 of the instant application indicate that a sparsified causal graph is based on a softmax function, Li is not explicitly disclosing a sparsified causal graph based on the softmax function. However, Yu discloses organizing neural network graph information and teaches an autonomous vehicle platform that implements sparsity constraints to reduce the number of irrelevant nodes within the graph it creates.
Therefore it would have been obvious to a person having ordinary skill in the art before the effective filing date of the instant application to have modified the driver-centric risk assessment: risk object identification via causal inference with intent-aware driving models of Li to provide for organizing neural network graph information, as taught in Yu, to provide one or more deep learning accelerator ("DLA") which may include, without limitation, one or more Tensor processing units ("TPUs") that may be configured to provide an additional ten trillion operations per second for deep learning applications and inferencing. (At Yu ¶ [0179])
Claim(s) 3, 13, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Yu, as applied to claims 1, 11, and 16, and in further view of Gong et al. (US 2024/0104338 A1, “Gong”).
Regarding claims 3, 13, and 17, the combination of Li and Yu does not explicitly teach wherein the processor generates the sparsified causal graph based on regularized Bernoulli distribution. However, Gong discloses modelling causation in machine learning and teaches:
wherein the processor generates the sparsified causal graph based on regularized Bernoulli distribution. (Unfortunately from the Bayes' rule, the exact graph posterior is intractable due to the large combinatorial space of all DAGs. To overcome this challenge, we adopt variational inference, which uses variational distribution approximate the true posterior. We choose independent Bernoulli distribution for each directed edge – See at least ¶ [0202]; Examiner notes that a DAG is a sparsified casual graph – See at least ¶ [0058]-[0059])
Therefore it would have been obvious to a person having ordinary skill in the art before the effective filing date of the instant application to have modified the driver-centric risk assessment: risk object identification via causal inference with intent-aware driving models of Li and Yu to provide for modelling causation in machine learning, as taught in Gong, to predict the best order in which to apply treatments, the best timing in which to apply one or more treatments, and/or to predict the likely timing of an effect of one or more treatments in an autonomous vehicle or robot (At Gong ¶ [0149])
Claim(s) 5, 6, 15, 19, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Yu, as applied to claims 1, 11, and 16, and in further view of Caldwell et al. (US 2023/0041975 A1, “Caldwell”).
Regarding claim 5 and 19, the combination of Li and Yu does not explicitly teach wherein the processor generates a coarse trajectory prediction for one or more of the agents within the environment based on one or more of the agent future features. However, Caldwell discloses vehicle trajectory control using a tree search and teaches:
wherein the processor generates a coarse trajectory prediction for one or more of the agents within the environment based on one or more of the agent future features. (The planning component 112 may use the perception data received from perception component 110 and/or a path received from the guidance system 114, to determine one or more trajectories, control motion of the vehicle 102 to traverse a path or route, and/or otherwise control operation of the vehicle 102, though any such operation may be performed in various other components (e.g., localization may be performed by a localization component, which may be based at least in part on perception data) – See at least ¶ [0038] In some examples, the path determined by the guidance system may be a coarse path. For example, the coarse path may identify a position, heading, velocity, and/or curvature of approach for the vehicle to track at a 1 second or 500 millisecond interval – See at least ¶ [0120])
In summary, Li and Yu discloses maneuvering autonomous vehicles through an environment based on the movements of the dynamic objects in the environment. While Li and Yu both discuss trajectories, they do not explicitly teach that the trajectories are “coarse” trajectories. However, Caldwell discloses an autonomous vehicle navigating through an environment where coarse trajectories are used to cut down on operational demands and then those viable trajectories are translated into a smooth trajectory, i.e., a more detailed trajectory for automation control. This allows the vehicle to consider more potential trajectories without high the computational demands.
Therefore it would have been obvious to a person having ordinary skill in the art before the effective filing date of the instant application to have modified the driver-centric risk assessment: risk object identification via causal inference with intent-aware driving models of Li and Yu to provide for vehicle trajectory control using a tree search, as taught in Caldwell, to increase the number of scenarios the autonomous vehicle can safely and efficaciously navigate, e.g., without stopping, without stuttering, without the need to request help from a teleoperator, and/or by decreasing a likelihood of an impact occurring, particularly for aberrant circumstances but also for normative driving conditions. (At Caldwell ¶ [0012])
Regarding claim 6 and 20, the combination of Li and Yu does not explicitly teach, but Caldwell further teaches:
wherein the processor generates the trajectory prediction for the target agent based on the coarse trajectory prediction for one or more of the agents. (For example, the planning component 112 may determine a route for the vehicle 102 from a first location to a second location; determine a smooth trajectory from a coarse trajectory received from the guidance system 114; generate, substantially simultaneously and based at least in part on the path and perception data and/or simulated perception data (which may further include predictions regarding detected objects in such data), a plurality of potential trajectories for controlling motion of the vehicle 102 in accordance with a receding horizon technique (e.g., 1 micro-second, half a second, 2 seconds, 5 seconds, 10 seconds, or any other near-term time period) to control the vehicle to traverse the route (e.g., in order to avoid any of the detected objects); and select one of the potential trajectories as a trajectory 118 of the vehicle 102 that may be used to generate a drive control signal that may be transmitted to the controller(s) 116 for actuating drive components of the vehicle 102 – See at least ¶ [0038])
Therefore it would have been obvious to a person having ordinary skill in the art before the effective filing date of the instant application to have modified the driver-centric risk assessment: risk object identification via causal inference with intent-aware driving models of Li and Yu to provide for vehicle trajectory control using a tree search, as taught in Caldwell, to increase the number of scenarios the autonomous vehicle can safely and efficaciously navigate, e.g., without stopping, without stuttering, without the need to request help from a teleoperator, and/or by decreasing a likelihood of an impact occurring, particularly for aberrant circumstances but also for normative driving conditions. (At Caldwell ¶ [0012])
Regarding claim 15, the combination of Li and Yu does not explicitly teach, but Caldwell further teaches:
generating a coarse trajectory prediction for one or more of the agents within the environment based on one or more of the agent future features; and (The planning component 112 may use the perception data received from perception component 110 and/or a path received from the guidance system 114, to determine one or more trajectories, control motion of the vehicle 102 to traverse a path or route, and/or otherwise control operation of the vehicle 102, though any such operation may be performed in various other components (e.g., localization may be performed by a localization component, which may be based at least in part on perception data) – See at least ¶ [0038] In some examples, the path determined by the guidance system may be a coarse path. For example, the coarse path may identify a position, heading, velocity, and/or curvature of approach for the vehicle to track at a 1 second or 500 millisecond interval – See at least ¶ [0120])
generating the trajectory prediction for the target agent based on the coarse trajectory prediction for one or more of the agents. (For example, the planning component 112 may determine a route for the vehicle 102 from a first location to a second location; determine a smooth trajectory from a coarse trajectory received from the guidance system 114; generate, substantially simultaneously and based at least in part on the path and perception data and/or simulated perception data (which may further include predictions regarding detected objects in such data), a plurality of potential trajectories for controlling motion of the vehicle 102 in accordance with a receding horizon technique ( e.g., 1 micro-second, half a second, 2 seconds, 5 seconds, 10 seconds, or any other near-term time period) to control the vehicle to traverse the route ( e.g., in order to avoid any of the detected objects); and select one of the potential trajectories as a trajectory 118 of the vehicle 102 that may be used to generate a drive control signal that may be transmitted to the controller(s) 116 for actuating drive components of the vehicle 102 – See at least ¶ [0038])
Therefore it would have been obvious to a person having ordinary skill in the art before the effective filing date of the instant application to have modified the driver-centric risk assessment: risk object identification via causal inference with intent-aware driving models of Li and Yu to provide for vehicle trajectory control using a tree search, as taught in Caldwell, to increase the number of scenarios the autonomous vehicle can safely and efficaciously navigate, e.g., without stopping, without stuttering, without the need to request help from a teleoperator, and/or by decreasing a likelihood of an impact occurring, particularly for aberrant circumstances but also for normative driving conditions. (At Caldwell ¶ [0012])
Claim(s) 7, 9, and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Yu, as applied to claim 1, and in further view of Velickovic et al. (US 2021/0383228 A1, “Velickovic”).
Regarding claim 7, Li does not explicitly teach, but Yu further teaches:
wherein the processor generates the sparsified causal graph based on an adjacency matric and []. (FIG. 2 illustrates example matrices 200 of a graph neural network, according to at least one embodiment. In at least one embodiment, a graph neural network 202 has nodes {vi, v2 , v3 , v4 , v5 , v6 , v7 , v8 ] and edges between nodes, as illustrated in FIG. 2. In at least one embodiment, an adjacency matrix 204 of graph neural network 202 indicates which nodes are adjacent to which other nodes (e.g., which pairs of nodes have edges connecting them). In at least one embodiment, where adjacency matrix 204 has a non-zero value (e.g., 1) at row "i" and colunm "j," there is an edge between node v, and node vJ" In at least one embodiment, for example, row 1 of adjacency matrix 204 has all zero values except for at colunm 2, indicating that there is an edge between node v1 and node v2 – See at least ¶ [0075] and Fig. 2)
Therefore it would have been obvious to a person having ordinary skill in the art before the effective filing date of the instant application to have modified the driver-centric risk assessment: risk object identification via causal inference with intent-aware driving models of Li to provide for organizing neural network graph information, as taught in Yu, to provide one or more deep learning accelerator ("DLA") which may include, without limitation, one or more Tensor processing units ("TPUs") that may be configured to provide an additional ten trillion operations per second for deep learning applications and inferencing. (At Yu ¶ [0179])
The combination of Li and Yu does not explicitly teach that the generated sparsified casual graph is based on an adjacency matric and a sparse self-attention. However, Velickovic discloses generating prediction outputs using dynamic graphs and teaches:
wherein the processor generates the sparsified causal graph (The system described in this specification can learn to generate “sparse” graphs, i.e., that include only a limited number of edges, e.g., that include a number of edges that is equal to the number of nodes in the graph – See at least ¶ [0035]) based on an adjacency matric (The prediction system can initialize the current set of edges at time step (0) with a default set of dynamic edges, a e.g., a set of edges where each node points only to itself. The default set of edges can also optionally include a set of predefined static edges which the system cannot modify. The current set of edges can be represented by an adjacency matrix, e.g., a square binary matrix of zeros and ones with dimensionality equal to the number of nodes – See at least ¶ [0062]) and sparse self-attention. (With reference to the example of FIG. 1, the system generates an updated edge set using a self-attention (“self-attn”) operation to update the edges from nodes with masking output equal to zero – See at least ¶ [0068])
In summary, Yu discloses generating a sparsified casual graph based on an adjacency matrix. The combination of Li and Yu does not explicitly teach generating the sparsified casual graph based on an adjacency matrix and a sparse self-attention. However, Velickovic discloses generating prediction outputs using dynamic graphs and teaches generating sparse graphs using both adjacency matrices and spare self-attention.
Therefore it would have been obvious to a person having ordinary skill in the art before the effective filing date of the instant application to have modified the driver-centric risk assessment: risk object identification via causal inference with intent-aware driving models of Li and Yu to provide for the generating prediction outputs using dynamic graphs, as taught in Velickovic, to allow the system to be trained to perform subsequent tasks over fewer training time steps (thereby reducing computational resource consumption), and to improve the robustness and generalizability of the system. (At Velickovic ¶ [0036])
Regarding claim 9, the combination of Li and Yu does not explicitly teach, but Velickovic further teaches:
wherein the encoder includes one or more encoder layers and in each encoder layer, a message is only passed from each agent's parents to each agent itself. (The encoder neural network can have any appropriate neural network architecture that enables it to perform its described function, i.e., processing the input node features of each node and current node embedding of each node to generate a respective node feature representation for each node. In particular, each encoder neural network can include any appropriate types of neural network layers (e.g., fully-connected layers, attention-layers, convolutional layers , etc.) in any appropriate numbers (e.g., 1 layer, 5 layers, or 25 layers), and connected in any appropriate configuration (e.g., as a linear sequence of layers) – See at least ¶ [0085])
Therefore it would have been obvious to a person having ordinary skill in the art before the effective filing date of the instant application to have modified the driver-centric risk assessment: risk object identification via causal inference with intent-aware driving models of Li and Yu to provide for the generating prediction outputs using dynamic graphs, as taught in Velickovic, to allow the system to be trained to perform subsequent tasks over fewer training time steps (thereby reducing computational resource consumption), and to improve the robustness and generalizability of the system. (At Velickovic ¶ [0036])
Regarding claim 10, the combination of Li and Yu does not explicitly teach, but Velickovic further teaches:
wherein the decoder includes one or more decoder layers and in each decoder layer, a message is only passed from each agent's parents to each agent itself. (The decoder neural network can have any appropriate neural network architecture that enables it to perform its described function, i.e., processing the pooled embedding to generate a prediction output. In particular, the decoder neural network can have any appropriate types of neural network layers (e.g., fully-connected layers, attention-layers, etc.) in any appropriate numbers (e.g., 1 layer, 5 layers, or 25 layers), and connected in any appropriate configuration (e.g. , as a linear sequence of layers) – See at least ¶ [0102])
Therefore it would have been obvious to a person having ordinary skill in the art before the effective filing date of the instant application to have modified the driver-centric risk assessment: risk object identification via causal inference with intent-aware driving models of Li and Yu to provide for the generating prediction outputs using dynamic graphs, as taught in Velickovic, to allow the system to be trained to perform subsequent tasks over fewer training time steps (thereby reducing computational resource consumption), and to improve the robustness and generalizability of the system. (At Velickovic ¶ [0036])
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Chiu et al. (US 2022/0198813 A1) discloses systems and methods for efficient visual navigation for robotic systems. Chiu further teaches building sparse graphs using adjacency matrices.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHASE L COOLEY whose telephone number is (303)297-4355. The examiner can normally be reached Monday-Thursday 7-5MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Aniss Chad can be reached at 571-270-3832. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHASE L COOLEY/Examiner, Art Unit 3662
Prosecution Insights