DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on January 7, 2016 has been entered.
Response to Arguments
The amendment filed January 7, 2026 has been entered with the RCE filed the same day. Claims 1, 5, 11, 12, and 14 have been amended. Claim 15 is new. The remaining claims are in original or previously presented form. Therefore, claims 1-15 are pending in the application. Claims 1, 12, and 14 are the independent claims.
The applicant’s Remarks, filed January 7, 2026, has been fully considered. The applicant argues, under the heading “INTERVIEW SUMMARY,” that during the interview on December 4, 2025, “the Examiner indicated that the added features of amended claim 1 do not appear to be taught by Nister [Nister et al. (US2019/0243371)], and that the features of new dependent claim 15 appear to further distinguish the claimed invention from Nister. The Examiner stated that he needs additional time to review Nister and conduct another search before confirming his position on the art rejections.” This might have been better phrased as: “the applicant understood the examiner to state that…” Although it is possible that that is what the examiner stated, the examiner does not recall stating that the added features of amended claim 1 do not appear to be taught by Nister, nor that the features of new dependent claim 15 appear to further distinguish the claimed invention from Nister. The examiner’s interview summary also does not reflect that conclusion.
The examiner respectfully submits that Nister teaches claim 1 as currently amended. Claim 1 now recites:
A computer-implemented method for collision threat assessment of a vehicle, the computer-implemented method comprising:
obtaining context information for a surrounding of the vehicle including information about a road user, wherein:
the context information includes dynamic context information, and
the dynamic context information represents information about the road user including its position and velocity;
determining ego occupancy information for a plurality of possible future locations of the vehicle at a plurality of future points in time based on the context information, wherein:
determining ego occupancy information is performed by a trained artificial neural network, and
the trained artificial neural network has been trained based on training data including traffic situations of a plurality of moving road users;
determining road user occupancy information for a plurality of possible future locations of the road user at the plurality of future points in time based on the context information;
fusing the ego occupancy information and the road user occupancy information to obtain fused occupancy information at each future point in time; and
determining a collision threat value based on the fused occupancy information.
The examiner believes that Nister teaches these limitations. The examiner believes that Nister teaches the limitation that reads: “determining ego occupancy information is performed by a trained artificial neural network”. The previous clause defined “ego occupancy information” as “for a plurality of possible future locations of the vehicle at a plurality of future points in time based on the context information”.
This can be seen in Nister Fig. 3F in which the ego vehicle (or host vehicle) is vehicle 102. The system of Nister uses a neural network to generate the trajectory 320 and place it in the context of trajectory 322 and 324 of “objects” 106A and 106B. The same is true of Fig. 7A. Fig. 7A shows much the same thing as Fig. 3F, except that in certain scenarios the paths of the two vehicles overlap in region 740. This trajectory generation and overlap determination is performed using a neural network. See paragraph 0206 for using “one or more machine learning modes, such as neural networks” to determine the “particular path” that “an actor may be likely to follow”. See paragraph 0099 for teaching what “actors” are, they are “e.g., the vehicle 102 and the objects 106”. See paragraph 0085 for “machine learning models, such as neural networks…maybe used for determining the states of the actors.” Note that “actors” includes the host vehicle 102, as taught in paragraph 0099. See paragraph 0129 for using a “neural network” to perceive a latency or lag between a command and a response to a command. “In any example, the shape (e.g., length, width, heigh, etc.) of the trajectory(ies) of the claimed sets for the actors (e.g., the vehicle 102 and/or the objects 106) may be adjusted (e.g., lengthened, widened, etc.) to account for latency, lag, or reaction time.” See paragraph 0260 for “the neural network can take as its input at least some subset of parameters, such as bounding box dimensions,” etc. Thus the “shape…of the trajectory(ies)” in Fig. 7A is generated using a neural network.
Furthermore, the examiner submits that entire context of Nister is that it employs machine learning using a neural network “[in] contrast to conventional systems,” as recited in paragraph 0007. Paragraphs 0003-0006 make much of the limitations of conventional systems in their ability to “achieve autonomous driving levels 3-5,” as paragraph 0003 recites. Paragraph 0052 goes on to state that the “current system” can determine a state of a vehicle, determine points in space-time that the vehicle occupies, and generate a virtual representation of the points in two- or three-dimensions, including the ”vehicle-occupied trajectory(ies)”. How does the system do this? With a neural network. It seems to the examiner that this is so obvious as to annoyingly go unstated in some sections. Yet it’s unreasonable to think that the system does not use a neural network to generate vehicle-occupied trajectories. One reason for this is that the assignee of Nister is Nvidia, and Nvidia applications are well-known for using Nvidia-made GPUs to run neural networks. The host vehicle, as shown in Fig. 12, is complete with many GPUs and a neural network 1192. See paragraph 0276 for teaching that “multiple neural networks may be run simultaneously, as is required for Level 3, 4, or 5 driving.” So paragraph 0003 teaches that the “conventional systems” are limited in their ability to “achieve autonomous driving levels 3-5,” and then paragraph 0276 teaches that the present system runs multiple neural networks. The strong implication is that the contents of the disclosure are performed using neural networks. Paragraph 0319 even teaches using “deep-learning infrastructure” to check the “AI in the vehicle 102” to make sure it is working. In other words, Nister even teaches using a neural network to check a neural network.
Furthermore, the disclosure uses the term “layer” in the context of “the autonomous driving software stack,” many times. For example, paragraph 0056, refers to a “planning layer” and a “control layer”. In the vehicle control art, references to layers in this context refer to neural networks which have layers. Again, this is a strong indication that the system of Nister, including trajectory generation, is performed using a neural network. The examiner does not think that it can be proven that the system does not use a neural network to do trajectory generation. There is a discussion of a “control model” using differential equations in paragraphs 0086-0088. Yet even if these equations are what are used for determining trajectories of the host vehicle, which the examiner does not concede, the system still uses a neural network to account for flaring of those trajectories in order to take account of lag, latency, and other variables. Therefore, even if a baseline formula is fed into the neural network, the vehicle-occupied trajectory is generated by a neural network. That vehicle-occupied trajectory is what is shown in Fig. 7A.
Another reason the examiner does not think that the formulas in paragraph 0086-0088 are the end of the story is that paragraph 0206 teaches that the system can use “machine learning models, such as neural networks” to learn over time the “particular path” that “an actor may be likely to follow”. When Nister uses the term “learning models” it means neural networks. Paragraph 0085 teaches using “machine learning models, such as neural networks,” which are “used to determine the states of the actors.” Again, the “actors” includes the host vehicle and nearby objects. These “states” are the locations of actors, and other properties of the actors, including locations and properties at times in the future, according to paragraph 0131. See also paragraph 0083, which uses the term “state trajectories”.
See also Nister, paragraph 0091 for “one or more neural networks” being “used to identify the side of the road and/or to aid in maneuvering the vehicle 102 to the side of the road.”
The examiner also believes that Nister meets the limitations in the clause in present claim 1 that recites: “the trained artificial neural network has been trained based on training data including traffic situations of a plurality of moving road users”. Nister paragraph 0317 teaches that the server “may be used to train machine learning models (e.g., neural networks) based on training data. The training data may be generated by vehicles, and/or may be generated in a simulation…” See paragraph 0319 for teaching that a host vehicle can send “a sequence of images and/or objects” to the machine learning systems. See paragraph 0180 and 0316.
Nister also teaches that the neural network is trained. See paragraph 0275 for teaching that the neural network can read street signs even those that the system “has not been specifically trained.” This implies that the neural network is trained to read at least some street signs. The next paragraph teaches that the “neural network…has been trained”. Paragraph 0278 teaches that the CNN, which is a convolutional neural network “is trained to identify the relative closing speed of the emergency vehicle” and to identify emergency vehicles in general. There are many more instances of teaching that the neural network in Nister is trained.
The applicant argues in the Remarks on page 9 that “Nister appears silent with respect to using the machine learning model to determine future positions of vehicle 102, as required by amended claim 1. See Nister at [0085].” Brackets in original. Nister paragraph 0085 teaches that the host vehicle uses “neural networks…to deter the states of the actors.” The paragraph then states, “For example” and neural network can determine the state of objects 106. The paragraph goes on to state that the neural networks are used for “a variety of functions” though these are explicitly non-limiting examples.
Although the examiner believes that Nister teaches the limitations of present claim 1 and does so above a preponderance of the evidence standard, the examiner admits that Nister does not teach the limitations as clearly as he generally likes. While the examiner still thinks Nister meets the limitations, the examiner also brings up Caldwell et al. (US2021/0046924), a Zoox application cited in the “Additional Art” section of the Non-Final dated June 10, 2025.
Caldwell states in paragraph 0132 that “all of the components discussed herein may include any models, techniques, and/or machine learning techniques. For example, in some instances, the components in the memory 718 (and in the memory 734, discussed below) may be implemented as a neural network.” The next paragraph discusses that a neural network uses connected layers. Note that Fig. 7 shows that memory 718 includes the “planning component 724” and the “localization component 720” as well as the “prediction component 732”. The memory 734 is part of the server and includes the action cost component 738. See paragraph 0079 for teaching that the actions 204-210 of the host vehicle 202 may be generated by the “planning component 724”. This is the planning component in memory 718 that paragraph 0132 utilizes a neural network. See paragraph 0124 for the planning component determining various routes and vehicle trajectories.
Not only does this reference show that models and layers generally refer to neural networks, when in the context of such systems, but Caldwell also meets the limitations of claim 1. Caldwell teaches in paragraph 0046, determining the trajectories of both the host vehicle 102 and objects 104. Paragraph 0053 teaches that the trajectories include the “estimated position (e.g., estimated location) of the vehicle 102 and an estimated position of the object(s) 104 at a time in the future.” Fig. 2 in Caldwell shows these trajectories, and they include the entire space the host vehicle occupies and the space the objects occupy.
See Caldwell paragraph 0082 for teaching that “predicted object trajectories may be determined utilizing…machine learning techniques.” See paragraph 0125 for the prediction component using machine learning techniques.
See Caldwell paragraph 0083 for stating that Fig. 2 shows that the system may “simulate future states (e.g., estimated states) by projecting the vehicle 202 and objects 222(1) and 222(2) forward in the environment for a period of time”.
See Caldwell paragraph 0165 for teaching that the probability of a collision between the host vehicle and a nearby object “may be determined utilizing machine learning techniques.”
See Caldwell Fig. 2 and paragraph 0046 for teaching that “the vehicle computing system may determine one or more actions 110 for the vehicle 102 operating in the environment with the detected objects 104.” Paragraph 0047 teaches that these actions 110 can be varied as “sub-actions,” such as 110(2), for example.
See Caldwell paragraph 0154 for predicting the trajectory of an object using a “machine learned model trained to output a predicted trajectory of the selected object.”
See Caldwell paragraph 0165 for “In some examples, machined learned models may be trained to determine a probability of collision based on training data comprising scenarios in which vehicles and objects interact in an environment.” See paragraph 0056 for teaching that “a machine learning model may be trained utilizing training data comprising scenarios in which vehicles 102 and objects 104 did or did not collide.” The examine submits that essentially everything from host vehicle trajectory generation, to object trajectory generation, to collision determination is performed using trained neural networks in Caldwell. The examiner submits that Caldwell teaches the limitations of present claim 1.
Although the examiner believes that Nister is good enough to anticipate present claim 1, the examiner will cite Caldwell et al. (US2021/0046924) to teach some of the amended limitations because Caldwell is more explicit. Due to the applicant’s amendments the grounds for rejection have changed. Please see the rejections below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-7 and 9-15 are rejected under 35 U.S.C. 103 as being unpatentable over Nister et al. (US2019/0243371) in view of Caldwell et al. (US2021/0046924).
Regarding claim 1, Nister teaches:
A computer-implemented method for collision threat assessment of a vehicle, the computer-implemented method comprising (see Fig. 8):
obtaining context information for a surrounding of the vehicle including information about a road user (in the present published disclosure, Schaefer (US2024/0166204 A1), paragraph 0030 teaches that “context information contains information about the surrounding of the vehicle such as the position and velocity of other road users in the vicinity of the vehicle and can be obtained by onboard sensors such as radar, lidar, optical cameras, etc.” With that in mind, see Nister, paragraphs 0060 and 0063), wherein:
the context information includes dynamic context information, and the dynamic context information represents information about the road user including its position and velocity (see Nister Fig. 3F and paragraph 0114 for the teaching that the system of the host vehicle 102 can determining its own trajectory and those of others and situate them “in an environment 326…the trajectories may occupy a three-dimensional space…in space-time within the environment 326.” In other words, the host vehicle situates its own trajectory in the same three-dimensional space as other road users. See also paragraph 0115 which teaches that the host vehicle 102 can generate its own profile (or range of profiles in some embodiments), and the profiles of other vehicles, such as 106A and 106B. In the example of Fig. 3F “no overlap or intersection” exists. This means that the host vehicle is aware of where it is in the context of the other vehicles and their profiles.);
determining ego occupancy information for a plurality of possible future locations of the vehicle at a plurality of future points in time based on the context information (see Figs. 3F, 4E, and 6A. Nister teaches projecting trajectories of the ego vehicle and other road users into the future and determining if they will overlap. Then adjusting the ego vehicle’s commands accordingly. In relation to Fig. 6A, see paragraph 0154, noting that the ego vehicle is vehicle 102. Note that in Fig. 6A, the future position of host vehicle could be along area 602A, 602B, or 602C depending on what path the vehicle takes in the future. This is also discussed in paragraph 0052. See also Fig. 8, step B804), wherein:
determining road user occupancy information for a plurality of possible future locations of the road user at the plurality of future points in time based on the context information (see Fig. 6A and paragraph 0154. See also Fig. 10, step B1004.);
fusing the ego occupancy information and the road user occupancy information to obtain fused occupancy information at each future point in time (see Fig. 6A. See paragraph 0054 for the teaching that “The system may then monitor the vehicle-occupied trajectory(ies) in view of the object-occupied trajectories to determine if an intersection or overlap occurs”. See paragraph 0099 for the system being able to “determine each of the points in the occupied set”. One of the main teachings of Nister is that the disclosed method of extrapolating potential 3D shapes of the ego and nearby vehicles in what Nister calls “time slices,” as discussed in Fig. 9A and paragraph 0193. Nister determines if there is an “overlap region” at any time within the “space-time” projected in the disclosure. One overlap is shown as region 740 in Fig. 7A and discussed in paragraph 0163.); and
determining a collision threat value based on the fused occupancy information (in the present disclosure see paragraph 0038 for “a collision threat value” that “may for example correspond to the probability that the ego vehicle and the at least other road user will collide at least one future point in time.” With that in mind, see Nister paragraph 0056 for determining a “likelihood of a collision”. See paragraph 0092 for determining that “a likelihood of a collision is above a threshold risk level”.).
Yet Nister does not explicitly further teach as well as Caldwell:
determining ego occupancy information is performed by a trained artificial neural network, and
the trained artificial neural network has been trained based on training data including traffic situations of a plurality of moving road users.
However, Caldwell teaches:
determining ego occupancy information is performed by a trained artificial neural network (see Caldwell paragraph 0132 for the teaching that “all of the components discussed herein may include any models, techniques, and/or machine learning techniques. For example, in some instances, the components in the memory 718 (and in the memory 734, discussed below) may be implemented as a neural network.” The next paragraph discusses that a neural network uses connected layers. Note that Fig. 7 shows that memory 718 includes the “planning component 724” and the “localization component 720” as well as the “prediction component 732”. The memory 734 is part of the server and includes the action cost component 738. See paragraph 0079 for teaching that the actions 204-210 of the host vehicle 202 may be generated by the “planning component 724”. This is the planning component in memory 718 that paragraph 0132 utilizes a neural network. See paragraph 0124 for the planning component determining various routes and vehicle trajectories. Caldwell teaches in paragraph 0046, determining the trajectories of both the host vehicle 102 and objects 104. Paragraph 0053 teaches that the trajectories include the “estimated position (e.g., estimated location) of the vehicle 102 and an estimated position of the object(s) 104 at a time in the future.” Fig. 2 in Caldwell shows these trajectories, and they include the entire space the host vehicle occupies and the space the objects occupy. See Caldwell paragraph 0082 for teaching that “predicted object trajectories may be determined utilizing…machine learning techniques.” See paragraph 0125 for the prediction component using machine learning techniques. See Caldwell paragraph 0083 for stating that Fig. 2 shows that the system may “simulate future states (e.g., estimated states) by projecting the vehicle 202 and objects 222(1) and 222(2) forward in the environment for a period of time”. See Caldwell paragraph 0165 for teaching that the probability of a collision between the host vehicle and a nearby object “may be determined utilizing machine learning techniques.” See Caldwell Fig. 2 and paragraph 0046 for teaching that “the vehicle computing system may determine one or more actions 110 for the vehicle 102 operating in the environment with the detected objects 104.” Paragraph 0047 teaches that these actions 110 can be varied as “sub-actions,” such as 110(2), for example. See Caldwell paragraph 0154 for predicting the trajectory of an object using a “machine learned model trained to output a predicted trajectory of the selected object.”), and
the trained artificial neural network has been trained based on training data including traffic situations of a plurality of moving road users (see Caldwell paragraph 0165 for “In some examples, machined learned models may be trained to determine a probability of collision based on training data comprising scenarios in which vehicles and objects interact in an environment.” See paragraph 0056 for teaching that “a machine learning model may be trained utilizing training data comprising scenarios in which vehicles 102 and objects 104 did or did not collide.” The examine submits that essentially everything from host vehicle trajectory generation, to object trajectory generation, to collision determination is performed using trained neural networks in Caldwell.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Nister, to add the additional features of determining ego occupancy information is performed by a trained artificial neural network, and the trained artificial neural network has been trained based on training data including traffic situations of a plurality of moving road users, as taught by Caldwell. The motivation for doing so would be to “safely make forward progress while balancing passenger comfort, road rules, and norms of driving, as recognized by Caldwell (see paragraphs 0016 and 0023.).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Regarding claim 2, Nister and Caldwell teach the computer-implemented method of claim 1.
Nister further teaches:
The computer-implemented method of claim 1, further comprising:
filtering context information by selecting a subset of the context information (in the present disclosure, paragraph 0046 teaches that filtering context information can include filtering dynamic and static context information. For example, filtering dynamic context information, broadly and reasonably means filtering out moving objects detected in sensor data, meaning, filtering out other road users. So paragraph 0046 recites that filtering out dynamic context information includes “predicting the ego vehicle’s future position assuming no other road users are on the road….the ego vehicle may still follow the road or lane [static context information] but it does not see [other pedestrians or vehicles]”. In another example given in paragraph 0046 no ambiguity over the host vehicle’s future position is included (i.e., it is filtered out) because the host vehicle is commanded by a route guidance system and thus where the host vehicle is projected to go has no ambiguity.
With that in mind, see Nister paragraph 0075 for performing obstacle avoidance in a way that is in a “separate layer from the rules of the road layer.” The system can “ignore traffic laws, rules of the road, and courteous driving norms in order to ensure that collisions do not occur between the [host] vehicle 102 and any objects.” This is filtering out static context information which is a subset of the context information. The system does not stop projecting potential trajectories of all traffic actors into the future when it begins executing an avoidance maneuver. Nothing in the disclosure suggests that. Rather, the disclosure teaches that the forward projections are determined multiple times per second. So paragraph 0075 teaches that the system “may ensure that the vehicle 102 is only performing safe actions,” i.e., not running into traffic actors. See also paragraphs 0080 for the obstacle avoidance components that may operate separately. See paragraph 0081 for this separate layer still including the forward projections discussed in the rejection of claim 1. According to paragraph 0081 the obstacle avoidance components that may operate separately can still include the forward determiners that extrapolate the actors’ trajectories into the future. Yet when there is a potential collision, the system will filter out the traffic rules layer. This is filtering out static context information in the language of the present clause. See also paragraph 0073 for the teaching that the system “may use a forward prediction model that takes control [of the ego vehicle] as an input variable”. In other words, the system essentially eliminates the doubt about where the ego vehicle wants to go because it knows that based on the control component’s commands to the ego vehicle which are input into the model.),
wherein determining ego occupancy information and determining road user occupancy information are performed based on the selected subset of the context information (see paragraph 0075. See also paragraph 0081 for the system still having a claimed set determiner 136, trajectory generator 138 that still determines the state of each actor.).
Regarding claim 3, Nister and Caldwell teach the computer-implemented method of claim 1.
Nister further teaches:
The computer-implemented method of claim 1 wherein:
the plurality of possible future locations of the vehicle and the road user are organized as a grid-map (see paragraph 0071 for “a dynamic occupancy grid”. See also Fig. 7B. See also Figs. 9A and 9B. The base of each figure is a grid, and each block is an occupied area on the grid by an actor at a given space-time slice.); and
the ego occupancy information and the road user occupancy information are overlapped in the grid-map to obtain the fused occupancy information (see paragraph 0071 and Figs. 9A and 9B. The examples in these figures do not show overlap of the various actors but there can be overlap, as shown in Fig. 6A, which is essentially a 2D version of 9A in an example with overlap. See also Fig. 7A, where overlap is shown as region 740, according to paragraph 0163.).
Regarding claim 4, Nister and Caldwell teach the computer-implemented method of claim 1.
Nister further teaches:
The computer-implemented method of claim 1 further comprising
triggering an Advanced Driver Assistance Systems (ADAS) functionality in response to the collision threat value exceeding a predetermined threshold at a future point in time (see paragraph 0056 for determining a “likelihood of a collision”. See paragraph 0092 for determining that “a likelihood of a collision is above a threshold risk level” and then executing a “safety procedure” to avoid or minimize the harm. See Fig. 11C for ADAS System 1138. See paragraph 0298 for the ADAS including a crash warning, and collision warning systems. See paragraph 0311 for using the ADAS warnings when a collision is perceived by the system.).
Regarding claim 5, Nister and Caldwell teach the computer-implemented method of claim 1.
Nister further teaches:
The computer-implemented method of claim 1 wherein:
the context information includes static context information; and the static context information represents information about the surrounding of the vehicle (see paragraph 0180 for the system identifying “traffic lights, traffic signs, stop lines,” etc. See paragraph 0177 for the system understanding “traffic rules”. See paragraph 0075 for the system (usually) obeying “traffic laws, rules of the road”.).
Regarding claim 6, Nister and Caldwell teach the computer-implemented method of claim 5.
Nister further teaches:
The computer-implemented method of claim 5 wherein:
the static context information is represented at least in part by at least one of map data or traffic rules (see paragraph 0075 for the system (usually) obeying “traffic laws, rules of the road”.).
Regarding claim 7, Nister and Caldwell teach the computer-implemented method of claim 1.
Nister further teaches:
The computer-implemented method of claim wherein
the dynamic context information additionally represents information about in the present disclosure, paragraph 0032 teaches that the context information can include “information about the [ego] vehicle itself, such as its current speed and direction.” Paragraph 0019, in the context of dynamic context information, teaches that the ego vehicle can obtain maneuver information from the vehicle’s direction indicator,” i.e., turn signal indicator, or the vehicle’s “onboard route guidance system” or the autonomous vehicle’s planned maneuvers.” Present claim 1 recites “obtaining context information for a surrounding of the [ego] vehicle” and “determining ego occupancy information…based on the context information”. With that in mind, see Nister paragraph 0072 for an autonomous lane change planner that determines if a “lane change is requested” and then checks the lane change “against the lane graph” to see if it “is safe and doesn’t require heavy braking”. Reasonably, as seen in Fig. 5A, if ego vehicle 102 proposes to change lanes but overlap with another road user will occur, the system will not execute the lane change at that time. So in Nister, the ego vehicle’s system knows the ego vehicle’s position and velocity and future potential trajectory based on dynamic context information including its own dynamics.).
Regarding claim 9, Nister and Caldwell teach the computer-implemented method of claim 1.
Nister further teaches:
The computer-implemented method of claim 1 wherein:
obtaining context information for the vehicle includes obtaining planned maneuver information relating to a planned maneuver of the vehicle (see paragraph 0073 for the teaching that the system “may use a forward prediction model that takes control [of the ego vehicle] as an input variable”. In other words, the system essentially eliminates the doubt about where the ego vehicle wants to go because it knows that based on the control component’s commands to the ego vehicle which are input into the model.); and
determining ego occupancy information is additionally based on the planned maneuver information (see paragraph 0073 for taking the control commands of the ego vehicle and using for the “forward prediction model” that “produces predictions” of the future positions of the ego vehicle and other road actors.).
Regarding claim 10, Nister and Caldwell teach the computer-implemented method of claim 1.
Nister further teaches:
The computer-implemented method of claim 1 further comprising
obtaining context information for the surrounding of the vehicle including information about a plurality of road users (see Fig. 3F for a plurality of road users).
Regarding claim 11, Nister and Caldwell teach the computer-implemented method of claim 1.
Nister further teaches:
The computer-implemented method of claim 1, wherein:
determining road user occupancy information is performed by the trained artificial neural network (see Nister Fig. 3F in which the ego vehicle (or host vehicle) is vehicle 102 and the objects 106A and 106B are the road users. The system of Nister uses a neural network to generate the trajectory 320 and place it in the context of trajectories 322 and 324 of “objects” 106A and 106B. The same is true of Fig. 7A. Fig. 7A shows much the same thing as Fig. 3F, except that in certain scenarios the paths of the two vehicles overlap in region 740. This trajectory generation and overlap determination is performed using a neural network. See paragraph 0206 for using “one or more machine learning modes, such as neural networks” to determine the “particular path” that “an actor may be likely to follow”. See paragraph 0099 for teaching what “actors” are, they are “e.g., the vehicle 102 and the objects 106”. See paragraph 0085 for “machine learning models, such as neural networks…maybe used for determining the states of the actors.” Note that “actors” includes the host vehicle 102, as taught in paragraph 0099. Paragraph 0052 states that the system can determine a state of a vehicle, determine points in space-time that the vehicle occupies, and generate a virtual representation of the points in two- or three-dimensions, including the ”vehicle-occupied trajectory(ies)”.). .
Regarding claim 12, Nister teaches:
An apparatus comprising (see Fig. 1):
a computer-readable medium storing instruction (see Fig. 11C for processor 1110); and
at least one processor configured to execute the instructions, wherein the instructions include (see paragraph 0057, last sentence.):
obtaining context information for a surrounding of a vehicle including information about a road user (for the remainder of the rejection, see the rejection of claim 1 which is substantially similar.), wherein:
the context information includes dynamic context information, and
the dynamic context information represents information about the road user including its position and velocity;
determining ego occupancy information for a plurality of possible future locations of the vehicle at a plurality of future points in time based on the context information, wherein:
determining ego occupancy information is performed by a trained artificial neural network, and the trained artificial neural network has been trained based on training data including traffic situations of a plurality of moving road users;
determining road user occupancy information for a plurality of possible future locations of the road user at the plurality of future points in time based on the context information;
fusing the ego occupancy information and the road user occupancy information to obtain fused occupancy information at each future point in time; and
determining a collision threat value based on the fused occupancy information.
Regarding claim 13, Nister and Caldwell teach the apparatus of claim 12.
Nister further teaches:
The apparatus of claim 12, wherein
a sensor system including a plurality of sensors configured to provide sensor data (see Nister Fig. 11B and paragraphs 0060 and 0063),
wherein the context information is determined based at least in part on the sensor data (see paragraphs 0060 and 0063).
Regarding claim 14, Nister teaches:
A non-transitory computer-readable medium comprising instructions including (see paragraph 0183 for instructions stored in memory):
obtaining context information for a surrounding of a vehicle including information about a road user (for the remainder of the rejection, see the rejection of claim 1 which is substantially similar), wherein:
the context information includes dynamic context information, and
the dynamic context information represents information about the road user including its position and velocity;
determining ego occupancy information for a plurality of possible future locations of the vehicle at a plurality of future points in time based on the context information, wherein:
determining ego occupancy information is performed by a trained artificial neural network, and
the trained artificial neural network has been trained based on training data including traffic situations of a plurality of moving road users;
determining road user occupancy information for a plurality of possible future locations of the road user at the plurality of future points in time based on the context information;
fusing the ego occupancy information and the road user occupancy information to obtain fused occupancy information at each future point in time; and
determining a collision threat value based on the fused occupancy information.
Regarding claim 15, Nister and Caldwell teach the computer-implemented method of claim 1.
Nister further teaches:
The computer-implemented method of claim 1 wherein:
the ego occupancy information includes at least one ego occupancy value (see Figs. 6A and 6B and paragraphs 0155-0156 for various possible trajectories that the host vehicle could take and where those trajectories would generate an occupancy area of the ego vehicle. The system can check for overlap or potential collision and reject some trajectories as unsafe. As can be seen in Fig. 6B, ego vehicle trajectory 602F and 602E will overlap with object trajectory 604B, but ego trajectory 602D will not cause an overlap.),
the road user occupancy information includes at least one road user occupancy value (see Fig. 7A for overlapping region 740. See paragraph 0260 for a neural-network generated “probability” or confidence value in a bounding box or ground plane estimate of an object. And “3D location estimates of the object”. See also Figs. 6A and 6B as discussed in the previous bullet.),
Yet Nister does not further teach:
The computer-implemented method of claim 1 wherein:
fusing the ego occupancy information and the road user occupancy information includes multiplying the at least one ego occupancy value and the at least one road user occupancy value to obtain at least one fused occupancy value, and
the fused occupancy information includes the at least one fused occupancy value.
However, Caldwell teaches:
The computer-implemented method of claim 1 wherein:
fusing the ego occupancy information and the road user occupancy information includes multiplying the at least one ego occupancy value and the at least one road user occupancy value to obtain at least one fused occupancy value (in the present published disclosure, paragraph 0034 teaches that in step 120 “ego occupancy information relates to the possibility of the ego vehicle being located at a particular location at a particular future point in time. A larger occupancy information value indicates a higher probability of being at a particular location compared to a lower occupancy value. Paragraph 0037 teaches that “In step 140 the ego occupancy information and the road user occupancy information is fused to obtain fused occupancy information at each future point in time.” This can determine if there is a “risk of collision.” The paragraph goes on to teach that “In another example, the ego occupancy information and the road user occupancy information is multiplied and the result is compared against a threshold. In general, any mathematical operation that leads to an estimate of the collision risk is suitable.” Paragraph 0038 teaches “the ego occupancy information and the road user occupancy information are fused by a mathematical operation (such as e.g. multiplication) that results in a single value, this value may represent the collision threat value.” Paragraph 0010 teaches that “Fusing the individual predictions may for examples be done by combining the individual predictions, e.g., by adding or multiplying.”
With that in mind, see Caldwell paragraph 0023 for the system being able to “determine a cost associated with each estimated state, such as based on the estimated positions of the vehicle and the object relative to one another.” One factor in the overall cost may be safety “avoiding a collision between the vehicle and the object”. Paragraph 0024 teaches that the safety may include “a likelihood of collision (e.g. probability of collision) between the vehicle and the object.” It can be calculated by distance between the vehicles, converging trajectories that will “substantially intersect”, or something similar. It “may be based on threshold values associated” with these factors. Paragraph 0033 teaches that the “safety cost may be multiplied by a factor”. This provides a ranking by which safety can be ranked against other factors such as forward progress and comfort. The system wants to maintain safety, but at some point the host vehicle cannot simply sit still, it most make progress toward its destination.), and
the fused occupancy information includes the at least one fused occupancy value (see the above bullet noting that the safety value in Caldwell is a fused occupancy value.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Nister and Caldwell, to add the additional features fusing the ego occupancy information and the road user occupancy information includes multiplying the at least one ego occupancy value and the at least one road user occupancy value to obtain at least one fused occupancy value, and the fused occupancy information includes the at least one fused occupancy value, as taught by Caldwell. The motivation for doing so would be to “safely make forward progress while balancing safety with passenger comfort, road rules, and norms of driving, as recognized by Caldwell (see paragraphs 0016 and 0023.).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Note that Nister Figs. 6A and 6B are similar to aspects of Fig. 2 in which there are host vehicle 202 trajectories that will intersect with the object 222 and some that will not.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Nister in view of Caldwell in further view of Narang et al. (U.S. 11,124,204 B1).
Regarding claim 8, Nister and Caldwell teach the computer-implemented method of claim 1.
Yet Nister and Caldwell do not explicitly further teach:
A computer-implemented method further comprising:
filtering out road users by selecting a subset of road users in the surrounding of the vehicle,
wherein determining the ego occupancy information is performed based on the selected subset of road users.
However, Narang teaches:
A computer-implemented method further comprising:
filtering out road users by selecting a subset of road users in the surrounding of the vehicle (see Fig. 3C for a traffic situation labeled “Localized Environmental Representation”. The system is similar to Nester in that it is related to trajectory prediction of vehicle and trajectory generation of an ego vehicle. See Narang col. col. 1, lines 25-43 for the system having a “predefined driving goal” that is “continuously constrained by both driving rules of the road and human driving conventions”. The system considers surrounding objects as well as “map information such as: road boundaries, location of stop signs,” according to col. 24, lines 9-27 and lines 28-58. The system can determine a “safety tunnel,” which according to col. 24, lines 9-27 and lines 28-58 is essentially where the host vehicle can drive safely in the near future based on static and dynamic context information. According to col. 25, lines 11-29 “The safety tunnel can optionally be used to select which static and dynamic objects are within the safety tunnel, wherein only those objects are used for consideration and/or further processing (e.g., in determining the localized environmental representation, in determining a latent space representation, etc.). In some variations, for instance, localized dynamic and static object selectors (e.g., in the computing system) select the relevant surrounding objects based on the action output from the 1.sup.st learning module, its associated safety tunnel, as well as any information about these objects such as their location, distance from the ego vehicle, speed, and direction of travel (e.g., to determine if they will eventually enter the safety tunnel). Additionally or alternatively, relevant static and dynamic objects can be determined in absence of and/or independently from a safety tunnel (e.g., just based on the selected action, based on a predetermined set of action constraints, etc.), all static and dynamic objects can be considered, and/or S224 can be otherwise suitably performed.” See col. 25, lines 38-42 for teaching that in step S224 “The set of inputs can include any or all of the inputs described above…and/or any suitable set of combination of inputs.” This “includes any or all of: dynamic object information (e.g., within the safety tunnel) and their predicted paths; static object information (e.g., within the safety tunnel); one or more uncertainty estimates…a map” etc.
The reason to do this is described in col. 25, line 54 through col. 26, line 3. This section teaches that “The set of inputs are preferably used to determine a localized environmental representation, which takes into account the information collected…thereby producing a more targeted, relevant, and localized environmental representation for the agent based on the action selected, which is equivalently referred to herein as a localized environmental representation.” This reduces the amount of information processed.
When does S224 occur? See col. 23, lines 52-67 for the teaching that it can be done after S214. But it can also be done after S210 or S212, concurrently with S214, or “in response to S205.” According to Fig. 11, S205 is “receiving a set of inputs”. So “in response” to that, the system can run step S224 and remove or filter some inputs out. According to col. 25, lines 38-42 step S224 includes selecting or filtering “any or all” inputs, which “includes any or all of: dynamic object information (e.g., within the safety tunnel) and their predicted paths; static object information”. See Fig. 11 for the end result being generating a vehicle trajectory in S220.),
wherein determining the ego occupancy information is performed based on the selected subset of road users (see Fig. 11 for the end result being generating a vehicle trajectory in S220.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Nister and Caldwell, to add the additional features of filtering out road users by selecting a subset of road users in the surrounding of the vehicle, wherein determining the ego occupancy information is performed based on the selected subset of road users, as taught by Narang. The motivation for doing so would be to produce a more targeted and relevant set of vehicle actors in the environment of the host vehicle in order to reduce processing by the computer, as recognized by Narang (see col. 25, line 54 through col. 26, line 3.).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Caldwell at least strongly teaches towards this in paragraph 0061 which discusses computing valued based on the “nearest neighbors”.
Additional Art
The prior art made of record here, though not relied upon, is considered pertinent to the present disclosure.
Refaat et al. (U.S. 11,950,166) teaches at least using a neural network to determine vehicle occupancy. See Fig. 5 below.
PNG
media_image1.png
546
796
media_image1.png
Greyscale
Alvarez et al. (U.S. 11,702,105). Teaches at least using neural networks to determine trajectories and potential collisions. See Fig. 1 below.
PNG
media_image2.png
588
890
media_image2.png
Greyscale
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL M. ROBERT whose telephone number is (571)270-5841. The examiner can normally be reached M-F 7:30-4:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hunter Lonsberry can be reached at 571-272-7298. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL M. ROBERT/Primary Examiner, Art Unit 3665