DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This action is in response to amendments and remarks filed on 05/18/2026. Claim(s) 1-3, 7-9, 11-13, 17, and 19-23 have been amended. Claim(s) 4-6, 10, 14-16, 18, and 25 have been cancelled. Claim(s) 26-27 have been added. Claim(s) 1-3,7-9,11-13,17,19-24 and 26-27 are pending examination. This action is made final.
Response to Arguments
Applicant presents the following argument(s) regarding the previous office action:
Applicant asserts that the 103 rejections of claims 1-3, 7-9, 11-13, and 17-25 is improper. Applicant asserts that at least the independent claims 1, 11, and 20 are allowable over the prior art in light of the amendments.
Applicant’s arguments with respect to claim(s) 1-3, 7-9, 11-13, and 17-25 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Regarding applicant’s argument A, the examiner finds it moot. The applicant’s amendments to independent claims 1, 11, and 20 do not overcome the rejections under the prior art of Wray in combination with Fergusson. Applicant’s arguments rely on newly amended limitations to point to how their claim overcomes the prior art. After further search and consideration, the examiner would rely on newly cited portions of Wray to teach the amended limitations. These newly cited portions would teach new limitations as amended. Therefore claims 1, 11, and 20 would remain rejected as detailed below. The dependent claims would remain rejected at least due to their dependence on rejected independent claims. A more detailed explanation and mapping can be found below in the section titled, “Claim Rejections – 35 USC 103.”
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 1-3, 7-9, 11-13, 17, 19-24, and 26-27 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wray (US PG Pub 2019/0329771) in view of Fergusson (US Pat, 10,899,345).
Regarding claim 1, Wray teaches a computer-implemented method comprising: ([0037] teaches a computer method) receiving sensor data associated with a period of operation in an environment by at least one sensor of a vehicle; ([0196], [0205]-[0206], [0234], and at least [0236] teach a vehicle receiving data from a sensor of said vehicle)
processing the sensor data and thereby deriving (i) vehicle information that indicates a past and future trajectory of the vehicle during the period of operation, ([0184] teaches the system deriving the past and future locations of the vehicle. [0237] further teaches determining the future trajectory of the vehicle. The system describes these as scenarios and each scenario represents a potential vehicle state, wherein the state is a representation of the world around the vehicle, i.e. potential objects, and a scenario is a specific way the vehicle navigates a scene, i.e. an intersection) (ii) respective agent information for each agent object detected in the vehicle's surrounding environment during the period of operation that indicates ([0209] teaches the system identifying the current spatiotemporal information for an external moving object, i.e. agent. [0198] teaches the system may categorize the agent information to make a determination of how it may interact with the vehicle, i.e. a state change. [0240] further teaches the system determining the future trajectory of an agent vehicle) and (iii) respective non-agent information for each non-agent object detected in the vehicle's surrounding environment during the period of operation; ([0097] teaches the system identifying any object around the vehicle)
segmenting the period of operation into a sequence of contiguous decision units of the vehicle using the vehicle information, the respective agent information, the respective non-agent information, and at least one interaction prediction model that is configured to predict a likelihood of the vehicle's decision-making being impacted by a detected agent or non-agent object during a future time horizon, (0141]-[0146] teach the system segmenting a scenario into a series of states and state changes. These states/state changes represent distinct points in time during which the vehicle makes a change in its actions for the scenario presented to the vehicle. [0141]-[0148] teaches the use of a Markov decision process to evaluate the current and future actions of a vehicle based on the vehicle’s state and the respective environmental state. The vehicle states are determined as discrete spatial-temporal locations that are differentiated by the changes represented to achieve the new state. [0225]-[0228] teaches the system determining the likelihood that an agent will impact the vehicle’s decision making process) wherein each respective decision unit in the sequence of contiguous decision units comprises a respective continuous interval of time spanning a plurality of successive time points during which there is no change in which agents and non-agent objects are considered to be relevant to the vehicle’s decision-making, ([0141]-[0148], [0166], and [0223] teach the system monitoring the “temporal location” these locations include environmental information for each time period. As each temporal location is a slice in time there is no change to what is relevant to a vehicle for a given time period. [0235] further teaches that for each temporal location information about the relevant objects is recorded. The states and temporal locations are not limited to a single specific time point, rather they are the time during which the vehicle is in a given state for as long as it needs.) wherein each transition between contiguous decision units is defined by a time point when there is a change in which agent and non-agent objects are considered to be relevant to the vehicle's decision-making, ([0148]-[0151] teach that state transitions, i.e. changes in decision units, can be identified based on changes to the operational environment around a vehicle if the weather changes, if a new vehicle arrives, if a pedestrian walks into the environment. All of these changes alter how the vehicle interacts with the world and would change what is relevant to a vehicle) and wherein the segmenting involves evaluating whether each respective time point of a series of time points during the period of operation comprises a boundary point between a pair of contiguous decision units ([0141] teaches the system identifying an action that moves it from one state to another) by:
evaluating whether there has been a change to which agent or non-agent objects are relevant to the vehicle's decision-making at the respective time by:
identifying a respective set of objects that were detected within the vehicle’s surrounding environment at the respective time point; ([0223]-[0224] and [0235]-[0240] teach the system sensing and identifying a respective set of objects around the ego vehicle at a given time period as a series of scenarios as the vehicle moves through the environment)
determining whether any individual agent object and any individual non-agent object in the respective set of objects is considered to be relevant to the vehicle’s decision-making ([0225]-[0226] teaches the system determining that a detected object at a respective time period is relevant to the vehicle operation. This can be done by identifying that the surrounding object is on a collision course with the ego vehicle) using the at least one interaction prediction model along with (i) a portion of the vehicle information associated with the respective time point, (ii) respective agent information for any agent object detected in the vehicle's surrounding environment at the respective time point, and (iii) respective non-agent information for any non-agent object detected in the vehicle's surrounding environment at the respective time point; ([0225]-[0231] teaches that the system is constantly evaluating its surroundings and that it may determine that an external object, or itself has changed in a way that would impact the decision making process)
based on the determining, identifying a respective subset of objects considered to be relevant to the vehicle’s decision-making at the respective time (Fig. 7 and [0232]-[0241] teaches the system determining which of a detected subset of objects, i.e. “pedestrians” are considered relevant to the vehicle’s travels and thus its decision making)
comparing the respective subset of objects considered to be relevant to the vehicle’s decision-making at the respective time to a subset of objects considered to be relevant to the vehicle’s decision-making at a preceding time in the series of time points; ([0133], [0209], [0236], and [0256] teach the system continuously updating the observed data in regards to the surroundings and ensuring considered which objects are relevant at each respective time period, [0184] teaches the system identifying stale data, that is data that is no longer deemed to be relevant to the given operation) and
based on the evaluating, determining either that (i) the respective time point defines a change in decision unit of the vehicle and comprises a boundary point between a pair of contiguous decision units if there has been a change to which agent or non-agent objects are relevant to the vehicle's decision-making at the respective time point or (ii) the respective time point does not define a change in decision unit of the vehicle and does not comprise a boundary point between a pair of contiguous decision units if there has not been a change to which agent or non-agent objects are relevant to the vehicle's decision-making at the respective time point; ([0235]-[0240] teaches the system may determine for each temporal period whether or not the environment around the vehicle has changed and based on the changes update to the next temporal location which includes a new vehicle state and new vehicle action relevant to the environment) and
generating a respective data representation of each respective decision unit in the sequence of contiguous decision units comprising a discrete searchable data structure that encodes the vehicle's interaction with (i) any agent object determined to be relevant to the vehicle's decision-making during the respective decision unit and (ii) any non-agent object determined to be relevant to the vehicle's decision-making during the respective decision unit ([0075]-[0079] teach the system storing the series of scenarios, each scenario is determined to have the operational environment and the information relevant to the vehicle as a state. The scenarios are stored in a way that the computer can access them when needed. [0137] teaches the further idea of generating state specific scenarios that model the exact environment and surroundings of a vehicle. These states are generated and stored in a way that the vehicle can then access them by using present information to find a matching scenario, i.e. a searchable data structure) wherein the discrete searchable data structure of at least a subset of the respective decision units encodes the vehicle's interaction with multiple different agent or non-agent objects determined to be relevant to the vehicle's decision-making during the respective decision unit, ([0101] teaches the system having a scenario in which there are multiple pedestrians around the vehicle, i.e. multiple agents relevant to the vehicle’s decision making) and
causing the respective data representation of each respective decision unit to be stored within a searchable data repository. ([0117], [0190], [0195], and [0213] all teach the system storing data representing the environment around the vehicle for a given scenario)
Wray does not teach respective agent information for each agent object detected in the vehicle's surrounding environment during the period of operation that indicates a respective past.
However, Fergusson teaches “respective agent information for each agent object detected in the vehicle's surrounding environment during the period of operation that indicates a respective past.” (Col. 18, lines 38-41; teach determining the past trajectory of an external agent vehicle)
It would have been prima facie obvious to one of ordinary skill in the art, before the effective filing date, to incorporate the teachings of Wray in view of Fergusson; and have a reasonable expectation of success. Both teach vehicle perception and control systems. Understanding where an agent comes from can allow the system to understand what may happen next. If a vehicle knows that an agent comes from one direction and can understand the route it takes, the vehicle can then determine what may be an unlikely way for the agent to go. This allows for optimal route planning. As Fergusson teaches in Col. 1, Background, a vehicle with a better perception system allows for safer decision making. This keeps drivers safe.
Claims 11 and 20 are substantially similar and would be rejected for the same rationale as above.
Regarding claim 2, Wray teaches the computer-implemented method of claim 1, wherein discrete searchable data structure of each respective decision unit further encodes (i) vehicle information that indicates a past and future trajectory of the vehicle during the respective decision unit (Fig. 7 and [0235]-[0237] teaches the system may can have a model of a scenario, the scenario can have the trajectory information of the vehicle) and (ii) one or both of (a) agent information that indicates a past and future trajectory of at least one agent object that is determined to be relevant to the vehicle's decision-making during the respective decision unit, or (b) non-agent information for at least one non-agent object that is determined to be relevant to the vehicle's decision-making during the respective decision unit. (Fig. 7 and [0235]-[0239] teaches the system will have the state information of the agent and non-agents around the vehicle including their respective trajectory information at different temporal locations)
Claim 12 is substantially similar and would be rejected for the same rationale as above.
Regarding claim 3, Wray teaches the computer-implemented method of claim 2, wherein, within the discrete searchable data structure of each respective decision unit: (i) the encoded vehicle information that indicates the past and future trajectory of the vehicle during the respective decision unit includes confidence information indicating an estimated accuracy of the past and future trajectory of the vehicle ([0126]-[0128] teach the system having a probability of how likely the vehicle is to take a given trajectory based on the given scenario information. [0158] further teaches the system determining that the observations around the vehicle are accurate.) and (ii) the encoded agent information that indicates the past and future trajectory of the at least one agent object that is determined to be relevant to the vehicle's decision-making during the respective decision unit includes confidence information indicating an estimated accuracy of the past and future trajectory of the at least one agent object. (0131], [0154] and [0158] teach the system have determined a probability, or confidence, that the modeled information is accurate and that the trajectories of the vehicle and/or agents are correct)
Claim 13 is substantially similar and would be rejected for the same rationale as above.
Regarding claim 7, Wray teaches the computer-implemented method of claim 1, further comprising: based on a selected decision unit included in the sequence of contiguous decision units, generating at least one synthetic scene comprising a predicted evolution in time. ([0080] teaches the system having to create a compound scenario, which is when multiple scenarios are combined. Fig. 8 and [0261] and [0272]-[0273] teach the system determining alternative versions of each scenario for a given time frame.)
Claim 17 is substantially similar and would be rejected for the same rationale as above.
Regarding claim 8, Wray teaches the computer-implemented method of claim 7, wherein generating the at least one synthetic scene comprises: utilizing a predictive model to generate, for the selected decision unit, at least one predicted evolution in time of one or both of (i) vehicle information that indicates a past and future trajectory of the vehicle during the selected decision unit (Fig. 8 and [261 teach the system generating alterative trajectory data for the vehicle during a temporal location) or (ii) agent information that indicates a past and future trajectory of at least one agent object determined to be relevant to the vehicle's decision-making during the selected decision unit. (Fig. 8 and [0268] and [0272]-[0273] teach the system generating alternative trajectories for each of the agent trajectories at various temporal locations and modeling the scenarios over different time points)
Regarding claim 9, Wray teaches the computer-implemented method of claim 1, further comprising: based on the respective data representations of (i) a first decision unit included in the sequence of contiguous decision units and (ii) a second decision unit included in the sequence of contiguous decision units, generating a data representation of a new synthetic scene ([0080] teaches the system having to create a compound scenario, which is when multiple scenarios are combined. Fig. 8 and [0261] and [0272]-[0273] teach the system determining alternative versions of each scenario for a given time frame. Additionally, [0250]-[0256] teaches the system taking a series of decision units, i.e. scenarios, and instantiating them at the same time which would be analogous to a representation of a new scenario) comprising: a discrete searchable data structure that encodes a synthetic combination of (i) the vehicle’s interaction with any agent of non-agent object determined to be relevant to the vehicle’s decision-making during the first decision unit and (ii) the vehicle’s interaction with any agent or non-agent object determined to be relevant to the vehicle’s decision-making during the second decision unit. ([0080] teaches the system having to create a compound scenario, which is when multiple scenarios are combined. As further taught in [0137] the system can generate a distinct scenario for a time by combining scenarios. This scenario can include information relevant to the vehicle as it travels between multiple states in time and how the vehicle interacts with its surroundings.)
Claim 19 is substantially similar and would be rejected for the same rationale as above.
Regarding claim 21, Wray teaches the computer-implemented method of claim 1, wherein a change to which agent or non-agent objects are relevant to the vehicle's decision-making at the respective time point comprises one of: a change from (i) no agent or non-agent object determined to be relevant to the vehicle's decision-making at a prior time point to (ii) at least one agent or non-agent object determined to be relevant to the vehicle's decision-making at the respective time point; ([0260]-[0270] teaches the system monitoring potential agents to determine the relevance of the agents, this includes updating the environmental information to include changes to the number of agents detected around the vehicle)
a change from (i) at least one agent or non-agent object determined to be relevant to the vehicle's decision-making at a prior time point to (ii) no agent or non-agent object determined to be relevant to the vehicle's decision-making at the respective time point; ([0260]-[0270] teaches the system monitoring potential agents to determine the relevance of the agents, this includes updating the environmental information to include changes to the number of agents detected around the vehicle) or
a change from (i) a first set of one or more agent or non-agent objects determined to be relevant to the vehicle's decision-making at a prior time point to (ii) a second set of one or more agent or non-agent objects determined to be relevant to the vehicle's decision-making at the respective time point, wherein the first set one or more agent or non-agent objects differs from the second set of one or more agent or non-agent objects. ([0260]-[0270] teaches the system monitoring potential agents to determine the relevance of the agents, this includes updating the environmental information to include changes to the number of agents detected around the vehicle)
Regarding claim 22, Wray teaches the computer-implemented method of claim 1, wherein at least one given decision unit included in the sequence of contiguous decision units comprises a continuous interval of time during which no agent or non-agent object is determined to be relevant to the vehicle's decision-making, and wherein the respective representation of the given decision unit comprises a discrete searchable data structure indicating that the vehicle did not interact with any agent or non-agent object determined to be relevant to the vehicle's decision-making during the given decision unit. ([0147] teaches the changes in vehicle state at each temporal location are used as an analogous structure to the decision unit in the current application. [0150] teaches that changes in state information can be reflective of no interaction between a vehicle and agents, the given example being a change in state based on the vehicle arriving at an intersection. This is a change in state without the requisite interaction with an agent)
Regarding claim 23, Wray teaches the computer-implemented method of claim 1, wherein determining whether any individual agent object and any individual non-agent object in the respective set of objects is considered to be relevant to the vehicle’s decision-making comprises: inputting, into the at least one interaction prediction model, (i) the portion of the vehicle information associated with the respective time point, (ii) the respective agent information for any agent object detected in the vehicle's surrounding environment at the respective time point, and (iii) respective non-agent information for any non-agent object detected in the vehicle's surrounding environment at the respective time point; ([0149]-[0151] teaches the system determining the probability that a vehicle’s future decision making will be impacted by changes in the operating environment of the area. This prediction is further explained in the example given in [0226]-[0230] in which the vehicle system inputs information regarding location and trajectory of the vehicle and agents/non-agents to determine the likelihood of interference between the two)
for any agent object detected in the vehicle's surrounding environment, using the at least one interaction prediction model to predict a respective likelihood of the vehicle's decision-making being impacted by the agent object during the future time point horizon; ([0149]-[0151] teaches the system determining the probability that a vehicle’s future decision making will be impacted by changes in the operating environment of the area) and
for any non-agent object detected in the vehicle's surrounding environment, using the at least one interaction prediction model to predict a respective likelihood of the vehicle's decision- making being impacted by the non-agent object during the future time point horizon. ([0149]-[0151] teaches the system determining the probability that a vehicle’s future decision making will be impacted by changes in the operating environment of the area)
Regarding claim 24, Wray teaches the computer-implemented method of claim 1, wherein the at least one interaction prediction model utilizes a confidence level associated with sensor data for the detected agent or non-agent object to predict the likelihood of the vehicle's decision-making being impacted by the detected agent or non-agent object during the future time horizon. ([0149]-[0151] teaches the system determining the probability that a vehicle’s future decision making will be impacted by changes in the operating environment of the area)
Regarding claim 26, Wray teaches the computer-implemented method of claim 1, further comprising: performing a search of the discrete searchable data structures stored within the searchable data repository to identify decision units involving a given type of interaction between objects; ([0082] and [0109]-[0110] teaches the system looking for scenarios that are correspond to the given interaction a vehicle is in) and based on performing the search, identifying one or more decision units from the sequence of contiguous decision units that involve the given type of interaction between objects. ([0109]-[0110] teach the system identifying the types of interactions that the vehicle is experiencing between itself and the objects around the vehicle)
Regarding claim 27, Wray teaches the computer-implemented method of claim 1, wherein the discrete searchable data structure of each respective decision unit omits (i) agent information for any agent object that was detected within the surrounding environment of the vehicle but is not determined to be relevant to the vehicle's decision-making during the respective decision unit and (ii) non- agent information for any non-agent object that was detected within the surrounding environment of the vehicle but is not determined to be relevant to the vehicle's decision-making during the respective decision unit. ([0113]-[0114] teaches the system having a variable threshold that can omit information from elements it deems are not essential to the current operation of the vehicle from the scenario system)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Russell (US PG Pub 2020/0004256) teaches controlling a vehicle in an autonomous driving mode. For example, sensor data identifying a plurality of objects may be received. Pairs of objects of the plurality of objects may be identified. For each identified pair of objects of the plurality of objects, a similarity value which indicates whether the objects of that identified pair of objects can be responded to by the vehicle as a group may be determined. The objects of one of the identified pairs of objects may be clustered together based on the similarity score. The vehicle may be controlled in the autonomous mode by responding to each object in the cluster in a same way.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS STRYKER whose telephone number is (571)272-4659. The examiner can normally be reached Monday-Friday 7:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Christian Chace can be reached at (571) 272-4190. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.S./Examiner, Art Unit 3665 /CHRISTIAN CHACE/Supervisory Patent Examiner, Art Unit 3665
Prosecution Insights