DETAILED ACTION
This is a first Office Action on the merits and is responsive to the originally filed application papers. Claims filed on 07/22/2025 are being examined. Claims 2-21 are being considered and further pending examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS(s)) submitted on 09/04/2025 is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 17-19 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Gate et al (US 11595619 B1) henceforth referred to as Gate.
Regarding claim 17, the combination of Gate and Lockwood teaches One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising (col 10 : “In general, the routines executed to implement the various implementations described herein, whether implemented as part of an operating system or a specific application, component, program, object, module or sequence of instructions, or even a subset thereof, will be referred to herein as “program code.” Program code typically comprises one or more instructions that are resident at various times in various memory and storage devices, and that, when read and executed by one or more processors, perform the steps necessary to execute steps or elements embodying the various aspects of the invention. Moreover, while the invention has and hereinafter will be described in the context of fully functioning computers and systems, it will be appreciated that the various implementations described herein are capable of being distributed as a program product in a variety of forms, and that the invention applies equally regardless of the particular type of computer readable media used to actually carry out the distribution. Examples of computer readable media include tangible, non-transitory media such as volatile and non-volatile memory devices, floppy and other removable disks, solid state drives, hard disk drives, magnetic tape, and optical disks (e.g., CD-ROMs, DVDs, etc.), among others.”):
receiving, at a computing device external to a vehicle, first data associated with an environment of the vehicle, the first data indicating a flagger is located within a path of the vehicle; (col 7 line 5-8 : “A primary sensor system 130 may include various sensors suitable for collecting information from a vehicle's surrounding environment for use in controlling the operation of the vehicle.”, col 13 line 37-40 : “It may also be desirable to propagate data collected during a teleoperations session (e.g., data related to lane closures, detected construction or incidents, etc.) to other vehicles in a fleet.”, col 15 line 20-26 : “A non-exclusive list of potential conditions that may be sensed and used to trigger monitoring includes construction zones, temporary traffic devices, authorities directing traffic, loading or unloading school buses, emergency vehicles, the placement of signs, cones, or flares, crossing guards, funeral processions, collisions, inoperative traffic devices, etc.”, col 15 line 10-12 : “As illustrated in block 252, an autonomous vehicle may initially detect a condition that triggers a desire for teleoperations monitoring.”)
based at least in part on receiving the first data, initiating a guidance session with the vehicle (col 15 line 32-36 : “For example, it may be desirable in some implementations to automatically initiate teleoperations sessions with a vehicle in order to reassess certain conditions that have previously been detected, whether by the same autonomous vehicle or by other autonomous vehicles.”, col 16 line 4-11 : “Once a session has been initiated, the autonomous vehicle may initiate streaming of session data in block 258. In some implementations, the session data can include autonomy data, sensor data, telemetry data and video data. The teleoperations system may receive the streamed session data and present the session data to the operator via the teleoperations user interface in block 260.”);
generating, during the guidance session, guidance data configured to assist the vehicle in navigating through the environment, wherein during the guidance session the guidance data comprises one of:
first guidance data indicative of a teleoperator continuously activating an input device and instructing the vehicle to navigate through the environment, or
second guidance data indicative of the teleoperator releasing the input device and instructing the vehicle to stop in the environment (col 16-17 : “Next, in block 264, the vehicle receives operator input, and block 266 validates the operator input. As noted above, in the illustrated implementation, an indirect control methodology is utilized whereby a teleoperations operator generally provides commands representing recommendations to an autonomous vehicle, which are then processed by the vehicle to implement those commands if appropriate. A recommendation refers to a high level directive provided to an autonomous vehicle. Based on the recommendation that is a high level directive, the autonomous vehicle control system may determine and execute a particular operation to control the autonomous vehicle rather than directly controlling the autonomous vehicle with the recommendation. For example, instead of providing, from a teleoperations system to the autonomous vehicle control system, commands specifying specific velocities, steering angles, and brake controls, recommendations such as “change lanes,” “follow a suggested path,” and “disregard a sensed condition” can be provided from a teleoperations system to an autonomous vehicle control system such that the autonomous vehicle control system can determine a particular operation such as specific velocities, steering angles, and brake controls to implement the recommendation. In these implementations, by providing the recommendations rather than the specific commands, less data may be transferred between the teleoperations system and the autonomous vehicle control system such that fewer network resources may be used.”, where indication of a teleoperator operating an input device is an indication that a teleoperator was continuously activating an input for at least some moment of time as it would be impossible for a teleoperator to operate an input device for only a singular instant of time.);
sending the guidance data to the vehicle (col 17 : “In these implementations, by providing the recommendations rather than the specific commands, less data may be transferred between the teleoperations system and the autonomous vehicle control system such that fewer network resources may be used.”); and
receiving second data indicating that the vehicle has performed an action that deviates from the guidance data (col 15 : “As illustrated in block 252, an autonomous vehicle may initially detect a condition that triggers a desire for teleoperations monitoring.”, col 17 : “In some implementations, it may also be desirable to delay or prohibit execution of commands until it is deemed safe for a vehicle to do so, e.g., based upon any perception constraints for the autonomous vehicle. For example, if a vehicle were to receive a directive to drive around an obstruction in a lane by driving into a lane that is primarily used for traffic in the opposite direction (e.g., to drive on the other side of a two-lane road), the vehicle may delay executing the directive until the perception component of the autonomous vehicle determines that the lane is free of traffic and/or pedestrians.”, where data transmitted for teleoperations monitoring occurring during the teleoperations session in which the command is sent would be data that would indicate when the vehicle disregarded/delayed execution of the directive.).
Regarding claim 18, the Gate teaches The one or more non-transitory computer-readable media as recited in claim 17, further Gate teaches wherein the action comprises performing a stop at an intermediate location between a first location within a zone associated with the flagger and a second location outside the zone associated with the flagger (col 17 : “In some implementations, it may also be desirable to delay or prohibit execution of commands until it is deemed safe for a vehicle to do so, e.g., based upon any perception constraints for the autonomous vehicle. For example, if a vehicle were to receive a directive to drive around an obstruction in a lane by driving into a lane that is primarily used for traffic in the opposite direction (e.g., to drive on the other side of a two-lane road), the vehicle may delay executing the directive until the perception component of the autonomous vehicle determines that the lane is free of traffic and/or pedestrians.”, where stopping to delay the directive at any location after the directive has been transmitted would be an intermediate location between a first location within a zone associated with the flagger (such as the zone including where the flagger was first detected) and a second location outside the zone as there will always be a zone outside of the selected zone).
Regarding claim 19, the combination of Gate and Lockwood teaches The one or more non-transitory computer-readable media as recited in claim 17, further Gate teaches the operations further comprising:
prior to generating the guidance data, verifying that the first data indicates the flagger is located in the environment (col 23 : “In this regard, the context data may represent practically any data that may be relevant to the current state of the vehicle and its surroundings, and furthermore relevant in some manner to any decisions that may be asked of a teleoperations system operator during a teleoperations session—particularly decisions associated with verifying any conditions detected by a detector in the autonomous vehicle.”, where verification occurs before a teleoperator would select to transmit guidance data.).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2-4, 6-8, 10, 12-13, and 20-21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gate et al (US 11595619 B1) henceforth referred to as Gate and further in view of Lockwood et al (US 10386836 B2) henceforth referred to as Lockwood.
Regarding claim 2 Gate teaches A system comprising (col 5 line 56-58 : “The various implementations discussed hereinafter are generally directed to a teleoperations system for use with an autonomous vehicle.”):
one or more processors; and
one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising (col 6 line 64-67 – col 7 line 1-4 : “In the illustrated implementation, autonomous control over vehicle 100 (which may include various degrees of autonomy as well as selectively autonomous functionality) is primarily implemented in a primary vehicle control system 120, which may include one or more processors 122 and one or more memories 124, with each processor 122 configured to execute program code instructions 126 stored in a memory 124.”):
receiving first sensor data from one or more sensors associated with a vehicle, the first sensor data associated with an environment of the vehicle (col 7 line 5-8 : “A primary sensor system 130 may include various sensors suitable for collecting information from a vehicle's surrounding environment for use in controlling the operation of the vehicle.”);
detecting, based at least in part on the first sensor data, a first object within the environment, the first object located within a first path of the vehicle (col 15 : “As illustrated in block 252, an autonomous vehicle may initially detect a condition that triggers a desire for teleoperations monitoring. As will become more apparent below, a condition may include, for example, a detection by a detector in a perception component, a request from a planner component for assistance with selection of a suitable scenario or path given current circumstances, a request initiated by a vehicle occupant, an error condition detected in the autonomous vehicle, or in other situations where teleoperations assistance may be desired. A non-exclusive list of potential conditions that may be sensed and used to trigger monitoring includes construction zones, temporary traffic devices, authorities directing traffic, loading or unloading school buses, emergency vehicles, the placement of signs, cones, or flares, crossing guards, funeral processions, collisions, inoperative traffic devices, etc.”);
based at least in part on detecting the first object, initiating a guidance session with a teleoperator in part by sending first data to a computing device, wherein, during the guidance session, the teleoperator provides guidance data by interacting with an input device associated with the computing device (col 15 : “A non-exclusive list of potential conditions that may be sensed and used to trigger monitoring includes construction zones, temporary traffic devices, authorities directing traffic, loading or unloading school buses, emergency vehicles, the placement of signs, cones, or flares, crossing guards, funeral processions, collisions, inoperative traffic devices, etc.”, col 15 : “Now returning to FIG. 4, in response to detection of the condition, block 254 may communicate with the teleoperations service to request a session, which in block 256 initiates the session and connects the vehicle with an operator via a teleoperations user interface.”, col 16 : “Thereafter, once presented with the session data, an operator may (i) generate one or more inputs that attempt to address the condition that triggered the session and (ii) communicate those inputs to the vehicle in block 262.”);
receiving, during the guidance session, the guidance data indicative of the teleoperator activating the input device (col 12 line 33-42 : “A teleoperations operator user interface 226 is coupled to module 212 to provide a user interface through which an operator, e.g., a human operator, may communicate with a vehicle 202 during a session. The user interface may be implemented in any number of suitable manners, and may utilize text, graphics, video, audio, virtual or augmented reality, keyboard input, mouse input, touch input, voice input, gesture input, etc. Dedicated or customized controls and/or indicators may also be used in some implementations.”, col 16 : “Thereafter, once presented with the session data, an operator may (i) generate one or more inputs that attempt to address the condition that triggered the session and (ii) communicate those inputs to the vehicle in block 262.);
based at least in part on receiving the guidance data, causing the vehicle to navigate past a first location in the environment (col 16 : “The inputs may request additional data from a vehicle, may include one or more commands that instruct the vehicle to take a certain action, or may otherwise interact with the vehicle in order to address the condition.”, col 16 : “Next, in block 264, the vehicle receives operator input, and block 266 validates the operator input. As noted above, in the illustrated implementation, an indirect control methodology is utilized whereby a teleoperations operator generally provides commands representing recommendations to an autonomous vehicle, which are then processed by the vehicle to implement those commands if appropriate. A recommendation refers to a high level directive provided to an autonomous vehicle. Based on the recommendation that is a high level directive, the autonomous vehicle control system may determine and execute a particular operation to control the autonomous vehicle rather than directly controlling the autonomous vehicle with the recommendation. For example, instead of providing, from a teleoperations system to the autonomous vehicle control system, commands specifying specific velocities, steering angles, and brake controls, recommendations such as “change lanes,” “follow a suggested path,” and “disregard a sensed condition” can be provided from a teleoperations system to an autonomous vehicle control system such that the autonomous vehicle control system can determine a particular operation such as specific velocities, steering angles, and brake controls to implement the recommendation.”);
receiving second sensor data from the one or more sensors associated with the vehicle (col 16 line 4-11 : “Once a session has been initiated, the autonomous vehicle may initiate streaming of session data in block 258. In some implementations, the session data can include autonomy data, sensor data, telemetry data and video data. The teleoperations system may receive the streamed session data and present the session data to the operator via the teleoperations user interface in block 260.”). However, Gate does not explicitly teach detecting, based at least in part on the second sensor data, a second object located within a second path of the vehicle;
determining a potential collision between the vehicle and the second object; and
based at least in part on the potential collision, causing the vehicle to override the guidance data and stop at an intermediate location between the first location and a second location outside a zone associated with the first object.
However, in a similar field of endeavor (collision avoidance for autonomous vehicles), Lockwood teaches detecting, based at least in part on the second sensor data, a second object located within a second path of the vehicle (col 11-12 : “The example vehicle systems 202 also include one or more of a planner 214, an object data calculator 216, an object classifier 218, a collision predictor system 220, a kinematics calculator 222, and a safety system actuator 224. The vehicle systems 202 may be configured to access one or more data stores including, but not limited to, an object type data store 226. The object type data store 226 may include data representing object types associated with object classifications for objects detected in the environment 100.”);
determining a potential collision between the vehicle and the second object (col 13 line 8-13 : “In some examples, the collision predictor system 220 may be configured to use the data representing the object type, the predicted object behavior, the data representing the trajectory of the object, and/or the data representing the trajectory of the vehicle 102, to predict a collision between the vehicle 102 and the object.”, col 14 line 27-32 : “In some examples, the collision predictor system 220 may be used to predict a collision between the vehicle 102 and an object in the environment 100 based on the object type, whether the object is moving, the trajectory of the vehicle 102, the predicted path of the object obtained from the planner 214.”; and
based at least in part on the potential collision, causing the vehicle to override the guidance data and stop at an intermediate location between the first location and a second location outside a zone associated with the first object (col 13 line 46-63 : “In some examples, the safety system actuator 224 may be configured to activate one or more safety systems of the autonomous vehicle 102 when a collision is predicted by the collision predictor 220 and/or the occurrence of other safety related events, such as, for example, an emergency maneuver by the vehicle 102, such as hard braking or a sharp acceleration. The safety system actuator 224 may be configured to activate an interior safety system (e.g., including seat belt pre-tensioners and/or air bags), an exterior safety system (e.g., including warning sounds and/or warning lights), the drive system 232 configured to execute an emergency maneuver to avoid a collision, and/or any combination thereof. For example, the drive system 232 may receive data for causing a steering system of the vehicle 102 to change the travel direction of the vehicle 102, and a propulsion system of the vehicle 102 to change the speed of the vehicle 102 to alter the trajectory of vehicle 102 from an initial trajectory to a trajectory for avoiding a collision.”, as the detection of an object and determination of a possible collision can only occur before, during, or after the vehicle in any case where an potential collision is predicted before or after passing of the first location the guidance data would be overridden to stop at a location between two locations outside a zone associated with the first object).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to modify the system of Gate with the collision predictor and safety system actuator of Lockwood to improve safety by avoiding predicted collisions while navigating the vehicle.
Regarding claim 3, the combination of Gate and Lockwood teaches The system as recited in claim 2, further Gate teaches wherein during the guidance session,
the guidance data comprises one of: first guidance data indicative of the teleoperator continuously activating the input device and instructing the vehicle to navigate through the zone, or
second guidance data indicative of the teleoperator releasing the input device and instructing the vehicle to stop in the environment (col 12 line 33-42 : “A teleoperations operator user interface 226 is coupled to module 212 to provide a user interface through which an operator, e.g., a human operator, may communicate with a vehicle 202 during a session. The user interface may be implemented in any number of suitable manners, and may utilize text, graphics, video, audio, virtual or augmented reality, keyboard input, mouse input, touch input, voice input, gesture input, etc. Dedicated or customized controls and/or indicators may also be used in some implementations.”, col 16 : “Thereafter, once presented with the session data, an operator may (i) generate one or more inputs that attempt to address the condition that triggered the session and (ii) communicate those inputs to the vehicle in block 262.).
Regarding claim 4, the combination of Gate and Lockwood teaches The system as recited in claim 2, further Gate teaches the operations further comprising:
based at least in part on causing the vehicle to override the guidance data, transmitting second data to the computing device indicating the potential collision (col 14 : “For the interface with perception component 240, teleoperations control module 208 may be configured to receive from the perception component actors and/or tracks of actors detected in the environment, detections by various detectors 246 implemented in the perception component, and other perception-related data. All of such data may be communicated by module 208 to a teleoperations system as autonomy data. In addition, as will become more apparent below, module 208 may also communicate to perception component 240 any detector overrides generated by a teleoperations system, e.g., to indicate when a detector 246 has likely generated a false positive detection.”).
Regarding claim 6, it recites a method with limitations substantially the same as claim 2 above, therefore it is rejected for the same reason.
Regarding claim 7, the combination of Gate and Lockwood teaches The method as recited in claim 6, further Lockwood teaches further comprising:
detecting, based at least in part on the second sensor data, a second object located within a second path of the vehicle (col 11-12 : “The example vehicle systems 202 also include one or more of a planner 214, an object data calculator 216, an object classifier 218, a collision predictor system 220, a kinematics calculator 222, and a safety system actuator 224. The vehicle systems 202 may be configured to access one or more data stores including, but not limited to, an object type data store 226. The object type data store 226 may include data representing object types associated with object classifications for objects detected in the environment 100.”); and
determining, based at least in part on the second sensor data, a potential collision between the vehicle and the second object (col 13 line 8-13 : “In some examples, the collision predictor system 220 may be configured to use the data representing the object type, the predicted object behavior, the data representing the trajectory of the object, and/or the data representing the trajectory of the vehicle 102, to predict a collision between the vehicle 102 and the object.”, col 14 line 27-32 : “In some examples, the collision predictor system 220 may be used to predict a collision between the vehicle 102 and an object in the environment 100 based on the object type, whether the object is moving, the trajectory of the vehicle 102, the predicted path of the object obtained from the planner 214.”),
wherein causing the vehicle to stop at the intermediate location is based at least in part on determining the potential collision (col 13 line 46-63 : “In some examples, the safety system actuator 224 may be configured to activate one or more safety systems of the autonomous vehicle 102 when a collision is predicted by the collision predictor 220 and/or the occurrence of other safety related events, such as, for example, an emergency maneuver by the vehicle 102, such as hard braking or a sharp acceleration. The safety system actuator 224 may be configured to activate an interior safety system (e.g., including seat belt pre-tensioners and/or air bags), an exterior safety system (e.g., including warning sounds and/or warning lights), the drive system 232 configured to execute an emergency maneuver to avoid a collision, and/or any combination thereof. For example, the drive system 232 may receive data for causing a steering system of the vehicle 102 to change the travel direction of the vehicle 102, and a propulsion system of the vehicle 102 to change the speed of the vehicle 102 to alter the trajectory of vehicle 102 from an initial trajectory to a trajectory for avoiding a collision.”, as the detection of an object and determination of a possible collision can only occur before, during, or after the vehicle in any case where an potential collision is predicted before or after passing of the first location the guidance data would be overridden to stop at a location between two locations outside a zone associated with the first object).
Regarding claim 8, the combination of Gate and Lockwood teaches The method as recited in claim 6, further Gate teaches further comprising:
determining, based at least in part on the second sensor data, that a state of the first object has changed, the state of the first object indicating that the vehicle is to stop (col 16 : “The sensor data may include, for example, data captured from various sensors such as radar, LIDAR, etc. In some implementations, the sensor data can also include image or video data obtained from one or more cameras.”, col 20-21 : “school bus detectors that may be used to detect the presence of a school bus having its stop arm extended (indicating that vehicles must stop to allow passengers to board or get off of the bus)”),
further Lockwood teaches wherein causing the vehicle to stop at the intermediate location is based at least in part on determining that the state of the first object has changed (col 13 line 46-63 : “In some examples, the safety system actuator 224 may be configured to activate one or more safety systems of the autonomous vehicle 102 when a collision is predicted by the collision predictor 220 and/or the occurrence of other safety related events, such as, for example, an emergency maneuver by the vehicle 102, such as hard braking or a sharp acceleration. The safety system actuator 224 may be configured to activate an interior safety system (e.g., including seat belt pre-tensioners and/or air bags), an exterior safety system (e.g., including warning sounds and/or warning lights), the drive system 232 configured to execute an emergency maneuver to avoid a collision, and/or any combination thereof. For example, the drive system 232 may receive data for causing a steering system of the vehicle 102 to change the travel direction of the vehicle 102, and a propulsion system of the vehicle 102 to change the speed of the vehicle 102 to alter the trajectory of vehicle 102 from an initial trajectory to a trajectory for avoiding a collision.”, where determination of a potential collision is a factor of the environment and the states of objects in the environment, so a potential collision would be detected based at least in part on a determination of the environment including the state of the first object having changed.).
16. Regarding claim 10, the combination of Gate and Lockwood teaches The method as recited in claim 6, further Gate teaches wherein the guidance data is first guidance data, the first guidance data indicative of the teleoperator continuously activating the input device (col 12 line 33-42 : “A teleoperations operator user interface 226 is coupled to module 212 to provide a user interface through which an operator, e.g., a human operator, may communicate with a vehicle 202 during a session. The user interface may be implemented in any number of suitable manners, and may utilize text, graphics, video, audio, virtual or augmented reality, keyboard input, mouse input, touch input, voice input, gesture input, etc. Dedicated or customized controls and/or indicators may also be used in some implementations.”, col 16 : “Thereafter, once presented with the session data, an operator may (i) generate one or more inputs that attempt to address the condition that triggered the session and (ii) communicate those inputs to the vehicle in block 262.”, where indication of a teleoperator operating an input device is an indication that a teleoperator was continuously activating an input for at least some moment of time as it would be impossible for a teleoperator to operate an input device for only a singular instant of time.).
17. Regarding claim 12, the combination of Gate and Lockwood teaches The method as recited in claim 6, further Gate teaches further comprising:
prior to receiving the guidance data from the computing device, receiving a verification that the first object is within the environment (col 23 : “In this regard, the context data may represent practically any data that may be relevant to the current state of the vehicle and its surroundings, and furthermore relevant in some manner to any decisions that may be asked of a teleoperations system operator during a teleoperations session—particularly decisions associated with verifying any conditions detected by a detector in the autonomous vehicle.”, where verification occurs before a teleoperator would select to transmit guidance data.).
18. Regarding claim 13, the combination of Gate and Lockwood teaches The method as recited in claim 6, further Gate teaches wherein the first object is associated with multiple states, and wherein the vehicle is configured to perform a first action when the first object is in a first state, and to perform a second action when the first object is in a second state (col 22 : “Control 365 is associated with a school bus detector that detects when a school bus is stopped or slowing with its stop arm extended, and control 366 is associated with an emergency vehicle detector that detects when an emergency vehicle is in the vicinity with its lights and/or siren on.”, col 23 : “For example, where it is known that a school bus detector has been triggered, and that the school bus is located straight ahead of the autonomous vehicle, the context data that may be communicated for presentation to a teleoperations system operator may focus on data associated with the track or path of the school bus, data associated with the track or path of the autonomous vehicle, data associated with the speed of the autonomous vehicle, video data associated with any forward-facing camera feeds, or any other data that may be of use to a teleoperations system operator when attempting to verify whether the school bus does in fact have its stop arm extended and is either stopped or is in the process of stopping to load or unload passengers.”, where an object with two states is at least a bus with a stop arm and in the case of the stop arm being extended the teleoperations session is started, otherwise when not extended to vehicle continues to operate as it was previously.).
19. Regarding claim 20, the combination of Gate and Lockwood teaches The one or more non-transitory computer-readable media as recited in claim 17, however Gate does not explicitly teach wherein the second data indicates that the vehicle determined a potential collision with a second object.
However, in a similar field of endeavor (collision avoidance for autonomous vehicles), Lockwood teaches wherein the second data indicates that the vehicle determined a potential collision with a second object. (col 13 line 8-13 : “In some examples, the collision predictor system 220 may be configured to use the data representing the object type, the predicted object behavior, the data representing the trajectory of the object, and/or the data representing the trajectory of the vehicle 102, to predict a collision between the vehicle 102 and the object.”, col 14 line 27-32 : “In some examples, the collision predictor system 220 may be used to predict a collision between the vehicle 102 and an object in the environment 100 based on the object type, whether the object is moving, the trajectory of the vehicle 102, the predicted path of the object obtained from the planner 214.”.
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to modify the system of Gate with the collision predictor and safety system actuator of Lockwood to improve safety by avoiding predicted collisions while navigating the vehicle.
20. Regarding claim 21, the combination of Gate and Lockwood teaches The one or more non-transitory computer-readable media as recited in claim 17, further Lockwood teaches wherein the second data indicates that the vehicle determined that a state of the flagger has changed (col 13 line 46-63 : “In some examples, the safety system actuator 224 may be configured to activate one or more safety systems of the autonomous vehicle 102 when a collision is predicted by the collision predictor 220 and/or the occurrence of other safety related events, such as, for example, an emergency maneuver by the vehicle 102, such as hard braking or a sharp acceleration. The safety system actuator 224 may be configured to activate an interior safety system (e.g., including seat belt pre-tensioners and/or air bags), an exterior safety system (e.g., including warning sounds and/or warning lights), the drive system 232 configured to execute an emergency maneuver to avoid a collision, and/or any combination thereof. For example, the drive system 232 may receive data for causing a steering system of the vehicle 102 to change the travel direction of the vehicle 102, and a propulsion system of the vehicle 102 to change the speed of the vehicle 102 to alter the trajectory of vehicle 102 from an initial trajectory to a trajectory for avoiding a collision.”, where determination of a potential collision is a factor of the environment and the states of objects in the environment, so a potential collision would be detected based at least in part on a determination of the environment including the state of the first object having changed.).
21. Claim(s) 5 and 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gate in view of Lockwood and further in view of Pedersen et al (US 11016485 B2) henceforth referred to as Pedersen.
22. Regarding claim 5, the combination of Gate and Lockwood teaches The system as recited in claim 2, further Gate teaches the operations further comprising:
based at least in part on detecting the first object, determining policy data for the first object, the policy data defining a set of rules for navigating the vehicle through the environment including the first object (col 15 line 10-26 : “As illustrated in block 252, an autonomous vehicle may initially detect a condition that triggers a desire for teleoperations monitoring. As will become more apparent below, a condition may include, for example, a detection by a detector in a perception component, a request from a planner component for assistance with selection of a suitable scenario or path given current circumstances, a request initiated by a vehicle occupant, an error condition detected in the autonomous vehicle, or in other situations where teleoperations assistance may be desired. A non-exclusive list of potential conditions that may be sensed and used to trigger monitoring includes construction zones, temporary traffic devices, authorities directing traffic, loading or unloading school buses, emergency vehicles, the placement of signs, cones, or flares, crossing guards, funeral processions, collisions, inoperative traffic devices, etc.”, col 15 line 32-36 : “For example, it may be desirable in some implementations to automatically initiate teleoperations sessions with a vehicle in order to reassess certain conditions that have previously been detected, whether by the same autonomous vehicle or by other autonomous vehicles.”, where the policy associated with the flagged intended to navigate the vehicle is the initiation of the teleoperations session.). However, the combination does not explicitly teach based at least in part on determining the policy data, causing the vehicle to stop at the first location at least a threshold distance from the first object.
However, in a similar field of endeavor (processes for transition to remote control), Pedersen teaches a system based at least in part on determining the policy data, causing the vehicle to stop at the first location at least a threshold distance from an obstruction situation (col 2 line 60-67 – col 3 line 1-3 : “When the AV encounters an exception situation, the AV can stop and request assistance from a tele-operator. For example, when the AV encounters an obstruction (e.g., a construction site, a stopped vehicle, etc.) in a roadway, the AV might not go around the obstruction if doing so means that the AV will travel through an area that is physically safe but is restricted by traffic regulations. Accordingly, a tele-operator (e.g., a human operator, a vehicle manager) can be tasked with assisting the AV in negotiating its problematic situation by, for example, mapping a path (i.e., a trajectory) for the AV around the obstruction.”, col 15 line 8-13 : “For example, when the adjusted drivable area, accounting for static objects, contains a static blockage (e.g., an obstruction situation), the trajectory planning module 404 adjusts the discrete-time speed plan such that the AV comes to a stop a prescribed distance before the static blockage.”, col 15 line 8-13 : “For example, when the adjusted drivable area, accounting for static objects, contains a static blockage (e.g., an obstruction situation), the trajectory planning module 404 adjusts the discrete-time speed plan such that the AV comes to a stop a prescribed distance before the static blockage.”, as Gate teaches identification of a first object and Pedersen teaches determining an obstruction situation in which to stop the vehicle at a threshold distance away from while waiting to receive and instruction from a remote computing device, the combination of Gate and Pedersen, using the obstruction situation as the point at which a first object is identified teaches the limitation of determining, based at least in part on a rule associated with the first object, a stopping location that is at least a threshold distance from the first object as the rule associated would then be to determine a stopping point a threshold distance away while waiting to receive instruction from the remote computing device.).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to modify the combination of Gate and Lockwood with the teachings of Pedersen to increase safety by stopping a vehicle in a safe location while awaiting guidance.
23. Regarding claim 15, it recites a method with limitations substantially the same to claim 5 above, therefore it is rejected for the same reason.
24. Regarding claim 16, the combination of Gate and Lockwood teaches The method as recited in claim 6, however the combination does not explicitly teach further comprising:
based at least in part on detecting the first object, causing the vehicle to stop at the first location at least a threshold distance from the first object.
However, in a similar field of endeavor (processes for transition to remote control), Pedersen teaches a system based at least in part on detecting the first object, causing the vehicle to stop at the first location at least a threshold distance from an obstruction situation (col 2 line 60-67 – col 3 line 1-3 : “When the AV encounters an exception situation, the AV can stop and request assistance from a tele-operator. For example, when the AV encounters an obstruction (e.g., a construction site, a stopped vehicle, etc.) in a roadway, the AV might not go around the obstruction if doing so means that the AV will travel through an area that is physically safe but is restricted by traffic regulations. Accordingly, a tele-operator (e.g., a human operator, a vehicle manager) can be tasked with assisting the AV in negotiating its problematic situation by, for example, mapping a path (i.e., a trajectory) for the AV around the obstruction.”, col 15 line 8-13 : “For example, when the adjusted drivable area, accounting for static objects, contains a static blockage (e.g., an obstruction situation), the trajectory planning module 404 adjusts the discrete-time speed plan such that the AV comes to a stop a prescribed distance before the static blockage.”, col 15 line 8-13 : “For example, when the adjusted drivable area, accounting for static objects, contains a static blockage (e.g., an obstruction situation), the trajectory planning module 404 adjusts the discrete-time speed plan such that the AV comes to a stop a prescribed distance before the static blockage.”, as Gate teaches identification of a first object and Pedersen teaches determining an obstruction situation in which to stop the vehicle at a threshold distance away from while waiting to receive and instruction from a remote computing device, the combination of Gate and Pedersen, using the obstruction situation as the point at which a first object is identified teaches the limitation of determining, based at least in part on a rule associated with the first object, a stopping location that is at least a threshold distance from the first object as the rule associated would then be to determine a stopping point a threshold distance away while waiting to receive instruction from the remote computing device.).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to modify the combination of Gate and Lockwood with the teachings of Pedersen to increase safety by stopping a vehicle in a safe location while awaiting guidance.
25. Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gate and Lockwood and further in view of Govardhanam (US 20220398412 A1) henceforth referred to as Govardhanam.
26. Regarding claim 9, the combination of Gate and Lockwood teaches The method as recited in claim 6, further Gate teaches the method further comprising:
indicating a classification of the first object, the classification indicating the first object is a person directing traffic (col 13 line 37-40 : “It may also be desirable to propagate data collected during a teleoperations session (e.g., dat1a related to lane closures, detected construction or incidents, etc.) to other vehicles in a fleet.”, col 15 line 20-26 : “A non-exclusive list of potential conditions that may be sensed and used to trigger monitoring includes construction zones, temporary traffic devices, authorities directing traffic, loading or unloading school buses, emergency vehicles, the placement of signs, cones, or flares, crossing guards, funeral processions, collisions, inoperative traffic devices, etc.”). However, the combination does not explicitly teach inputting the first sensor data into a machine-learned model; and
receiving, from the machine-learned model, an output indication the classification of the object.
However, in a similar field of endeavor (machine-learning for object classification) XXX teaches inputting the first sensor data into a machine-learned model (para [0025] : “FIG. 3 illustrates a block diagram of a process 300 for performing object classification. Process 300 begins with step 302 in which road data is received, e.g., by a training data augmentation system/process of the disclosed technology. The road data can include sensor data for a recorded driving scene. As discussed above with respect to FIGS. 2A and 2B, the road data can represent a driving scenario of a vehicle (AV), on a particular course through a 3D environment, such as those represented by scenarios 200, and 201. As such, the road data can include various objects, including other traffic participants or entities, and non-traffic participants and static objects, such as buildings, trees, and other map features.”); and
receiving, from the machine-learned model, an output indication the classification of the object (para [0031] : “At step 308, process 400 includes classifying the virtual object, e.g., at the first location using the modified scene data. By performing classification on the inserted (virtual) object, the accuracy of various object detection/classification models can be improved, with respect to that object type and insertion context. As such, the modified road data can be used as training data to improve object classification for any type of virtual object that is inserted into the recorded driving scene.”).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to modify the combination of Gate and Delbari with the machine learning of Govardhanam to improve the accuracy of object classification.
27. Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gate and Lockwood and further in view of Delbari et al (US 20200142395 A1) henceforth referred to as Delbari.
28. Regarding claim 11, the combination of Gate and Lockwood teaches The method as recited in claim 10, however the combination does not explicitly teach the method further comprising:
receiving, from the computing device, second guidance data indicative of the teleoperator releasing the input device and instructing the vehicle to stop in the environment; and
causing the vehicle to stop and remain stopped until receiving additional guidance data indicative of the teleoperator activating the input device.
However, in a similar field of endeavor (systems for teleoperation of autonomous vehicles), Delbari teaches receiving, from the computing device, second guidance data indicative of the teleoperator releasing the input device and instructing the vehicle to stop in the environment (para [0061] line 1-10 : “Next, in a decision block 407, the computer 110 determines whether the remote vehicle control 140 is activated. For example, the switch 161 can be disengaged, e.g., the user may release the switch 161 and/or the remote vehicle control 140. The computer 110 stops communicating with the remote vehicle control 140 when the switch 161 is disengaged, i.e., when the remote vehicle control 140 is deactivated. If the remote vehicle control 140 is deactivated, the process 400 continues in a block 414.”); and
causing the vehicle to stop and remain stopped until receiving additional guidance data indicative of the teleoperator activating the input device (para [0061] line 1-10 : “Next, in a decision block 407, the computer 110 determines whether the remote vehicle control 140 is activated. For example, the switch 161 can be disengaged, e.g., the user may release the switch 161 and/or the remote vehicle control 140. The computer 110 stops communicating with the remote vehicle control 140 when the switch 161 is disengaged, i.e., when the remote vehicle control 140 is deactivated. If the remote vehicle control 140 is deactivated, the process 400 continues in a block 414. Otherwise, the process 400 continues in a block 408.”, Fig. 4B, Fig. 4B shows block 414 engaging powertrain in neutral mode followed by a check of a policy where vehicle speed is used to determine if the powertrain should enter park mode, as the vehicle is stopped in part on receiving the second guidance data, it would be required that the vehicle receive additional guidance data for the vehicle to resume.).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to modify the combination of Gate and Lockwood with the system of Delbari for improved control or teleoperated vehicles.
29. Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gate in view of Lockwood and further in view of Brown et al (US 20220081003 A1) henceforth referred to as Brown.
30. Regarding claim 14, the combination of Gate and Lockwood teaches The method as recited in claim 6, further comprising:
determining that the first object is a flagger (col 13 line 37-40 : “It may also be desirable to propagate data collected during a teleoperations session (e.g., data related to lane closures, detected construction or incidents, etc.) to other vehicles in a fleet.”, col 15 line 20-26 : “A non-exclusive list of potential conditions that may be sensed and used to trigger monitoring includes construction zones, temporary traffic devices, authorities directing traffic, loading or unloading school buses, emergency vehicles, the placement of signs, cones, or flares, crossing guards, funeral processions, collisions, inoperative traffic devices, etc.”). However, the combination does not explicitly teach based at least in part on determining a classification of one or more additional objects in the zone.
However, in a similar field of endeavor (detection of vehicle environments), Brown teaches determining a construction zone based at least in part on determining a classification of one or more additional objects in the zone (para [0164] : “Based on comparing the sensor data 906a with the third portion of the map data 1510, the control subsystem 1400 determines whether a construction zone 902 is detected by identifying one or more objects 904 associated with the construction zone 902 that are not among the expected objects in the third portion of the map data 1510. In other words, the control subsystem 1400 determines that there is a construction zone 902 ahead of the lead AV 1602-1 if it detects the presence of one or more objects 604.”, where construction zones commonly have flaggers or authorities directing traffic and the combination of Gate’s identification of construction zones and/or authorities directing traffic along with the construction zone identification of Brown teaches identification of an authority directing traffic based at least in part on the identification of a construction zone and one or more additional objects.).
It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to modify the combination of Gate and Lockwood with the multi object identification of Brown to increase the safety and reliability of the classification systems.
Conclusion
31. Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID HATCH whose telephone number is (571)272-4518. The examiner can normally be reached on Monday-Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, James J Lee can be reached on 571-270-5965. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/D.H./Examiner, Art Unit 3668
/JAMES J LEE/Supervisory Patent Examiner, Art Unit 3668
Prosecution Insights