Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 5, 7, 10, 14, 16, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Levinson (US 9,507,346 B1), in view of Hamada (US6107937A).
Regarding Claim 1,
Disclosure by Levinson
Levinson teaches:
A method, comprising:
See at least: “In particular, a method may include receiving a teleoperation message...” (Abstract).
Rationale: Levinson explicitly discloses a method comprising steps, meeting the preamble.
obtaining, by a control host of a remote driving system,
See at least: “RECEIVE MESSAGE DATA AT A TELEOPERATOR COMPUTING DEVICE…” (Fig. 15, 1500).
Rationale: The teleoperator computing device is part of the teleoperator manager/service platform, which is the control host of the remote driving system.
a first state information of the remote driving system,
See at least: “message data 3619 may include any type of data, such as telemetry data or planner data, that may be suitable for facilitating teleoperations.” (Col. 39, ll. 2-4).
Rationale: Telemetry data or planner data contained in the message is first state information of the remote driving system.
wherein the remote driving system is configured to perform remote control on one or more vehicles
See at least: “A teleoperator 1108 may transmit via teleoperator computing device 1104 a selected trajectory 1137 to guided trajectory generator 1126.” (Col. 20, ll. 21-23).
Rationale: Transmitting a selected trajectory to the vehicle's planner to cause motion is performing remote control on a vehicle.
communicatively connected with the remote driving system;
See at least: “teleoperator manager 3607…configured to… establish communications between teleoperator computing device 3604 and autonomous vehicle 3630.” (Col. 39, ll. 5-9).
Rationale: Explicitly teaching to establish communications satisfies communicatively connected.
determining, by the control host, whether an abnormality occurs in the remote driving system based on the first state information;
See at least: “teleoperator manager 3807 determines that an autonomous vehicle service platform fails to receive either sufficient data or a periodic signal (e.g., heartbeat signal) that confirms communication… is present.” (Col. 46, ll. 24-28).
Rationale: The teleoperator manager (control host) determines a failure of a heartbeat signal (an abnormality) based on the absence of expected data (first state information).
determining, by the control host, abnormality information of the remote driving system in response to the occurrence of the abnormality in the remote driving system;
See at least: “The message data may indicate event attributes associated with a non-normative state of operation…” (Col. 23, ll. 25-26). “a teleoperation repository may be accessed to retrieve a first subset of recommendations…” (Col. 23, ll. 34-36).
Rationale: In response to message data indicating a non-normative state (abnormality), the system accesses a repository to retrieve recommendations. These recommendations constitute abnormality information determined by the control host.
and adjusting, by the control host, a state of the remote control based on the abnormality information,
See at least: “At 212, the selection of a candidate trajectory as a guided trajectory may be transmitted to the vehicle, which, in turn, implements the guided trajectory for resolving the condition by causing the vehicle to perform a teleoperator-specified maneuver” (Col. 8, 29-33).
Rationale: Transmitting the selected guided trajectory (a recommended course of action from the abnormality information) to the vehicle adjusts the state of the remote control from planning to executing a specific teleoperator-guided command.
wherein the abnormality information comprises a handling action for the abnormality,
See at least: “the first subset and the second subset of recommendations are combined to form a set of recommended courses of action for the autonomous vehicle.” (Col. 23, ll. 46-49).
Rationale: A recommended course of action is a handling action for the abnormality.
handling the abnormality according to the handling action;
See at least: “A teleoperator may provide instruction data 3817 to autonomous vehicle 3830 to travel one or more guided trajectories 3866.” (Col. 46, ll. 65-67).
Rationale: Providing instruction data to cause the vehicle to travel a guided trajectory (a selected course of action) is handling the abnormality according to the handling action.
and resuming the remote control in response to the abnormality being eliminated.
See at least: “autonomous vehicle 3830 may be able to resume autonomous vehicle services upon completion of successful safe-stop recovery processes.” (Col. 46, ll. 49-52).
Rationale: Resuming services upon successful completion of recovery processes is resuming the remote control in response to the abnormality being eliminated, as the recovery process addresses the fault condition.
Claim limitations Not Explicitly Disclosed by Levinson
Levinson does not explicitly disclose:
and wherein the adjusting the state of the remote control comprises: suspending the remote control;
Levinson describes invoking teleoperation and implementing safe-stops (Col. 46, ll. 38-40), which implies an interruption of normal control, but does not explicitly describe the step of suspending the remote control as part of the adjustment.
Disclosure by Hamada
Hamada teaches:
and wherein the adjusting the state of the remote control comprises: suspending the remote control;
See at least: “The transmission of the new control command is suspended…” (Col. 5, ll. 28-29).
Rationale: Hamada expressly teaches suspending the transmission of control commands, which is suspending the remote control.
Motivation to Combine Levinson and Hamada
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson and Hamada before them, to incorporate Hamada's explicit teaching of suspending control command transmission into Levinson's teleoperation system. Both references are directed to robust remote vehicle control systems. Levinson teaches a framework for detecting abnormalities (e.g., lost heartbeat) and determining recovery actions. Hamada teaches the specific safety measure of suspending control transmission. A skilled artisan, seeking to enhance the safety of Levinson's system during fault conditions, would have been motivated to integrate Hamada's suspension step. This would create a predictable and safer operational sequence: upon detecting an abnormality, the host would first suspend remote control (per Hamada), then execute the handling/recovery actions (per Levinson), and finally resume control once the abnormality is resolved.
Regarding Claim 5,
The combination of Levinson and Hamada establishes the method of Claim 1, which is the basis for Claim 5.
Disclosure by Levinson
Levinson teaches:
wherein the remote driving system comprises a data collection device
See at least: “RECEIVE SENSOR DATA ORIGINATING FROM SENSORS OF MULTIPLE MODALITIES IN ASSOCIATION IN AN AUTONOMOUS VEHICLE” (Fig. 5, 502)
Rationale: Under BRI, the recited remote driving system is a distributed system that remotely controls one or more communicatively connected vehicles and includes the vehicle-side sensing that provides operating information to the control host. Accordingly, Levinson’s vehicle sensors correspond to the claimed data collection device that collects environmental information, and the resulting sensor data corresponds to the claimed first data included in the first state information. To the extent the claim is construed to require the data collection device to be physically located at the control host, a PHOSITA would have found it obvious to additionally provide a host-side sensor/monitor to collect environmental/operational data for reliability and safety, and to determine an abnormality when that data fails to satisfy a preset condition (e.g., a threshold, consistency, or quality check).
for collecting an environmental information of an environment in which the remote driving system is located,
See at least: “SEGMENT FEATURES OF AN ENVIRONMENT IN WHICH THE AUTONOMOUS VEHICLE IS DISPOSED TO FORM SEGMENTED OBJECTS” (Fig. 24, 2406)
Rationale: Levinson expressly collects/processes information about the “environment in which the autonomous vehicle is disposed,” which maps to for collecting an environmental information of an environment in which the remote driving system is located,.
and the first state information comprises a first data
See at least: “MESSAGE DATA INDICATING EVENT ATTRIBUTES ASSOCIATED WITH A NON-NORMATIVE STATE OF OPERATION…” (Fig. 5, 1502)
Rationale: Levinson’s “message data” that indicates “event attributes” associated with the system’s operating state constitutes state information/data, mapping to and the first state information comprises a first data.
collected by the data collection device,
See at least: “RECEIVE SENSOR DATA ORIGINATING FROM SENSORS…” (Fig. 5, 502)
Rationale: Levinson’s “sensor data” is expressly “originating from sensors,” i.e., collected by the sensors, mapping to collected by the data collection device,.
and wherein the determining whether an abnormality occurs in the remote driving system comprises: determining that an abnormality occurs in the data collection device
See at least: “ANOMALY 3977” … “SENSOR 1”
Rationale: Levinson explicitly presents an “ANOMALY” associated with a “SENSOR ALERT,” which is an abnormality tied to the sensing/data-collection component, mapping to …determining that an abnormality occurs in the data collection device.
in response to the first data not satisfying a preset condition.
See at least: “DETERMINE A VALUE OF A CONFIDENCE LEVEL THAT EXCEEDS TO A RANGE OF CONFIDENCE LEVELS ASSOCIATED WITH NORMATIVE OPERATION OF THE AUTONOMOUS VEHICLE” (Fig. 5, 512)
Rationale: Levinson uses a “range…associated with normative operation” as a threshold/range condition; when the value “exceeds” that range, the data fails that normative condition, mapping to in response to the first data not satisfying a preset condition..
Motivation to Combine Levinson and Hamada
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson and Hamada before them, to incorporate Hamada’s teachings for managing remote-system operation in the presence of multiple devices and monitoring inputs into Levinson’s teleoperation architecture so as to improve reliability and safety of the remote driving system by ensuring that device-provided data used for system operation is properly collected and evaluated for abnormal conditions. Levinson teaches a remote driving/teleoperation framework that receives and uses sensor-derived and message data to characterize operating state and detect non-normative operation conditions, while Hamada teaches remote operation architectures that include input/monitoring devices and associated management of signals in a remote control setting. A skilled artisan would have been motivated to integrate Hamada’s remote-system device/monitoring teachings into Levinson’s remote driving system to strengthen robustness of remote operation, yielding predictable results in which data collected by a data collection device is used as part of system state information and abnormality conditions are determined when collected data fails to satisfy an expected condition (e.g., a threshold/range/consistency condition), thereby improving safe and reliable remote driving operation.
Regarding Claim 7,
The combination of Levinson and Hamada establishes the method of Claim 1, which is the basis for Claim 7.
Disclosure by Levinson
Levinson teaches:
wherein the remote driving system comprises a hardware device
See at least: “a communication interface 3313 (e.g., an Ethernet or wireless controller, a Bluetooth controller, NFC logic, etc.)” (Fig. 33, 3313, Col. 34, ll. 34-36)
Rationale: Levinson expressly discloses a physical “communication interface” (e.g., Ethernet/wireless/Bluetooth/NFC controller), which constitutes a “hardware device” within the remote driving system.
connected to a port of the control host,
See at least: “to facilitate communications via a port on communication link 3321” (Fig. 33, 3321, Col. 34, ll. 36-37)
Rationale: Levinson expressly discloses communications being facilitated “via a port,” which corresponds to the hardware device being connected through and operating via a port of the host computing platform (control host).
the first state information comprising a port information of the port,
See at least: “a periodic signal (e.g., heartbeat signal) that confirms communication via network 3806 is present” (Col. 46, ll. 24–28)
Rationale: Levinson expressly uses a “periodic signal (e.g., heartbeat signal)” as information that “confirms communication … is present,” which constitutes “port information” reflecting the port/link status, and thus is included within the system’s state information.
and wherein the determining whether an abnormality occurs in the remote driving system comprises: determining that a connection of the hardware device is abnormal
See at least: “initiate one or more actions to reestablish communications” (Col. 46, ll. 28–31)
Rationale: Levinson’s initiation of actions “to reestablish communications” reflects determining that the communications connection (i.e., the connection of the communication-interface hardware device via the port/link) is abnormal and requires recovery.
in response to the port information indicating that the port is not occupied.
See at least: “failure to receive a periodic signal (e.g., heartbeat signal) that confirms communication via network 3806 is present” (Col. 46, ll. 24–28)
Rationale: Levinson expressly ties the abnormal condition to “failure to receive” the heartbeat that “confirms communication … is present”; under BRI, a PHOSITA would understand this as port/link status information indicating no active occupant/endpoint on the port (i.e., the port is not occupied / link not present), thereby triggering the abnormality determination.
Motivation to Combine Levinson and Hamada
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson and Hamada before them, to incorporate Hamada’s multi-controller remote-operation teachings into Levinson’s remote vehicle teleoperation architecture because both references address remote control of a controlled device over a communication link, and integrating Hamada’s coordination of remote control devices into Levinson’s teleoperation framework would have yielded predictable results in improved operational robustness and fault handling when multiple remote control sources and link-state conditions are present.
Regarding Claim 10,
Disclosure by Levinson
Levinson discloses:
An electronic device,
See at least: “computing platform 3300 can be disposed in any device” (Col. 34, ll. 23-24)
Rationale: Levinson’s “computing platform 3300” corresponds to an electronic device.
wherein the electronic device is used as a control host of a remote driving system,
See at least: “system memory 3306 includes an autonomous vehicle service plat form module 3450 and/or its components (e.g., a teleoperator manager, a simulator, etc.), any of which, or one or more portions of which, can be configured to facilitate managing an autonomous vehicle service by implementing one or more functions described herein”
Rationale: The quoted “autonomous vehicle service platform module 3450 (e.g., a teleoperator manager …)” identifies a host-side platform/manager that operates as the control-side component of a remote driving system.
the electronic device comprising: a processor;
See at least: “processor 3304”
Rationale: Processor 3304 corresponds to a processor
and a memory communicatively connected with the processor;
See at least: “a bus 3302 … for communicating information, and a processor 3304 coupled with the bus 3302 … [and] a system memory 3306 coupled to the bus 3302” (Fig. 3)
Rationale: The quoted “processor 3304 coupled with the bus 3302” and “system memory 3306 coupled to the bus 3302” establishes the memory is communicatively connected with the processor via the bus.
wherein the memory stores instructions executable by the processor,
See at least: “system memory 3306 can include various modules that include executable instructions to implement functionalities described herein” (Col. 35, ll. 35-37)
Rationale: The quoted “system memory 3306 … include executable instructions” establishes instructions stored in memory that are executable by the processor.
wherein the instructions, when executed by the processor, are configured to cause the processor to perform operations comprising:
See at least: “processor 3304 executing one or more sequences of one or more instructions … may be read into system memory 3306” (Col. 34, ll. 52-54)
Rationale: The quoted “processor 3304 executing … instructions” and “read into system memory 3306” establishes execution of stored instructions to perform operations.
obtaining a first state information of the remote driving system,
See at least: “Message data may be received at a teleoperator computing device” (Col. 23, ll. 23-24)
Rationale: The quoted “message data may be received” corresponds to obtaining state information used for teleoperation.
wherein the remote driving system is configured to perform remote control on one or more vehicles communicatively connected with the remote driving system;
See at least: “At 212, the selection of a candidate trajectory as a guided trajectory may be transmitted to the vehicle, which, in turn, implements the guided trajectory … by causing the vehicle to perform a teleoperator-specified maneuver” (Col. 8, 29-33)
Rationale: Levinson establishes remote control of a vehicle via transmitted control content.
determining whether an abnormality occurs in the remote driving system based on the first state information;
See at least: “teleoperator manager 3807 determines that an autonomous vehicle service platform fails to receive either sufficient data or a periodic signal (e.g., heartbeat signal) that confirms communication … is present” (Col. 46, ll. 24-28)
Rationale: The quoted “determines … fails to receive either sufficient data or a periodic signal (e.g., heartbeat signal)” establishes determining an abnormality based on the presence/absence of received data/signal.
determining abnormality information of the remote driving system in response to the occurrence of the abnormality in the remote driving system;
See at least: “ACCESS A TELEOPERATION REPOSITORY TO RETRIEVE A FIRST SUBSET OF RECOMMENDATIONS” (Fig. 15, 1504)
Rationale: The quoted “retrieve … recommendations” provides information used after an abnormal/non-normative condition is identified.
and adjusting a state of the remote control based on the abnormality information,
See at least: “At 212, the selection of a candidate trajectory as a guided trajectory may be transmitted to the vehicle, which, in turn, implements the guided trajectory for resolving the condition” (Col. 8, ll. 29-32)
Rationale: The quoted “implements … for resolving the condition” establishes adjusting remote control state responsive to condition-handling information.
wherein the abnormality information comprises a handling action for the abnormality,
See at least: “recommendations are combined to form a set of recommended courses of action” (Col. 23, ll. 48-49)
Rationale: The quoted “recommended courses of action” corresponds to a handling action for the abnormality.
handling the abnormality according to the handling action;
See at least: “At 212, the selection of a candidate trajectory as a guided trajectory may be transmitted to the vehicle, which, in turn, implements the guided trajectory for resolving the condition by causing the vehicle to perform a teleoperator-specified maneuver” (Col. 8, ll. 29-33)
Rationale: The quoted “implements … for resolving the condition” and “causing the vehicle to perform a teleoperator-specified maneuver” corresponds to handling the abnormality according to the course of action.
and resuming the remote control in response to the abnormality being eliminated.
See at least: “resume autonomous vehicle services upon completion of successful safe-stop recovery processes” (Col. 46, ll. 50-52)
Rationale: The quoted “resume … upon completion of … recovery processes” corresponds to resuming after the abnormality is eliminated via recovery.
Claim limitations Not Explicitly Disclosed by Levinson
Levinson does not explicitly disclose:
suspending the remote control;
Disclosure by Hamada
Hamada discloses:
suspending the remote control;
See at least: “The transmission of the new control command is suspended” (Col. 5, ll. 28–29)
Rationale: The quoted “transmission … is suspended” corresponds to suspending remote control command transmission.
Motivation to Combine Levinson and Hamada
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson and Hamada before them, to incorporate Hamada’s explicit “transmission … is suspended” control-suspension technique into Levinson’s teleoperation control flow when “teleoperator manager 3807 determines” a communication/data abnormality, so that remote control commands are suspended during abnormality handling and then resumed after “successful … recovery processes,” yielding the predictable result of preventing unsafe or conflicting command transmission while the system resolves the abnormal condition.
Regarding Claim 14,
The combination of Levinson and Hamada establishes the electronic device of Claim 10, which is the basis for Claim 14.
Disclosure by Levinson
Levinson discloses:
wherein the remote driving system comprises a data collection device
See at least: “RECEIVE SENSOR DATA ORIGINATING FROM SENSORS OF MULTIPLE MODALITIES IN ASSOCIATION IN AN AUTONOMOUS VEHICLE” (Fig. 5, 502)
Rationale: Under BRI, the recited remote driving system is a distributed system that remotely controls one or more communicatively connected vehicles and includes the vehicle-side sensing that provides operating information to the control host. Accordingly, Levinson’s vehicle sensors correspond to the claimed data collection device that collects environmental information, and the resulting sensor data corresponds to the claimed first data included in the first state information. To the extent the claim is construed to require the data collection device to be physically located at the control host, a PHOSITA would have found it obvious to additionally provide a host-side sensor/monitor to collect environmental/operational data for reliability and safety, and to determine an abnormality when that data fails to satisfy a preset condition (e.g., a threshold, consistency, or quality check).
for collecting an environmental information of an environment in which the remote driving system is located,
See at least: “SEGMENT FEATURES OF AN ENVIRONMENT IN WHICH THE AUTONOMOUS VEHICLE IS DISPOSED TO FORM SEGMENTED OBJECTS” (Fig. 24, 2406)
Rationale: Levinson expressly collects/processes information about the “environment in which the autonomous vehicle is disposed,” which maps to for collecting an environmental information of an environment in which the remote driving system is located.
and the first state information comprises a first data
See at least: “MESSAGE DATA INDICATING EVENT ATTRIBUTES ASSOCIATED WITH A NON-NORMATIVE STATE OF OPERATION…” (Fig. 5, 1502)
Rationale: Levinson’s “message data” that indicates “event attributes” associated with the system’s operating state constitutes state information/data, mapping to and the first state information comprises a first data.
collected by the data collection device,
See at least: “RECEIVE SENSOR DATA ORIGINATING FROM SENSORS…” (Fig. 5, 502)
Rationale: Levinson’s “sensor data” is expressly “originating from sensors,” i.e., collected by the sensors, mapping to collected by the data collection device,.
and wherein the determining whether an abnormality occurs in the remote driving system comprises: determining that an abnormality occurs in the data collection device
See at least: “ANOMALY 3977” … “SENSOR 1”
Rationale: Levinson explicitly presents an “ANOMALY” associated with a “SENSOR ALERT,” which is an abnormality tied to the sensing/data-collection component, mapping to …determining that an abnormality occurs in the data collection device.
in response to the first data not satisfying a preset condition.
See at least: “DETERMINE A VALUE OF A CONFIDENCE LEVEL THAT EXCEEDS TO A RANGE OF CONFIDENCE LEVELS ASSOCIATED WITH NORMATIVE OPERATION OF THE AUTONOMOUS VEHICLE” (Fig. 5, 512)
Rationale: Levinson uses a “range…associated with normative operation” as a threshold/range condition; when the value “exceeds” that range, the data fails that normative condition, mapping to in response to the first data not satisfying a preset condition..
Motivation to Combine Levinson and Hamada
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson and Hamada before them, to incorporate Hamada’s teachings for managing remote-system operation in the presence of multiple devices and monitoring inputs into Levinson’s teleoperation architecture so as to improve reliability and safety of the remote driving system by ensuring that device-provided data used for system operation is properly collected and evaluated for abnormal conditions. Levinson teaches a remote driving/teleoperation framework that receives and uses sensor-derived and message data to characterize operating state and detect non-normative operation conditions, while Hamada teaches remote operation architectures that include input/monitoring devices and associated management of signals in a remote control setting. A skilled artisan would have been motivated to integrate Hamada’s remote-system device/monitoring teachings into Levinson’s remote driving system to strengthen robustness of remote operation, yielding predictable results in which data collected by a data collection device is used as part of system state information and abnormality conditions are determined when collected data fails to satisfy an expected condition (e.g., a threshold/range/consistency condition), thereby improving safe and reliable remote driving operation.
Regarding Claim 16,
The combination of Levinson and Hamada establishes the electronic device of Claim 10, which is the basis for Claim 16.
Disclosure by Levinson
Levinson discloses:
wherein the remote driving system comprises a hardware device
See at least: “a communication interface 3313 (e.g., an Ethernet or wireless controller, a Bluetooth controller, NFC logic, etc.)” (Fig. 33, 3313, Col. 34, ll. 34-36)
Rationale: Levinson expressly discloses a physical “communication interface” (e.g., Ethernet/wireless/Bluetooth/NFC controller), which constitutes a “hardware device” within the remote driving system.
connected to a port of the control host,
See at least: “to facilitate communications via a port on communication link 3321” (Fig. 33, 3321, Col. 34, ll. 36-37)
Rationale: Levinson expressly discloses communications being facilitated “via a port,” which corresponds to the hardware device being connected through and operating via a port of the host computing platform (control host).
the first state information comprising a port information of the port,
See at least: “a periodic signal (e.g., heartbeat signal) that confirms communication via network 3806 is present” (Col. 46, ll. 24–28)
Rationale: Levinson expressly uses a “periodic signal (e.g., heartbeat signal)” as information that “confirms communication … is present,” which constitutes “port information” reflecting the port/link status, and thus is included within the system’s state information.
and wherein the determining whether an abnormality occurs in the remote driving system comprises: determining that a connection of the hardware device is abnormal
See at least: “initiate one or more actions to reestablish communications” (Col. 46, ll. 28–31)
Rationale: Levinson’s initiation of actions “to reestablish communications” reflects determining that the communications connection (i.e., the connection of the communication-interface hardware device via the port/link) is abnormal and requires recovery.
in response to the port information indicating that the port is not occupied.
See at least: “failure to receive a periodic signal (e.g., heartbeat signal) that confirms communication via network 3806 is present” (Col. 46, ll. 24–28)
Rationale: Levinson expressly ties the abnormal condition to “failure to receive” the heartbeat that “confirms communication … is present”; under BRI, a PHOSITA would understand this as port/link status information indicating no active occupant/endpoint on the port (i.e., the port is not occupied / link not present), thereby triggering the abnormality determination.
Motivation to Combine Levinson and Hamada
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson and Hamada before them, to incorporate Hamada’s multi-controller remote-operation teachings into Levinson’s remote vehicle teleoperation architecture because both references address remote control of a controlled device over a communication link, and integrating Hamada’s coordination of remote control devices into Levinson’s teleoperation framework would have yielded predictable results in improved operational robustness and fault handling when multiple remote control sources and link-state conditions are present.
Regarding Claim 19,
Disclosure by Levinson
Levinson discloses:
A non-transitory computer-readable storage medium
See at least: "computing platform 3300 includes a bus 3302 ... system memory 3306 (e.g., RAM, etc.), storage device 3308 (e.g., ROM, etc.)" (Col. 34, ll. 28-32)
Rationale: The claim requires a non-transitory computer-readable storage medium. Levinson's "storage device 3308 (e.g., ROM)" is a specific type of non-volatile, non-transitory computer-readable storage medium for holding data and instructions.
storing computer instructions,
See at least: "system memory 3306 can include various modules that include executable instructions to implement functionalities described herein." (Col. 35, ll. 35-37)
Rationale: The claim requires the medium to be storing computer instructions. Levinson's system memory 3306 and storage device 3308 store "modules" that contain "executable instructions," which are computer instructions.
wherein the computer instructions are configured to enable a control host of a remote driving system to perform operations
See at least: "teleoperator manager 3607 ... may include logic, such as hardware or software, or a combination thereof, and may be disposed in a teleoperator computing device 3604 with which teleoperators 3608 may interact…" (Col. 37, ll. 44-48)
Rationale: The claim requires the computer instructions to be configured to enable a control host of a remote driving system to perform operations. Levinson's "teleoperator manager 3607," which includes software logic (i.e., computer instructions), is hosted on a teleoperator computing device 3604. This device acts as the control host for the remote driving system (i.e., the teleoperation system). The instructions enable it to perform operations, such as facilitating interactions with the autonomous vehicle controller, which is the core function of the claimed remote driving system.
comprising: obtaining a first state information of the remote driving system,
See at least: “Message data may be received at a teleoperator computing device” (Col. 23, ll. 23-24)
Rationale: The quoted “message data may be received” corresponds to obtaining state information used for teleoperation.
wherein the remote driving system is configured to perform remote control on one or more vehicles communicatively connected with the remote driving system;
See at least: “At 212, the selection of a candidate trajectory as a guided trajectory may be transmitted to the vehicle, which, in turn, implements the guided trajectory … by causing the vehicle to perform a teleoperator-specified maneuver” (Col. 8, 29-33)
Rationale: Levinson establishes remote control of a vehicle via transmitted control content.
determining whether an abnormality occurs in the remote driving system based on the first state information;
See at least: “teleoperator manager 3807 determines that an autonomous vehicle service platform fails to receive either sufficient data or a periodic signal (e.g., heartbeat signal) that confirms communication … is present” (Col. 46, ll. 24-28)
Rationale: The quoted “determines … fails to receive either sufficient data or a periodic signal (e.g., heartbeat signal)” establishes determining an abnormality based on the presence/absence of received data/signal.
determining abnormality information of the remote driving system in response to the occurrence of the abnormality in the remote driving system;
See at least: “ACCESS A TELEOPERATION REPOSITORY TO RETRIEVE A FIRST SUBSET OF RECOMMENDATIONS” (Fig. 15, 1504)
Rationale: The quoted “retrieve … recommendations” provides information used after an abnormal/non-normative condition is identified.
and adjusting a state of the remote control based on the abnormality information,
See at least: “At 212, the selection of a candidate trajectory as a guided trajectory may be transmitted to the vehicle, which, in turn, implements the guided trajectory for resolving the condition” (Col. 8, ll. 29-32)
Rationale: The quoted “implements … for resolving the condition” establishes adjusting remote control state responsive to condition-handling information.
wherein the abnormality information comprises a handling action for the abnormality,
See at least: “recommendations are combined to form a set of recommended courses of action” (Col. 23, ll. 48-49)
Rationale: The quoted “recommended courses of action” corresponds to a handling action for the abnormality.
handling the abnormality according to the handling action;
See at least: “At 212, the selection of a candidate trajectory as a guided trajectory may be transmitted to the vehicle, which, in turn, implements the guided trajectory for resolving the condition by causing the vehicle to perform a teleoperator-specified maneuver” (Col. 8, ll. 29-33)
Rationale: The quoted “implements … for resolving the condition” and “causing the vehicle to perform a teleoperator-specified maneuver” corresponds to handling the abnormality according to the course of action.
and resuming the remote control in response to the abnormality being eliminated.
See at least: “resume autonomous vehicle services upon completion of successful safe-stop recovery processes” (Col. 46, ll. 50-52)
Rationale: The quoted “resume … upon completion of … recovery processes” corresponds to resuming after the abnormality is eliminated via recovery.
Claim limitations Not Explicitly Disclosed by Levinson
Levinson does not explicitly disclose:
suspending the remote control;
Disclosure by Hamada
Hamada discloses:
suspending the remote control;
See at least: “The transmission of the new control command is suspended” (Col. 5, ll. 28–29)
Rationale: The quoted “transmission … is suspended” corresponds to suspending remote control command transmission.
Motivation to Combine Levinson and Hamada
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson and Hamada before them, to incorporate Hamada’s explicit “transmission … is suspended” control-suspension technique into Levinson’s teleoperation control flow when “teleoperator manager 3807 determines” a communication/data abnormality, so that remote control commands are suspended during abnormality handling and then resumed after “successful … recovery processes,” yielding the predictable result of preventing unsafe or conflicting command transmission while the system resolves the abnormal condition.
Claims 2, 3, 11, and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Levinson (US 9,507,346 B1), in view of Hamada (US6107937A), and in view of Katrak (US 2008/0065292 A1)
Regarding Claim 2,
The combination of Levinson and Hamada establishes the method of Claim 1, which is the basis for Claim 2.
Disclosure by Levinson
Levinson teaches
further comprising:
See at least: "In particular, a method may include... identifying one or more courses of action to perform responsive to detecting the data specifying the event" (Abstract)
Rationale: Levinson describes the method as including additional steps performed in response to detected data.
Obtaining, by the control host, a second state information of a vehicle from the one or more vehicles,
See at least: "any autonomous vehicle in fleet of autonomous vehicles 3630a may generate message data 3619... message data 3619 may include any type of data, such as telemetry data..." (Col. 38, ll. 64-67 – Col. 39, ll. 1-3)
Rationale: The control host (teleoperator manager) obtains telemetry data—which constitutes state information—directly from the vehicles in the fleet.
Claim limitations Not Explicitly Disclosed by Levinson and Hamada
Levinson and Hamada do not explicitly disclose:
wherein the determining whether an abnormality occurs in the remote driving system comprises: determining whether a state of the remote driving system and a state of the vehicle are synchronized
based on the first state information and the second state information;
and determining that an abnormality occurs in the remote driving system
in response to the state of the remote driving system being out of synchronization with the state of the vehicle.
Disclosure by Katrak
Katrak discloses:
wherein the determining whether an abnormality occurs in the remote driving system comprises: determining whether a state of the remote driving system and a state of the vehicle are synchronized
See at least: "...checking a difference between a target road wheel angle and an actual road wheel angle..." ([0007])
Rationale: Katrak teaches a determination step that checks for a difference—which a PHOSITA would understand as a lack of synchronization—between a target system state and an actual measured vehicle state.
based on the first state information and the second state information;
See at least: "The processing logic element is configured to compare the first value and the second value as a function of vehicle speed." ([0047])
Rationale: Katrak teaches that the determination is based on the comparison of a first value and a second value, representing the two distinct state inputs.
and determining that an abnormality occurs in the remote driving system
See at least: "the AFS system is configured to revert to a mechanical mode if the comparison value exceeds the threshold." ([0049])
Rationale: Katrak teaches that exceeding a threshold in the comparison results in a fault state (reverting to mechanical mode), which constitutes determining that an abnormality occurs.
in response to the state of the remote driving system being out of synchronization with the state of the vehicle.
See at least: "...if the comparison value exceeds the threshold." ([0049])
Rationale: Katrak teaches that the fault determination is triggered specifically when the comparison value exceeds a threshold, which represents the states being out of synchronization.
Motivation to Combine Levinson, Hamada, and Katrak
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson, Hamada, and Katrak before them, to incorporate Katrak's synchronization-based fault detection into the remote driving system of Levinson. Levinson provides the foundational teleoperation method and the transfer of telemetry (state) data, while Katrak teaches a specific technique for identifying system abnormalities by comparing target states with actual measured vehicle states. A PHOSITA would recognize that in a remote driving context, ensuring that the host's control state and the vehicle's physical state remain synchronized is critical for safety; applying a threshold-based comparison as taught by Katrak provides a predictable way to detect communication lag or system errors (abnormalities) and trigger the handling actions already established in the Levinson/Hamada combination.
Regarding Claim 3,
The combination of Levinson and Hamada establishes the method of Claim 2, which is the basis for Claim 3.
Disclosure by Levinson
Levinson teaches:
wherein the remote driving system comprises a first control device,
See at least: "A teleoperator computing device 3604... includes a teleoperator manager 3607..." (Col. 49, Ln. 45)
Rationale: Levinson teaches a host-side teleoperator computing device which acts as a first control device in the remote driving system.
and the vehicle comprises a second control device,
See at least: "autonomous vehicle 3630... includes an autonomous vehicle controller 3647..." (Col. 49, Ln. 60)
Rationale: Levinson teaches a vehicle-side controller which acts as a second control device.
the first control device configured to control the second control device
See at least: "teleoperator computing device 3604... can be configured to initiate modification of trajectories (e.g., remotely) to influence navigation of autonomous vehicles." (Col. 49, Ln. 38)
Rationale: The host-side first control device transmits trajectory modifications to the vehicle-side second control device to influence its navigation.
to cause the vehicle to generate a corresponding driving behavior,
See at least: "Response data 107 then can be transmitted back to autonomous vehicle 109d to cause the vehicle to... safely cross a set of double lines" (Col. 5, Ln. 30)
Rationale: The host-side commands cause the vehicle to generate a corresponding driving behavior such as crossing lines or navigating an environment.
wherein the first state information comprises a first motion information of the first control device,
See at least: "teleoperator manager 3607... transmits the selection of the candidate trajectory as... a guided trajectory... to the vehicle." (Col. 53, Ln. 34)
Rationale: The host-side first control device (teleoperator manager) selects and issues a guided trajectory; this trajectory data originates at the host, constituting first motion information of the first control device.
and the second state information comprises a second motion information of the second control device,
See at least: "telemetry data 3619 may include... position data... to facilitate teleoperations." (Col. 49, Ln. 64)
Rationale: The vehicle-side second control device provides telemetry including position data, which constitutes second motion information of the second control device.
Claim limitations Not Explicitly Disclosed by Levinson and Hamada
Levinson does not explicitly teach:
and wherein the determining whether the state of the remote driving system and the state of the vehicle are synchronized comprises: determining that the second control device is not synchronized with the first control device
in response to a difference between the second motion information and the first motion information being greater than a first threshold.
Disclosure by Katrak
Katrak teaches:
and wherein the determining whether the state of the remote driving system and the state of the vehicle are synchronized comprises: determining that the second control device is not synchronized with the first control device
See at least: "The AFS system utilizes... supervisory control systems... to lower likelihood of fault detection errors by checking a difference between a target road wheel angle and an actual road wheel angle" ([0007])
Rationale: Katrak teaches a determination step that identifies a fault (abnormality) by checking for a difference between target and actual values; a PHOSITA would recognize that finding such a difference—representing a mismatch between intended host commands and actual vehicle response—is determining that the second control device is not synchronized with the first control device.
in response to a difference between the second motion information and the first motion information being greater than a first threshold.
See at least: "the processing logic element is configured to compare the first value and the second value... the AFS system is configured to revert to a mechanical mode if the comparison value exceeds the threshold." ([0047], [0049])
Rationale: Katrak teaches a logic element that compares a first value and a second value and identifies an abnormality specifically in response to a difference between the compared values being greater than a first threshold (the comparison value exceeding the threshold).
Motivation to Combine Levinson, Hamada, and Katrak
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson, Hamada, and Katrak before them, to incorporate Katrak's threshold-based comparison logic into the remote driving system of Levinson. Levinson provides the foundational architecture where a host (first control device) issues intended trajectory data and a vehicle (second control device) reports actual telemetry data. Katrak provides a generic, robust method for detecting system-wide faults by comparing target values against actual measured values and declaring an abnormality when the difference exceeds a threshold. A PHOSITA would implement this threshold-based comparison to ensure that the vehicle's reported position and motion accurately reflect the host's commands, recognizing that a difference exceeding a threshold is a predictable indicator that the two devices are not synchronized, thereby necessitating the handling actions (such as safe-stop recovery) already established in the Levinson/Hamada combination.
Regarding Claim 11,
The combination of Levinson and Hamada establishes the electronic device of Claim 10, which is the basis for Claim 11.
Disclosure by Levinson
Levinson discloses:
wherein the operations further comprise: obtaining a second state information
See at least: "any autonomous vehicle in fleet of autonomous vehicles 3630a may generate message data 3619... message data 3619 may include any type of data, such as telemetry data..." (Col. 38, ll. 64-67 – Col. 39, ll. 1-3)
Rationale: The verbatim text from Levinson discloses wherein the operations further comprise: obtaining a second state information as any autonomous vehicle in fleet of autonomous vehicles may generate message data... message data may include any type of data, such as telemetry data.
of a vehicle from the one or more vehicles,
See at least: "any autonomous vehicle in fleet of autonomous vehicles 3630a may generate message data 3619... message data 3619 may include any type of data, such as telemetry data..." (Col. 38, ll. 64-67 – Col. 39, ll. 1-3)
Rationale: The verbatim text from Levinson discloses of a vehicle from the one or more vehicles as any autonomous vehicle in fleet of autonomous vehicles may generate message data.
Claim limitations Not Explicitly Disclosed by Levinson
Levinson does not explicitly disclose the following claim limitations:
wherein the determining whether an abnormality occurs in the remote driving system comprises: determining whether a state of the remote driving system and a state of the vehicle are synchronized
based on the first state information and the second state information;
and determining that an abnormality occurs in the remote driving system in response to the state of the remote driving system being out of synchronization with the state of the vehicle.
Disclosure by Katrak
Katrak discloses:
wherein the determining whether an abnormality occurs in the remote driving system comprises: determining whether a state of the remote driving system and a state of the vehicle are synchronized
See at least: "...checking a difference between a target road wheel angle and an actual road wheel angle..." ([0007])
Rationale: Katrak discloses wherein the determining whether an abnormality occurs in the remote driving system comprises: determining whether a state of the remote driving system and a state of the vehicle are synchronized as checking a difference between a target road wheel angle and an actual road wheel angle.
based on the first state information and the second state information;
See at least: "The processing logic element is configured to compare the first value and the second value as a function of vehicle speed." ([0047])
Rationale: Katrak discloses based on the first state information and the second state information as The processing logic element is configured to compare the first value and the second value as a function of vehicle speed.
and determining that an abnormality occurs in the remote driving system in response to the state of the remote driving system being out of synchronization with the state of the vehicle.
See at least: "the AFS system is configured to revert to a mechanical mode if the comparison value exceeds the threshold." ([0049])
Rationale: The verbatim text from Katrak discloses and determining that an abnormality occurs in the remote driving system in response to the state of the remote driving system being out of synchronization with the state of the vehicle as the AFS system is configured to revert to a mechanical mode if the comparison value exceeds the threshold.
Motivation to Combine Levinson, Hamada, and Katrak
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson, Hamada, and Katrak before them, to incorporate Katrak’s comparison/threshold-based synchronization determination into the remote driving control host framework established by Levinson as modified by Hamada. Levinson provides the remote teleoperation architecture that obtains vehicle-originated operational state data for teleoperations, and Hamada provides the safety-oriented remote-control handling logic used when abnormal conditions occur in such remote control systems. Katrak teaches explicitly determining abnormal/fault conditions by comparing a first value and a second value and acting when a comparison value exceeds a threshold, which corresponds to determining whether states are synchronized and determining an abnormality when they are out of synchronization. A PHOSITA would have combined these teachings to achieve the predictable result of detecting a teleoperation abnormality when the remote/system state and vehicle state diverge beyond an acceptable threshold, and then applying the established abnormality-handling logic of the remote control host.
Regarding Claim 12,
The combination of Levinson, Hamada, and Katrak establishes the electronic device of Claim 11, which is the basis for Claim 12.
Disclosure by Levinson
Levinson discloses:
wherein the remote driving system comprises a first control device,
See at least: “teleoperator manager 3607” (Fig. 36)
Rationale: Identifies “teleoperator computing device 3604” within the teleoperation architecture, which corresponds to a first control device.
and the vehicle comprises a second control device,
See at least: “AV Controller 3647.” (Fig. 36)
Rationale: Identifies “AV Controller 3647” in the vehicle, which corresponds to the claimed vehicle that comprises a second control device.
the first control device configured to control the second control device
See at least: “a teleoperator may interact with user interface 3701 to provide implicit guidance or explicit guidance to modify trajectories of autonomous vehicle 3730” (Col. 42, ll. 55-58)
Rationale: Disclosure expressly teaches that the “teleoperator computing device” performs trajectory modification for the “autonomous vehicle,” which corresponds to the claimed the first control device configured to control the second control device.
to cause the vehicle to generate a corresponding driving behavior,
See at least: “a teleoperator may interact with user interface 3701 to provide implicit guidance or explicit guidance to modify trajectories of autonomous vehicle 3730” (Col. 42, ll. 55-58)
Rationale: The quoted disclosure expressly teaches providing instruction data to the autonomous vehicle modify its trajectory,” which corresponds to the claimed to cause the vehicle to generate a corresponding driving behavior.
wherein the first state information comprises a first motion information of the first control device,
See at least: “At 212, the selection of a candidate trajectory as a guided trajectory may be transmitted to the vehicle, which, in turn, implements the guided trajectory … by causing the vehicle to perform a teleoperator-specified maneuver.” (Col. 8, ll. 29-33)
Rationale: The quoted disclosure expressly teaches transmission of a “candidate trajectory as a guided trajectory” that the vehicle implements to perform a maneuver, which corresponds to the claimed wherein the first state information comprises a first motion information of the first control device,.
and the second state information comprises a second motion information of the second control device,
See at least: “Obstacle data 920, planner options data 924, and position data 926 are transmitted to a messaging service bridge 932, which, in accordance with message service configuration data 934, generates telemetry data 940 and query data 942, both of which are transmitted via data-centric messaging bus 972 into teleoperator application 901 as telemetry data 950 and query data 952” (Col. 18, ll. 11-18)
Rationale: The quoted disclosure expressly identifies “position data” included in exchanged message data from the vehicle side, which corresponds to the claimed and the second state information comprises a second motion information of the second control device,.
Claim limitations Not Explicitly Disclosed by Levinson and Hamada
Levinson and Hamada do not explicitly disclose:
and wherein the determining whether the state of the remote driving system and the state of the vehicle are synchronized comprises: determining that the second control device is not synchronized with the first control device
in response to a difference between the second motion information and the first motion information being greater than a first threshold.
Disclosure by Katrak
Katrak discloses:
and wherein the determining whether the state of the remote driving system and the state of the vehicle are synchronized comprises: determining that the second control device is not synchronized with the first control device
See at least: “checking a difference between a target road wheel angle and an actual road wheel angle.” (Abstract)
Rationale: The quoted disclosure expressly teaches “checking a difference” between a target value and an actual value, which corresponds to the claimed limitation.
in response to a difference between the second motion information and the first motion information being greater than a first threshold.
See at least: “THE FIRST COMPARISON VALUE EXCEEDS A THRESHOLD?” (FIG. 2, 218)
Rationale: The quoted disclosure expressly teaches a condition where a “COMPARISON VALUE EXCEEDS … A THRESHOLD,” which corresponds to the claimed in response to a difference between the second motion information and the first motion information being greater than a first threshold..
Motivation to Combine Levinson, Hamada, and Katrak
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson, Hamada, and Katrak before them, to incorporate Katrak’s explicit threshold-based “difference/compare” fault logic into the teleoperation control-host architecture of Levinson (as already modified by Hamada for abnormality-handling control-state management), because Levinson provides the remote-driving system in which a host issues motion guidance and receives vehicle motion/state data, and Katrak provides a directly compatible technique for declaring non-synchronization when a difference-derived “comparison value” exceeds a threshold, yielding the predictable result of detecting an abnormal synchronization condition between host-side intended motion information and vehicle-side actual motion information, and then applying the already-established abnormality-handling flow.
Claims 4, 9, 13, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Levinson (US 9,507,346 B1), in view of Hamada (US6107937A), and in view of Shimida (US 2014/0129049 A1)
Regarding Claim 4,
The combination of Levinson and Hamada establishes the method of Claim 1, which is the basis for Claim 4.
Disclosure by Levinson
Levinson teaches:
wherein the remote driving system
See at least: “teleoperator manager 3607…configured to… establish communications between teleoperator computing device 3604 and autonomous vehicle 3630.” (Col. 39, ll. 5-9)
Rationale: The verbatim text from Levinson teaches wherein the remote driving system as the teleoperator manager/service platform.
Claim limitations Not Explicitly Disclosed by Levinson
Levinson does not explicitly teach the following:
comprises a third control device
and a fourth control device,
the third control device configured to control a vehicle
from the one or more vehicles
to generate a first driving behavior
through a first control instruction,
and the fourth control device configured to control the vehicle
to generate a second driving behavior
through a second control instruction,
wherein the first driving behavior is opposite to the second driving behavior,
and the first state information comprises a flag
indicating whether the first control instruction and the second control instruction present,
and wherein the determining whether an abnormality occurs in the remote driving system comprises: determining that an operation of the third control device or the fourth control device is abnormal
in response to the simultaneous presence of the first control instruction and the second control instruction.
Disclosure by Hamada
Hamada teaches:
comprises a third control device
See at least: "a plurality of remote manipulation devices" (Abstract)
Rationale: The verbatim text from Hamada teaches comprises a third control device as one of the plurality of remote manipulation devices.
and a fourth control device,
See at least: "a plurality of remote manipulation devices" (Abstract)
Rationale: Hamada teaches and a fourth control device as another of the plurality of remote manipulation devices.
the third control device configured to control a vehicle
See at least: "the remote manipulation device... transmit a control signal to the controlled device" (Abstract)
Rationale: Hamada teaches the third control device configured to control a vehicle as the remote manipulation device transmitting a control signal to the controlled device.
from the one or more vehicles
See at least: "controlled devices" (Abstract)
Rationale: Hamada teaches from the one or more vehicles as the controlled devices.
to generate a first driving behavior
See at least: "transmit a control signal to the controlled device" (Abstract)
Rationale: The verbatim text from Hamada teaches to generate a first driving behavior as the control signal causing the controlled device to perform a behavior.
through a first control instruction,
See at least: "transmit a control signal" (Abstract)
Rationale: The verbatim text from Hamada teaches through a first control instruction as the control signal from one remote manipulation device.
and the fourth control device configured to control the vehicle
See at least: "the remote manipulation device... transmit a control signal to the controlled device" (Abstract)
Rationale: The verbatim text from Hamada teaches and the fourth control device configured to control the vehicle as another remote manipulation device transmitting a control signal to the controlled device.
to generate a second driving behavior
See at least: "transmit a control signal to the controlled device" (Abstract)
Rationale: The verbatim text from Hamada teaches to generate a second driving behavior as the control signal causing the controlled device to perform another behavior.
through a second control instruction,
See at least: "transmit a control signal" (Abstract)
Rationale: The verbatim text from Hamada teaches through a second control instruction as the control signal from another remote manipulation device.
and wherein the determining whether an abnormality occurs in the remote driving system comprises: determining that an operation of the third control device or the fourth control device is abnormal
See at least: "preventing the mixing of signals from remote manipulation devices" (Abstract)
Rationale: A PHOSITA would recognize that detecting and preventing “mixing of signals” from multiple remote manipulation devices necessarily entails identifying a conflict condition; treating such a detected conflict as an abnormal operation of at least one of the remote manipulation devices is an obvious safety implementation in a multi-controller remote control system.
in response to the simultaneous presence of the first control instruction and the second control instruction.
See at least: "the simultaneous transmission of control commands by more than one remote manipulation devices is avoided" (Abstract)
Rationale: Hamada teaches in response to the simultaneous presence of the first control instruction and the second control instruction as avoiding simultaneous transmission of control commands.
Motivation to Combine Levinson and Hamada
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson and Hamada before them, to incorporate Hamada's teaching of multiple remote control devices with conflict prevention into Levinson's teleoperation system for autonomous vehicles. Both references are directed to remote control systems managing commands to controlled devices. Levinson provides a framework for remote vehicle guidance via trajectories, while Hamada teaches handling multiple control sources to avoid signal mixing. A skilled artisan, seeking to enhance Levinson's system reliability by preventing conflicting commands from multiple teleoperators or sources, would have been motivated to integrate Hamada's multi-device management, including applying it to vehicle control and driving behaviors as in Levinson, yielding predictable results in safer remote driving by detecting and avoiding simultaneous conflicting instructions, where treating detected simultaneous commands as an abnormal operation is a predictable safety enhancement.
Claim limitations Not Explicitly Disclosed by the Combination of Levinson and Hamada
After combining the teachings of Levinson and Hamada, the following are not explicitly disclosed:
wherein the first driving behavior is opposite to the second driving behavior,
and the first state information comprises a flag
indicating whether the first control instruction and the second control instruction present
Disclosure by Shimida
Shimida teaches:
wherein the first driving behavior is opposite to the second driving behavior,
See at least: "limits drive force of a vehicle upon simultaneous performance of an accelerator operation and a brake operation" ([0009])
Rationale: Shimida teaches wherein the first driving behavior is opposite to the second driving behavior as accelerator operation and brake operation, which are opposite behaviors.
and the first state information comprises a flag
See at least: "Upon detection of the simultaneous performance, a drive force limit initiation flag is set on." ([0024])
Rationale: Shimida teaches and the first state information comprises a flag as the drive force limit initiation flag set upon detection of simultaneous performance.
indicating whether the first control instruction and the second control instruction present.
See at least: "detection of simultaneous performance of the accelerator operation and the brake operation" ([0023])
Rationale: A PHOSITA would recognize that the “accelerator operation” and “brake operation” correspond to contemporaneous control input signals; detecting their simultaneous performance is equivalent to detecting simultaneous presence of two control instruction signals, and the flag indicates whether both signals are present.
Motivation to Combine Levinson, Hamada, and Shimida
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson, Hamada, and Shimida before them, to incorporate Shimida's teaching of detecting and flagging simultaneous opposite control operations into the combined remote control system of Levinson and Hamada. All references pertain to control systems ensuring safe operation by handling conflicting inputs. The Levinson-Hamada combination manages remote commands to vehicles, while Shimida provides specific detection of conflicting accelerator/brake inputs as a safety override. A skilled artisan, aiming to improve safety in remote driving by addressing potential conflicting behaviors like simultaneous acceleration and braking (e.g., due to faults or errors in remote instructions from multiple control devices), would have been motivated to integrate Shimida's opposite instruction detection and flagging, including implementing the flag in the remote host's first state information as an obvious design choice, resulting in predictable enhancements for abnormality handling in teleoperated vehicles.
Regarding Claim 9,
The combination of Levinson and Hamada establishes the method of Claim 1, which is the basis for Claim 9.
Disclosure by Levinson
Levinson teaches:
wherein the abnormality information further comprises an alarm level of the abnormality,
See at least: “one or more events (e.g., associated with a range of confidence levels) for which teleoperation services may be invoked.” (Col. 45, ll. 61-63)
Rationale: Levinson expressly associates the event/abnormal condition with “a range of confidence levels,” which a PHOSITA would understand as an alarm level indicating severity/criticality of the abnormality.
and wherein the suspending the remote control comprises: suspending the remote control
See at least: “autonomous vehicle 3830 … had implemented a safe-stop trajectory …” (Col. 46, ll. 38-40)
Rationale: Levinson’s “safe-stop” implementation reflects interrupting ongoing operational control in response to abnormal conditions, consistent with suspending remote control under the method context.
in response to the alarm level
See at least: “events (e.g., associated with a range of confidence levels) for which teleoperation services may be invoked.” (Col. 46, ll. 62-62)
Rationale: Levinson expressly ties invoking teleoperation services to events associated with confidence levels, i.e., the action is taken in response to the alarm level.
Claim limitations Not Explicitly Disclosed by Levinson
Levinson does not explicitly teach:
and wherein the suspending the remote control comprises: suspending the remote control (as an explicit “suspending” step of the remote control state)
being greater than a second threshold.
Disclosure by Hamada
Hamada teaches:
and wherein the suspending the remote control comprises: suspending the remote control
See at least: “The transmission of the new control command is suspended …” (Col. 5, ll. 28–29)
Rationale: Hamada expressly teaches suspending transmission of the control command, which is suspending the remote control.
Motivation to Combine Levinson and Hamada
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson and Hamada before them, to incorporate Hamada’s explicit suspension of control-command transmission into Levinson’s teleoperation framework that invokes teleoperation services responsive to abnormal events associated with confidence levels, because both references address maintaining safe and reliable remote/assisted vehicle operation during abnormal conditions, and adding an explicit suspension step provides a predictable safety response that prevents further remote commands while the abnormal condition is handled.
Claim limitations Not Explicitly Disclosed by the Combination of Levinson and Hamada
After combining the teachings of Levinson and Hamada, the following are not explicitly disclosed:
being greater than a second threshold.
Disclosure by Shimida
Shimida teaches:
being greater than a second threshold.
See at least: “step S105 verifies whether the value of the first delay counter is greater than or equal to a defined threshold β.” ([0030])
Rationale: Shimida expressly teaches a “greater than or equal to … threshold” comparison, which corresponds to the alarm level being greater than a second threshold.
Motivation to Combine Levinson, Hamada, and Shimida
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson, Hamada, and Shimida before them, to implement Shimida’s explicit “greater than or equal to … threshold” decision logic in the Levinson/Hamada remote-control suspension workflow so that the suspension of remote control is triggered when the alarm level (confidence level associated with the abnormal event) exceeds a predetermined threshold, because Shimida provides a well-known, predictable threshold-trigger mechanism for initiating a protective action, and applying that mechanism to confidence-level-driven abnormality handling in teleoperation systems yields predictable improvements in reliability and safety.
Regarding Claim 13,
The combination of Levinson and Hamada establishes the electronic device of Claim 10, which is the basis for Claim 13.
Disclosure by Levinson
Levinson discloses:
wherein the remote driving system
See at least: “teleoperator manager 3607…configured to… establish communications between teleoperator computing device 3604 and autonomous vehicle 3630.” (Col. 39, ll. 5-9)
Rationale: The verbatim text from Levinson teaches wherein the remote driving system as the teleoperator manager/service platform.
Claim limitations Not Explicitly Disclosed by Levinson
Levinson does not explicitly disclose:
comprises a third control device
and a fourth control device,
the third control device configured to control a vehicle
from the one or more vehicles
to generate a first driving behavior
through a first control instruction,
and the fourth control device configured to control the vehicle
to generate a second driving behavior
through a second control instruction,
wherein the first driving behavior is opposite to the second driving behavior,
and the first state information comprises a flag
indicating whether the first control instruction and the second control instruction present,
and wherein the determining whether an abnormality occurs in the remote driving system comprises: determining that an operation of the third control device or the fourth control device is abnormal
in response to the simultaneous presence of the first control instruction and the second control instruction.
Disclosure by Hamada
Hamada discloses:
comprises a third control device
See at least: "a plurality of remote manipulation devices" (Abstract)
Rationale: The verbatim text from Hamada teaches comprises a third control device as one of the plurality of remote manipulation devices.
and a fourth control device,
See at least: "a plurality of remote manipulation devices" (Abstract)
Rationale: Hamada teaches and a fourth control device as another of the plurality of remote manipulation devices.
the third control device configured to control a vehicle
See at least: "the remote manipulation device... transmit a control signal to the controlled device" (Abstract)
Rationale: Hamada teaches the third control device configured to control a vehicle as the remote manipulation device transmitting a control signal to the controlled device.
from the one or more vehicles
See at least: "controlled devices" (Abstract)
Rationale: Hamada teaches from the one or more vehicles as the controlled devices.
to generate a first driving behavior
See at least: "transmit a control signal to the controlled device" (Abstract)
Rationale: The verbatim text from Hamada teaches to generate a first driving behavior as the control signal causing the controlled device to perform a behavior.
through a first control instruction,
See at least: "transmit a control signal" (Abstract)
Rationale: The verbatim text from Hamada teaches through a first control instruction as the control signal from one remote manipulation device.
and the fourth control device configured to control the vehicle
See at least: "the remote manipulation device... transmit a control signal to the controlled device" (Abstract)
Rationale: The verbatim text from Hamada teaches and the fourth control device configured to control the vehicle as another remote manipulation device transmitting a control signal to the controlled device.
to generate a second driving behavior
See at least: "transmit a control signal to the controlled device" (Abstract)
Rationale: The verbatim text from Hamada teaches to generate a second driving behavior as the control signal causing the controlled device to perform another behavior.
through a second control instruction,
See at least: "transmit a control signal" (Abstract)
Rationale: The verbatim text from Hamada teaches through a second control instruction as the control signal from another remote manipulation device.
and wherein the determining whether an abnormality occurs in the remote driving system comprises: determining that an operation of the third control device or the fourth control device is abnormal
See at least: "preventing the mixing of signals from remote manipulation devices" (Abstract)
Rationale: A PHOSITA would recognize that detecting and preventing “mixing of signals” from multiple remote manipulation devices necessarily entails identifying a conflict condition; treating such a detected conflict as an abnormal operation of at least one of the remote manipulation devices is an obvious safety implementation in a multi-controller remote control system.
in response to the simultaneous presence of the first control instruction and the second control instruction.
See at least: "the simultaneous transmission of control commands by more than one remote manipulation devices is avoided" (Abstract)
Rationale: Hamada teaches in response to the simultaneous presence of the first control instruction and the second control instruction as avoiding simultaneous transmission of control commands.
Motivation to Combine Levinson and Hamada
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson and Hamada before them, to incorporate Hamada's teaching of multiple remote control devices with conflict prevention into Levinson's teleoperation system for autonomous vehicles. Both references are directed to remote control systems managing commands to controlled devices. Levinson provides a framework for remote vehicle guidance via trajectories, while Hamada teaches handling multiple control sources to avoid signal mixing. A skilled artisan, seeking to enhance Levinson's system reliability by preventing conflicting commands from multiple teleoperators or sources, would have been motivated to integrate Hamada's multi-device management, including applying it to vehicle control and driving behaviors as in Levinson, yielding predictable results in safer remote driving by detecting and avoiding simultaneous conflicting instructions, where treating detected simultaneous commands as an abnormal operation is a predictable safety enhancement.
Claim limitations Not Explicitly Disclosed by the Combination of Levinson and Hamada
After combining the teachings of Levinson and Hamada, the following are not explicitly disclosed:
wherein the first driving behavior is opposite to the second driving behavior,
and the first state information comprises a flag
indicating whether the first control instruction and the second control instruction present
Disclosure by Shimida
Shimida discloses:
wherein the first driving behavior is opposite to the second driving behavior,
See at least: "limits drive force of a vehicle upon simultaneous performance of an accelerator operation and a brake operation" ([0009])
Rationale: Shimida teaches wherein the first driving behavior is opposite to the second driving behavior as accelerator operation and brake operation, which are opposite behaviors.
and the first state information comprises a flag
See at least: "Upon detection of the simultaneous performance, a drive force limit initiation flag is set on." ([0024])
Rationale: Shimida teaches and the first state information comprises a flag as the drive force limit initiation flag set upon detection of simultaneous performance.
indicating whether the first control instruction and the second control instruction present.
See at least: "detection of simultaneous performance of the accelerator operation and the brake operation" ([0023])
Rationale: A PHOSITA would recognize that the “accelerator operation” and “brake operation” correspond to contemporaneous control input signals; detecting their simultaneous performance is equivalent to detecting simultaneous presence of two control instruction signals, and the flag indicates whether both signals are present.
Motivation to Combine Levinson, Hamada, and Shimida
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson, Hamada, and Shimida before them, to incorporate Shimida's teaching of detecting and flagging simultaneous opposite control operations into the combined remote control system of Levinson and Hamada. All references pertain to control systems ensuring safe operation by handling conflicting inputs. The Levinson-Hamada combination manages remote commands to vehicles, while Shimida provides specific detection of conflicting accelerator/brake inputs as a safety override. A skilled artisan, aiming to improve safety in remote driving by addressing potential conflicting behaviors like simultaneous acceleration and braking (e.g., due to faults or errors in remote instructions from multiple control devices), would have been motivated to integrate Shimida's opposite instruction detection and flagging, including implementing the flag in the remote host's first state information as an obvious design choice, resulting in predictable enhancements for abnormality handling in teleoperated vehicles.
Regarding Claim 18,
The combination of Levinson and Hamada establishes the electronic device of Claim 10, which is the basis for Claim 18.
Disclosure by Levinson
Levinson discloses:
wherein the abnormality information further comprises an alarm level of the abnormality,
See at least: “one or more events (e.g., associated with a range of confidence levels) for which teleoperation services may be invoked.” (Col. 45, ll. 61-63)
Rationale: Levinson expressly associates the event/abnormal condition with “a range of confidence levels,” which a PHOSITA would understand as an alarm level indicating severity/criticality of the abnormality.
and wherein the suspending the remote control comprises: suspending the remote control
See at least: “autonomous vehicle 3830 … had implemented a safe-stop trajectory …” (Col. 46, ll. 38-40)
Rationale: Levinson’s “safe-stop” implementation reflects interrupting ongoing operational control in response to abnormal conditions, consistent with suspending remote control under the method context.
in response to the alarm level
See at least: “events (e.g., associated with a range of confidence levels) for which teleoperation services may be invoked.” (Col. 46, ll. 62-62)
Rationale: Levinson expressly ties invoking teleoperation services to events associated with confidence levels, i.e., the action is taken in response to the alarm level.
Claim limitations Not Explicitly Disclosed by Levinson
Levinson does not explicitly teach:
and wherein the suspending the remote control comprises: suspending the remote control (as an explicit “suspending” step of the remote control state)
being greater than a second threshold.
Disclosure by Hamada
Hamada discloses:
and wherein the suspending the remote control comprises: suspending the remote control
See at least: “The transmission of the new control command is suspended …” (Col. 5, ll. 28–29)
Rationale: Hamada expressly teaches suspending transmission of the control command, which is suspending the remote control.
Motivation to Combine Levinson and Hamada
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson and Hamada before them, to incorporate Hamada’s explicit suspension of control-command transmission into Levinson’s teleoperation framework that invokes teleoperation services responsive to abnormal events associated with confidence levels, because both references address maintaining safe and reliable remote/assisted vehicle operation during abnormal conditions, and adding an explicit suspension step provides a predictable safety response that prevents further remote commands while the abnormal condition is handled.
Claim limitations Not Explicitly Disclosed by the Combination of Levinson and Hamada
After combining the teachings of Levinson and Hamada, the following are not explicitly disclosed:
being greater than a second threshold.
Disclosure by Shimida
Shimida discloses:
being greater than a second threshold.
See at least: “step S105 verifies whether the value of the first delay counter is greater than or equal to a defined threshold β.” ([0030])
Rationale: Shimida expressly teaches a “greater than or equal to … threshold” comparison, which corresponds to the alarm level being greater than a second threshold.
Motivation to Combine Levinson, Hamada, and Shimida
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson, Hamada, and Shimida before them, to implement Shimida’s explicit “greater than or equal to … threshold” decision logic in the Levinson/Hamada remote-control suspension workflow so that the suspension of remote control is triggered when the alarm level (confidence level associated with the abnormal event) exceeds a predetermined threshold, because Shimida provides a well-known, predictable threshold-trigger mechanism for initiating a protective action, and applying that mechanism to confidence-level-driven abnormality handling in teleoperation systems yields predictable improvements in reliability and safety.
Claims 6 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Levinson (US 9,507,346 B1), in view of Hamada (US6107937A), and in view of Turk (US 2021/0001774 A1).
Regarding Claim 6,
The combination of Levinson and Hamada establishes the method of Claim 5, which is the basis for Claim 6.
Disclosure by Levinson
Levinson teaches:
wherein the data collection device comprises an image collection device,
See at least: “Visualization controller 3611 … is configured to process sensor-generated data (e.g., fused sensor data of one or more of … cameras …)”; (Col. 41, ll. 38-40)
Rationale: Levinson explicitly identifies “cameras” among the sensor modalities whose sensor-generated data is processed, which corresponds to the data collection device comprising an image collection device.
and the first data comprises an image,
See at least: “…to facilitate generation of imagery of high fidelity…” (Col. 41, ll. 49-50)
Rationale: Levinson explicitly uses “imagery,” which corresponds to first data comprising an image.
of a first control device
See at least: “…teleoperator manager 3607 … between one of teleoperators 3608 and … autonomous vehicle 3630 …” (Col. 37, ll. 65-67); “…communication between teleoperator computing device 3604 and autonomous vehicle 3630.” (Col. 38, ll. 30-31)
Rationale: Levinson expressly identifies “teleoperator computing device 3604” as a control-side device within the teleoperation architecture, which corresponds to a first control device.
for controlling the one or more vehicles
See at least: “…an example of explicit guidance includes … [actions] … that may be transmitted … as instruction data 3617.” (Col. 38–39 excerpt)
Rationale: Levinson expressly discloses transmitting “instruction data” from the teleoperator manager architecture to guide the vehicle, which corresponds to the first control device for controlling the one or more vehicles.
Claim limitations Not Explicitly Disclosed by Levinson and Hamada
Levinson and Hamada does not explicitly teach:
and wherein the determining that an abnormality occurs in the data collection device comprises: performing target detection on the image
to determine position coordinates
in the image;
and determining that a shooting angle of the image collection device is abnormal
in response to the position coordinates being not within a preset coordinate range.
Disclosure by Turk
Turk teaches:
and wherein the determining that an abnormality occurs in the data collection device comprises: performing target detection on the image
See at least: “Frames of image data … are processed to determine … of a detected object …” (Abstract)
Rationale: Turk expressly discloses that frames of image data are “processed” to determine information for a “detected object,” which corresponds to performing target detection on the image.
to determine position coordinates
See at least: “…based on positions of the detected object in frames of image data…” (Abstract)
Rationale: Turk expressly determines “positions” of the detected object in frames of image data; a PHOSITA would understand such “positions” in image data as position coordinates.
in the image;
See at least: “…positions of the detected object in frames of image data captured by the camera.” (Abstract)
Rationale: Turk expressly ties the “positions” to “frames of image data,” i.e., in the image.
and determining that a shooting angle of the image collection device is abnormal
See at least: “…determine … that the camera … is misaligned.” (Abstract)
Rationale: Turk expressly determines the camera is “misaligned,” which corresponds to determining that a shooting angle of the image collection device is abnormal.
in response to the position coordinates being not within a preset coordinate range.
See at least: “The determined object motion vector is compared to the determined vehicle motion vector to determine … that the camera … is misaligned.” (Abstract)
Rationale: Turk expressly determines misalignment “responsive to” a comparison outcome; a PHOSITA would recognize this as implementing a preset acceptance criterion (i.e., a preset coordinate range/allowed deviation) such that, when the detected-object position-derived quantities fall outside the preset allowable range, misalignment is determined.
Motivation to Combine Levinson, Hamada, and Turk
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson, Hamada, and Turk before them, to incorporate Turk’s image-based target detection and camera misalignment determination into the Levinson-Hamada remote driving method to improve reliability and safety of camera-derived information used during remote control. Levinson teaches a teleoperation/remote driving architecture that processes camera-derived data/imagery in support of remotely controlling an autonomous vehicle, and Hamada teaches remote control system management in which device-provided signals/information are managed to avoid improper or conflicting operation and to maintain reliable control. Turk teaches processing frames of image data to detect an object, determine positions of the detected object, and determine that a camera is misaligned based on a comparison outcome derived from the image-based positions. A PHOSITA would have been motivated to integrate Turk’s misalignment logic into the Levinson-Hamada remote driving system so that abnormal shooting angle/misalignment of the image collection device can be detected using image-derived positional information, thereby yielding predictable results in which camera faults are identified and remote driving operation is made more robust and safe.
Regarding Claim 15,
The combination of Levinson and Hamada establishes the electronic device of Claim 14, which is the basis for Claim 15.
Disclosure by Levinson
Levinson discloses:
wherein the data collection device comprises an image collection device,
See at least: “Visualization controller 3611 … is configured to process sensor-generated data (e.g., fused sensor data of one or more of … cameras …)”; (Col. 41, ll. 38-40)
Rationale: Levinson explicitly identifies “cameras” among the sensor modalities whose sensor-generated data is processed, which corresponds to the data collection device comprising an image collection device.
and the first data comprises an image,
See at least: “…to facilitate generation of imagery of high fidelity…” (Col. 41, ll. 49-50)
Rationale: Levinson explicitly uses “imagery,” which corresponds to first data comprising an image.
of a first control device
See at least: “…teleoperator manager 3607 … between one of teleoperators 3608 and … autonomous vehicle 3630 …” (Col. 37, ll. 65-67); “…communication between teleoperator computing device 3604 and autonomous vehicle 3630.” (Col. 38, ll. 30-31)
Rationale: Levinson expressly identifies “teleoperator computing device 3604” as a control-side device within the teleoperation architecture, which corresponds to a first control device.
for controlling the one or more vehicles
See at least: “…an example of explicit guidance includes … [actions] … that may be transmitted … as instruction data 3617.” (Col. 38–39 excerpt)
Rationale: Levinson expressly discloses transmitting “instruction data” from the teleoperator manager architecture to guide the vehicle, which corresponds to the first control device for controlling the one or more vehicles.
Claim limitations Not Explicitly Disclosed by Levinson and Hamada
Levinson and Hamada do not explicitly disclose:
and wherein the determining that an abnormality occurs in the data collection device comprises: performing target detection on the image
to determine position coordinates
in the image;
and determining that a shooting angle of the image collection device is abnormal
in response to the position coordinates being not within a preset coordinate range.
Disclosure by Turk
Turk discloses:
and wherein the determining that an abnormality occurs in the data collection device comprises: performing target detection on the image
See at least: “Frames of image data … are processed to determine … of a detected object …” (Abstract)
Rationale: Turk expressly discloses that frames of image data are “processed” to determine information for a “detected object,” which corresponds to performing target detection on the image.
to determine position coordinates
See at least: “…based on positions of the detected object in frames of image data…” (Abstract)
Rationale: Turk expressly determines “positions” of the detected object in frames of image data; a PHOSITA would understand such “positions” in image data as position coordinates.
in the image;
See at least: “…positions of the detected object in frames of image data captured by the camera.” (Abstract)
Rationale: Turk expressly ties the “positions” to “frames of image data,” i.e., in the image.
and determining that a shooting angle of the image collection device is abnormal
See at least: “…determine … that the camera … is misaligned.” (Abstract)
Rationale: Turk expressly determines the camera is “misaligned,” which corresponds to determining that a shooting angle of the image collection device is abnormal.
in response to the position coordinates being not within a preset coordinate range.
See at least: “The determined object motion vector is compared to the determined vehicle motion vector to determine … that the camera … is misaligned.” (Abstract)
Rationale: Turk expressly determines misalignment “responsive to” a comparison outcome; a PHOSITA would recognize this as implementing a preset acceptance criterion (i.e., a preset coordinate range/allowed deviation) such that, when the detected-object position-derived quantities fall outside the preset allowable range, misalignment is determined.
Motivation to Combine Levinson, Hamada, and Turk
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Levinson, Hamada, and Turk before them, to incorporate Turk’s image-based target detection and camera misalignment determination into the Levinson-Hamada remote driving method to improve reliability and safety of camera-derived information used during remote control. Levinson teaches a teleoperation/remote driving architecture that processes camera-derived data/imagery in support of remotely controlling an autonomous vehicle, and Hamada teaches remote control system management in which device-provided signals/information are managed to avoid improper or conflicting operation and to maintain reliable control. Turk teaches processing frames of image data to detect an object, determine positions of the detected object, and determine that a camera is misaligned based on a comparison outcome derived from the image-based positions. A PHOSITA would have been motivated to integrate Turk’s misalignment logic into the Levinson-Hamada remote driving system so that abnormal shooting angle/misalignment of the image collection device can be detected using image-derived positional information, thereby yielding predictable results in which camera faults are identified and remote driving operation is made more robust and safe.
Response to Arguments
Applicant's arguments filed 11/11/2025 have been fully considered but they are not persuasive.
Summary of the 103 rejections and cited references
Applicant’s remarks are acknowledged. The current Final Rejection is maintained under 35 U.S.C. 103 on the following grounds:
Claims 1, 5, 7, 10, 14, 16, and 19 are rejected as obvious over Levinson (US 9,507,346 B1) in view of Hamada (US 6,107,937 A).
Claims 2, 3, 11, and 12 are rejected as obvious over Levinson in view of Hamada, and further in view of Katrak (US 2008/0065292 A1).
Claims 4, 9, 13, and 18 are rejected as obvious over Levinson in view of Hamada, and further in view of Shimida (US 2014/0129049 A1).
Claims 6 and 15 are rejected as obvious over Levinson in view of Hamada, and further in view of Turk (US 2021/0001774 A1).
Accordingly, there is no rejection under 35 U.S.C. 102 in the present Final Rejection.
Arguments directed to Kenner (and prior art of record) are moot
Applicant’s argument asserts that Kenner is “vehicle-side,” does not monitor the “tele-panel side,” and that the other previously-cited references also do not cure this alleged deficiency.
However, Kenner (and the prior art discussed in Applicant’s argument) is not relied upon in the present Final Rejection. The present §103 rejections are instead based on Levinson in view of Hamada, and, for certain dependent claims, further in view of Katrak, Shimida, and/or Turk, as summarized above.
Therefore, Applicant’s Kenner-based position does not address the actual teachings and rationales supporting the current Final §103 rejections and is not persuasive.
The present 103 rejection expressly teaches remote/system-side monitoring by a control host, abnormality detection, suspension, handling, and resumption
Applicant argues that claim 1 requires monitoring/diagnosing the remote driving system side (control host side) and suspending remote control until the abnormality is handled. The present rejection meets these limitations with Levinson + Hamada:
Control host on the remote driving system side; obtaining first state information.Levinson teaches obtaining information at the remote-side teleoperation platform, including: “RECEIVE MESSAGE DATA AT A TELEOPERATOR COMPUTING DEVICE…” and “message data 3619 may include any type of data, such as telemetry data or planner data….”
Determining whether an abnormality occurs in the remote driving system based on the first state information.
Levinson teaches abnormality detection at the remote teleoperation manager/platform: “teleoperator manager 3807 determines that an autonomous vehicle service platform fails to receive either sufficient data or a periodic signal (e.g., heartbeat signal)….”
This directly addresses Applicant’s assertion that the art is only “vehicle-side”; Levinson’s “teleoperator manager” performs the monitoring/determination at the remote/teleoperation side.
Determining abnormality information and a handling action; handling the abnormality; resuming remote control.
Levinson teaches abnormality-related information and actions, including: “event attributes associated with a non-normative state of operation,” retrieving recommendations, “recommended courses of action,” and resuming after recovery: “resume autonomous vehicle services upon completion of successful safe-stop recovery processes.”
Suspending the remote control.
Where Levinson does not explicitly recite “suspending,” Hamada explicitly teaches suspension: “The transmission of the new control command is suspended.”
Motivation to combine remains as set forth in the Final Rejection.
The rejection explains that it would have been obvious to incorporate Hamada’s “transmission … is suspended” technique into Levinson’s teleoperation flow when an abnormality is detected, thereby suspending commands during abnormality handling and then resuming after “successful … recovery processes,” yielding predictable safety/reliability results.
Accordingly, Applicant’s position that the prior art is “silent on monitoring the remote driving system side and adjusting remote control when an abnormality occurs” is not persuasive as applied to the current reference set, which expressly teaches remote/teleoperation-side abnormality detection, handling, suspension, and resumption.
Dependent-claim patentability arguments are not persuasive because the base claims remain rejected, and the added limitations are taught by the applied secondary references
Claims 2, 3, 11, and 12 (Levinson + Hamada + Katrak).
Applicant argues dependent claims are allowable “by virtue of their dependencies on allowable claim 1.” Because claim 1 remains rejected, that premise fails.
Further, the Final Rejection explains that the additional “synchronization”/out-of-sync abnormality limitations are taught by Katrak, including:
“checking a difference between a target road wheel angle and an actual road wheel angle,”
“compare the first value and the second value,” and
reverting upon exceeding a threshold: “revert to a mechanical mode if the comparison value exceeds the threshold.”
Thus, the dependent limitations do not render the claims patentable over the applied combination.
Claims 4, 9, 13, and 18 (Levinson + Hamada + Shimida).
The Final Rejection applies Shimida to teach the additional “threshold” comparison, e.g., “greater than or equal to a defined threshold….”
Claims 6 and 15 (Levinson + Hamada + Turk).
The Final Rejection applies Turk to address the additional limitations recited in those claims, and the rejection is maintained on that basis.
For the reasons above, Applicant’s arguments—directed to Kenner and other prior art not relied upon in the present Final Rejection—are moot, and Applicant has not persuasively traversed the current 103 rejections based on Levinson + Hamada (and further in view of Katrak, Shimida, and/or Turk, as applicable).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OLUWABUSAYO ADEBANJO AWORUNSE whose telephone number is (571)272-4311. The examiner can normally be reached M - F (8:30AM - 5PM).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jelani Smith can be reached at (571) 270-3969. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/OLUWABUSAYO ADEBANJO AWORUNSE/Examiner, Art Unit 3662
/JELANI A SMITH/Supervisory Patent Examiner, Art Unit 3662