DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 7 and 14 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim recites mental processes and are further reviewed below.
101 Analysis – Step 1
Claim 7 is directed to a method of initializing a connection from a system to a vehicle (process). Therefore, Claim 7 is within at least one of the four statutory categories.
Claim 14 is directed to an apparatus (medium). Therefore, Claim 14 is within at least one of the four statutory categories.
101 Analysis – Step 2A, Prong I
Regarding Prong I of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed
to determine whether they recite subject matter that falls within one of the following groups of
abstract ideas: a) mathematical concepts, b) certain methods of organizing human activity,
and/or c) mental processes.
Claims 7 and 14 include limitations that recite an abstract idea (emphasized below) and Claim 7 will be used as a representative claim for the remainder of the 101 rejections.
Claim 7 recites a method, comprising:
initializing a connection from a remote operations system and to an autonomous vehicle, for remote operator assistance;
receiving, from the autonomous vehicle, sensor data associated with the autonomous vehicle, the sensor data associated with a first time;
generating, based on the sensor data, a predicted image associated with a second time subsequent to the first time, the predicted image depicting a predicted view at the second time; and
displaying the predicted image via the remote operations system.
The examiner submits that the foregoing bolded limitation constitutes a “mental
process” because under its broadest reasonable interpretation, the claim covers performance
of the limitation in the human mind. Specifically, the “receiving” step encompasses a user to gather information (e.g. sensor data) associated with the autonomous vehicle to operate the remote operations system. Accordingly, the claim recites at least one abstract idea.
101 Analysis – Step 2A, Prong II
Regarding Prong II of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether the claim, as a whole, integrates the abstract into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are
as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”):
For the following reason(s), the examiner submits that the above identified additional limitations (underlined above) do not integrate the above-noted abstract idea into a practical application.
Regarding the additional limitation of “initializing a connection…,” “generating…a predicted image…,” and “displaying the predicted image…” the examiner submits that these limitations are insignificant extra-solution activities that merely use a computer (remote operations system) to perform the process. In particular, the initializing step from a remote operations system is recited at a high level of generality (i.e. to create a connection from the system to the vehicle for communication). The generating step is recited at a high level of generality (i.e. general means of using sensor data for use in the displaying step). The displaying step is also recited at a high level of generality (i.e. general means of displaying the predicted image result from the sensor data in the receiving and generating steps). Additionally, the system is claimed generically and is operating in its ordinary capacity and does not use the judicial
exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the exception. The additional limitations are no more than mere instructions to apply the exception using a system.
Thus, taken alone, the additional elements do not integrate the abstract idea into a practical application. Further, looking at the additional limitation(s) as an ordered combination or as a whole, the limitation(s) add nothing that is not already present when looking at the elements taken individually. For instance, there is no indication that the additional elements, when considered as a whole, reflect an improvement in the functioning of a computer or an improvement to another technology or technical field, apply or use the above-noted judicial exception to affect a particular treatment or prophylaxis for a disease or medical condition, implement/use the above-noted judicial exception with a particular machine or manufacture that is integral to the claim, effect a transformation or reduction of a particular article to a different state or thing, or apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological
environment, such that the claim as a whole is not more than a drafting effort designed to monopolize the exception (MPEP § 2106.05). Accordingly, the additional limitation(s) do/does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
101 Analysis – Step 2B
Regarding Step 2B of the 2019 PEG, claim 17 does not include additional elements (considered both individually and as an ordered combination) that are sufficient to amount to significantly more than the judicial exception for the same reasons to those discussed above with respect to determining that the claim does not integrate the abstract idea into a practical application. As discussed above with respect to integration of the abstract idea into a practical
application, the additional limitations of “initializing a connection…,” “generating…a predicted image…,” and “displaying the predicted image…” the examiner submits that these limitations amount to no more than what is well-understood, routine and conventional activity. Hence, claim 7 is not patent eligible.
Further, Claim 14 is not patent eligible for the same reasons.
Dependent Claims 10- and 19 when analyzed as a whole, are held to be patent ineligible under 35 U.S.C. 101 because the additional recited limitation(s) fail(s) to establish that the claim(s) is/are not directed to an abstract idea. The additional elements, if any, in the dependent claims are not sufficient to amount to significantly more than the judicial exception for the same reasons as with Claims 7 and 14.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 7-8, 13-15, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Magzimof (US 20200062267 A1) in view of Tiwari (US 20180267558 A1).
Regarding claim 1, Magzimof teaches a remote operations system comprising:
at least one processor (one or more processors; [0008]); and
at least one non-transitory memory having stored thereon processor-executable instructions that (a non-transitory computer-readable storage medium that stores instructions executable by the one or more processors; [0008]), when executed by the at least one processor, configure the remote operations system to:
receive, from an autonomous vehicle, a request for remote operator assistance (the vehicle 102 may request teleoperation assistance from the teleoperation support module 130; [0022]);
receive, from the autonomous vehicle, data associated with the autonomous vehicle, the data associated with the autonomous vehicle including one or more of an image associated with a first time or occupancy data associated with an environment of the autonomous vehicle associated with the first time (The vehicle 102 may also comprise various sensors… allowing real-time acquisition of data on the vehicle environment 100, vehicle 102 components and occupants, that capture image data and other environmental data…; [0019]);
display the predicted view via the remote operations system (the remote support terminal 110 may display an AR speed hump and monitor the deceleration pattern employed by the operator 202; [0067]);
receive an input from a remote operator (In the case of teleoperation, the vehicle 102 wirelessly receives control inputs via the one or more networks 140 that control various components of the drive system; [0019]).
However, Magzimof does not teach the remote operations system to
generate, by a machine-learned model and based on the data associated with the autonomous vehicle, a predicted image associated with a second time subsequent to the first time, the predicted image depicting a predicted view at the second time;
transmit, based at least in part on the input, guidance to the autonomous vehicle, the guidance configured to be used by the autonomous vehicle as part of controlling the autonomous vehicle.
Tiwari teaches the remote operations system to
generate, by a machine-learned model and based on the data associated with the autonomous vehicle, a predicted image associated with a second time subsequent to the first time, the predicted image depicting a predicted view at the second time (The training mode includes correlating and/or synchronizing range finding data with imaging data and vehicle control monitoring data to produce a mapping of human driver actions to range finding sensor input (e.g. range data), in order to train a machine learning model; [0049]);
transmit, based at least in part on the input, guidance to the autonomous vehicle, the guidance configured to be used by the autonomous vehicle as part of controlling the autonomous vehicle (teleoperation mode can include collecting sensor data onboard and transmitting sensor data and/or derived data to a remote operator… the teleoperation mode can include performing any other suitable actions related to remote operation of the vehicle; [0062]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof to incorporate the teachings of Tiwari in order for the remote operations system to project predictive image data to the user, notifying the user of traffic and the surrounding environment for safety, and to control the autonomous functions of the vehicle.
Regarding claim 2, Magzimof teaches the remote operations system of claim 1, but does not teach the remote operations system being further configured to:
determine the second time based at least in part on determining latency associated with at least one of receiving the data associated with the autonomous vehicle or displaying the data associated with the autonomous vehicle via the remote operations system, wherein generating the predicted image is further based at least in part on the second time.
Tiwari teaches to determine the second time based at least in part on determining latency associated with at least one of receiving the data associated with the autonomous vehicle or displaying the data associated with the autonomous vehicle via the remote operations system, wherein generating the predicted image is further based at least in part on the second time (The system 100 can also function to reduce network requirements (e.g., bandwidth, latency, quality of service, etc.) by performing sensor fusion at the vehicle. The system 100 can also function to collect sensor data to be used in autonomous driving models; [0015]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof to incorporate the teachings of Tiwari in order to determine the latency of data received by the vehicle in autonomous mode and be able to display real-time data to the user.
Regarding claim 7, Magzimof teaches a method, comprising:
initializing a connection from a remote operations system and to an autonomous vehicle, for remote operator assistance (the teleoperation support module 130 communicates with a vehicle 102 to provide teleoperation or other support services in instances when extra assistance is desired; [0022]).
receiving, from the autonomous vehicle, sensor data associated with the autonomous vehicle, the sensor data associated with a first time (The vehicle 102 may depend on a reliable network connection for streaming video or other sensor data to the remote support server 120 and for receiving control inputs or data used by the vehicle 102 to navigate in a safe and efficient manner; [0020]);
displaying the predicted image via the remote operations system (the remote support terminal 110 may display an AR speed hump and monitor the deceleration pattern employed by the operator 202; [0067]).
However, Magzimof does not teach generating, based on the sensor data, a predicted image associated with a second time subsequent to the first time, the predicted image depicting a predicted view at the second time; and displaying the predicted image via the remote operations system.
Tiwari teaches generating, based on the sensor data, a predicted image associated with a second time subsequent to the first time, the predicted image depicting a predicted view at the second time (The training mode includes correlating and/or synchronizing range finding data with imaging data and vehicle control monitoring data to produce a mapping of human driver actions to range finding sensor input (e.g. range data), in order to train a machine learning model; [0049]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof to incorporate the teachings of Tiwari in order for the remote operations system to project predictive image data to the user, notifying the user of traffic and the surrounding environment for safety, and to control the autonomous functions of the vehicle.
Regarding claim 8, Magzimof teaches the method of claim 7, but does not teach the method further comprising:
determining the second time based at least in part on determining latency associated with at least one of receiving the sensor data from the autonomous vehicle or displaying the sensor data via the remote operations system, wherein generating the predicted image is further based at least in part on the second time.
Tiwari teaches determining the second time based at least in part on determining latency associated with at least one of receiving the sensor data from the autonomous vehicle or displaying the sensor data via the remote operations system, wherein generating the predicted image is further based at least in part on the second time (The system 100 can also function to reduce network requirements (e.g., bandwidth, latency, quality of service, etc.) by performing sensor fusion at the vehicle. The system 100 can also function to collect sensor data to be used in autonomous driving models; [0015]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof to incorporate the teachings of Tiwari in order to determine the latency of data received by the vehicle in autonomous mode and be able to display real-time data to the user.
Regarding claim 13, Magzimof teaches the method of claim 7, the method further comprising:
receiving an input from a remote operator (In the case of teleoperation, the vehicle 102 wirelessly receives control inputs via the one or more networks 140 that control various components of the drive system; [0019]).
However, Magzimof does not teach the method comprising: transmit, based at least in part on the input, guidance to the autonomous vehicle, the guidance configured to be used by the autonomous vehicle as part of controlling the autonomous vehicle in an environment.
Tiwari teaches the method comprising: transmit, based at least in part on the input, guidance to the autonomous vehicle, the guidance configured to be used by the autonomous vehicle as part of controlling the autonomous vehicle in an environment (teleoperation mode can include collecting sensor data onboard and transmitting sensor data and/or derived data to a remote operator… the teleoperation mode can include performing any other suitable actions related to remote operation of the vehicle; [0062]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof to incorporate the teachings of Tiwari in order for the remote operations system to project predictive image data to the user, notifying the user of traffic and the surrounding environment for safety, and to control the autonomous functions of the vehicle.
Regarding claim 14, Magzimof teaches one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors (a non-transitory computer-readable storage medium that stores instructions executable by the one or more processors; [0008]), cause the one or more processors to perform actions comprising:
initializing a connection from a remote operations system and to an autonomous vehicle, for remote operator assistance (the teleoperation support module 130 communicates with a vehicle 102 to provide teleoperation or other support services in instances when extra assistance is desired; [0022]).;
receiving, from the autonomous vehicle, sensor data associated with the autonomous vehicle, the sensor data associated with a first time (The vehicle 102 may depend on a reliable network connection for streaming video or other sensor data to the remote support server 120 and for receiving control inputs or data used by the vehicle 102 to navigate in a safe and efficient manner; [0020]);
displaying the predicted image via the remote operations system (the remote support terminal 110 may display an AR speed hump and monitor the deceleration pattern employed by the operator 202; [0067]).
However, Magzimof does not teach generating, based on the sensor data, a predicted image associated with a second time subsequent to the first time, the predicted image depicting a predicted view at the second time;
Tiwari teaches generating, based on the sensor data, a predicted image associated with a second time subsequent to the first time, the predicted image depicting a predicted view at the second time (The training mode includes correlating and/or synchronizing rangefinding data with imaging data and vehicle control monitoring data to produce a mapping of human driver actions to rangefinding sensor input (e.g. range data), in order to train a machine learning model; [0049]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof to incorporate the teachings of Tiwari in order for the remote operations system to project predictive image data to the user, notifying the user of traffic and the surrounding environment for safety, and to control the autonomous functions of the vehicle.
Regarding claim 15, Magzimof teaches the one or more non-transitory computer-readable media of claim 14, but does not teach wherein:
determining the second time based at least in part on determining latency associated with at least one of receiving the sensor data from the autonomous vehicle or displaying the sensor data via the remote operations system, wherein generating the predicted image is further based at least in part on the second time.
Tiwari teaches determining the second time based at least in part on determining latency associated with at least one of receiving the sensor data from the autonomous vehicle or displaying the sensor data via the remote operations system, wherein generating the predicted image is further based at least in part on the second time (The system 100 can also function to reduce network requirements (e.g., bandwidth, latency, quality of service, etc.) by performing sensor fusion at the vehicle. The system 100 can also function to collect sensor data to be used in autonomous driving models; [0015]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof to incorporate the teachings of Tiwari in order to determine the latency of data received by the vehicle in autonomous mode and be able to display real-time data to the user.
Regarding claim 20, Magzimof teaches the one or more non-transitory computer-readable media of claim 14, the actions further comprising:
receiving an input from a remote operator (In the case of teleoperation, the vehicle 102 wirelessly receives control inputs via the one or more networks 140 that control various components of the drive system; [0019]).
However, Magzimof does not teach the remote operations system to transmit, based at least in part on the input, guidance to the autonomous vehicle, the guidance configured to be used by the autonomous vehicle as part of controlling the autonomous vehicle in an environment.
Tiwari teaches the remote operations system to transmit, based at least in part on the input, guidance to the autonomous vehicle, the guidance configured to be used by the autonomous vehicle as part of controlling the autonomous vehicle in an environment (The training mode includes correlating and/or synchronizing rangefinding data with imaging data and vehicle control monitoring data to produce a mapping of human driver actions to rangefinding sensor input (e.g. range data), in order to train a machine learning model; [0049]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof to incorporate the teachings of Tiwari in order for the remote operations system to project predictive image data to the user, notifying the user of traffic and the surrounding environment for safety, and to control the autonomous functions of the vehicle.
Claims 3, 9, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Magzimof (US 20200062267 A1) in view of Tiwari (US 20180267558 A1) in further view of Ogawa (US 11490007 B2).
Regarding claim 3, Magzimof teaches the remote operations system of claim 2, but does not teach the remote operations system being further configured to:
receive additional sensor data including a second image captured at a third time subsequent the first time and within a threshold difference of time from the second time;
determine a similarity of the second image to the predicted image; and
in response to the similarity being below a threshold, discontinuing display of predicted images via the remote operations system.
Tiwari teaches the remote operations system being further configured to:
receive additional sensor data including a second image captured at a third time subsequent the first time and within a threshold difference of time from the second time (A static portion of image data is preferably a first set of data points (e.g., pixels) in the image data wherein the value (e.g., intensity, hue, etc.) is substantially static (e.g., temporally, between image frames, within a threshold temporal variance, etc.);
determine a similarity of the second image to the predicted image (…compared to a second set of data points (e.g., pixels) in the image data; see [0072]);
Tiwari does not teach in response to the similarity being below a threshold, discontinuing display of predicted images via the remote operations system.
Ogawa teaches in response to the similarity being below a threshold, discontinuing display of predicted images via the remote operations system (when the importance level of each of all of the areas or a value obtained by adding together importance levels of the respective areas has become less than or equal to a predetermined threshold value, the automatic image capturing mode is cancelled; Col 30, lines 42-46).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof in view of Tiwari to incorporate the teachings of Ogawa in order for the remote operations system to adjust the predicted environmental data in comparison to real-time data and reduce the possibility of having divergencies in the data, enabling safer autonomous driving and vehicle navigation.
Regarding claim 9, Magzimof teaches the method of claim 8, but does not teach the method further comprising:
receiving additional sensor data including a second image captured at a third time subsequent the first time and within a threshold difference of time from the second time;
determining a similarity of the second image to the predicted image; and
in response to the similarity being below a threshold, discontinuing display of predicted images via the remote operations system.
Tiwari teaches
receiving additional sensor data including a second image captured at a third time subsequent the first time and within a threshold difference of time from the second time (The system 100 can collect and/or generate data that can be streamed off the vehicle (e.g., via a wireless communication link), stored in an onboard memory subsystem, processed onboard and then streamed and/or stored asynchronously (e.g., at a time point later than a collection time point or time period), and/or manipulated in any other suitable manner; [0020]).
determining a similarity of the second image to the predicted image (…compared to a second set of data points (e.g., pixels) in the image data; see [0072]).
Whereas Ogawa teaches
in response to the similarity being below a threshold, discontinuing display of predicted images via the remote operations system (when the importance level of each of all of the areas or a value obtained by adding together importance levels of the respective areas has become less than or equal to a predetermined threshold value, the automatic image capturing mode is cancelled; Col 30, lines 42-46).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof in view of Tiwari to incorporate the teachings of Ogawa in order for the remote operations system to adjust the predicted environmental data in comparison to real-time data and reduce the possibility of having divergencies in the data, enabling safer autonomous driving and vehicle navigation.
Regarding claim 16, Magzimof teaches the one or more non-transitory computer-readable media of claim 15, but does not teach the actions further comprising:
receiving additional sensor data including a second image captured at a third time subsequent the first time and within a threshold difference of time from the second time;
determining a similarity of the second image to the predicted image; and
in response to the similarity being below a threshold, discontinuing display of predicted images via the remote operations system.
Tiwari teaches
receiving additional sensor data including a second image captured at a third time subsequent the first time and within a threshold difference of time from the second time (A static portion of image data is preferably a first set of data points (e.g., pixels) in the image data wherein the value (e.g., intensity, hue, etc.) is substantially static (e.g., temporally, between image frames, within a threshold temporal variance, etc.);
determining a similarity of the second image to the predicted image (…compared to a second set of data points (e.g., pixels) in the image data; see [0072]).
Tiwari does not teach in response to the similarity being below a threshold, discontinuing display of predicted images via the remote operations system.
Ogawa teaches
in response to the similarity being below a threshold, discontinuing display of predicted images via the remote operations system (when the importance level of each of all of the areas or a value obtained by adding together importance levels of the respective areas has become less than or equal to a predetermined threshold value, the automatic image capturing mode is cancelled; Col 30, lines 42-46).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof in view of Tiwari to incorporate the teachings of Ogawa in order for the remote operations system to adjust the predicted environmental data in comparison to real-time data and reduce the possibility of having divergencies in the data, enabling safer autonomous driving and vehicle navigation.
Claims 4-6, 10-12, and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Magzimof (US 20200062267 A1) in view of Tiwari (US 20180267558 A1) in further view of Huang (US 20190302761 A1).
Regarding claim 4, Magzimof in view of Tiwari teaches the remote operations system of claim 1, but both do not teach wherein generating the predicted image based on the image comprises:
generating, by a diffusion model and based at least in part on the image, latent variable data, wherein the latent variable data is associated with the second time; and generating, by a decoder and based at least in part on the latent variable data, the predicted image.
Huang teaches generating, by a diffusion model (machine learning model(s) 504; [0137] and Fig. 5A) and based at least in part on the image, latent variable data, wherein the latent variable data is associated with the second time; and generating, by a decoder and based at least in part on the latent variable data, the predicted image (once received by the remote control system 106, the sensor data (e.g., encoded sensor data) may be decoded by decoder 142 of the remote control system 106…The sensor data may include image data from camera(s), LIDAR data from LIDAR sensor(s), RADAR data from RADAR sensor(s), and/or other data types from other sensor(s) 110; [0087]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof and Tiwari to incorporate the teachings of Huang in order for the predicted views output by the diffusion model and decoder be considered by the remote operator to improve vehicle safety by controlling the vehicle based on information that is closer to the real-time state of the vehicle.
Regarding claim 5, Magzimof in view of Tiwari teaches the remote operations system of claim 4, but do not teach wherein generating the predicted image is based on map data and an object trajectory of the occupancy data associated with an object in the environment associated with the autonomous vehicle.
Huang teaches wherein generating the predicted image is based on map data (provide mapping outputs (e.g., map data) that may be localized by the localization manager based on a particular location of the vehicle 102; [0067]) and an object trajectory of the occupancy data associated with an object in the environment associated with the autonomous vehicle (location of other vehicles (e.g., an occupancy grid), information about objects and status of objects as perceived by the controller(s) 736; [0169]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof and Tiwari to incorporate the teachings of Huang in order to assist in generating the predicted view for the remote operation system as to improve safety in the autonomous vehicle.
Regarding claim 6, Magzimof in view of Tiwari teaches the remote operations system of claim 4, wherein the diffusion model is configured to perform a denoising algorithm based at least in part on the image to generate the latent variable data (Block S220 can also function to transform the surroundings data to make the data suited for downstream processing (e.g., compress the data, remove unnecessary portions of the data, denoise the data, etc.); Tiwari [0070]).
However, Magzimof in view of Tiwari do not teach the diffusion model itself. Huang teaches the diffusion model (machine learning model(s) 504; [0137] and Fig. 5A).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof and Tiwari to incorporate the teachings of Huang in order to help generate the latent variable data and produce quality video streaming of the surrounding environment on the display when driving.
Regarding claim 10, Magzimof in view of Tiwari teaches the method of claim 7, but does not teach wherein generating the predicted image is based on map data and occupancy data associated with an object in an environment associated with the autonomous vehicle.
Huang teaches wherein generating the predicted image is based on map data (provide mapping outputs (e.g., map data) that may be localized by the localization manager based on a particular location of the vehicle 102; [0067]) and occupancy data associated with an object in an environment associated with the autonomous vehicle (location of other vehicles (e.g., an occupancy grid), information about objects and status of objects as perceived by the controller(s) 736; [0169]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof and Tiwari to incorporate the teachings of Huang in order to assist in generating the predicted view for the remote operation system as to improve safety in the autonomous vehicle.
Regarding claim 11, Magzimof in view of Tiwari teaches the method of claim 7, but do not teach wherein generating the predicted image based on the sensor data comprises:
generating, by a diffusion model and based at least in part on an image included in the sensor data, latent variable data, wherein the latent variable data is associated with the second time; and
generating, by a decoder and based at least in part on the latent variable data, the predicted image.
Huang teaches generating, by a diffusion model (machine learning model(s) 504; [0137] and Fig. 5A) and based at least in part on an image included in the sensor data, latent variable data, wherein the latent variable data is associated with the second time; and generating, by a decoder and based at least in part on the latent variable data, the predicted image (once received by the remote control system 106, the sensor data (e.g., encoded sensor data) may be decoded by decoder 142 of the remote control system 106…The sensor data may include image data from camera(s), LIDAR data from LIDAR sensor(s), RADAR data from RADAR sensor(s), and/or other data types from other sensor(s) 110; [0087]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof and Tiwari to incorporate the teachings of Huang in order for the predicted views output by the diffusion model and decoder be considered by the remote operator to improve vehicle safety by controlling the vehicle based on information that is closer to the real-time state of the vehicle.
Regarding claim 12, Magzimof in view of Tiwari teaches the method of claim 11, wherein the diffusion model is configured to perform a denoising algorithm based at least in part on the image to generate the latent variable data (Block S220 can also function to transform the surroundings data to make the data suited for downstream processing (e.g., compress the data, remove unnecessary portions of the data, denoise the data, etc.); Tiwari [0070]).
However, Magzimof in view of Tiwari do not teach the diffusion model itself. Huang teaches wherein the diffusion model (machine learning model(s) 504; [0137] and Fig. 5A).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof and Tiwari to incorporate the teachings of Huang in order to help generate the latent variable data and produce quality video streaming of the surrounding environment on the display when driving.
Regarding claim 17, Magzimof in view of Tiwari teaches the one or more non-transitory computer-readable media of claim 14, but do not teach wherein generating the predicted image based on the sensor data comprises:
generating, by a diffusion model and based at least in part on an image included in the sensor data, latent variable data, wherein the latent variable data is associated with the second time; and
generating, by a decoder and based at least in part on the latent variable data, the predicted image.
Huang teaches generating, by a diffusion model (machine learning model(s) 504; [0137] and Fig. 5A) and based at least in part on an image included in the sensor data, latent variable data, wherein the latent variable data is associated with the second time; and
generating, by a decoder and based at least in part on the latent variable data, the predicted image (once received by the remote control system 106, the sensor data (e.g., encoded sensor data) may be decoded by decoder 142 of the remote control system 106…The sensor data may include image data from camera(s), LIDAR data from LIDAR sensor(s), RADAR data from RADAR sensor(s), and/or other data types from other sensor(s) 110; [0087]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof and Tiwari to incorporate the teachings of Huang in order for the predicted views output by the diffusion model and decoder be considered by the remote operator to improve vehicle safety by controlling the vehicle based on information that is closer to the real-time state of the vehicle.
Regarding claim 18, Magzimof in view of Tiwari teaches the one or more non-transitory computer-readable media of claim 17,
wherein the diffusion model is configured to perform a denoising algorithm based at least in part on the image to generate the latent variable data ((Block S220 can also function to transform the surroundings data to make the data suited for downstream processing (e.g., compress the data, remove unnecessary portions of the data, denoise the data, etc.); Tiwari [0070]).
However, Magzimof in view of Tiwari do not teach the diffusion model itself. Huang teaches the diffusion model (machine learning model(s) 504; [0137] and Fig. 5A).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof and Tiwari to incorporate the teachings of Huang in order to help generate the latent variable data and produce quality video streaming of the surrounding environment on the display when driving.
Regarding claim 19, Magzimof in view of Tiwari teaches the one or more non-transitory computer-readable media of claim 14, but do not teach
wherein the predicted image is based on map data and occupancy data associated with an object in an environment associated with the autonomous vehicle.
Huang wherein the predicted image is based on map data provide mapping outputs (e.g., map data) that may be localized by the localization manager based on a particular location of the vehicle 102; [0067]) and occupancy data associated with an object in an environment associated with the autonomous vehicle (location of other vehicles (e.g., an occupancy grid), information about objects and status of objects as perceived by the controller(s) 736; [0169]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Magzimof and Tiwari to incorporate the teachings of Huang in order to assist in generating the predicted view for the remote operation system as to improve safety in the autonomous vehicle.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KENDRICK K LY whose telephone number is (571)272-5831. The examiner can normally be reached M-F 9:00-16:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Helal Algahaim can be reached at (571) 270-5227. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KENDRICK K LY/Examiner, Art Unit 3666
/HELAL A ALGAHAIM/SPE , Art Unit 3666
Prosecution Insights