DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1-15 of U.S. Application No. 18/731,115 filed on 05/31/2024 have been examined.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-15 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al.(US10442443B1)in view of Carion et al.(End-to-End Object Detection with Transformers_2020), and in further view of Kocic et al. (An End-to-End Deep Neural Network for Autonomous Driving Designed for Embedded Automotive Platforms_2019), hereinafter referred to as Li, Carion and Kocic respectively.
Regarding claim 1, Li discloses computer-implemented method of providing driver assistance alerts to a driver (“According to one or more embodiments, a computer program product includes a computer readable storage device comprising one or more computer executable instructions, which, when executed by a controller cause the controller to provide driver performance feedback for a driver of a vehicle.” [Col.3 ln 12-26]), the method including:
receiving environmental data for a sequence of driving states including at least video from a camera, returns from an optical sensor, and location data from a GNSS receiver, wherein the camera, the optical sensor, and the GNSS receiver are coupled to a processor carried by a vehicle (“The automated driving system may use on-board sensors, cameras, a global positioning system (GPS), and telecommunications to obtain environment information in order to make judgments regarding safety-critical situations and operate/warn appropriately by effectuating control at some automation level.” [Col.4 ln 48-53]);
wherein the end-to-end neural network is trained to generate prescriptive steering and speed control actions in response to a present driving state (“In other words, differences between the two predicted sets of metrics are computed as the deviation metrics. For example, the vehicle speeds, vehicle acceleration, and vehicle steering direction values in the two maneuver controls are compared and respective differences are computed. Further, a driver notification is generated corresponding to the computed deviation metrics, at 268 (see FIG. 3). The notification can be a combination of visual, audio, and haptic feedback provided to the driver.” [Col.8 ln 8-20] see also (Col. 1 ln 42-62 & Col.3 ln 27-41]);
analyzing hidden layer data and output data from the end-to-end neural network to estimate collision avoidance data (“Based on the predictions for the vehicle operation using the driver input, the automated driver system 122 determines if the driver input creates a hazardous situation, at 264. For example, the automated driver system 122 determines if a hazardous situation is created based on a predicted time of collision being below a predetermined threshold.” [Col.7 ln 50-67]),
wherein the collision avoidance data includes, at least: one or more detected objects within the video from the camera(“The automated driving system may use on-board sensors, cameras,” [Col.4 ln 48-50] and “The sensor measurements can include measurements from one or more on-board and/or off-board measurement systems. For example, measurements may be received via the sensors 124, and or via a vehicle-to-vehicle communication (V2X) system. The measurements may be received in a wired/wireless manner. The measurements can include a location of the vehicle 120, a distance from an object (including another vehicle) within a predetermined vicinity, an estimated time of collision with the object, a vehicle speed, a vehicle steering direction, a vehicle acceleration/deceleration, and any other such sensor measurements.” [Col.6 ln 48-58]),
a directional cue, wherein the directional cue is a projection overlay based on the prescriptive steering control actions onto a heads up display (“The notification includes one or more visual notifications depicted in FIG. 4A. Consider the case where the suggested vehicle speed, i.e. first vehicle speed 405 from the first control maneuver from the automated driving system 122 is 0 MPH. Accordingly, the automated driving system 122 computes a first glide slope 410 that represents the first deceleration from the first maneuver control for the vehicle 120. The first deceleration is determined based on the current vehicle speed to reach the first vehicle speed 405 given the predicted time of collision with a remote object or a location to stop because of a stop sign, traffic light, etc. The DVI 180 displays the computed glide slope 410.” [Col.9 ln 16-38]),
and a risk metric that quantifies a dissimilarity between the generated prescriptive steering and speed control actions, and received driver steering and speed control actions (“In one or more examples, the audio/visual notification can vary based on a magnitude of the difference computed between the parameters from the first maneuver control and the second maneuver control. For example, the color of the second glide slope 420 and/or the curve 425 representing the variance between the first glide slope 410 and the second glide slope 420 can change based on the difference between the two glide slopes (410, 420). For example, the color may be green (or any other first color) if the difference is below a predetermined threshold and red (or any other second color) if the difference exceeds the predetermined threshold.” [Col. 9 ln 55-65]);
and presenting, to the driver, a user interface including driver assistance alerts based on the collision avoidance data (“In one or more examples, the audio/visual notification can vary based on a magnitude of the difference computed between the parameters from the first maneuver control and the second maneuver control. For example, the color of the second glide slope 420 and/or the curve 425 representing the variance between the first glide slope 410 and the second glide slope 420 can change based on the difference between the two glide slopes (410, 420). For example, the color may be green (or any other first color) if the difference is below a predetermined threshold and red (or any other second color) if the difference exceeds the predetermined threshold.” [Col. 9 ln 55-65]).
Li does not explicitly teach processing the environmental data as input to a transformer-based end-to-end neural network.
However, Carion does teach processing the environmental data as input to a transformer-based end-to-end neural network (“Fig. 1. DETR directly predicts (in parallel) the final set of detections by combining a common CNN with a transformer architecture. During training, bipartite matching uniquely assigns predictions with ground truth boxes.” [Pg.2 Par.1] and “Our DEtection TRansformer (DETR, see Fig. 1) predicts all objects at once, and is trained end-to-end with a set loss function which performs bipartite matching between predicted and ground-truth objects. DETR simplifies the detection pipeline by dropping multiple hand-designed components that encode prior knowledge, like spatial anchors or non-maximal suppression.” [Pg.2 par.4]). Both Li and Carion teach methods of providing driver assistance alerts to a driver. However, Carion explicitly teaches processing the environmental data as input to a transformer-based end-to-end neural network.
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the driver assistance method of Li to also include processing the environmental data as input to a transformer-based end-to-end neural network, as taught by Carion, with a reasonable expectation of success. Doing so improves safety of autonomous vehicle operation (With regard to this reasoning, see at least [Carion, Pg.2]).
Li does not explicitly teach extracted from attention weights of the hidden layer data to generate an attention map
However, Kocic does teach extracted from attention weights of the hidden layer data to generate an attention map (“Introducing more hidden layers to a deep neural network helps with parameters efficiency. It is likely to get much more performance with pure parameters by going deeper rather than wider. In addition to this, deep neural networks applied to images are very efficient, since images tend to have a hierarchical structure that deep models naturally capture. The lower layers of deep neural networks capture simple features like lines or edges. Further layers extract more complicated features like geometric shapes, and the last layers are extracting objects. Since the aim of our work was to drive a vehicle in a representative track, the features needed to be extracted were not objects, rather they were the simple features or geometric shapes. For that reason, for our final model, we have chosen three convolutional layers followed with one flattened layer and two fully connected layers, as will be discussed in more detail in the further text.” [Pg.12 par.2-3). Both Li and Kocic teach methods of providing driver assistance alerts to a driver. However, Kocic explicitly teaches extracted from attention weights of the hidden layer data to generate an attention map.
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the driver assistance method of Li to also include extracted from attention weights of the hidden layer data to generate an attention map, as taught by Kocic, with a reasonable expectation of success. Doing so improves safety of autonomous vehicle operation (With regard to this reasoning, see at least [Kocic, Pg.12]).
Regarding claim 2, Li discloses The computer-implemented method of claim 1, wherein the directional cue projected onto the heads up display within the user interface is a dynamic whisker arrow indicating a prescriptive vehicle orientation relative to a current vehicle orientation based on the generated prescriptive steering control actions (“One or a combination of the following could be used to indicate various types of driver performance issues: steering wheel (LEDs, haptics, resistance); haptic seat (direction, intensity, pattern); haptic pedals (buzz, short pulses); visual cues (HUD, center stack, etc.); and audio.” [Col.12 ln 4-16] and “In one or more examples, the LEDs 460 in the steering wheel 530 form a 360 degree display to facilitate the steering wheel 530 to be used as an awareness device. In one or more examples, the steering wheel 530 has any other type of 360 degree display embedded that facilitates the steering wheel 530 to be used as an awareness device to provide the one or more visual notifications.” [Col.10 ln 34-41].
Regarding claim 3, Li discloses The computer-implemented method of claim 1, wherein obtaining the risk metric further includes:
calculating a cross entropy between the generated prescriptive steering and speed control actions and current driver steering and speed control actions and standardizing the cross entropy calculation to generate a risk metric output (“The automated driving system 122 using the technical solutions described herein (FIG. 2) determines whether the driver is following a velocity glide slope to safely and efficiently stop at the light in time. The reference velocity glide slope is calculated by the automated driving system (first maneuver control). The actual input from the driver is compared with the reference glide slope and the difference, if any, is notified to the driver.” [Col.9 ln 3-15]),
wherein the risk metric output is a proxy for an imminent collision risk and increases proportionally as the current driver steering and speed control actions deviate further from the generated prescriptive steering and speed control actions (“The notification includes one or more visual notifications depicted in FIG. 4A. Consider the case where the suggested vehicle speed, i.e. first vehicle speed 405 from the first control maneuver from the automated driving system 122 is 0 MPH. Accordingly, the automated driving system 122 computes a first glide slope 410 that represents the first deceleration from the first maneuver control for the vehicle 120. The first deceleration is determined based on the current vehicle speed to reach the first vehicle speed 405 given the predicted time of collision with a remote object or a location to stop because of a stop sign, traffic light, etc.” [Col.9 ln 17-28]).
Regarding claim 4, Li discloses The computer-implemented method of claim 3, further including, in response to the risk metric output:
maintaining a manual driving mode while the risk metric output is less than a pre-determined threshold value, wherein the manual driving mode includes permitting the vehicle to apply the current driver steering and speed control input actions (“Further, the controller receives a second maneuver control from the driver. The controller, in response to the first maneuver control being different from the second maneuver control, generates a driver notification that is indicative of the first maneuver control from the automated driving system and operates the vehicle using the second maneuver control.” [Col.2 ln 33-39]),
and engaging in an autonomous driving mode when the risk metric output is equal to or greater than the pre-determined threshold value, wherein the autonomous driving mode includes causing the vehicle to apply the prescriptive steering and speed control input actions (“automated driving systems provide at least some aspects of a safety-critical control function (e.g., steering, throttle, or braking occur without direct driver input). In one or more examples, the automated driving system provides safety warnings to drivers but does not perform a control function. The automated driving system may use on-board sensors, cameras, a global positioning system (GPS), and telecommunications to obtain environment information in order to make judgments regarding safety-critical situations and operate/warn appropriately by effectuating control at some automation level.” [Col.4 ln 43-54]).
Regarding claim 5, Li discloses The computer-implemented method of claim 3, further including categorizing the risk metric into risk levels by defining a particular risk level as including risk metric outputs within a pre-determined range between a lower boundary value and an upper boundary value (“In one or more examples, the visual notifications described herein are accompanied with audio notifications via the speakers of the DVI 180. For example, if the speed limit is exceeded by a predetermined threshold, the DVI 180 can provide an audible warning such as a beep, a spoken warning, and the like. The presented auditory cues could also be adapted in terms of frequency, duty cycle or intensity to indicate the detected deviation level.” [Col.9 ln 46-54]).
Regarding claim 6, Li discloses The computer-implemented method of claim 5, wherein the user interface presents, to the driver, a quantitative risk including the risk metric output or a categorical risk including the risk level based on the risk metric value (“FIG. 4B depicts an example notification provided to a driver according to one or more embodiments. The notification includes one or more visual notifications depicted in FIG. 4B. The visual notification, alternatively or in addition to the display notifications via the DVI 180, can further include using one or more light emitting diodes (LEDs) 460 embedded in the steering wheel 530.” [Col.9 ln 66-67 & Col.10 ln 1-5]).
Regarding claim 7, Li discloses The computer-implemented method of claim 1, wherein the driver assistance alerts include one or more of a visual display, an audio signal, or a haptic signal (“The notification can be a combination of visual, audio, and haptic feedback provided to the driver.” [Col.8 ln 19-20]).
Regarding claim 8, Li discloses The computer-implemented method of claim 1, wherein the user interface further presents the generated prescriptive steering and speed control actions to the driver (“For example, if the speed limit is exceeded by a predetermined threshold, the DVI 180 can provide an audible warning such as a beep, a spoken warning, and the like. The presented auditory cues could also be adapted in terms of frequency, duty cycle or intensity to indicate the detected deviation level.” [Col.9 ln 46-54]).
Regarding claim 9, Li discloses The computer-implemented method of claim 1,
Li does not explicitly teach wherein the environmental data is pre-processed prior to being provided to the end-to-end neural network, the pre- processing further including: tokenizing the environmental data for the present driving state to generate environmental data tokens, mapping the environmental data tokens to a reduced dimensional vector space to produce environmental data embeddings, and adding the environmental data embeddings to positional embeddings to generate input embeddings for the end-to-end neural network, wherein the positional embeddings preserve spatial information for the environmental data tokens.
However, Carion does teach wherein the environmental data is pre-processed prior to being provided to the end-to-end neural network, the pre- processing further including: tokenizing the environmental data for the present driving state to generate environmental data tokens, mapping the environmental data tokens to a reduced dimensional vector space to produce environmental data embeddings, and adding the environmental data embeddings to positional embeddings to generate input embeddings for the end-to-end neural network, wherein the positional embeddings preserve spatial information for the environmental data tokens (“Transformers were first used in auto-regressive models, following early sequence-to-sequence models [43], generating output tokens one by one. However, the prohibitive inference cost (proportional to output length, and hard to batch) lead to the development of parallel sequence generation, in the domains of audio [28], machine translation [9,11], word representation learning [7], and more recently speech recognition [6]. We also combine transformers and parallel decoding for their suitable trade-off between computational cost and the ability to perform the global computations required for set prediction.” [Pg.4 Par.2]). Both Li and Carion teach methods of providing driver assistance alerts to a driver. However, Carion explicitly teaches tokenizing the environmental data for the present driving state to generate environmental data token.
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the driver assistance method of Li to also include tokenizing the environmental data for the present driving state to generate environmental data token, as taught by Carion, with a reasonable expectation of success. Doing so improves safety of autonomous vehicle operation (With regard to this reasoning, see at least [Carion, Pg.4]).
Regarding claim 10, Li discloses The computer-implemented method of claim 9,
Li does not explicitly teach wherein the transformer-based end-to-end neural network is a transformer model trained for end-to-end autonomous driving.
However, Carion does teach wherein the transformer-based end-to-end neural network is a transformer model trained for end-to-end autonomous driving “Our DEtection TRansformer (DETR, see Fig. 1) predicts all objects at once, and is trained end-to-end with a set loss function which performs bipartite matching between predicted and ground-truth objects. DETR simplifies the detection pipeline by dropping multiple hand-designed components that encode prior knowledge, like spatial anchors or non-maximal suppression. Unlike most existing detection methods, DETR doesn't require any customized layers, and thus can be reproduced easily in any framework that contains standard ResNet [14] and Transformer [16] classes.” [Pg.2 Par.3], Both Li and Carion teach methods of providing driver assistance alerts to a driver. However, Carion explicitly teaches wherein the transformer-based end-to-end neural network is a transformer model trained for end-to-end autonomous driving.
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the driver assistance method of Li to also include wherein the transformer-based end-to-end neural network is a transformer model trained for end-to-end autonomous driving, as taught by Carion, with a reasonable expectation of success. Doing so improves safety of autonomous vehicle operation (With regard to this reasoning, see at least [Carion, Pg.2]).
Li does not explicitly teach and the transformer model processing the environmental data further includes: processing the generated input embeddings combined with compressed embeddings from nine or more earlier driving states over at least three seconds and generating, as output, a compressed embedding for the present driving state and prescriptive steering and speed control actions in response to the present driving state
However, Kocic does teach and the transformer model processing the environmental data further includes: processing the generated input embeddings combined with compressed embeddings from nine or more earlier driving states over at least three seconds and generating, as output, a compressed embedding for the present driving state and prescriptive steering and speed control actions in response to the present driving state (“For training the deep neural network, images acquired from all three cameras were used--central, left, and right; Figure 9. Using three cameras for gathering data for training deep neural network for autonomous drive was inspired by [1]. All of these images captured a similar scene but from slightly different positions. Advantages of using three cameras instead of one central camera are three times more data, and better performance for steering back to the center scenario when the vehicle starts drifting off to the side. The steering angle was paired with three images from a particular frame corresponding to the central camera, and the images from the left and right cameras had a field of view shifted on the left or right side of the road, respectively, which means that the steering angle for the left and right images is not correct. To overcome this, the correction factor for the steering angle measurement had been added or subtracted from the left and right images, respectively. The correction factor was one of the hyperparameters to be fine-tuned during the training process.” [Pg.9 Par.5]). Both Li and Kocic teach methods of providing driver assistance alerts to a driver. However, Kocic explicitly teaches generating a compressed embedding for the present driving state and prescriptive steering and speed control actions in response to the present driving state.
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the driver assistance method of Li to also include generating a compressed embedding for the present driving state and prescriptive steering and speed control actions in response to the present driving state, as taught by Kocic, with a reasonable expectation of success. Doing so improves safety of autonomous vehicle operation (With regard to this reasoning, see at least [Kocic, Pg.9]).
Regarding claim 11, Li discloses The computer-implemented method of claim 10,
Li does not explicitly teach further including extracting a set of attention weights from the transformer model, and generating, using the positional embeddings, an attention map including a projection of the extracted attention weights
However, Kocic does teach further including extracting a set of attention weights from the transformer model, and generating, using the positional embeddings, an attention map including a projection of the extracted attention weights (“Introducing more hidden layers to a deep neural network helps with parameters efficiency. It is likely to get much more performance with pure parameters by going deeper rather than wider. In addition to this, deep neural networks applied to images are very efficient, since images tend to have a hierarchical structure that deep models naturally capture. The lower layers of deep neural networks capture simple features like lines or edges. Further layers extract more complicated features like geometric shapes, and the last layers are extracting objects. Since the aim of our work was to drive a vehicle in a representative track, the features needed to be extracted were not objects, rather they were the simple features or geometric shapes. For that reason, for our final model, we have chosen three convolutional layers followed with one flattened layer and two fully connected layers, as will be discussed in more detail in the further text.” [Pg.12 par.2-3),
wherein:
a magnitude of a particular attention weight increases proportionally to an importance of the particular attention weight in generating the prescriptive steering and speed actions, and an object is implicitly detected within a region of an area of real space surrounding the vehicle based on (i) a comparison of an average attention weight value within the region and another average attention weight value within one or more adjacent regions and (ii) the positional embeddings (“The core gradient of deep neural networks are convolutional networks [12] Convolutional neural networks (CNN) are a specialized kind of neural network for processing data that has a known grid-like topology. CNNs combine three architectural ideas: local representative fields, shared weights, and spatial or temporal sub-sampling, which leads to some degree of shift, scale, and distortion invariance. Convolutional neural networks are designed to process data with multiple arrays (e.g., color image, language, audio spectrogram, and video), and benefit from the properties of such signals: local connections, shared weights, pooling, and the use of many layers. For that reason, CNNs are most commonly applied to analyzing visual imagery.” [Pg.3 Par.4]). Both Li and Kocic teach methods of providing driver assistance alerts to a driver. However, Kocic explicitly teaches extracting a set of attention weights from the transformer model, and generating, using the positional embeddings, an attention map including a projection of the extracted attention weights, wherein: a magnitude of a particular attention weight increases proportionally to an importance of the particular attention weight in generating the prescriptive steering and speed actions, and an object is implicitly detected within a region of an area of real space surrounding the vehicle based on (i) a comparison of an average attention weight value within the region and another average attention weight value within one or more adjacent regions and (ii) the positional embeddings.
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the driver assistance method of Li to also include extracting a set of attention weights from the transformer model, and generating, using the positional embeddings, an attention map including a projection of the extracted attention weights, wherein: a magnitude of a particular attention weight increases proportionally to an importance of the particular attention weight in generating the prescriptive steering and speed actions, and an object is implicitly detected within a region of an area of real space surrounding the vehicle based on (i) a comparison of an average attention weight value within the region and another average attention weight value within one or more adjacent regions and (ii) the positional embeddings, as taught by Kocic, with a reasonable expectation of success. Doing so improves safety of autonomous vehicle operation (With regard to this reasoning, see at least [Kocic, Pg.3]).
Regarding claim 12, Li discloses The computer-implemented method of claim 11, wherein presenting, via the user interface, the one or more detected objects within the heads up display further includes color-coding attention weights within the attention map enabling visual identification of implicitly detected objects and projecting an overlay of the color-coded attention map onto a heads up display (“The DVI 180 displays the computed glide slope 410. Further, based on the driver input, a second glide slope 420 is computed using the second deceleration from the second maneuver control. The second glide slope 420 can be above or below the first glide slope 410 based on driver input to the brake pedal. In one or more examples, the DVI 180 displays the second glide slope 420 along with a curve 425 representing a difference in the suggested glide slope 410 from the automated driving system and the actual glide slope 420 based on the driver's input.” [Col.9 ln 28-38]).
Regarding claim 13, Li discloses The computer-implemented method of claim 1, further including storing a history of the video from the camera and the driver assistance alerts presented to the driver within a driving database, wherein the driving database is available for additional data analysis and data auditing after a driving activity is completed (“he method may be executed by the control system 126, in one or more examples. In one or more examples, the method may be executed by executing one or more computer executable instructions that are stored on a computer readable storage device. The method includes receiving one or more sensor measurements, at 210. The sensor measurements can include measurements from one or more on-board and/or off-board measurement systems.” See at least [Col.6 ln 40-60]).
Regarding claim 14, Li discloses A computer-implemented method of training a neural network to generate driver assistance alert data, the method including:
receiving environmental data for a sequence of driving states resulting from human driving, including at least video from a camera, returns from an optical sensor, and location data from a GNSS receiver, wherein the camera, the optical sensor, and the GNSS receiver are coupled to a processor carried by a vehicle (“The automated driving system may use on-board sensors, cameras, a global positioning system (GPS), and telecommunications to obtain environment information in order to make judgments regarding safety-critical situations and operate/warn appropriately by effectuating control at some automation level.” [Col.4 ln 48-53]);
Li does not explicitly teach processing the environmental data as input to imitation training of a transformer-based end-to-end neural network, including training the end-to-end neural network to generate prescriptive steering and speed control actions whereby attention weights of the trained, transformer-based end-to-end neural network are available to be extracted from hidden layer data and indicate areas of the video that contribute most significantly to the generated prescriptive steering and speed control actions
However, Kocic does teach processing the environmental data as input to imitation training of a transformer-based end-to-end neural network, including training the end-to-end neural network to generate prescriptive steering and speed control actions whereby attention weights of the trained, transformer-based end-to-end neural network are available to be extracted from hidden layer data and indicate areas of the video that contribute most significantly to the generated prescriptive steering and speed control actions (“For training the deep neural network, images acquired from all three cameras were used--central, left, and right; Figure 9. Using three cameras for gathering data for training deep neural network for autonomous drive was inspired by [1]. All of these images captured a similar scene but from slightly different positions. Advantages of using three cameras instead of one central camera are three times more data, and better performance for steering back to the center scenario when the vehicle starts drifting off to the side. The steering angle was paired with three images from a particular frame corresponding to the central camera, and the images from the left and right cameras had a field of view shifted on the left or right side of the road, respectively, which means that the steering angle for the left and right images is not correct. To overcome this, the correction factor for the steering angle measurement had been added or subtracted from the left and right images, respectively. The correction factor was one of the hyperparameters to be fine-tuned during the training process.” [Pg.9 Par.5]). Both Li and Kocic teach methods of providing driver assistance alerts to a driver. However, Kocic explicitly teaches processing the environmental data as input to imitation training of a transformer-based end-to-end neural network, including training the end-to-end neural network to generate prescriptive steering and speed control actions whereby attention weights of the trained, transformer-based end-to-end neural network are available to be extracted from hidden layer data and indicate areas of the video that contribute most significantly to the generated prescriptive steering and speed control actions.
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the driver assistance method of Li to also include processing the environmental data as input to imitation training of a transformer-based end-to-end neural network, including training the end-to-end neural network to generate prescriptive steering and speed control actions whereby attention weights of the trained, transformer-based end-to-end neural network are available to be extracted from hidden layer data and indicate areas of the video that contribute most significantly to the generated prescriptive steering and speed control actions, as taught by Kocic, with a reasonable expectation of success. Doing so improves safety of autonomous vehicle operation (With regard to this reasoning, see at least [Kocic, Pg.9]).
Regarding claim 15, Li discloses The method of claim 14,
whereby the normalized risk metric onto a heads up display (“The DVI 180 displays the computed glide slope 410. Further, based on the driver input, a second glide slope 420 is computed using the second deceleration from the second maneuver control. The second glide slope 420 can be above or below the first glide slope 410 based on driver input to the brake pedal. In one or more examples, the DVI 180 displays the second glide slope 420 along with a curve 425 representing a difference in the suggested glide slope 410 from the automated driving system and the actual glide slope 420 based on the driver's input.” [Col.9 ln 28-38]).
Li does not explicitly teach further including configuring a system including the end-to-end neural network and further including a risk metric generator, including: training parameters of the risk metric generator to generate a normalized risk metric that quantifies a dissimilarity between the generated prescriptive steering and speed control actions and received driver steering and speed control actions that vary from the generated prescriptive steering and speed control actions
However, Kocic does teach further including configuring a system including the end-to-end neural network and further including a risk metric generator, including: training parameters of the risk metric generator to generate a normalized risk metric that quantifies a dissimilarity between the generated prescriptive steering and speed control actions and received driver steering and speed control actions that vary from the generated prescriptive steering and speed control actions(“For training the deep neural network, images acquired from all three cameras were used--central, left, and right; Figure 9. Using three cameras for gathering data for training deep neural network for autonomous drive was inspired by [1]. All of these images captured a similar scene but from slightly different positions. Advantages of using three cameras instead of one central camera are three times more data, and better performance for steering back to the center scenario when the vehicle starts drifting off to the side. The steering angle was paired with three images from a particular frame corresponding to the central camera, and the images from the left and right cameras had a field of view shifted on the left or right side of the road, respectively, which means that the steering angle for the left and right images is not correct. To overcome this, the correction factor for the steering angle measurement had been added or subtracted from the left and right images, respectively. The correction factor was one of the hyperparameters to be fine-tuned during the training process.” [Pg.9 Par.5]). Both Li and Kocic teach methods of providing driver assistance alerts to a driver. However, Kocic explicitly teaches configuring a system including the end-to-end neural network and further including a risk metric generator, including: training parameters of the risk metric generator to generate a normalized risk metric that quantifies a dissimilarity between the generated prescriptive steering and speed control actions and received driver steering and speed control actions that vary from the generated prescriptive steering and speed control actions.
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the driver assistance method of Li to also include configuring a system including the end-to-end neural network and further including a risk metric generator, including: training parameters of the risk metric generator to generate a normalized risk metric that quantifies a dissimilarity between the generated prescriptive steering and speed control actions and received driver steering and speed control actions that vary from the generated prescriptive steering and speed control actions, as taught by Kocic, with a reasonable expectation of success. Doing so improves safety of autonomous vehicle operation (With regard to this reasoning, see at least [Kocic, Pg.9]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AHMED ALKIRSH whose telephone number is (703) 756-4503. The examiner can normally be reached M-F 9:00 am-5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, FADEY JABR can be reached on (571) 272-1516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AA/Examiner, Art Unit 3668
/Fadey S. Jabr/Supervisory Patent Examiner, Art Unit 3668