DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This is the First Office Action on the merits.
Claims 1-20 are currently pending and addressed below.
Information Disclosure Statement
The information disclosure statement (IDS) filed on 03/22/2025 has been considered. An initialed copy of the IDS is enclosed herewith.
Claim Objections
Claim 6 is objected to because of the following informalities: the claim has no period at the end. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-7, 9-16 and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable
Over Bajpayee et al. (US 20230177839 A1) in view of Bianconicini et al. (US 20230077516 A1).
Regarding claim 1, and similarly with respect to claims 14 and 18, Bajpayee et al. discloses A
method comprising: receiving, by at least one processor of a computing device in communication with a camera mounted on or in a vehicle, an image captured by the camera; and ([0006] “deep learning based operational domain verification using camera-based inputs for autonomous systems and applications.” and [0022] “a processor comprising one or more circuits may compute data representative of operational domain conditions using a neural network and based at least in part on image data generated using one or more image sensors. . . operational domain conditions may include one or more of the following: an amount of camera blindness, a blindness classification corresponding to the amount of camera blindness, an illumination level, a path surface condition, a visibility distance, a scene type classification, and a distance to a scene corresponding to the scene type classification.”)
executing, by the at least one processor, a machine learning model using the image as
input to generate a plurality of environmental indicators for an environment of the vehicle by: ([0022] “a processor comprising one or more circuits may compute data representative of operational domain conditions using a neural network and based at least in part on image data generated using one or more image sensors. . . operational domain conditions may include one or more of the following: an amount of camera blindness, a blindness classification corresponding to the amount of camera blindness, an illumination level, a path surface condition, a visibility distance, a scene type classification, and a distance to a scene corresponding to the scene type classification” and [0035] “a blindness classification of “blocked” may be associated with one or more blindness sub-classifications, such as sun, fog, water, mist, snow, frozen pane, day, night, broken lens, self-glare, mud, paper, leaf, etc. In this way, blindness sub-classifications may further delineate a cause for the blindness in the sensor data 102. In some examples, the blindness sub-classifications may include further sub-classification blindness causes, such as to indicate a level of compromised visibility (e.g., heavy, moderate, light, etc.). The machine learning model(s) 104 may similarly have as many output channels as the number of blindness classifications and/or corresponding blindness sub-classifications that the machine learning model(s) is trained to predict.”)
inserting, by the at least one processor, the image into a first set of layers of the machine
learning model to generate an image embedding of the image; inserting, by the at least one processor, the image embedding into a second set of layers of the machine learning model to generate a weather prediction embedding; ([0034] “The blindness classification 110 may represent the cause for blindness associated with one or more pixels of the sensor data 101 with detected blindness. The sensor data 102 may include any number of different blindness classifications. For example, a single blindness region(s) or pixel may have one, two, or more associated blindness classifications without departing from the scope of the present disclosure. In some examples, the machine learning model(s) 104 may output a number of channels corresponding to a number of classifications the machine learning model(s) is trained to predict (e.g., each channel may correspond to one blindness classification). For example, the blindness classification 110 may include one or more of blocked, blurred, reflection, open, ego-vehicle, sky, frame label, etc., and the number of output channels may correspond to the number of classifications desired. The blindness classification 110 may include sub-classifications, including but not limited to, rain, glare, broken lens, light, mud, paper, person, etc.”, [0039] “The machine learning model(s) 104 may further be trained to predict the path surface condition 114 from the sensor data 102, such as dry, wet, snowy, icy, etc., as well as the illumination level 112, e.g. the amount of light perceived by the road user.”, [0040] “such as where the machine learning model(s) 104 includes a DNN, the machine learning model(s) 104 may include any number of layers. One or more of the layers may include an input layer. The input layer may hold values associated with the sensor data 402. One or more layers may include convolutional layers. The convolutional layers may compute the output of neurons that are connected to local regions in an input layer, each neuron computing a dot product between their weights and a small region they are connected to in the input volume. A result of the convolutional layers may be another volume, with one of the dimensions based on the number of filters applied (e.g., the width, the height, and the number of filters, such as 32×32×12, if 12 were the number of filters).”, and [0058])
inserting, by the at least one processor, the weather prediction embedding into each set
of environment prediction layers in a plurality of sets of environment prediction layers, wherein each set of environment prediction layers outputs a probability distribution for a respective environmental indicator based on the weather prediction embedding; and ([0040] “the machine learning model(s) 104 includes a DNN, the machine learning model(s) 104 may include any number of layers. One or more of the layers may include an input layer. The input layer may hold values associated with the sensor data 402. One or more layers may include convolutional layers. The convolutional layers may compute the output of neurons that are connected to local regions in an input layer, each neuron computing a dot product between their weights and a small region they are connected to in the input volume. A result of the convolutional layers may be another volume, with one of the dimensions based on the number of filters applied (e.g., the width, the height, and the number of filters, such as 32×32×12, if 12 were the number of filters).” and [0050] “ machine learning model(s) 104A for implementing an example process for determining illumination level, path surface condition, visibility distance, scene classification, distance to a scene, and/or sensor blindness and/or its cause(s), in accordance with some embodiments of the present disclosure. For example, in one or more embodiments, the machine learning model(s) 104A may include a CNN that may include one or more trunks, an encoder decoder architecture, and/or one or more output heads (e.g. a blindness mask head 142, an illumination head 144, a path surface head 146, a blindness cause head 148, and a scene classification head 150). . . e.g. the path surface head 146, may be used to compute a path surface condition, and/or a fifth output head, e.g. the scene head 150, may be used to compute a scene type classification (e.g. a probability of a type of scene 152) and/or a distance to the scene corresponding to the scene type classification (e.g. a distance to a scene 154).” and see at least [0039]) generating, by the at least one processor, the plurality of environmental indicators for the environment of the vehicle based on the probability distributions. ([0114] “a neural network that outputs a measure of confidence for each object detection. Such a confidence value may be interpreted as a probability, or as providing a relative “weight” of each detection compared to other detections. This confidence value enables the system to make further decisions regarding which detections should be considered as true positive detections rather than false positive detections. For example, the system may set a threshold value for the confidence and consider only the detections exceeding the threshold value as true positive detections.”, and see at least [0058])
However, Bajpayee et al. may be alleged to not explicitly teach inserting, by the at least one
processor, the image embedding into a second set of layers of the machine learning model to generate a weather prediction embedding; . . . wherein each set of environment prediction layers outputs a probability distribution for a respective environmental indicator based on the weather prediction embedding;
Bianconicini et al. teaches inserting, by the at least one processor, the image embedding
into a second set of layers of the machine learning model to generate a weather prediction embedding; . . . wherein each set of environment prediction layers outputs a probability distribution for a respective environmental indicator based on the weather prediction embedding; (Figure 1A and 1D, [0016] “FIGS. 1A-1F are diagrams of an example 100 associated with utilizing machine learning to detect adverse conditions. As shown in FIGS. 1A-1F, example 100 includes an adverse environment detection system 105 associated with a vehicle device 110 and a multi-task model 115. The multi-task model 115 may be configured to detect a current road condition, a current weather condition, and a current visibility condition, as described in greater detail herein.”, [0026] “The second neural network block 125b may be configured to receive the output of the backbone 120 and to process the output to determine a weather condition of the road depicted in the image. In some implementations, the second neural network block 125b may include a multi-layer perceptron. The multi-layer perceptron may include a plurality of sequential, linear layers. For example, as shown in FIG. 1A, the multi-layer perceptron may include linear layer 130c and linear layer 13d. The quantity of linear layers may be the same as, or different than, the quantity of linear layers of the multi-layer perceptron of the first neural network block 125a.”, [0029] “the quantity of neurons included in the last linear layer (e.g., linear layer 130d) of the second neural network block 125b is based on a task associated with the second neural network block 125b. As an example, the second neural network block 125b may be configured to classify an image based on three classifications of weather conditions (e.g., a clear weather condition, an overcast weather condition, and a snowy weather condition). The last linear layer (e.g., linear layer 130d) of the second neural network block 125b may include three neurons based on the second neural network block 125b being configured to classify an image based on three classifications of weather conditions.”, [0041] “an output of each first linear layer (e.g., the second quantity of combinations of features) may be provided, as an input, to the respective second linear layers (e.g., linear layer 130b, linear layer, 130d, and linear layer 130f). The second linear layers may process the second quantity of combinations of features to generate an output indicating a road condition associated with the image, a weather condition associated with the image, and a visibility condition associated with the image, respectively.”) and generating, by the at least one processor, the plurality of environmental indicators for the environment of the vehicle based on the probability distributions. ([0043] “the adverse environment detection system 105 may determine respective scores associated with the road condition, the weather condition, and/or the visibility condition. The scores may indicate a likelihood that the road condition, the weather condition, and/or the visibility condition will contribute to an adverse road condition. For example, the adverse environment detection system 105 may determine a first score for a dry road condition that is lower than a second score determined for a wet road condition based on the wet road condition being more likely to contribute to an adverse road condition relative to the dry road condition.” and [0044] “The adverse environment detection system 105 may determine the classification for the current environmental condition associated with the vehicle based on the respective scores determined for the road condition, the weather condition, and/or the visibility condition. For example, the adverse environment detection system 105 may determine whether a combination of the respective scores (e.g., a sum of the respective scores, a weighted sum of the respective scores, an average of the respective scores, and/or the like) satisfies one or more criteria (e.g., is greater than a score threshold, is less than a score threshold, crosses a score threshold, and/or the like). The adverse environment detection system 105 may determine the classification for the current environmental condition associated with the vehicle based on whether the combination of the respective scores satisfies the one or more criteria.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date
of the claimed invention with reasonable expectations of success to modify the invention of Bajpayee et al. to incorporate multiple layers to generate a probability for a respective environmental indicator(s) as taught by Bianconcini et al. for the purpose of “accurately determine adverse weather conditions that may affect an operation of a vehicle”. ([0012], Bianconcini et al.)
With the additional limitation of claim 18, Bajpayee et al. discloses selecting, by the at least
one processor, an environmental indicator for at least one of the plurality of sets of environment prediction layers responsive to the environmental indicator exceeding a consistency threshold. ([0114] “The DLA may be used to run any type of network to enhance control and driving safety, including for example, a neural network that outputs a measure of confidence for each object detection. Such a confidence value may be interpreted as a probability, or as providing a relative “weight” of each detection compared to other detections. This confidence value enables the system to make further decisions regarding which detections should be considered as true positive detections rather than false positive detections. For example, the system may set a threshold value for the confidence and consider only the detections exceeding the threshold value as true positive detections.”)
Regarding claim 2, and similarly with respect to claim 15, Bajpayee et al. in view of
Bianconcini et al. discloses The method of claim 1,
Bajpayee et al. discloses transmitting, by the at least one processor, the plurality of
environmental indicators to a second processor controlling the vehicle, the second processor controlling the vehicle according to the plurality of environmental indicators. ([0171] “The server(s) 678 may be used to train machine learning models (e.g., neural networks) based on training data. The training data may be generated by the vehicles . . . , the machine learning models may be used by the vehicles (e.g., transmitted to the vehicles over the network(s) 690, and/or the machine learning models may be used by the server(s) 678 to remotely monitor the vehicles.” , [0065] “determining, based at least in part on the data, an operational level of an ego-machine corresponding to the environment. For example, the operational level may be determined based on one or more of the camera blindness, a blindness classification corresponding to the camera blindness, an illumination level, a path surface condition, a visibility distance, a scene type classification, and/or a distance to a scene corresponding to the scene type classification” and [0066] “The method 500, at block B506, includes controlling an operation of the ego-machine according to the operational level. For example, where the scene type classification is determined to be a toll plaza and/or the distance to the scene is determined to be 100 yards, the ego-machine may decelerate, in accordance with embodiments of the present disclosure. In other examples, where the visibility distance is determined to be low, the ego-machine may decelerate or perform other safety maneuvers (stopping, navigating to a safe or designated area, etc.).”)
Regarding claim 3, and similarly with respect to claims 16 and 19, Bajpayee et al. in view of
Bianconcini et al. discloses The method of claim 2,
Bajpayee et al. discloses wherein the second processor controls the vehicle according to
the plurality of environmental indicators by: adjusting one or more vehicle control parameters based on the plurality of environmental indicators; and selecting an action to control the vehicle based on the adjusted one or more vehicle control parameters. ([0036] “An operational level of the machine may be adjusted based on the predicted visibility distance. For example, where the machine learning model(s) 104 determines the visibility distance is low—e.g., 20 meters or less—the corresponding sensor data may only be relied upon for Level 0 (no automation) or Level 1 (driver assistance) tasks, or may be relied upon only for predictions that are within 20 meters of the machine (e.g., and predictions beyond 20 meters may be disregarded, or more have a lower associated confidence). Similarly, as another example, where an estimated visibility distance is high—e.g., 1000 meters or more—the corresponding sensor data may be relied upon for the full performance of Level 3 (conditional driving automation) and Level 4 (high driving automation) tasks, or may be relied upon for predictions corresponding to locations within 1000 meters of the machine. . . ”, [0065] “determining, based at least in part on the data, an operational level of an ego-machine corresponding to the environment. For example, the operational level may be determined based on one or more of the camera blindness, a blindness classification corresponding to the camera blindness, an illumination level, a path surface condition, a visibility distance, a scene type classification, and/or a distance to a scene corresponding to the scene type classification” , [0066] “controlling an operation of the ego-machine according to the operational level. For example, where the scene type classification is determined to be a toll plaza and/or the distance to the scene is determined to be 100 yards, the ego-machine may decelerate, in accordance with embodiments of the present disclosure. In other examples, where the visibility distance is determined to be low, the ego-machine may decelerate or perform other safety maneuvers (stopping, navigating to a safe or designated area, etc.)”, see at least [0037], [0052] and [0171])
Regarding claim 5, Bajpayee et al. in view of Bianconcini et al. discloses The method of
claim 1,
Bajpayee et al. discloses wherein the plurality of sets of environment prediction layers
comprises a first set of environment prediction layers predicting a visibility condition and a second set of environment prediction layers predicting a road condition. (figure 1, [0023] “Deep neural network (DNN) processing may be used to determine an amount of camera blindness, a blindness classification corresponding to the amount of camera blindness, an illumination level, a path surface condition, a visibility distance, a scene type classification, and a distance to a scene corresponding to the scene type classification. Sensor data (e.g., representative of image(s)) may be received from one or more sensors (e.g., cameras) disposed on or otherwise associated with a vehicle. The sensor data may be applied to a neural network (e.g., a DNN, such as a convolutional neural network (CNN)) that may be trained to determine the amount of camera blindness, an illumination level, a path surface condition, a visibility distance, a scene type classification, and a distance to a scene corresponding to the scene type classification associated with the sensor data.”, [0032] “The machine learning model(s) 104 may use as input one or more images or other data representations (e.g., LIDAR point clouds, RADAR representations, etc.) as represented by the sensor data 102 to generate output(s) 106. In a non-limiting example, the machine learning model(s) 104 may take, as input, an image(s) represented by the sensor data 102 (e.g., after pre-processing) to generate the output(s) 106, e.g., camera blindness 108, blindness classification(s) 110, illumination level 112, path surface condition 114, visibility distance, scene-type classification 118, and/or distance to a scene 120.”, [0035] “blindness classification of “blocked” may be associated with one or more blindness sub-classifications, such as sun, fog, water, mist, snow, frozen pane, day, night, broken lens, self-glare, mud, paper, leaf, etc. In this way, blindness sub-classifications may further delineate a cause for the blindness in the sensor data 102. In some examples, the blindness sub-classifications may include further sub-classification blindness causes, such as to indicate a level of compromised visibility (e.g., heavy, moderate, light, etc.). The machine learning model(s) 104 may similarly have as many output channels as the number of blindness classifications and/or corresponding blindness sub-classifications that the machine learning model(s) is trained to predict.” and [0040] “here the machine learning model(s) 104 includes a DNN, the machine learning model(s) 104 may include any number of layers. One or more of the layers may include an input layer. The input layer may hold values associated with the sensor data 402. One or more layers may include convolutional layers. The convolutional layers may compute the output of neurons that are connected to local regions in an input layer, each neuron computing a dot product between their weights and a small region they are connected to in the input volume. A result of the convolutional layers may be another volume, with one of the dimensions based on the number of filters applied (e.g., the width, the height, and the number of filters, such as 32×32×12, if 12 were the number of filters)”)
Regarding claim 6, Bajpayee et al. in view of Bianconcini et al. discloses The method of
claim 5,
Bianconcini et al. teaches wherein the plurality of sets of environment prediction layers
further comprises a third set of environment prediction layers predicting a windshield condition (Abstract, [0013] “an adverse environment detection system that utilizes machine learning to detect adverse conditions based on a current road condition, a current weather condition, and a current visibility condition. For example, the adverse environment detection system may receive an image of a road scene associated with a vehicle. The adverse environment detection system may determine a set of features associated with the image based on providing image data corresponding to the image to a backbone portion of a multi-task model. The set of features may be provided to a first processing layer, a second processing layer, and a third processing layer of the multi-task model. The first, second, and third processing layers may process the set of features, in parallel, to determine a current road condition, a current weather condition, and a current visibility condition, respectively. The adverse environment detection system may cause information relating to the current road condition, the current weather condition, and the current visibility condition to be provided to user device of a user, such as user device of a driver of the vehicle and/or a user device of a fleet manager associated with the vehicle”, [0031] “The third neural network block 125c (e.g., the linear layer 130e and the linear layer 130f) may comprise a multi-layer neural network that is configured to generate an output indicating a visibility condition based on the set of features. For example, the third neural network block 125c may generate an output indicating a poor visibility condition and/or a good visibility condition, as described in greater detail herein. The poor visibility condition may indicate that rain drops, snowflakes, and/or dirt, among other examples, are present on a windshield of the vehicle and/or may affect the visibility of the driver of the vehicle. The good visibility condition may indicate that rain drops, snowflakes, and/or dirt, among other examples, are not present on the windshield of the vehicle.”) and a fourth set of environment prediction layers predicting a side-of-road condition. ([0034] “The multi-task model 115 may be trained based on images depicting road scenes of vehicles. The multi-task model 115 may be trained to determine, based on an image of a road scene, a road condition, a weather condition, and a visibility condition with which the road scene is associated, a confidence score that reflects a measure of confidence that the road condition is accurate for this image, a confidence score that reflects a measure of confidence that the weather condition is accurate for this image, and/or a confidence score that reflects a measure of confidence that the visibility condition is accurate for this image. In some implementations, the multi-task model 115 may be trained to determine the road condition, the weather condition, and the visibility condition in parallel.” and [0038] “the second neural network block 125b, and the third neural network block 125c may process the set of features, in parallel, to generate a first output indicating a road condition associated with the image, a second output indicating a weather condition associated with the image, and a third output indicating a visibility condition associated with the image, respectively.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date
of the claimed invention with reasonable expectations of success to modify the invention of Bajpayee et al. in combination with Bianconcini et al. to incorporate additional environment prediction (e.g. road condition and windshield condition) as taught by Bianconcini et al. for the purpose of “providing information indicating the likelihood of an occurrence of a harsh driving event to the vehicle device 110 and/or a user device associated with the vehicle . . .”, increasing safety. ([0046], Bianconcini et al.)
Regarding claim 7, Bajpayee et al. in view of Bianconcini et al. discloses The method of
claim 1,
Bajpayee et al. discloses receiving, by the at least one processor, a first set of images from the camera; executing, by the at least one processor, the machine learning model using each image of the set of images as input to generate a second plurality of environmental indicators for the environment of the vehicle for the image of the first set of images; determining, by the at least one processors, that the second plurality of environmental indicators satisfy a consistency threshold based on a number of each of the second plurality of environmental indicators generated based on the first set of images; and ([0032] “The machine learning model(s) 104 may use as input one or more images or other data representations (e.g., LIDAR point clouds, RADAR representations, etc.) as represented by the sensor data 102 to generate output(s) 106. In a non-limiting example, the machine learning model(s) 104 may take, as input, an image(s) represented by the sensor data 102 (e.g., after pre-processing) to generate the output(s) 106, e.g., camera blindness 108, blindness classification(s) 110, illumination level 112, path surface condition 114, visibility distance, scene-type classification 118, and/or distance to a scene 120.” and [0052] “the outputs 106 may undergo post-processing by a post processor 122. The post processor 122 may determine drivability corresponding to the sensor data 102. For example, the post processor 122 may perform temporal smoothing of the outputs 106 of the machine learning model(s) 104. Temporal smoothing may be used in some embodiments to improve stability of the system by reducing false positives based on a single frame—by incorporating prior predictions of the machine learning model(s) 104 corresponding to temporally adjacent frame—to smooth and reduce noise in the output of the machine learning model(s) 104. In some examples, values computed by the machine learning model(s) 104 for a current instance of the sensor data 102 may be weighed against values computed by the machine learning model(s) 104 for one or more prior instances of the sensor data 102. Where the sensor data 102 is image data representative of images, for example, the outputs 106 computed by the machine learning model(s) 104 for a current or most recent image may be weighed against the outputs 106 computed by the machine learning model(s) 104 for one or more temporally adjacent—e.g., previous and/or sequential—images. As such, final values (e.g., for the camera blindness 108, the blindness classification(s) 110, the illumination level 112, the path surface condition 114, the visibility distance 116, the scene type classification 118, and the distance to the scene 120) corresponding to an instance of the sensor data 102 may be determined by weighting prior values associated with one or more other instances of the sensor data 102 against current values associated with the instance of the sensor data 102.”, [0058] “the training engine 128 may include a loss function(s) 132 to compare the outputs 106 to the ground truth data 130 (e.g., corresponding to one or more of the amount of camera blindness, blindness classification corresponding to camera blindness, illumination level, path surface condition, visibility distance, scene type classification, and distance to a scene corresponding to scene type classification). In embodiments, the machine learning model(s) 104 may be trained with sensor data 102 using multiple iterations until the value of the loss function(s) 132 is below a predetermined threshold. Any type of loss function may be used, such as cross entropy loss, mean squared error, mean absolute error, mean bias error, and/or other loss function types. In some examples, gradients of the loss function 132 may be iteratively computed with respect to training parameters. An optimizer, such as an Adam optimizer, stochastic gradient descent, or any other type of optimization algorithm may be used to optimize the loss function 132 while training the machine learning model(s) 104. The machine learning model(s) 104 may be trained iteratively until the training parameters converge to optimum, desired, or accepted values.”)
responsive to determining the second plurality of environmental indicators satisfy the
consistency threshold, generating, by the at least one processor, a record identifying the second plurality of environmental indicators. ([0114] “The DLA may be used to run any type of network to enhance control and driving safety, including for example, a neural network that outputs a measure of confidence for each object detection. Such a confidence value may be interpreted as a probability, or as providing a relative “weight” of each detection compared to other detections. This confidence value enables the system to make further decisions regarding which detections should be considered as true positive detections rather than false positive detections. For example, the system may set a threshold value for the confidence and consider only the detections exceeding the threshold value as true positive detections.”)
Regarding claim 9, Bajpayee et al. in view of Bianconcini et al. discloses The method of
claim 1.
Bajpayee et al. discloses further comprising: receiving by the at least one processor, a plurality of environmental labels for a training image, each label corresponding to a different set of the plurality of sets of environment prediction layers; ([0056] “Referring back to FIG. 1A, in embodiments, the machine learning model(s) 104 may be trained both with information supplied by the sensor data 102 and by information supplied by a training engine 128. In examples, the training engine 128 may include ground truth data 130. In some embodiments, the ground truth data 130 may include labels or annotations corresponding to the sensor data 102. For example, for each instance of the sensor data 102A, the labels or annotations may correspond to one or more of the amount of camera blindness, blindness classification corresponding to camera blindness, illumination level, path surface condition, visibility distance, scene type classification, and distance to a scene corresponding to scene type classification.”) executing, by the at least one processor, the machine learning model using the training image as input to obtain an output environmental indicator distribution from each of the plurality of sets of environment prediction layers; and training, by the at least one processor, each of the plurality of sets of environment prediction layers according to the label that corresponds to the set of environment prediction layers and the output environmental indicator distribution from the set of environment prediction layers. ([0049] “In embodiments, determining an operational level of the ego-machine, e.g. operations 124, may include weighting the camera blindness, the blindness classification, the illumination level, the path surface condition, the visibility distance, the scene type classification, and the distance to a scene according to pre-defined weights. In other embodiments, the camera blindness may be associated with a highest weight of the pre-defined weights.” [0058] In embodiments, the training engine 128 may include a loss function(s) 132 to compare the outputs 106 to the ground truth data 130 (e.g., corresponding to one or more of the amount of camera blindness, blindness classification corresponding to camera blindness, illumination level, path surface condition, visibility distance, scene type classification, and distance to a scene corresponding to scene type classification.”)
Regarding claim 10, Bajpayee et al. in view of Bianconcini et al. discloses The method of
claim 9,
Bajpayee et al. discloses further comprising: training by the at least one processor, the
second set of layers, by: calculating, by the at least one processor, a difference for each of the plurality of sets of environment prediction layers based on the label that corresponds to the set of environment prediction layers and the output environmental indicator distribution from the set of environment prediction layers; ([0056] “the machine learning model(s) 104 may be trained both with information supplied by the sensor data 102 and by information supplied by a training engine 128. In examples, the training engine 128 may include ground truth data 130. In some embodiments, the ground truth data 130 may include labels or annotations corresponding to the sensor data 102. For example, for each instance of the sensor data 102A, the labels or annotations may correspond to one or more of the amount of camera blindness, blindness classification corresponding to camera blindness, illumination level, path surface condition, visibility distance, scene type classification, and distance to a scene corresponding to scene type classification. The labels and annotations may be generated within a drawing program (e.g., an annotation program), a computer aided design (CAD) program, a labeling program, another type of program suitable for generating the annotations”, [0058] “the training engine 128 may include a loss function(s) 132 to compare the outputs 106 to the ground truth data 130 (e.g., corresponding to one or more of the amount of camera blindness, blindness classification corresponding to camera blindness, illumination level, path surface condition, visibility distance, scene type classification, and distance to a scene corresponding to scene type classification). In embodiments, the machine learning model(s) 104 may be trained with sensor data 102 using multiple iterations until the value of the loss function(s) 132 is below a predetermined threshold. Any type of loss function may be used, such as cross entropy loss, mean squared error, mean absolute error, mean bias error, and/or other loss function types. In some examples, gradients of the loss function 132 may be iteratively computed with respect to training parameters. An optimizer, such as an Adam optimizer, stochastic gradient descent, or any other type of optimization algorithm may be used to optimize the loss function 132 while training the machine learning model(s) 104. The machine learning model(s) 104 may be trained iteratively until the training parameters converge to optimum, desired, or accepted values.”)
aggregating, by the at least one processor, the calculated differences for the plurality of
sets of environment prediction layers according to a loss function to obtain an aggregated difference; and using, by the at least one processor, a back-propagation technique with the aggregated difference on the second set of prediction layers. (0058] “the training engine 128 may include a loss function(s) 132 to compare the outputs 106 to the ground truth data 130 (e.g., corresponding to one or more of the amount of camera blindness, blindness classification corresponding to camera blindness, illumination level, path surface condition, visibility distance, scene type classification, and distance to a scene corresponding to scene type classification). In embodiments, the machine learning model(s) 104 may be trained with sensor data 102 using multiple iterations until the value of the loss function(s) 132 is below a predetermined threshold. Any type of loss function may be used, such as cross entropy loss, mean squared error, mean absolute error, mean bias error, and/or other loss function types. In some examples, gradients of the loss function 132 may be iteratively computed with respect to training parameters. An optimizer, such as an Adam optimizer, stochastic gradient descent, or any other type of optimization algorithm may be used to optimize the loss function 132 while training the machine learning model(s) 104. The machine learning model(s) 104 may be trained iteratively until the training parameters converge to optimum, desired, or accepted values.”)
Regarding claim 11, Bajpayee et al. in view of Bianconcini et al. discloses The method of
claim 10 ,
Bajpayee et al. discloses wherein aggregating the calculated differences for the plurality of
sets of environment prediction layers comprises aggregating, by the at least one processor, the calculated differences according to weights assigned to the plurality of sets of environment prediction layers. (0058] “the training engine 128 may include a loss function(s) 132 to compare the outputs 106 to the ground truth data 130 (e.g., corresponding to one or more of the amount of camera blindness, blindness classification corresponding to camera blindness, illumination level, path surface condition, visibility distance, scene type classification, and distance to a scene corresponding to scene type classification). In embodiments, the machine learning model(s) 104 may be trained with sensor data 102 using multiple iterations until the value of the loss function(s) 132 is below a predetermined threshold. Any type of loss function may be used, such as cross entropy loss, mean squared error, mean absolute error, mean bias error, and/or other loss function types. In some examples, gradients of the loss function 132 may be iteratively computed with respect to training parameters. An optimizer, such as an Adam optimizer, stochastic gradient descent, or any other type of optimization algorithm may be used to optimize the loss function 132 while training the machine learning model(s) 104. The machine learning model(s) 104 may be trained iteratively until the training parameters converge to optimum, desired, or accepted values.”)
Regarding claim 12, Bajpayee et al. in view of Bianconcini et al. discloses The method of
claim 1,
Bajpayee et al. discloses further comprising: generating, by the at least one processor, a
record based on the plurality of environmental indicators; and transmitting, by the at least one processor, the record to a remote computing device across a network. ([0170] “The server(s) 678 may receive, over the network(s) 690 and from the vehicles, image data representative of images showing unexpected or changed road conditions, such as recently commenced road-work . . .” and [00171] “The server(s) 678 may be used to train machine learning models (e.g., neural networks) based on training data. The training data may be generated by the vehicles . . . Once the machine learning models are trained, the machine learning models may be used by the vehicles (e.g., transmitted to the vehicles over the network(s) 690, and/or the machine learning models may be used by the server(s) 678 to remotely monitor the vehicles.”)
Regarding claim 13, Bajpayee et al. in view of Bianconcini et al. discloses The method of
claim 1,
Bajpayee et al. discloses wherein executing the machine learning model comprises:
selecting, by the at least one processor, an environmental indicator from the plurality of potential environmental indicators responsive to the environmental indicator having a probability distribution that satisfies a condition. ([0051] “one or more of the first, second, third, fourth, fifth, and/or sixth heads of the CNN of the machine learning model(s) 104 may include at least one fully connected layer. For example, the illumination head 144, the path surface head 146, the blindness cause head 148, and the scene head 150, may each include at least one fully connected layer. Each neuron in the fully connected layer(s) may be connected to each of the neurons in the previous volume. The fully connected layer may compute class scores, and the resulting volume may be 1×1×n, where n is equivalent to the number of classes. In some non-limiting embodiments, one or more of the heads of the CNN of the machine learning model(s) 104 may include one or more down-sampling layers and one or more up-sampling layers. For example, the blindness mask head 142 of the CNN of the machine learning model(s) 104 may include a series of down-sampling layers (e.g., pooling layers) to facilitate image feature extraction followed by up-sampling layers to facilitate global context feature extraction. In embodiments, one or more pooling layers may perform a down sampling operation along the spatial dimensions (e.g., the height and the width), which may result in a smaller volume than the input of the pooling layer (e.g., 16×16×12 from the 32×32×12 input volume). In examples, each of the illumination head 144, the path surface head 146, the blindness cause head 148, and the scene head 150, may include a pooling layer. In further examples, the scene head 150 may output the probability of a type of scene 152, such as a tunnel, a construction zone, a blocked lane, or a toll plaza, using a sigmoid function. In other examples, the scene head 150 may also output the distance to a scene 154 using a ReLU layer”, [0114] “The DLA may be used to run any type of network to enhance control and driving safety, including for example, a neural network that outputs a measure of confidence for each object detection. Such a confidence value may be interpreted as a probability, or as providing a relative “weight” of each detection compared to other detections. This confidence value enables the system to make further decisions regarding which detections should be considered as true positive detections rather than false positive detections. For example, the system may set a threshold value for the confidence and consider only the detections exceeding the threshold value as true positive detections. . .” and see at least [0050])
Claims 4 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Bajpayee et
al. (US 20230177839 A1) in view of Bianconcini et al. (US 20230077516 A1) and further in view of Takeda (US 20170270799 A1).
Regarding claim 4, and similarly with respect to claim 17, Bajpayee et al. in view of
Bianconcini et al. discloses The method of claim 3,
However, Bajpayee et al. in view of Bianconcini et al. fails to explicitly disclose wherein an
adjustment to the one or more vehicle control parameters comprises increasing a vehicle braking distance value.
Takeda teaches wherein an adjustment to the one or more vehicle control parameters
comprises increasing a vehicle braking distance value.([0096] “determines whether or not the weather conditions, road surface conditions, visibility conditions, or the like include conditions that would increase the braking distance of the vehicle M. The margin amount setting section 114 changes the size of the vehicle-in-front rearward margin MGA so as to increase the size of the vehicle-in-front rearward margin MGA in cases in which the weather conditions, road surface conditions, visibility conditions, or the like include conditions that would increase the braking distance of the vehicle M. For example, the margin amount setting section 114 changes the size of the vehicle-in-front rearward margin MGA so as to increase the size of the vehicle-in-front rearward margin MGA in rainy or snowy conditions, in conditions in which there is a possibility of ice on the road surface, or in conditions in which visibility is poor.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date
of the claimed invention with reasonable expectations of success to modify the invention of Bajpayee et al. in combination with Bianconcini et al. to incorporate increasing the braking distance based on road surface conditions and visibility conditions as taught by Takeda for the purpose of “increase[ing] in safety in various situations of the vehicle M and the nearby vehicles.” ([0108], Takeda)
Claims 8 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Bajpayee et
al. (US 20230177839 A1) in view of Bianconcini et al. (US 20230077516 A1) and further in view of Tariq et al. (US 20190392242 A1).
Regarding claim 8, and similarly with respect to claim 20, Bajpayee et al. in view of
Bianconcini et al. discloses The method of claim 7,
Bajpayee et al. discloses responsive to detecting a change between the second plurality of
environmental indicators and the third plurality of environmental indicators, generating, by the at least one processor, a second record identifying the third plurality of environmental indicators. ([0170] “The server(s) 678 may receive, over the network(s) 690 and from the vehicles, image data representative of images showing unexpected or changed road conditions, such as recently commenced road-work. The server(s) 678 may transmit, over the network(s) 690 and to the vehicles, neural networks 692, updated neural networks 692, and/or map information 694, including information regarding traffic and road conditions. The updates to the map information 694 may include updates for the HD map 622, such as information regarding construction sites, potholes, detours, flooding, and/or other obstructions. In some examples, the neural networks 692, the updated neural networks 692, and/or the map information 694 may have resulted from new training and/or experiences represented in data received from any number of vehicles in the environment, and/or based on training performed at a datacenter (e.g., using the server(s) 678 and/or other servers”)
However, Bajpayee et al. in combination with Bianconcini et al. fails to explicitly disclose
wherein the first set of images has a first size, and further comprising: receiving, by the at least one processor subsequent to receiving the first set of images, a second set of images from the camera; executing, by the at least one processor, the machine learning model using each image of the second set of images as input to generate a third plurality of environmental indicators for the environment of the vehicle for the image of the second set of images; responsive to the second set of images having a second size equal to the first size, determining, by the at least one processor, the third plurality of environmental indicators satisfy the consistency threshold; and responsive to detecting a change between the second plurality of environmental indicators and the third plurality of environmental indicators, generating, by the at least one processor, a second record identifying the third plurality of environmental indicators.
Tariq et al. teaches wherein the first set of images has a first size, and further comprising:
receiving, by the at least one processor subsequent to receiving the first set of images, a second set of images from the camera; ([0084] “the first stage may include providing a first batch of images to the ML model for training, and the second stage may include providing a second batch of images to the ML model for training. That is to say, different batches of data may be used to subsequently train the model.”)
executing, by the at least one processor, the machine learning model using each image of
the second set of images as input to generate a third plurality of environmental indicators for the environment of the vehicle for the image of the second set of images; ([0087] “the first batch of images and/or the second batch of images may be data balanced to ensure that the ML model is being trained to accurately detect objects of different types. ML models that are continuously exposed to one object classification may slowly “forget” (e.g., become less accurate in detecting) objects of a different type to which they are exposed less frequently. Furthermore, where few examples exist in a dataset of one particular class (e.g., if cars heavily dominate a dataset with respect to bicycles), the ML model may optimize losses by simply stating that there are no bicycles in any image. Therefore, in some instances, the first batch of images and/or the second batch of images may include a first predefined number of images that are associated with a first object classification (e.g., each of the first predefined number of images include at least one pedestrian) and a second predefined number of images that are associated with a second object classification (e.g., each of the second predefined number of images include at least one biker), though any number of classes and relative weights are contemplated . . . training the ML model may include receiving an image, determining an object classification associated with the image, and associating the image with a training queue that is associated with the object classification. At training time, the batches may be composed from outputs of multiple queues, where each queue is associated with a different object classification and/or image condition (e.g., night/day, obstruction). By ensuring that the ML model is continuously training on a variety of object classifications, the ML model is less likely to “forget” an object classification.” and [0090] “a first ML model that indicates the accuracy of the first ML model to classify (e.g., produce an ROI and/or confidence score for) an object over apparent object size in an image. To produce this response curve, the techniques may include providing, as input to the first ML model, a plurality of objects (represented in one or more images); receiving, as output from the first ML model, a plurality of ROIs and/or confidence scores associated with the plurality of objects; and determining an accuracy for each of the ROIs and/or confidence scores (e.g., by comparing to a ground truth, by computing a degree of alignment for each ROI to a respective ground truth area.”)
responsive to the second set of images having a second size equal to the first size, ([0087]
“the first batch of images and/or the second batch of images may be data balanced to ensure that the ML model is being trained to accurately detect objects of different types.”) determining, by the at least one processor, the third plurality of environmental indicators satisfy the consistency threshold; and ([0023] “the ML model may additionally, or alternatively, output confidence scores (or confidence information) associated with each of the multiple ROIs. For example, the ML model may detect the presence of an object from a portion of the image, and may generate an ROI to indicate where that object is in the image. The ML model may additionally, or alternatively, produce a confidence score that essentially indicates how confident the ML model is that the ML model actually identified a salient object in the image and/or how well the ROI fits the object. For example, a confidence score may be a value between 0 and 1, where 0 represents an indication by the ML model that the ML model is not at all confident that an object appears in the ROI and where 1 indicates that the ML model is strongly confident that an object appears in the ROI, although other permutations are contemplated. In other words, the ML model outputs an indication of where it thinks an object might be and a score that indicates how confident the ML model is that it correctly identified an object and/or how well the ROI points out where the object is in the image.”, and [0091] “a batch of thousands or, even, hundreds of thousands of images may be provided, as input, to the first ML model so that this response curve may be determined. Once the first ML model has processed the batch, the techniques may include identifying a range of ROI sizes associated with accuracies that meet or exceed a threshold accuracy, and associating the range of sizes with the first ML model. This range of sizes may indicate that the first ML model determines “good” ROIs for objects in that range of sizes. . .”, and see at least [0100])
It would have been obvious to one of ordinary skill in the art before the effective filling date
of the claimed invention with reasonable expectations of success to modify the invention of Bajpayee et al. in combination with Bianconcini et al. to incorporate computer vision training techniques, including data balanced image sets, as taught by Tariq et. al. for the purpose of “increasing the accuracy of object detection and decreasing the compute time for obtaining accurate object identifications so that objects may be detected in real time for use in applications such as autonomous vehicle control” ([0021], Tariq).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MISA HUYNH NGUYEN whose telephone number is (571)270-5604. The examiner can normally be reached Monday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anne Antonucci can be reached at (313) 446-6519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MISA H NGUYEN/Examiner, Art Unit 3666
/ANNE MARIE ANTONUCCI/Supervisory Patent Examiner, Art Unit 3666
