DETAILED ACTION
This action is written in response to the remarks and amendments dated 12/23/25. This action is made final. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
In view of the Applicant’s arguments—as well as the latest claim amendments—the Examiner withdraws all outstanding rejections under §101.
The Applicants argue that the previous art of record does not anticipate or render obvious the claims as currently amended. The Examiner provides updated prior art rejections below necessitated by the current amendments.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
The following are the references relied upon in the rejections below:
Kwon (US 2020/0218979 A1)
Meyer (Meyer, Gregory P., et al. "Lasernet: An efficient probabilistic 3d object detector for autonomous driving." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.)
Ros Sanchez (US 2019/0317510 A1)
Urtasun (US 2021/0276587 A1)
Villegas (US 2020/0082248 A1)
Zehr (US 10,963,939 B1)
Claims 1, 6 and 22-23 are rejected under 35 U.S.C. 103 as being unpatentable over Ros Sanchez and Zehr.
Regarding claims 1 and 22-23, Ros Sanchez discloses a computer-implemented method (and a related system and non-transitory media) of modelling a perception system, the perception system configured to receive sensor data and interpret the sensor data to generate actual perception outputs, the method comprising:
receiving a plurality of input samples, wherein each input sample comprises sensor data and is associated with one or more training perception ground truths pertaining to one or more ground truth objects;
[0014] “FIG. 5 illustrates a sensor data image captured by a vehicle, according to aspects of the present disclosure.”
[0060] “A method 700 of FIG. 7 begins at block 702, in which a detection module Mi is trained according to a set of training data De." (Emphasis added.)
providing the sensor data of each input sample to the perception system to be modelled, wherein the perception system interprets the sensor data, in order to generate one or more actual perception outputs for the input sample; and
Fig. 5 (reproduced below).
PNG
media_image1.png
602
566
media_image1.png
Greyscale
‘actual perception outputs’ :: bounding boxes in fig. 5.
training a function approximator to model the perception system, including:
for each input sample, inputting the training perception ground truths to the function approximator, wherein the function approximator computes one or more predicted perception values by processing the training perception ground truths but not the sensor data from which the actual perception outputs are generated, and
See fig. 6 (reproduced below).
PNG
media_image2.png
274
796
media_image2.png
Greyscale
[0055] “FIG. 6 is a block diagram of an autonomous agent system configured for full-stack verification using system simulated sensor information, according to aspects of the present disclosure. In this configuration, an autonomous agent system 600 (or alternatively an autonomous vehicle system) is configured to replace an object detector module (e.g., 460) with a neural network 620 (e.g., a surrogate function 1) trained to generate simulated sensor information data in response to ground truth information 610. Using the neural network 620 in place of an object detector module (e.g., 460) enables replacing of a sensory signal input (e.g., 412). Instead, the ground truth information 610 is provided to the neural network 620 through a ground truth (GT) abstract signal 612. “
[0062] “At block 708, a surrogate function Φ is learned based on a set of outputs of the detection module Mi along with an expected response and a vector of measurements. For example, the set of outputs of the detection module is: M(oi)=ŷi, (e.g., a noisy output ŷi) along with the expected response yi (e.g., ground truth), and the vector of measurements mi are used to learn a surrogate function φθ (yi,mi), where θ represents a set of learned parameters that define the behavior of φ. This process may be carried out using minimization by standard optimization techniques based on gradient decent and risk minimization methods.”
[0063] “In this example, a given module Mi (e.g., a sensor module Mi) is used to learn a surrogate function φθ that does not depend on the sensory raw data oi. That is, the function φ is trained to mimic the signature (behavior) of M starting from an ideal estimation (ground truth). In other words, the function φ learns the noise model associated to Mi.”
adapting parameters of the function approximator, so as to match the corresponding predicted perception values corresponding to the input samples to the actual perception outputs for each of the input samples;
[0057] “In this aspect of the present disclosure, the neural network 620 is trained to provide a surrogate function 1 representing a noise model associated with an object detection module (e.g., 460), trained according to an object detection training data set. In operation, ground truth information 610 is fed to the neural network 620. The ground truth information 610 may represent two-dimensional (2D) and/or three-dimensional (3D) bounding boxes of detected objects in an area surrounding the autonomous vehicle 602. In response to the ground truth information 610 provided in the GT abstract signal 612, the neural network 620 generates a first abstract signal 622 according to the surrogate function 1, adding noise to the 2D bounding boxes of the ground truth information 610. The combination of the noise and 2D bounding boxes of the ground truth information 610 provides simulated sensory information for full-stack verification of the autonomous agent system 600.”
[0062] “At block 708, a surrogate function Φ is learned based on a set of outputs of the detection module Mi along with an expected response and a vector of measurements. For example, the set of outputs of the detection module is: M(oi)=ŷi, (e.g., a noisy output ŷi) along with the expected response yi (e.g., ground truth), and the vector of measurements mi are used to learn a surrogate function φθ (yi,mi), where θ represents a set of learned parameters that define the behavior of φ. This process may be carried out using minimization by standard optimization techniques based on gradient decent and risk minimization methods. Here, £ is an appropriate loss function, such as cross-entropy or lp-norm (see Equation (1)). In this aspect of the present disclosure, the purpose of φ is to learn the noise model associated with the output of Mi starting at an ideal prediction (ground truth).”
wherein the training perception ground truths associated with at least one of the input samples comprise first and second training perception ground truths pertaining to first and second ground truth objects respectively, …
Id.
Zehr discloses the following further limitation which Ros Sanchez does not disclose:
wherein at least one of the corresponding predicted perception values is computed from both the first and second training perception ground truths for modelling correlations between the first and second ground truth objects.
Claim 4, “determining correlations between the plurality of items, based at least in part on the respective determined attributes;”.
At the time of filing, it would have been obvious to a person of ordinary skill to apply the techniques disclosed by Zehr for measuring correlations between objects with the computer vision system of Ros Sanchez because the former can facilitate improved object recognition performance by leveraging information about co-occurring objects (eg a bicycle and a bicycle rider, or a tractor-trailer vehicle). Both disclosures pertain to machine vision.
Regarding claim 6, Ros Sanchez discloses the further limitation wherein:
the function approximator has a neural net architecture; and
[0069] “A method 900 begins in block 902, in which a neural network is trained to learn a noise model associated with an object detection module of an autonomous agent system.”
the function approximator optionally has a convolutional neural network (CNN) architecture, wherein the training perception ground truths are spatially encoded in at least one input tensor and the predicted perception values are spatially encoded in at least one output tensor computed from the at least one input tensor.
[This limitation is explicitly optional.]
Claims 2, 4, 11 and 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Ros Sanchez, Zehr and Villegas.
Regarding claim 2, Villegas discloses the following further limitation which neither Ros Sanchez/Zehr discloses wherein:
the at least one predicted perception value encodes:
one or more perception output distributions for sampling predicted perception outputs for one or more predicted perceived objects, or one or more predicted perception outputs for one or more predicted perceived objects; and
[0029] “The sensor data 102 may be used by an object detector 104 to detect, identify, and/or track objects in the environment. For example, a machine learning model(s) (e.g., a deep neural network trained for object detection), a computer vision algorithm, and/or another type of object detection algorithm may be used to detect objects in the environment. Locations, speeds, velocities, acceleration, yaw angles, and/or other information about each of the objects may be determined using the object detector 104.”
FIG. 1 shows that the output of the “object detector 104” which corresponds to “one or more predicted perceived objects” is passed to a “temporal encoder 106” and “spatial encoder 108” which performs “sequence encoding 110” and “conv. encoding 112”.
[0031] “The temporal encoder 106 may be used to learn dynamics of vehicle motion at each interval, frame, and/or time step. In some examples, the temporal encoder 106 may include a recurrent neural network (RNN), such as a long short-term memory (LSTM) network, that may receive object information for an object of the objects—provided as information relative to the vehicle 500, in embodiments—and generate an encoded state for each object.”
the function approximator optionally has an architecture such that [[the]] a number of predicted perceived objects for each input sample is not constrained to match [[the]] a number of ground truth objects for that input sample.
[0030] “In some examples, the object detector 104 and/or another detector may use the sensor data 102 to determine additional information about the objects and/or the environment. For example, with respect to objects, brake lights, blinkers, flashing lights, and/or other indicators of the object may be detected.”
[0032] “Although four objects 204 are illustrated in FIG. 2A, this is not intended to be limiting, and any number of objects may be present and/or tracked at any given time depending on the embodiment. For example, as described herein, each of the objects 204 within a certain distance (e.g., radius, straight-line distance, etc.) to the vehicle 500 (e.g., the object 204C) may be tracked, in addition to the vehicle 500 itself (e.g., where relative information is used).”
“ground truth objects” :: ‘objects’ in [0030]
At the time of filing, it would have been obvious to a person of ordinary skill to have replaced the machine-learned perception-prediction simulation model taught by Ros Sanchez/Zehr with an object detector with an architecture that does not constrain the number of input and output results and encodes perceived outputs as taught by Villegas because this would be a substitute of one known element (machine-learned perception-prediction simulation model) with another (an object detector) to obtain predictable results (predicting perceived objects) (MPEP 2141(III)(B) Simple substitution of one known element for another to obtain predictable results).
Regarding claim 4, Villegas discloses the following further limitation which neither Ros Sanchez/Zehr disclose wherein the number of predicted perceived objects is defined by the corresponding predicted perception values, or the corresponding predicted perception values encode at least one distribution for determining the number of predicted perceived objects via sampling.
[0099] “ Similarly, in some implementations, the prediction submodel (e.g., a multi-layer perceptron, etc.) of the machine-learned perception-prediction simulation model 210 can be utilized to determine the simulated prediction data of the simulated perception-prediction output 212 based at least in part on the simplified scenario data and the simulated perception data βsim(Wt, M). In some implementations, this simulated prediction data can include a simulated set of future states for each bounding box {tilde over (b)} i ∈ βsim(Wt, M) by determining one or more feature vectors respectively descriptive of the one or more simulated bounding boxes βsim(Wt, M).”
The obviousness analysis of claim 2 applies equally here.
Regarding claim 11, Villegas discloses the following further limitation which neither Ros Sanchez/Zehr discloses wherein a number of perceived objects of the output tensor is different than a number of ground truth objects encoded in the input tensor.
[0030] “In some examples, the object detector 104 and/or another detector may use the sensor data 102 to determine additional information about the objects and/or the environment. For example, with respect to objects, brake lights, blinkers, flashing lights, and/or other indicators of the object may be detected.”
[0032] “Although four objects 204 are illustrated in FIG. 2A, this is not intended to be limiting, and any number of objects may be present and/or tracked at any given time depending on the embodiment. For example, as described herein, each of the objects 204 within a certain distance (e.g., radius, straight-line distance, etc.) to the vehicle 500 (e.g., the object 204C) may be tracked, in addition to the vehicle 500 itself (e.g., where relative information is used).”
The obviousness analysis of claim 2 applies equally here.
Regarding claim 15, Villegas discloses the following further limitation which neither Ros Sanchez/Zehr discloses wherein:
the at least one predicted perception value encodes:
one or more perception output distributions for sampling predicted perception outputs for one or more predicted perceived objects, or one or more predicted perception outputs for one or more predicted perceived objects; and
[0029] “The sensor data 102 may be used by an object detector 104 to detect, identify, and/or track objects in the environment. For example, a machine learning model(s) (e.g., a deep neural network trained for object detection), a computer vision algorithm, and/or another type of object detection algorithm may be used to detect objects in the environment. Locations, speeds, velocities, acceleration, yaw angles, and/or other information about each of the objects may be determined using the object detector 104.”;
FIG. 1 shows that the output of the “object detector 104” which corresponds to “one or more predicted perceived objects” is passed to a “temporal encoder 106” and “spatial encoder 108” which performs “sequence encoding 110” and “conv. encoding 112””
[0031] “The temporal encoder 106 may be used to learn dynamics of vehicle motion at each interval, frame, and/or time step. In some examples, the temporal encoder 106 may include a recurrent neural network (RNN), such as a long short-term memory (LSTM) network, that may receive object information for an object of the objects—provided as information relative to the vehicle 500, in embodiments—and generate an encoded state for each object.”
the at least one predicted perception value encodes a predicted perception output or perception output distribution for an output object that is dependent on both the first and second perception ground truths for the first and second ground truth objects.
[0029] “The sensor data 102 may be used by an object detector 104 to detect, identify, and/or track objects in the environment. For example, a machine learning model(s) (e.g., a deep neural network trained for object detection), a computer vision algorithm, and/or another type of object detection algorithm may be used to detect objects in the environment. Locations, speeds, velocities, acceleration, yaw angles, and/or other information about each of the objects may be determined using the object detector 104.”
FIG. 1 shows that the output of the “object detector 104” which corresponds to “one or more predicted perceived objects” is passed to a “temporal encoder 106” and “spatial encoder 108” which performs “sequence encoding 110” and “conv. encoding 112”.
[0031] “The temporal encoder 106 may be used to learn dynamics of vehicle motion at each interval, frame, and/or time step. In some examples, the temporal encoder 106 may include a recurrent neural network (RNN), such as a long short-term memory (LSTM) network, that may receive object information for an object of the objects—provided as information relative to the vehicle 500, in embodiments—and generate an encoded state for each object.”
Regarding claim 16, Kwon discloses the following further limitation which neither Ros Sanchez/Zehr discloses comprising the step of generating the first and second perception ground truths based on a simulated scenario running in a simulator.
[0055] “An output of the object detector 214 may be locations of bounding shapes (e.g., bounding boxes, rectangles, squares, circles, polygons, etc.) corresponding to the objects or obstacles represented by the image data. Once the bounding shape locations and dimensions are known with respect to a particular image, additional sensor data 102—such as LIDAR data and/or RADAR data, as non-limiting examples—may be used to determine distances to the objects or obstacles corresponding to the respective bounding shapes.”; [0070] “Once a final distance value(s) has been selected for an object 306, one or more pixels of the image 302 may be encoded with the final depth value(s) to generate the ground truth depth map 222. In some non-limiting embodiments, to determine the one or more pixels to be encoded for the object 306, each of the pixels associated with the bounding shape 304 and/or the cropped bounding shape 308 may be encoded with the final distance value(s).”
[0056] “For example, because conversions from world-space outputs of one or more LIDAR sensors, RADAR sensors, SONAR sensors, etc. to image-space outputs of one or more cameras may be known, and because the locations and dimensions of bounding shapes within the image-space may be known, the distances (e.g., LIDAR distances 216, RADAR distances 218, etc.) may be correlated automatically with the objects and/or obstacles within the images. As an example, where a distance(s) to one or more objects in world-space is determined to correspond to a region of image-space associated with a bounding shape of an object, the distance(s) may be associated with the object for the purposes of ground truth encoding 110. In some examples, a single distance value may be correlated to each object”.
[0071] “Ground truth encoding 110 with respect to the predictions of the object detection(s) 116 may include labeling, or annotating, the sensor data 102 (e.g., images, depth maps, point clouds, etc.) with bounding shapes and/or corresponding class labels (e.g., vehicle, pedestrian, building, airplane, watercraft, street sign, etc.). As such, the ground truth annotations or labels may be compared, using loss function(s) 108, to the predictions of the object detection(s) 116 by the machine learning model(s) 104 to update and optimize the machine learning model(s) 104 for predicting locations of objects and/or obstacles.”
Regarding claim 17, Villegas discloses the following further limitation which neither Ros Sanchez/Zehr discloses wherein the function approximator has a convolutional neural network (CNN) architecture, wherein the first and second perception ground truths are spatially encoded in the same a particular input tensor, wherein function approximator computes therefrom an output tensor that encodes one or more perceived object objects, such that a number of predicted objects is not constrained to match the number of ground truth objects.
[0030] “In some examples, the object detector 104 and/or another detector may use the sensor data 102 to determine additional information about the objects and/or the environment. For example, with respect to objects, brake lights, blinkers, flashing lights, and/or other indicators of the object may be detected.”
[0032] “Although four objects 204 are illustrated in FIG. 2A, this is not intended to be limiting, and any number of objects may be present and/or tracked at any given time depending on the embodiment. For example, as described herein, each of the objects 204 within a certain distance (e.g., radius, straight-line distance, etc.) to the vehicle 500 (e.g., the object 204C) may be tracked, in addition to the vehicle 500 itself (e.g., where relative information is used).”
Claims 7, 13 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Ros Sanchez, Zehr and Uratsun.
Regarding claim 7, Urtasun discloses the following further limitation which Ros Sanchez/Zehr do not disclose:
wherein one or more ground truth perception layers of the at least one input tensor spatially encode the one or more training perception ground truths for each of the one or more ground truth object(s) objects;
[0021] “ Rather than utilizing raw sensor data, simplified scenario data can be obtained that is associated with or otherwise represents an autonomous vehicle driving scenario. In some implementations, the simplified scenario can be or otherwise include an intermediate output associated with a scenario for a perception and a prediction system of an autonomous vehicle. As an example, autonomous vehicle perception and prediction systems can generally receive sensor data (e.g., LIDAR sensors, ultrasonic sensors, image sensors, etc.) and generate intermediate outputs that represent or otherwise describe an environment and any actors about the autonomous vehicle (e.g., objects, transportation network segments, weather, etc.).”
FIG 4. displays that the simplified scenario data is utilized for “determining simulated perception data using a perception submodel”
[0035]-[0036] “A loss function can be evaluated that includes a perception loss term and a prediction loss term. The perception loss term can evaluate a difference between the simulated perception data and ground truth perception data…In some implementations, with box regression a smooth l1 loss can be utilized….”
[0038] “One or more parameters of the machine-learned perception-prediction simulation model can be adjusted based at least in part on the loss function…The one or more parameters can be updated to reduce the difference evaluated by the loss function”
Villegas discloses the following further limitation which neither Ros Sanchez/Zehr/Uratsun discloses:
wherein one or more predicted perception layers of the at least one output tensor are computed from at least the ground truth perception layers, and matched to one or more actual perception layers that spatially encode the actual perception outputs, by optimizing a regression loss defined on the predicted and actual perception layers;
[0045] “Referring again to FIG. 1, and with respect to convolutional encoding 112, for each object, a spatial arrangement (e.g., grid 212 for the object 204C of FIG. 2A) may be defined relative to a current object being predicted…The spatial arrangement corresponding to the object to be predicted may then be applied to a machine learning model, such as a convolutional neural network (CNN), to generate a spatial encoding.”
[0047] “One or more of the layers of the CNN 242 may include an input layer. The input layer(s) may hold values associated with the tensor generated from the spatial arrangement.”
[0051] “One or more of the layers may include a fully connected layer. Each neuron in the fully connected layer(s) may be connected to each of the neurons in the previous volume. The fully connected layer may compute class scores, and the resulting volume may be 1×1×number of classes.”
wherein, optionally, a ground truth classification layer of the at least one input tensor encodes a ground truth map of the ground truth objects, and
This limitation is optional.
wherein, optionally, a predicted classification layer of the at least one output tensor is computed from at least the ground truth classification layer, and matched to an actual object map by optimizing a classification loss defined on the predicted classification layer and the actual object maps, wherein the predicted classification layer is used to mask the perception layers of the at least one output tensor when optimizing the regression loss.
This limitation is optional.
At the time of filing, it would have been obvious to a person of ordinary skill to have replaced the machine-learned perception-prediction simulation model taught by Ros Sanchez/Zehr/Uratsun with an object detector with an architecture that does not constrain the number of input and output results and encodes perceived outputs as taught by Villegas because this would be a substitute of one known element (machine-learned perception-prediction simulation model) with another (an object detector) to obtain predictable results (predicting perceived objects) (MPEP 2141(III)(B) Simple substitution of one known element for another to obtain predictable results).
Regarding claim 13, Urtasun discloses the following further limitation which Ros Sanchez/Zehr do not disclose wherein, for each input sample:
one or more confounder values are inputted to the function approximator with the training perception ground truths, the confounder values representing one or more physical conditions in which the sensor data of the training sample was captured for learning an effect of those the conditions on the actual perception outputs; and/or
[This limitation and the following limitation constitute a Markush group. The Examiner maps only the second limitation.]
an indication of environmental structure in which the sensor data was captured is inputted with the training perception ground truths for modelling the effect of the environmental structure on the actual perception outputs.
[0064] “In some implementations, the sensor(s) 135 can include one or more internal sensors. The internal sensor(s) can be configured to acquire sensor data 155….”
[0065] “In some implementations, the sensor data 155 can be indicative of one or more objects within the surrounding environment of the vehicle 105. The object(s) can include, for example, vehicles, pedestrians, bicycles, and/or other objects.”
[0069] “The vehicle computing system 110…can process the sensor data 155…to obtain perception data 175A. The vehicle computing system 110 can generate perception data 175A that is indicative of one or more states (e.g., current and/or past state(s)) of one or more objects that are within a surrounding environment of the vehicle 105…The vehicle computing system 110 can utilize one or more algorithms and/or machine-learned model(s) that are configured to identify object(s) based at least in part on the sensor data 155. This can include, for example, one or more neural networks trained to identify object(s) within the surrounding environment of the vehicle 105 and the state data associated therewith. The perception data 175A can be utilized for the prediction function 175B of the autonomy computing system 140.”
At the time of filing, it would have been obvious to a person of ordinary skill to apply the technique disclosed by Uratsun to the Ros Sanchez/Zehr system because the former would provide for effective encoding of environmental information, which can prove useful in subsequent control decision making.
Regarding claim 19, Uratsun discloses the following further limitation which neither Ros Sanchez/Zehr discloses wherein the simulated scenario is performed to test performance of a robotic system in a presence of perception error, the robotic system controlling a simulated agent of the simulated scenario based on the corresponding predicted perception values, wherein a test oracle assesses the performance of the robotic system based on a behaviour of the simulated agent, without applying the perception system to the simulated scenario.
[0002] “The present disclosure relates generally to vehicle services and, more particularly, safety testing using simulated autonomous vehicle systems”; FIG. 5 step 506 “providing the simulated perception-prediction output to a motion planning system of the autonomous vehicle” corresponds to “controlling a simulated agent of the simulated scenario based on the corresponding predicted perception values” in which “perception-prediction output” corresponds to “the corresponding predicted perception values” and “the autonomous vehicle” corresponds to “a simulated agent”; FIG. 5 step 502 states “obtaining simplified scenario data associated with a simulated scenario…of an autonomous vehicle” and “safety testing using simulated autonomous vehicle systems” corresponds to “the simulated scenario is performed to test performance of a robotic system in the presence of perception error”.
[0042] “The motion plan can describe a series of movements over time that, when implemented by the vehicle, are configured to avoid the obstruction. The computing system can then determine a safety metric that evaluates a level of performance associated with the series of movements described by the motion plan (e.g., a level of safety, a level of efficiency, etc.).”
[0043] “The computing system can evaluate a loss function that evaluates a difference between the motion plan output of the autonomous vehicle and a ground truth label. The ground truth label can be or otherwise describe an optimal motion plan for the autonomous vehicle based on the simulated perception-prediction output.”
Regarding claim 20, Uratsun discloses the following further limitation wherein the test oracle applies a set of predetermined safety rules, to determine whether the simulated agent exhibited unsafe behaviour in the simulated scenario under the control of the robotic system being tested.
[0002] “The present disclosure relates generally to vehicle services and, more particularly, safety testing using simulated autonomous vehicle systems”
[0042] “The motion plan can describe a series of movements over time that, when implemented by the vehicle, are configured to avoid the obstruction. The computing system can then determine a safety metric that evaluates a level of performance associated with the series of movements described by the motion plan (e.g., a level of safety, a level of efficiency, etc.).”
[0043] “The computing system can evaluate a loss function that evaluates a difference between the motion plan output of the autonomous vehicle and a ground truth label. The ground truth label can be or otherwise describe an optimal motion plan for the autonomous vehicle based on the simulated perception-prediction output.”
Claims 8-9, 12 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Ros Sanchez, Zehr, Uratsun and Kwon.
Regarding claim 8, Kwon discloses the following further limitation which Ros Sanchez/Zehr/Uratsun do not disclose wherein the first and second training perception ground truths pertaining to the first and second ground truth objects are spatially encoded in the same a particular input tensor, wherein the first perception ground truth is redundantly encoded in each pixel of multiple pixels of the particular input tensor within a first ground truth object region, and the second perception ground truth is redundantly encoded in each pixel of multiple pixels of the particular input tensor within a second ground truth object region.
[0070] “Once a final distance value(s) has been selected for an object 306, one or more pixels of the image 302 may be encoded with the fmal depth value(s) to generate the ground truth depth map 222. In some non-limiting embodiments, to determine the one or more pixels to be encoded for the object 306, each of the pixels associated with the bounding shape 304 and/or the cropped bounding shape 308 may be encoded with the final distance value(s). However, in such examples, where two or more bounding shapes 304 and/or cropped bounding shapes 308 at least partially overlap (e.g., one bounding shape 304 occludes another), using each of the pixels of the bounding shape 304 and/or the cropped bounding shape 308 may result in one or more of the objects 306 not being represented sufficiently in the ground truth depth map 222. As such, in some embodiments, a shape—such as a circle or ellipse—may be generated for each object. The shape, in some examples, may be centered at a centroid of the bounding shape 304 and/or the cropped bounding shape 308. By generating a circle or ellipse, the potential for occlusion leading to lack of representation of an object 306 in the ground truth depth map 222 may be reduced, thereby increasing the likelihood that each of the objects 306 are represented in the ground truth depth map 222. As a result, the ground truth depth map 222 may represent the ground truth distance(s) encoded onto an image—e.g., a depth map image. The ground truth depth map 222—or depth map image—may then be used as ground truth for training the machine learning model(s) 104 to predict distances to objects using images generated by one or more cameras. As such, the machine learning model(s) 104 may be trained to predict—in deployment—a depth map corresponding to the objects and/or obstacles depicted in images captured by the vehicle 2100 (and/or another vehicle type, a robot, a simulated vehicle, a water vessel, an aircraft, a drone, etc.)”
“a depth map” includes “one or more of the objects 306” and it encodes the pixels corresponding to each object, this corresponds to “the first and second training perception ground truths pertaining to the first and second ground truth objects are spatially encoded in the same input tensor an object 306” corresponds to “first perception ground truth”;
a “bounding shape 304 and/or cropped bounding shape 308” corresponds to “a first ground truth object region one or more of objects 306” corresponds to “second perception ground truth bounding shapes 304” corresponds to “a second ground truth object region”
At the time of filing, it would have been obvious to a person of ordinary skill to utilized encoded bounding boxes as disclosed by Urtasun and Villegas redundantly for each pixel for each object as taught by Kwon because doing so would ensure safe operation of a vehicle by enabling correct detection between distances of a vehicle and its surrounding environment (See Kwon [0003]).
Regarding claim 9, Kwon discloses the following further limitation which Ros Sanchez/Zehr/Uratsun do not disclose wherein the first training perception ground truth comprises first ground truth 3D bounding box coordinates, which are used to determine the first ground truth object region of the particular input tensor, and which are redundantly encoded as numerical values in the ground truth perception layers at each pixel of the first ground truth object region;
[0055] “An output of the object detector 214 may be locations of bounding shapes (e.g., bounding boxes, rectangles, squares, circles, polygons, etc.) corresponding to the objects or obstacles represented by the image data. Once the bounding shape locations and dimensions are known with respect to a particular image, additional sensor data 102—such as LIDAR data and/or RADAR data, as non-limiting examples—may be used to determine distances to the objects or obstacles corresponding to the respective bounding shapes.”
[0070] “Once a final distance value(s) has been selected for an object 306, one or more pixels of the image 302 may be encoded with the final depth value(s) to generate the ground truth depth map 222. In some non-limiting embodiments, to determine the one or more pixels to be encoded for the object 306, each of the pixels associated with the bounding shape 304 and/or the cropped bounding shape 308 may be encoded with the final distance value(s).”
wherein the second training perception ground truth comprises second ground truth 3D bounding box coordinates, which are used to determine the second ground truth object region of the particular input tensor, and which are redundantly encoded as numerical values in the ground truth perception layers at each pixel of the first second ground truth object region.
Id. at [0055]
Regarding claim 12, Kwon discloses the following further limitation which neither Ros Sanchez/Zehr discloses further comprising the step of generating the training perception ground truths for each input sample via manual, automatic or semi-automatic annotation.
[0071] “Ground truth encoding 110 with respect to the predictions of the object detection(s) 116 may include labeling, or annotating, the sensor data 102 (e.g., images, depth maps, point clouds, etc.) with bounding shapes and/or corresponding class labels (e.g., vehicle, pedestrian, building, airplane, watercraft, street sign, etc.). As such, the ground truth annotations or labels may be compared, using loss function(s) 108, to the predictions of the object detection(s) 116 by the machine learning model(s) 104 to update and optimize the machine learning model(s) 104 for predicting locations of objects and/or obstacles.”
At the time of filing, it would have been obvious to a person of ordinary skill to have incorporated annotating (as taught by Kwon) the sensor data mention by Ros Sanchez/Zehr because this would ensure safe operation of a vehicle by enabling correct detection between distances of a vehicle and its surrounding environment (Kwon [0003]).
Regarding claim 14, Kwon discloses the following further limitation which neither Ros Sanchez/Zehr discloses comprising realistically simulating perception outputs of the modelled perception system by:
receiving a ground truth set that encodes at least a first perception ground truth for a first ground truth object and a second perception ground truth for a second ground truth object; and
[0055] “An output of the object detector 214 may be locations of bounding shapes (e.g., bounding boxes, rectangles, squares, circles, polygons, etc.) corresponding to the objects or obstacles represented by the image data. Once the bounding shape locations and dimensions are known with respect to a particular image, additional sensor data 102—such as LIDAR data and/or RADAR data, as non-limiting examples—may be used to determine distances to the objects or obstacles corresponding to the respective bounding shapes.”.
[0070] “Once a final distance value(s) has been selected for an object 306, one or more pixels of the image 302 may be encoded with the final depth value(s) to generate the ground truth depth map 222. In some non-limiting embodiments, to determine the one or more pixels to be encoded for the object 306, each of the pixels associated with the bounding shape 304 and/or the cropped bounding shape 308 may be encoded with the final distance value(s).”
processing the ground truth set, by the trained function approximator, and thereby computing one or more corresponding predicted perception values for the ground truth set, without applying the perception system, wherein at least one of the corresponding predicted perception values is computed from both the first and second perception ground truths for modelling correlations between the first and second ground truth objects.
[0055] “An output of the object detector 214 may be locations of bounding shapes (e.g., bounding boxes, rectangles, squares, circles, polygons, etc.) corresponding to the objects or obstacles represented by the image data. Once the bounding shape locations and dimensions are known with respect to a particular image, additional sensor data 102—such as LIDAR data and/or RADAR data, as non-limiting examples—may be used to determine distances to the objects or obstacles corresponding to the respective bounding shapes.”
[0056] “For example, because conversions from world-space outputs of one or more LIDAR sensors, RADAR sensors, SONAR sensors, etc. to image-space outputs of one or more cameras may be known, and because the locations and dimensions of bounding shapes within the image-space may be known, the distances (e.g., LIDAR distances 216, RADAR distances 218, etc.) may be correlated automatically with the objects and/or obstacles within the images. As an example, where a distance(s) to one or more objects in world-space is determined to correspond to a region of image-space associated with a bounding shape of an object, the distance(s) may be associated with the object for the purposes of ground truth encoding 110. In some examples, a single distance value may be correlated to each object”
[0071] “Ground truth encoding 110 with respect to the predictions of the object detection(s) 116 may include labeling, or annotating, the sensor data 102 (e.g., images, depth maps, point clouds, etc.) with bounding shapes and/or corresponding class labels (e.g., vehicle, pedestrian, building, airplane, watercraft, street sign, etc.). As such, the ground truth annotations or labels may be compared, using loss function(s) 108, to the predictions of the object detection(s) 116 by the machine learning model(s) 104 to update and optimize the machine learning model(s) 104 for predicting locations of objects and/or obstacles.”
Claims 10 is rejected under 35 U.S.C. 103 as being unpatentable over Ros Sanchez, Zehr and Meyer.
Regarding claim 10, Meyer discloses the following further limitation which neither Ros Sanchez/Zehr discloses wherein the output tensor encodes distribution parameters for sampling 3D bounding boxes for one or more predicted objects.
P. 12677, Abstract, “Our approach uses a fully convolutional network to predict a multimodal distribution over 3D boxes for each point and then it efficiently fuses these distributions to generate a prediction for each object.”
At the time of filing, it would have been obvious to a person of ordinary skill to replace the encoded state disclosed by Ros Sanchez/Zehr with encoded distribution parameters as disclosed by Meyer for each bounding box for each object. This would be a substitute of one known element (encoded states) with another (encoded distributions) to obtain predictable results (encoding perceived objects).
Claims 18 is rejected under 35 U.S.C. 103 as being unpatentable over Ros Sanchez, Zehr, Villegas and Kwon.
Regarding claim 18, Kwon discloses the following further limitation which neither Ros Sanchez/Zehr/Villegas discloses wherein the first and second perception ground truths comprise first and second bounding boxes respectively, which are used to determine first and second ground truth object regions of the particular input tensor respectively, and wherein coordinates of the first and second bounding boxes are spatially encoded with redundancy in each pixel of multiple pixels of the first and second ground truth object regions respectively.
[0055] “An output of the object detector 214 may be locations of bounding shapes (e.g., bounding boxes, rectangles, squares, circles, polygons, etc.) corresponding to the objects or obstacles represented by the image data. Once the bounding shape locations and dimensions are known with respect to a particular image, additional sensor data 102—such as LIDAR data and/or RADAR data, as non-limiting examples—may be used to determine distances to the objects or obstacles corresponding to the respective bounding shapes.”.
[0070] “Once a final distance value(s) has been selected for an object 306, one or more pixels of the image 302 may be encoded with the final depth value(s) to generate the ground truth depth map 222. In some non-limiting embodiments, to determine the one or more pixels to be encoded for the object 306, each of the pixels associated with the bounding shape 304 and/or the cropped bounding shape 308 may be encoded with the final distance value(s).”
Additional Relevant Prior Art
The following references were identified by the Examiner as being relevant to the disclosed invention, but are not relied upon in any particular prior art rejection:
Redmon discloses using bounding boxes to detect perceived objects and matching perceived objects with ground truth data. (Redmon, Joseph, et al. "You only look once: Unified, real-time object detection." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.)
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Vincent Gonzales whose telephone number is (571) 270-3837. The examiner can normally be reached on Monday-Friday 7 a.m. to 4 p.m. MT. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang, can be reached at (571) 270-7092.
Information regarding the status of an application may be obtained from the USPTO Patent Center.
/Vincent Gonzales/Primary Examiner, Art Unit 2124
Prosecution Insights