DETAILED ACTION
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4-7, 9-11, 13, 16-19 are rejected under 35 U.S.C. 103 as being unpatentable over Xu (WO 2019060125 A1, as provided in IDS), in view of Douillard (US 20170124781 A1, as provided in IDS), and further in view of Chang (US 20200311979 A1, as provided in IDS).
Re Claim 1, Xu discloses an image processing method (see Xu: e.g., Fig. 2, -- a first processing algorithm 212 configured to receive the point cloud 204. In some implementations of this disclosure, the first processing algorithm 212 may include an artificial neural network (e.g., a convolutional neural network) configured to receive the point cloud and analyze the points. For example, the first processing algorithm 212 may be a PointNet network. PointNet is a deep network architecture that receives raw point cloud data and learns both global and local point features. PointNet has been used conventionally for classification, part segmentation, and semantic segmentation purposes. For purposes of this disclosure, however, the first processing algorithm 212 may be configured to produce a feature vector associated with the point cloud. For instance, when PointNet is used as the first processing algorithm 212, feature vectors may be produced at one of several layers before a prediction layer. The process 200 may extract one or more of these feature vectors, as illustrated at 214. The feature vectors 214 may be purely geometric feature vectors, associated only with the location of the points in the point cloud 206.--, in [0020]-[0028]), comprising:
obtaining an image from a camera located on a vehicle (see Xu: e.g., -- [0010] Autonomous vehicle systems may include an array of different types of sensors to detect, track and identify objects and/or attributes of objects. For instance, sensors, such as LIDAR and RADAR, ultrasonic transducers, depth cameras, and the like can provide three- dimensional information about objects in an environment and sensors such as conventional cameras can provide two-dimensional information about the environment. For instance, a LIDAR system may have a light emitter and a light sensor, with the light emitter including one or more lasers that direct highly focused light toward an object or surface which reflects the light back to the light sensor…. image capture devices may provide 2D image data, such as RGB image data, greyscale image data, or otherwise, about the environment.--, in [0010]);
Xu does not however explicitly disclose determining a location of the vehicle in a map,
Douillard discloses determining a location of the vehicle in a map (see Douillard: e.g., -- Autonomous vehicle controller 147 may be further configured to determine a local pose (e.g., local position) of an autonomous vehicle 109 and to detect external objects relative to the vehicle….A localizer (not shown) of autonomous vehicle controller 147 can determine a local pose at the geographic location 111. As such, the localizer may use acquired sensor data, such as sensor data associated with surfaces of buildings 115 and 117, which can be compared against reference data, such as map data (e.g., 3D map data, including reflectance data) to determine a local pose.--, in [0052]-[0054], and, --Further, autonomous vehicle controller 347a may receive any other sensor data 356, as well as reference data 339. In some cases, reference data 339 includes map data (e.g., 3D map data, 2D map data, 4D map data (e.g., including Epoch Determination)) and route data (e.g., road network data, including, but not limited to, RNDF data (or similar data), MDF data (or similar data), etc.
[0063] Localizer 368 is configured to receive sensor data from one or more sources, such as GPS data 352, wheel data, IMU data 354, Lidar data 346a, camera data 340a, radar data 348a, and the like, as well as reference data 339 (e.g., 3D map data and route data). Localizer 368 integrates (e.g., fuses the sensor data) and analyzes the data by comparing sensor data to map data to determine a local pose (or position) of bidirectional autonomous vehicle 330. According to some examples, localizer 368 may generate or update the pose or position of any autonomous vehicle in real-time or near real-time. Note that localizer 368 and its functionality need not be limited to “bi-directional” vehicles and can be implemented in any vehicle of any type.--, in [0062]-[0063], and, -- the perception data may include an obstacle map specifying static and dynamic objects located in the vicinity of an autonomous vehicle, whereas the localizer data may include a local pose or position. In operation, planner 364 generates numerous trajectories, and evaluates the trajectories, based on at least the location of the autonomous vehicle against relative locations of external dynamic and static objects….- one or more cameras 474, one or more radars 476, one or more global positioning system (“GPS”) data receiver-sensors, one or more inertial measurement units (“IMUs”) -, in [0065]-[0068]);
Xu and Douillard are combinable as they are in the same field of endeavor: image processing of image data captured by the camera of a vehicle and analysis of detected objects. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify Xu’s method using Douillard’s teachings by including determining a location of the vehicle in a map is obtained to Xu’s determining location in order to detect external objects relative to the vehicle (see Douillard: e.g. in [0052]-[0054], [0062]-[0063], and [0065]-[0068]);
Xu as modified by Douillard however still do not explicitly disclose selecting two points located at two distances from the location of the vehicle; and determining positions of the two points on the image by projecting a set of coordinates of the two points to the image;
Chang discloses selecting two points located at two distances from the location of the vehicle; and determining positions of the two points on the image by projecting a set of coordinates of the two points to the image (see Chang: e.g., Fig. 6, -- World coordinates of two reference points and image coordinates of two projection points corresponding to the two reference points are obtained. A plurality of coordinate transformation parameters relative to transformation between any image coordinates and any world coordinates corresponding to a camera are calculated according only to the world coordinates of the two reference points, the image coordinates of the two projection points, and world coordinates of the camera. A second image having an object image corresponding to an object is obtained through a camera. World coordinates of the object are positioned according to the coordinate transformation parameters.--, in [0008], and, -- The camera coordinate system is a three-dimensional coordinate system formed by treating the center point of the camera lens as the origin. In the camera coordinate system, the directions of the three axes in the three-dimensional coordinate system is defined corresponding to the left-handed coordinate system or the right-handed coordinate system.--, in [0021]-[0023]; and, --In step S210, the processor 130 obtains world coordinates of two reference points and image coordinates of two projection points corresponding to the two reference points. Further, in step S220, the processor 130 calculates a plurality of coordinate transformation parameters relative to transformation between any image coordinates and any world coordinates corresponding to the camera 110 according only to the world coordinates of the two reference points, the image coordinates of the two projection points, and world coordinates of the camera 110.--, in [0032], and [0044]-[0047]);
Xu (as modified by Douillard) and Chang are combinable as they are in the same field of endeavor: image processing of image data captured by the camera and analysis of detected objects . Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify Xu (as modified by Douillard)’s method using Chang’s teachings by including selecting two points located at two distances from the location of the vehicle; and determining positions of the two points on the image by projecting a set of coordinates of the two points to the image to Xu (as modified by Douillard)’s determining location of detected object in order to determine a location of the object of interest in the environment as world coordinates of the object (see Chang: e.g. in [0008], and [0021]-[0023], [0032], and [0044]-[0047]);
Xu as modified by Douillard and Chang further disclose cropping a portion of the image corresponding to a region that is identified based on the positions of the two points on the image (see Xu: e.g., Fig. 2, -- a machine learning algorithm is applied to the image data and the point cloud data, to estimate parameters for a three- dimensional bounding box associated with one or more objects in the environment. For instance, a first feature vector associated with the image data, for example, associated with a cropped image corresponding to the object of interest, and a second feature vector associated with the point cloud data may be input to the machine learning algorithm. The machine learning algorithm may output parameters of the three-dimensional bounding box. The parameters may include eight points in a coordinate system, the eight points representing the eight comers of the three-dimensional bounding box. An example machine learning algorithm used to recover the parameters is an artificial neural network (ANN), which may be a Convolutional Neural Network (CNN).--, in [0012]; and, -- the first processing algorithm 212 may be configured to produce a feature vector associated with the point cloud. For instance, when PointNet is used as the first processing algorithm 212, feature vectors may be produced at one of several layers before a prediction layer. The process 200 may extract one or more of these feature vectors, as illustrated at 214. The feature vectors 214 may be purely geometric feature vectors, associated only with the location of the points in the point cloud 206….. The second processing algorithm 216 may be configured to receive the vehicle image 210 and produce one or more appearance feature vectors 218 associated with the vehicle image 210. --, in [0022]-[0023]; also see: -- three-dimensional bounding box 226, defined by eight comers 228, is illustrated in FIG. 2. In another example, the ANN 222 may predict a center location, orientation, and three dimensional extents of such a bounding box. In such a manner, the ANN 222 may constrain the output to retain a rectangular volume shape…. provides a global architecture that directly regresses coordinates descriptive of a bounding box. FIG. 3 is a pictorial representation of a process 300, which, like the process 200, also determines parameters of a three-dimensional bounding box using a point cloud 302 and a cropped image 304 associated with an object.--, in [0024]-[0026]; and, -- one such object in the environment 100 is a vehicle 102. The environment 100 is associated with a coordinate system 104. The coordinate system 104 may be either global or local. In a global coordinate system, any point expressed in the coordinate system 104 is an absolute coordinate. Alternatively, in a local coordinate system points are expressed relative to an arbitrarily defined origin (such as a center of an autonomous vehicle as it travels through the environment), which may move in a global coordinate system.--, in [0017], [0024], and [0035], and, -- both the point cloud and the bounding box objective may be cropped (or otherwise altered to leave only the data within the two-dimensional bounding box in the image and related points in the point cloud (e.g., by reprojection using a known transformation between the two sensors)) and rotated to be centered along an axis of the sensors, e.g., a Z-axis.--, in [0042]; also see Chang: e.g., Fig. 6, -- World coordinates of two reference points and image coordinates of two projection points corresponding to the two reference points are obtained. A plurality of coordinate transformation parameters relative to transformation between any image coordinates and any world coordinates corresponding to a camera are calculated according only to the world coordinates of the two reference points, the image coordinates of the two projection points, and world coordinates of the camera. A second image having an object image corresponding to an object is obtained through a camera. World coordinates of the object are positioned according to the coordinate transformation parameters.--, in [0008], and, -- The camera coordinate system is a three-dimensional coordinate system formed by treating the center point of the camera lens as the origin. In the camera coordinate system, the directions of the three axes in the three-dimensional coordinate system is defined corresponding to the left-handed coordinate system or the right-handed coordinate system.--, in [0021]-[0023];see Douillard: e.g., -- an obstacle obscuring a roadway may be viewed as an event, as well as a reduction or loss of communication. An event may include traffic conditions or congestion, as well as unexpected or unusual numbers or types of external objects (or tracks) that are perceived by a perception engine. An event may include weather-related conditions (e.g., loss of friction due to ice or rain) or the angle at which the sun is shining (e.g., at sunset), such as low angle to the horizon that cause sun to shine brightly in the eyes of human drivers of other vehicles.--, in [0058]).
Re Claim 4, Xu as modified by Douillard and Chang further disclose wherein the positions of the two points on the image and a pre- defined left bound position and a pre-defined right bound position is used to obtain a cropping area of the region (see Xu: e.g., Fig. 2, and, -- a machine learning algorithm is applied to the image data and the point cloud data, to estimate parameters for a three- dimensional bounding box associated with one or more objects in the environment. For instance, a first feature vector associated with the image data, for example, associated with a cropped image corresponding to the object of interest, and a second feature vector associated with the point cloud data may be input to the machine learning algorithm. The machine learning algorithm may output parameters of the three-dimensional bounding box. The parameters may include eight points in a coordinate system, the eight points representing the eight comers of the three-dimensional bounding box. An example machine learning algorithm used to recover the parameters is an artificial neural network (ANN), which may be a Convolutional Neural Network (CNN).--, in [0012]; also see Chang: e.g., --in step S510 of FIG. 5, the processor 130 obtains an output image (a first image) from the camera 110 and obtains resolution information corresponding to the output image to obtain position of an image central point p.sub.C according to the resolution information. (43) Specifically, the resolution information corresponds to the width and height resolution of the output image. Therefore, after obtaining the width and height resolution of the image, the processor 130 transfers the location of image into coordinates according to the resolution information and analyzes the image. In this way, the processor 130 may further determine the image coordinates of the two projection points. For instance, in the image with the resolution being 1920×1080, the upper left corner being the origin, the horizontal axis extending from left to right, and the vertical axis extending from top to bottom, the position of the image central point is (960,540). If position of a projection point p.sub.A are (u′.sub.A,v′.sub.A) and position of a projection point p.sub.B are (u′.sub.B, v′.sub.B), in this embodiment, the processor 130 further sets the image coordinates of the image central point to be (0,0), the horizontal axis to be extending from left to right, and the vertical axis to be extending from bottom to top. That is, the coordinates of the upper left corner of the image are changed from (0, 0) to (−960, 540), the image coordinates of the projection point p.sub.A are (u.sub.A,v.sub.A)=(u′.sub.A−960,−v′.sub.A+540), and the image coordinates of the projection point p.sub.B are (u.sub.B,v.sub.B)=(u′.sub.B−960,−v′.sub.B+540).--, in [0051]-[0052]; and, : -- A reference distance from the lens central point P.sub.O of the camera to the first reference point P.sub.A is d.sub.1, a reference distance from the lens central point P.sub.O of the camera to the second reference point P.sub.B is d.sub.2, and a reference distance from the first reference point P.sub.A to the second reference point P.sub.B is d.sub.3. The lens central point P.sub.O of the camera vertically projects to the ground to form a vertical intersection point P.sub.Q, and a height from the lens central point P.sub.O of the camera to the vertical intersection point P.sub.Q is h.--, in [0053]).
Re Claim 5, Xu as modified by Douillard and Chang further disclose wherein in response to the vehicle being located within a pre- determined distance of the region, the positions of the two points are determined by using camera pose information of the camera to project the set of coordinates of the two points to the image (see Xu: e.g., Fig. 2, and, -- a machine learning algorithm is applied to the image data and the point cloud data, to estimate parameters for a three- dimensional bounding box associated with one or more objects in the environment. For instance, a first feature vector associated with the image data, for example, associated with a cropped image corresponding to the object of interest, and a second feature vector associated with the point cloud data may be input to the machine learning algorithm. The machine learning algorithm may output parameters of the three-dimensional bounding box. The parameters may include eight points in a coordinate system, the eight points representing the eight comers of the three-dimensional bounding box. An example machine learning algorithm used to recover the parameters is an artificial neural network (ANN), which may be a Convolutional Neural Network (CNN).--, in [0012]; also see Chang: e.g., --in step S510 of FIG. 5, the processor 130 obtains an output image (a first image) from the camera 110 and obtains resolution information corresponding to the output image to obtain position of an image central point p.sub.C according to the resolution information. (43) Specifically, the resolution information corresponds to the width and height resolution of the output image. Therefore, after obtaining the width and height resolution of the image, the processor 130 transfers the location of image into coordinates according to the resolution information and analyzes the image. In this way, the processor 130 may further determine the image coordinates of the two projection points. For instance, in the image with the resolution being 1920×1080, the upper left corner being the origin, the horizontal axis extending from left to right, and the vertical axis extending from top to bottom, the position of the image central point is (960,540). If position of a projection point p.sub.A are (u′.sub.A,v′.sub.A) and position of a projection point p.sub.B are (u′.sub.B, v′.sub.B), in this embodiment, the processor 130 further sets the image coordinates of the image central point to be (0,0), the horizontal axis to be extending from left to right, and the vertical axis to be extending from bottom to top. That is, the coordinates of the upper left corner of the image are changed from (0, 0) to (−960, 540), the image coordinates of the projection point p.sub.A are (u.sub.A,v.sub.A)=(u′.sub.A−960,−v′.sub.A+540), and the image coordinates of the projection point p.sub.B are (u.sub.B,v.sub.B)=(u′.sub.B−960,−v′.sub.B+540).--, in [0051]-[0052]; and, : -- A reference distance from the lens central point P.sub.O of the camera to the first reference point P.sub.A is d.sub.1, a reference distance from the lens central point P.sub.O of the camera to the second reference point P.sub.B is d.sub.2, and a reference distance from the first reference point P.sub.A to the second reference point P.sub.B is d.sub.3. The lens central point P.sub.O of the camera vertically projects to the ground to form a vertical intersection point P.sub.Q, and a height from the lens central point P.sub.O of the camera to the vertical intersection point P.sub.Q is h.--, in [0053]).
Re Claim 6, Xu as modified by Douillard and Chang further disclose wherein the set of coordinates are three-dimension (3D) world coordinates of the two points, and wherein the 3D world coordinates are obtained using the map and the location of the vehicle (see Chang: e.g., Fig. 6, -- World coordinates of two reference points and image coordinates of two projection points corresponding to the two reference points are obtained. A plurality of coordinate transformation parameters relative to transformation between any image coordinates and any world coordinates corresponding to a camera are calculated according only to the world coordinates of the two reference points, the image coordinates of the two projection points, and world coordinates of the camera. A second image having an object image corresponding to an object is obtained through a camera. World coordinates of the object are positioned according to the coordinate transformation parameters.--, in [0008], and, -- The camera coordinate system is a three-dimensional coordinate system formed by treating the center point of the camera lens as the origin. In the camera coordinate system, the directions of the three axes in the three-dimensional coordinate system is defined corresponding to the left-handed coordinate system or the right-handed coordinate system.--, in [0021]-[0023]; and, --In step S210, the processor 130 obtains world coordinates of two reference points and image coordinates of two projection points corresponding to the two reference points. Further, in step S220, the processor 130 calculates a plurality of coordinate transformation parameters relative to transformation between any image coordinates and any world coordinates corresponding to the camera 110 according only to the world coordinates of the two reference points, the image coordinates of the two projection points, and world coordinates of the camera 110.--, in [0032], and [0044]-[0047]; also see Douillard: e.g., -- Autonomous vehicle controller 147 may be further configured to determine a local pose (e.g., local position) of an autonomous vehicle 109 and to detect external objects relative to the vehicle….A localizer (not shown) of autonomous vehicle controller 147 can determine a local pose at the geographic location 111. As such, the localizer may use acquired sensor data, such as sensor data associated with surfaces of buildings 115 and 117, which can be compared against reference data, such as map data (e.g., 3D map data, including reflectance data) to determine a local pose.--, in [0052]-[0054], and, --Further, autonomous vehicle controller 347a may receive any other sensor data 356, as well as reference data 339. In some cases, reference data 339 includes map data (e.g., 3D map data, 2D map data, 4D map data (e.g., including Epoch Determination)) and route data (e.g., road network data, including, but not limited to, RNDF data (or similar data), MDF data (or similar data), etc.
[0063] Localizer 368 is configured to receive sensor data from one or more sources, such as GPS data 352, wheel data, IMU data 354, Lidar data 346a, camera data 340a, radar data 348a, and the like, as well as reference data 339 (e.g., 3D map data and route data). Localizer 368 integrates (e.g., fuses the sensor data) and analyzes the data by comparing sensor data to map data to determine a local pose (or position) of bidirectional autonomous vehicle 330. According to some examples, localizer 368 may generate or update the pose or position of any autonomous vehicle in real-time or near real-time. Note that localizer 368 and its functionality need not be limited to “bi-directional” vehicles and can be implemented in any vehicle of any type.--, in [0062]-[0063], and, -- the perception data may include an obstacle map specifying static and dynamic objects located in the vicinity of an autonomous vehicle, whereas the localizer data may include a local pose or position. In operation, planner 364 generates numerous trajectories, and evaluates the trajectories, based on at least the location of the autonomous vehicle against relative locations of external dynamic and static objects….- one or more cameras 474, one or more radars 476, one or more global positioning system (“GPS”) data receiver-sensors, one or more inertial measurement units (“IMUs”) -, in [0065]-[0068]).
Re Claim 7, Xu as modified by Douillard and Chang further disclose wherein the two points are located in a spatial region in front of the vehicle (see Xu: e.g., -- The second processing algorithm 216 may be configured to produce the feature vector(s) 218 associated with the vehicle image 210. In some examples, the process 200 may extract the feature vector(s) 218 from one of several layers of the residual learning network, before a prediction layer. For instance, the second processing algorithm 216 may be a ResNet-101 CNN and the feature vector 218 may be produced by the final residual block and averaged across feature map locations.--, in [0023], also see: Fig. 2, and, -- a machine learning algorithm is applied to the image data and the point cloud data, to estimate parameters for a three- dimensional bounding box associated with one or more objects in the environment. For instance, a first feature vector associated with the image data, for example, associated with a cropped image corresponding to the object of interest, and a second feature vector associated with the point cloud data may be input to the machine learning algorithm. The machine learning algorithm may output parameters of the three-dimensional bounding box. The parameters may include eight points in a coordinate system, the eight points representing the eight comers of the three-dimensional bounding box. An example machine learning algorithm used to recover the parameters is an artificial neural network (ANN), which may be a Convolutional Neural Network (CNN).--, in [0012]; and, -- the first processing algorithm 212 may be configured to produce a feature vector associated with the point cloud. For instance, when PointNet is used as the first processing algorithm 212, feature vectors may be produced at one of several layers before a prediction layer. The process 200 may extract one or more of these feature vectors, as illustrated at 214. The feature vectors 214 may be purely geometric feature vectors, associated only with the location of the points in the point cloud 206….. The second processing algorithm 216 may be configured to receive the vehicle image 210 and produce one or more appearance feature vectors 218 associated with the vehicle image 210. --, in [0022]-[0023]; also see: -- three-dimensional bounding box 226, defined by eight comers 228, is illustrated in FIG. 2. In another example, the ANN 222 may predict a center location, orientation, and three dimensional extents of such a bounding box. In such a manner, the ANN 222 may constrain the output to retain a rectangular volume shape…. provides a global architecture that directly regresses coordinates descriptive of a bounding box. FIG. 3 is a pictorial representation of a process 300, which, like the process 200, also determines parameters of a three-dimensional bounding box using a point cloud 302 and a cropped image 304 associated with an object.--, in [0024]-[0026]).
Re Claim 9, Xu as modified by Douillard and Chang further disclose wherein the location of the vehicle is associated with a time when the image is obtained see Douillard: e.g., -- Autonomous vehicle controller 147 may be further configured to determine a local pose (e.g., local position) of an autonomous vehicle 109 and to detect external objects relative to the vehicle….A localizer (not shown) of autonomous vehicle controller 147 can determine a local pose at the geographic location 111. As such, the localizer may use acquired sensor data, such as sensor data associated with surfaces of buildings 115 and 117, which can be compared against reference data, such as map data (e.g., 3D map data, including reflectance data) to determine a local pose.--, in [0052]-[0054], and, --Further, autonomous vehicle controller 347a may receive any other sensor data 356, as well as reference data 339. In some cases, reference data 339 includes map data (e.g., 3D map data, 2D map data, 4D map data (e.g., including Epoch Determination)) and route data (e.g., road network data, including, but not limited to, RNDF data (or similar data), MDF data (or similar data), etc.
[0063] Localizer 368 is configured to receive sensor data from one or more sources, such as GPS data 352, wheel data, IMU data 354, Lidar data 346a, camera data 340a, radar data 348a, and the like, as well as reference data 339 (e.g., 3D map data and route data). Localizer 368 integrates (e.g., fuses the sensor data) and analyzes the data by comparing sensor data to map data to determine a local pose (or position) of bidirectional autonomous vehicle 330. According to some examples, localizer 368 may generate or update the pose or position of any autonomous vehicle in real-time or near real-time. Note that localizer 368 and its functionality need not be limited to “bi-directional” vehicles and can be implemented in any vehicle of any type.--, in [0062]-[0063], and, -- the perception data may include an obstacle map specifying static and dynamic objects located in the vicinity of an autonomous vehicle, whereas the localizer data may include a local pose or position. In operation, planner 364 generates numerous trajectories, and evaluates the trajectories, based on at least the location of the autonomous vehicle against relative locations of external dynamic and static objects….- one or more cameras 474, one or more radars 476, one or more global positioning system (“GPS”) data receiver-sensors, one or more inertial measurement units (“IMUs”) -, in [0065]-[0068]).
Re Claim 10, Xu as modified by Douillard and Chang further disclose wherein the obtaining the image, the determining the location of the vehicle, the selecting the two points, the determining the positions of the two points, and the cropping the portion of the image are performed as the vehicle is being driven (see Xu: e.g., Fig. 2, an, -- a machine learning algorithm is applied to the image data and the point cloud data, to estimate parameters for a three- dimensional bounding box associated with one or more objects in the environment. For instance, a first feature vector associated with the image data, for example, associated with a cropped image corresponding to the object of interest, and a second feature vector associated with the point cloud data may be input to the machine learning algorithm. The machine learning algorithm may output parameters of the three-dimensional bounding box. The parameters may include eight points in a coordinate system, the eight points representing the eight comers of the three-dimensional bounding box. An example machine learning algorithm used to recover the parameters is an artificial neural network (ANN), which may be a Convolutional Neural Network (CNN).--, in [0012]; and, -- the first processing algorithm 212 may be configured to produce a feature vector associated with the point cloud. For instance, when PointNet is used as the first processing algorithm 212, feature vectors may be produced at one of several layers before a prediction layer. The process 200 may extract one or more of these feature vectors, as illustrated at 214. The feature vectors 214 may be purely geometric feature vectors, associated only with the location of the points in the point cloud 206….. The second processing algorithm 216 may be configured to receive the vehicle image 210 and produce one or more appearance feature vectors 218 associated with the vehicle image 210. --, in [0022]-[0023]; also see: -- three-dimensional bounding box 226, defined by eight comers 228, is illustrated in FIG. 2. In another example, the ANN 222 may predict a center location, orientation, and three dimensional extents of such a bounding box. In such a manner, the ANN 222 may constrain the output to retain a rectangular volume shape…. provides a global architecture that directly regresses coordinates descriptive of a bounding box. FIG. 3 is a pictorial representation of a process 300, which, like the process 200, also determines parameters of a three-dimensional bounding box using a point cloud 302 and a cropped image 304 associated with an object.--, in [0024]-[0026]; also see: [0019] -- Once image data is received from an image capture device, various algorithms (such as Single Shot Detector Multibox, Fast-CNN, Faster-R CNN, overfeat, region based fully -connected networks, etc.) may be applied to identify objects in the image, and in some implementations, two-dimensional bounding boxes. These algorithms may be selected to only identify certain object classes. For example, the algorithm may detect only cars, pedestrians, animals, or any combination thereof, though detection of any number of object classes is contemplated. As illustrated in FIG. 1, such an algorithm has detected an object, here, the vehicle, and has identified a corresponding two-dimensional bounding box 114. The two-dimensional bounding box 114 is rectangular and is dimensioned and positioned so as to completely encompass the vehicle image 112 within the image 110. In an alternate embodiment, the image 1 10 is captured by at least one stereo camera, RGBD camera, and/or depth camera. Use of multiple cameras may allow for recovery of depth information through the use of multiple view geometry. In such embodiments, depth information from stereo or RGBD cameras is used to aid detection of objects in image 1 10 for segmenting the image 1 10 and creating the two-dimensional bounding box 1 14.--,
{above the image processing algorithm of “detecting only cars, pedestrians, animals, or any combination thereof” is applied on the image data, which is received from an image capture device, and is the cropped image data as disclosed in [0012]}; and,-- receive an image captured from an image capture device; detect an object in the image; crop the image to form a cropped image including the object--, in [0091]; also see Douillard: e.g., -- Localizer 368 is configured to receive sensor data from one or more sources, such as GPS data 352, wheel data, IMU data 354, Lidar data 346a, camera data 340a, radar data 348a, and the like, as well as reference data 339 (e.g., 3D map data and route data). Localizer 368 integrates (e.g., fuses the sensor data) and analyzes the data by comparing sensor data to map data to determine a local pose (or position) of bidirectional autonomous vehicle 330. According to some examples, localizer 368 may generate or update the pose or position of any autonomous vehicle in real-time or near real-time. Note that localizer 368 and its functionality need not be limited to “bi-directional” vehicles and can be implemented in any vehicle of any type. Therefore, localizer 368 (as well as other components of AV controller 347a) may be implemented in a “uni-directional” vehicle or any non-autonomous vehicle. According to some embodiments, data describing a local pose may include one or more of an x-coordinate, a y-coordinate, a z-coordinate (or any coordinate of any coordinate system, including polar or cylindrical coordinate systems, or the like), a yaw value, a roll value, a pitch value (e.g., an angle value), a rate (e.g., velocity), altitude, and the like.--, in [0063]-[0065]).
Re Claim 11, Xu as modified by Douillard and Chang further disclose wherein the positions of the two points are coordinates of pixels associated with the positions in the image (see Xu: e.g., Fig. 2, and, -- a machine learning algorithm is applied to the image data and the point cloud data, to estimate parameters for a three- dimensional bounding box associated with one or more objects in the environment. For instance, a first feature vector associated with the image data, for example, associated with a cropped image corresponding to the object of interest, and a second feature vector associated with the point cloud data may be input to the machine learning algorithm. The machine learning algorithm may output parameters of the three-dimensional bounding box. The parameters may include eight points in a coordinate system, the eight points representing the eight comers of the three-dimensional bounding box. An example machine learning algorithm used to recover the parameters is an artificial neural network (ANN), which may be a Convolutional Neural Network (CNN).--, in [0012]; see Mao: e.g., -- the sensor subsystems use various technologies to measure and detect information about the environment. For example, one or more LIDAR subsystems may emit electromagnetic radiation and determine the locations of objects in the environment based on attributes of reflections of the emitted radiation that vary with the distance of the object from the vehicle. One or more camera subsystems may capture images of the environment. The sensor subsystems can provide their measurements as sensor data to a sensor subsystem interface and pre-processor, e.g., interface 306. [0067] The sensor data acquired by the sensor subsystems may include indications of multiple objects within a pre-defined distance (e.g., a sensing range) of the vehicle. At stage 604, the system (e.g., interface 306) selects one as an object of interest to be classified. The object of interest may be selected using any suitable criteria, such as a prominence of the object in the sensor data, a proximity of the object to the vehicle--, in [0066]-[0067]; also see Chang: e.g., --in step S510 of FIG. 5, the processor 130 obtains an output image (a first image) from the camera 110 and obtains resolution information corresponding to the output image to obtain position of an image central point p.sub.C according to the resolution information. (43) Specifically, the resolution information corresponds to the width and height resolution of the output image. Therefore, after obtaining the width and height resolution of the image, the processor 130 transfers the location of image into coordinates according to the resolution information and analyzes the image. In this way, the processor 130 may further determine the image coordinates of the two projection points. For instance, in the image with the resolution being 1920×1080, the upper left corner being the origin, the horizontal axis extending from left to right, and the vertical axis extending from top to bottom, the position of the image central point is (960,540). If position of a projection point p.sub.A are (u′.sub.A,v′.sub.A) and position of a projection point p.sub.B are (u′.sub.B, v′.sub.B), in this embodiment, the processor 130 further sets the image coordinates of the image central point to be (0,0), the horizontal axis to be extending from left to right, and the vertical axis to be extending from bottom to top. That is, the coordinates of the upper left corner of the image are changed from (0, 0) to (−960, 540), the image coordinates of the projection point p.sub.A are (u.sub.A,v.sub.A)=(u′.sub.A−960,−v′.sub.A+540), and the image coordinates of the projection point p.sub.B are (u.sub.B,v.sub.B)=(u′.sub.B−960,−v′.sub.B+540).--, in [0051]-[0052]; and, : -- A reference distance from the lens central point P.sub.O of the camera to the first reference point P.sub.A is d.sub.1, a reference distance from the lens central point P.sub.O of the camera to the second reference point P.sub.B is d.sub.2, and a reference distance from the first reference point P.sub.A to the second reference point P.sub.B is d.sub.3. The lens central point P.sub.O of the camera vertically projects to the ground to form a vertical intersection point P.sub.Q, and a height from the lens central point P.sub.O of the camera to the vertical intersection point P.sub.Q is h.--, in [0053]).
Re Claims 13, and 16-19, claims 13, and 16-19 are the corresponding apparatus claim to claims 1, and 4-7 respectively. Thus, claims 13, and 16-19 are rejected for the same reasons as for claims 1, and 4-7 respectively. Furthermore, Xu as modified by Douillard and Chang further disclose an apparatus comprising least one processor and a memory having instructions stored thereupon, the instructions upon execution by the at least one processor configures the apparatus to implement the method (see Xu: e.g., -- the subject matter presented herein may be implemented as a computer process, a computer-controlled apparatus, a computing system, or an article of manufacture, such as a computer-readable storage medium. While the subject matter described with respect to the methods 400, 500 are presented in the general context of operations that may be executed on and/or with one or more computing devices, those skilled in the art will recognize that other implementations may be performed in combination with various program/controller modules. Generally, such modules include routines, programs, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types.--, in [0037]).
Claims 2-3, 8, 12, 14-15, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Xu as modified by Douillard and Chang, and further in view of Mao (US 20200202145 A1, as provided in IDS).
Re Claim 2, Xu as modified by Douillard and Chang however still do not explicitly disclose the two points correspond to a first distance and a second distance from the location of the vehicle;
Mao discloses the two points correspond to a first distance and a second distance from the location of the vehicle (see Mao: e.g., -- determination of object depths for objects shown in the image based on differences in spatial orientations/offsets of the cameras' image sensors. With respect to LIDAR and RADAR, the raw sensor data can indicate a distance, a direction, and an intensity of reflected radiation.--, in [0038], and, -- the feature vector 322 based on a location of the object of interest in the environment, i.e., a location of the object represented by the object patches in first neural network inputs 316a-n. In some implementations, the system (e.g., interface 308) selects the feature vector 322 that corresponds to the region of the environment where the object of interest is located. If the object of interest spans multiple regions, the system may select a feature vector 322 that corresponds to the region of the environment where the greatest portion of the object is located…. an interface and pre-processor subsystem may obtain sensor data for a portion of an environment within sensing range of a vehicle, detect an object of interest near the vehicle, determine a bounding box (e.g., a rectangular box) around the object, and extract the content of the bounding box to form a patch for the object of interest. The bounding box may be drawn tightly around the object of interest. --, in [0058]-[0062]),
Xu (as modified by Douillard and Chang) and Mao are combinable as they are in the same field of endeavor: image processing of image data captured by the camera of a vehicle and analysis of detected objects. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify Xu (as modified by Douillard and Chang)’s method using Mao’s teachings by including the two points correspond to a first distance and a second distance from the location of the vehicle to Xu (as modified by Douillard and Chang)’s determining location such as feature vectors in order to determine a location of the object of interest in the environment (see Mao: e.g. in [0058]-[0062]).
Re Claim 3, Xu as modified by Douillard, Chang and Mao and further disclose wherein the first distance is an upper bound location of the region, and the second distance is a lower bound location of the region (see Xu: e.g., Fig. 2, and, -- a machine learning algorithm is applied to the image data and the point cloud data, to estimate parameters for a three- dimensional bounding box associated with one or more objects in the environment. For instance, a first feature vector associated with the image data, for example, associated with a cropped image corresponding to the object of interest, and a second feature vector associated with the point cloud data may be input to the machine learning algorithm. The machine learning algorithm may output parameters of the three-dimensional bounding box. The parameters may include eight points in a coordinate system, the eight points representing the eight comers of the three-dimensional bounding box. An example machine learning algorithm used to recover the parameters is an artificial neural network (ANN), which may be a Convolutional Neural Network (CNN).--, in [0012]; see Mao: e.g., -- the sensor subsystems use various technologies to measure and detect information about the environment. For example, one or more LIDAR subsystems may emit electromagnetic radiation and determine the locations of objects in the environment based on attributes of reflections of the emitted radiation that vary with the distance of the object from the vehicle. One or more camera subsystems may capture images of the environment. The sensor subsystems can provide their measurements as sensor data to a sensor subsystem interface and pre-processor, e.g., interface 306. [0067] The sensor data acquired by the sensor subsystems may include indications of multiple objects within a pre-defined distance (e.g., a sensing range) of the vehicle. At stage 604, the system (e.g., interface 306) selects one as an object of interest to be classified. The object of interest may be selected using any suitable criteria, such as a prominence of the object in the sensor data, a proximity of the object to the vehicle--, in [0066]-[0067]; also see Chang: e.g., --in step S510 of FIG. 5, the processor 130 obtains an output image (a first image) from the camera 110 and obtains resolution information corresponding to the output image to obtain position of an image central point p.sub.C according to the resolution information. (43) Specifically, the resolution information corresponds to the width and height resolution of the output image. Therefore, after obtaining the width and height resolution of the image, the processor 130 transfers the location of image into coordinates according to the resolution information and analyzes the image. In this way, the processor 130 may further determine the image coordinates of the two projection points. For instance, in the image with the resolution being 1920×1080, the upper left corner being the origin, the horizontal axis extending from left to right, and the vertical axis extending from top to bottom, the position of the image central point is (960,540). If position of a projection point p.sub.A are (u′.sub.A,v′.sub.A) and position of a projection point p.sub.B are (u′.sub.B, v′.sub.B), in this embodiment, the processor 130 further sets the image coordinates of the image central point to be (0,0), the horizontal axis to be extending from left to right, and the vertical axis to be extending from bottom to top. That is, the coordinates of the upper left corner of the image are changed from (0, 0) to (−960, 540), the image coordinates of the projection point p.sub.A are (u.sub.A,v.sub.A)=(u′.sub.A−960,−v′.sub.A+540), and the image coordinates of the projection point p.sub.B are (u.sub.B,v.sub.B)=(u′.sub.B−960,−v′.sub.B+540).--, in [0051]-[0052]; and, : -- A reference distance from the lens central point P.sub.O of the camera to the first reference point P.sub.A is d.sub.1, a reference distance from the lens central point P.sub.O of the camera to the second reference point P.sub.B is d.sub.2, and a reference distance from the first reference point P.sub.A to the second reference point P.sub.B is d.sub.3. The lens central point P.sub.O of the camera vertically projects to the ground to form a vertical intersection point P.sub.Q, and a height from the lens central point P.sub.O of the camera to the vertical intersection point P.sub.Q is h.--, in [0053]).
Re Claim 8, Xu as modified by Douillard and Chang further disclose wherein the two distances are two different pre-determined distances (see Xu: -- Measurement of the LIDAR system may be represented as three- dimensional LIDAR data having coordinates (e.g., Cartesian, polar, etc.) corresponding to positions or distances captured by the LIDAR system. For example, the LIDAR data may include point cloud data comprising a plurality of points in the environment.--, in [0010]; see Mao: e.g., -- the sensor subsystems use various technologies to measure and detect information about the environment. For example, one or more LIDAR subsystems may emit electromagnetic radiation and determine the locations of objects in the environment based on attributes of reflections of the emitted radiation that vary with the distance of the object from the vehicle. One or more camera subsystems may capture images of the environment. The sensor subsystems can provide their measurements as sensor data to a sensor subsystem interface and pre-processor, e.g., interface 306. [0067] The sensor data acquired by the sensor subsystems may include indications of multiple objects within a pre-defined distance (e.g., a sensing range) of the vehicle. At stage 604, the system (e.g., interface 306) selects one as an object of interest to be classified. The object of interest may be selected using any suitable criteria, such as a prominence of the object in the sensor data, a proximity of the object to the vehicle--, in [0066]-[0067], see Douillard: e.g., -- Autonomous vehicle controller 147 may be further configured to determine a local pose (e.g., local position) of an autonomous vehicle 109 and to detect external objects relative to the vehicle--, in [0052]-[0054], and, -- the perception data may include an obstacle map specifying static and dynamic objects located in the vicinity of an autonomous vehicle, whereas the localizer data may include a local pose or position. In operation, planner 364 generates numerous trajectories, and evaluates the trajectories, based on at least the location of the autonomous vehicle against relative locations of external dynamic and static objects….- one or more cameras 474, one or more radars 476, one or more global positioning system (“GPS”) data receiver-sensors, one or more inertial measurement units (“IMUs”) -, in [0065]-[0068]). See the similar obviousness and motivation statements addressed above for claim 2 as discussed above.
Re Claim 12, Xu as modified by Douillard and Chang further disclose wherein the region is pre-defined in the map (see Xu: e.g., Fig. 2, and, -- a machine learning algorithm is applied to the image data and the point cloud data, to estimate parameters for a three- dimensional bounding box associated with one or more objects in the environment. For instance, a first feature vector associated with the image data, for example, associated with a cropped image corresponding to the object of interest, and a second feature vector associated with the point cloud data may be input to the machine learning algorithm. The machine learning algorithm may output parameters of the three-dimensional bounding box. The parameters may include eight points in a coordinate system, the eight points representing the eight comers of the three-dimensional bounding box. An example machine learning algorithm used to recover the parameters is an artificial neural network (ANN), which may be a Convolutional Neural Network (CNN).--, in [0012]; also see Douillard: e.g., -- Localizer 368 is configured to receive sensor data from one or more sources, such as GPS data 352, wheel data, IMU data 354, Lidar data 346a, camera data 340a, radar data 348a, and the like, as well as reference data 339 (e.g., 3D map data and route data). Localizer 368 integrates (e.g., fuses the sensor data) and analyzes the data by comparing sensor data to map data to determine a local pose (or position) of bidirectional autonomous vehicle 330. According to some examples, localizer 368 may generate or update the pose or position of any autonomous vehicle in real-time or near real-time. Note that localizer 368 and its functionality need not be limited to “bi-directional” vehicles and can be implemented in any vehicle of any type. Therefore, localizer 368 (as well as other components of AV controller 347a) may be implemented in a “uni-directional” vehicle or any non-autonomous vehicle. According to some embodiments, data describing a local pose may include one or more of an x-coordinate, a y-coordinate, a z-coordinate (or any coordinate of any coordinate system, including polar or cylindrical coordinate systems, or the like), a yaw value, a roll value, a pitch value (e.g., an angle value), a rate (e.g., velocity), altitude, and the like.--, in [0063]-[0065], and also see Mao: e.g., -- the sensor subsystems use various technologies to measure and detect information about the environment. For example, one or more LIDAR subsystems may emit electromagnetic radiation and determine the locations of objects in the environment based on attributes of reflections of the emitted radiation that vary with the distance of the object from the vehicle. One or more camera subsystems may capture images of the environment. The sensor subsystems can provide their measurements as sensor data to a sensor subsystem interface and pre-processor, e.g., interface 306. [0067] The sensor data acquired by the sensor subsystems may include indications of multiple objects within a pre-defined distance (e.g., a sensing range) of the vehicle. At stage 604, the system (e.g., interface 306) selects one as an object of interest to be classified. The object of interest may be selected using any suitable criteria, such as a prominence of the object in the sensor data, a proximity of the object to the vehicle--, in [0066]-[0067]; also see Chang: e.g., --in step S510 of FIG. 5, the processor 130 obtains an output image (a first image) from the camera 110 and obtains resolution information corresponding to the output image to obtain position of an image central point p.sub.C according to the resolution information. (43) Specifically, the resolution information corresponds to the width and height resolution of the output image. Therefore, after obtaining the width and height resolution of the image, the processor 130 transfers the location of image into coordinates according to the resolution information and analyzes the image. In this way, the processor 130 may further determine the image coordinates of the two projection points. For instance, in the image with the resolution being 1920×1080, the upper left corner being the origin, the horizontal axis extending from left to right, and the vertical axis extending from top to bottom, the position of the image central point is (960,540). If position of a projection point p.sub.A are (u′.sub.A,v′.sub.A) and position of a projection point p.sub.B are (u′.sub.B, v′.sub.B), in this embodiment, the processor 130 further sets the image coordinates of the image central point to be (0,0), the horizontal axis to be extending from left to right, and the vertical axis to be extending from bottom to top. That is, the coordinates of the upper left corner of the image are changed from (0, 0) to (−960, 540), the image coordinates of the projection point p.sub.A are (u.sub.A,v.sub.A)=(u′.sub.A−960,−v′.sub.A+540), and the image coordinates of the projection point p.sub.B are (u.sub.B,v.sub.B)=(u′.sub.B−960,−v′.sub.B+540).--, in [0051]-[0052]; and, : -- A reference distance from the lens central point P.sub.O of the camera to the first reference point P.sub.A is d.sub.1, a reference distance from the lens central point P.sub.O of the camera to the second reference point P.sub.B is d.sub.2, and a reference distance from the first reference point P.sub.A to the second reference point P.sub.B is d.sub.3. The lens central point P.sub.O of the camera vertically projects to the ground to form a vertical intersection point P.sub.Q, and a height from the lens central point P.sub.O of the camera to the vertical intersection point P.sub.Q is h.--, in [0053]). See the similar obviousness and motivation statements addressed above for claim 2 as discussed above.
Re Claims 14-15, and 20, claims 14-15, and 20 are the corresponding apparatus claim to claims 2-3, and 8 respectively. Thus, claims 14-15, and 20 are rejected for the same reasons as for claims 2-3, and 8 respectively. Furthermore, Xu as modified by Douillard, Chang and Mao further disclose an apparatus comprising least one processor and a memory having instructions stored thereupon, the instructions upon execution by the at least one processor configures the apparatus to implement the method (see Xu: e.g., -- the subject matter presented herein may be implemented as a computer process, a computer-controlled apparatus, a computing system, or an article of manufacture, such as a computer-readable storage medium. While the subject matter described with respect to the methods 400, 500 are presented in the general context of operations that may be executed on and/or with one or more computing devices, those skilled in the art will recognize that other implementations may be performed in combination with various program/controller modules. Generally, such modules include routines, programs, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types.--, in [0037]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: DAEHLER (US 202000020117 A1) discloses cropping an image based on a width, height and center of a first vehicle in the image to determine an image patch, estimating a 3D pose of the first vehicle based on inputting the image patch and the width, height and center of the first vehicle into a deep neural network, and, operating a second vehicle based on the estimated 3D pose. The estimated 3D pose can include an estimated 3D position, an estimated roll, an estimated pitch and an estimated yaw of the first vehicle with respect to a 3D coordinate system (see [0008]).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEI WEN YANG whose telephone number is (571)270-5670. The examiner can normally be reached on 8:00 - 5:00 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEI WEN YANG/Primary Examiner, Art Unit 2662
Prosecution Insights