Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-4 and 9-12 are rejected under 35 U.S.C. 102(a) (1) as being anticipated by Pham et al (U.S. Pub No: 2020/0341466 A1).
Regarding claim 1, Pham discloses a method comprising: obtaining input data representing a dashed line associated with a drivable surface (see page 2, paragraph, [0026] In deployment, sensor data (e.g., images, videos, etc.) received and/or generated (input data), using sensors (e.g., cameras, RADAR sensors, LIDAR sensors, etc.) located or otherwise disposed on an autonomous or semi-autonomous vehicle. The sensor data may be applied to a neural network (e.g., a deep neural network (DNN), such as a convolutional neural network (CNN)) that is trained to identify areas of interest pertaining to “road markings”, road (drivable surface) boundaries, intersections, and/or the like (e.g., raised pavement markers, rumble strips, colored lane dividers, sidewalks, cross-walks, turn-offs, etc.) represented by the sensor data, as well as semantic and/or directional information pertaining thereto. More specifically, the neural network may be designed to compute key points corresponding to segments of an intersection (e.g., corresponding to lanes, bike paths, etc., and/or corresponding to features therein—such as cross walks, intersection entry points, intersection exit points, etc.), and to generate outputs identifying, for each key point, a width of a lane corresponding to the key point, a directionality of the lane, a heading corresponding to the lane, semantic information corresponding to the key point (e.g., crosswalk, crosswalk entry, crosswalk exit, intersection entry, intersection exit, etc., or a combination thereof), and/or other information. In some examples, the computed key points may be denoted by pixels represented by the sensor data where center and/or end points of intersection features are located.
Also, page 5, paragraph, [0045] further, labels for the lanes 204, 206, and 208 may further be annotated with a corresponding heading direction, as indicated by arrows 202A-202V. The heading direction may represent a direction of the traffic pertaining to a certain lane. In some examples, the heading directions may be associated with a center (or key) point of its corresponding lane label. For example, heading direction may be associated with a center point of lane. The different classification labels may be represented in FIG. 2A by different line types e.g., solid lines, “dashed lines”, etc. to represent different classifications. However, this is not intended to be limiting, and any visualization of the lane labels and their classifications may include different shapes, patterns, fills, colors, symbols, and/or other identifiers to illustrate differences in classification labels for features (e.g., lanes) in the images);
generating a representation associated with the dashed line based at least on intensity values associated with points corresponding to the input data (see page 3, paragraphs, [0027] and [0029], during training, the DNN may be trained with images or other sensor data representations labeled or annotated with line segments representing lanes, crosswalks, entry-lines, exit-lines, bike lanes, etc., and may further include semantic information corresponding thereto. The labeled “line segments” (dashed lines), and semantic information may then be used by a ground truth encoder to generate key point heat maps, offset vector maps, directional vector maps, heading vector maps, lane width “intensity maps”, lane counts, lane classifications, and/or other information corresponding to the intersection as determined from the annotations. In some examples, key points may also include corner or end points of the line segments corresponding to lanes, which may be inferred from the center points, the line direction vectors, and/or width information, or may be computed directly using one or more vector fields corresponding to the end point key points. [0029] the 2D line directional vector fields and the 2D heading direction vector fields may be encoded by assigning a directional vector to the center key point, and then assigning the same directional vector to each pixel within a defined radius of the location of the center key point. The lane widths may be encoded by assigning an “intensity value” equal to the lane width normalized by image width, and the same intensity value may be assigned to other pixels within a defined radius. In some examples, the end point key points may be encoded using an offset vector field, such that the DNN learns to compute the location of not only the center key points, but also the end point key points. Using this information, the width of the lanes may be directly determined from the end point key points (e.g., a distance from one end point to the other), the directionality of the line segments extending across the lanes may be determined from the end point key points (e.g., the direction may be defined by a line from one end point to the other end point);
determining, based at least on the representation, a relationship between at least a first portion of the dashed line and a second portion of the dashed line; and determining, based at least on the relationship, information associated with one or more components of the dashed line, the information including at least one or more locations associated with the one or more components (see above, also page 5, paragraph, [0041] The lane (or line) label(s) may include annotations, or other label types, corresponding to features or “areas of interest” corresponding to the intersection. In some examples, an intersection structure may be defined as a set of “line segments” corresponding to lanes, crosswalks, entry-lines, exit-lines, bike lanes, etc., in the sensor data. The line segments may be generated as polylines, with a center of each polyline defined as the center for the corresponding line segment. The classification(s) may be generated for each of the images (or other data representations) and/or for each one or more of the line segments and centers in the images represented by the sensor data used for training the machine learning model(s). The number of classification(s) may correspond to the number and/or types of features that the machine learning model(s) is trained to predict, or to the number of lanes and/or types of features in the respective image (dashed lines). Depending on the embodiment, the classification(s) may correspond to classifications or tags corresponding to the feature type, such as but not limited to, crosswalk, crosswalk entry, crosswalk exit, intersection entry.
Also pages 10-11, paragraphs, [0083-0084] and [0086-0089] the lane shapes may be determined by decoding the intensity maps to determine the width of the lanes and/or decoding the direction vector fields 112A to determine a directionality of a line segment corresponding to a center key point. As such, the location of the center key point, the directionality, and the width may be leveraged to determine the lane shapes (e.g., by extending the line segment from the center key point according to the directionality and up to a width of the line segment such as half the width from the center key point along the directionality in one direction, and then half the width from the center key point along the directionality in the opposite direction). For example, the lane headings may be determined as the normal to a line segment generated “between” the known locations of the left edge key point and the right edge key point. In other examples, the lane headings may be determined by decoding the heading vector fields 112B to determine the angle corresponding to the heading direction of the lane. In some examples, the lane headings may leverage the lane types 510 and/or other semantic information output by the network, as described herein. In some examples, the path generator 516 may implement curve fitting in order to determine final shapes that most accurately reflect a natural curve of the potential paths. Any known curve fitting algorithms may be used, such as but not limited to, polyline fitting, polynomial fitting, and/or clothoid fitting. The shape of the potential paths may be determined based on the locations of the key points 506 and the corresponding heading vectors (e.g., angles) (e.g., lane headings 514) associated with the key points to be connected. In some examples, the shape of a potential path may be aligned with a tangent of the heading vector at the location of the key points to be connected. The curve fitting process may be repeated for all key points that may potentially be connected to each other to generate all possible paths the vehicle 800 may take to navigate the intersection. In some examples, non-feasible paths may be removed from consideration based on traffic rules and physical restrictions associated with such paths. The remaining potential paths may be determined to be feasible 3D paths or trajectories that the vehicle 800 may take to traverse the intersection. In some embodiments, the path generator 516 may use a matching algorithm to connect the key points 506 and generate the potential paths for the vehicle to navigate the intersection. In such examples, matching scores may be determined for each pair of key points based on the location of the key points, lane headings 514 corresponding to the key points (e.g., two key points corresponding to different directions of travel will not be connected), and the shape of the fitted curve between the pair of key points. Each key point corresponding to an intersection entry may be connected to multiple key points corresponding to an intersection exit thereby generating a plurality of potential paths for the vehicle. In some examples, a linear matching algorithm such as Hungarian matching algorithm may be used. In other examples, a non-linear matching algorithm such as spectral matching algorithm may be used to connect a pair of key points. Now referring to FIG. 6A, FIG. 6A illustrates an example intersection structure prediction 600A (e.g., output(s) 106 of FIG. 5) generated using a neural network (e.g., machine learning model(s) 104 of FIG. 5), in accordance with some embodiments of the present disclosure. The prediction 600A includes a visualization of predicted line segments corresponding to lane classifications 604, 606, 608, 610, and 612, for each lane detected in the sensor data (e.g., sensor data 102 of FIG. 5). For example, the lane classification 604 may correspond to an entrance to a pedestrian crossing lane type, the lane classification 606 may correspond to an entrance to an intersection and/or an exit from a pedestrian crossing lane type, the lane classification 608 may correspond to an exit from a pedestrian crossing lane type, the lane classification 610 may correspond to an exit from an intersection and/or an entrance to a pedestrian crossing lane type, and the lane classifications 612 may correspond to a non-drivable lane type. Each line segment may also be associated with a center key point 506 and/or a corresponding heading direction(s) (e.g., key points and associated vectors 602A-602V). As such, the intersection structure and pose may be represented by a set of line segments with corresponding line classifications, key points, and/or heading directions).
Regarding claim 2, Pham discloses the method of claim 1, wherein the input data is first feature data obtained from mapping data, the method further comprising: generating second feature data associated with the one or more components, the second feature data including the one or more locations; and causing the mapping data to be updated to include the second feature data (see claim 1, also abstract, in various examples, live perception from sensors of a vehicle may be leveraged to generate potential paths for the vehicle to “navigate” an intersection in real-time or near real-time. For example, a deep neural network (DNN) may be trained to compute various outputs such as heat maps corresponding to key points associated with the intersection, vector fields corresponding to directionality, heading, and offsets with respect to lanes, “intensity maps” corresponding to widths of lanes, and/or classifications corresponding to line segments of the intersection. The outputs may be decoded and/or otherwise post-processed to reconstruct an intersection or key points corresponding thereto and to determine proposed or potential paths for navigating the vehicle through the intersection.
Also, page 7, paragraphs, [0055-0057] in some non-limiting embodiments, the offset 126 vector fields may be encoded by assigning, for each pixel in an offset vector field, a vector pointing to the closest ground truth key “point pixel location”. In this way, smaller encoding channels may be used even when the 2D encoding channels have a different spatial resolution than the input image. This allows the machine learning model(s) to train and predict intersection structure and pose in a computationally less expensive manner because the smaller encoding channels may be used without losing information due to down-sampling of images during processing by the machine learning model(s). In some examples, ground truth data 122 for a “number of features” (e.g., lanes) per classification(s) may be encoded directly using a simple count. The machine learning model(s) may then be trained to predict the number of “features per classification”(s) directly. [0056] In some embodiments, the “intensity map”, may be implemented to encode lane widths as determined from the lane label(s) 118A corresponding to segments of the lane(s). For example, once the lane widths are determined, the lane width for the lane segment corresponding to each key point may be encoded by assigning an intensity value equal to the lane width normalized by image width (e.g., also in image-space) to the key point. In some examples, the same intensity value may be assigned to other pixels within a defined radius of the associated key point, similar to as described herein with respect to the direction vector fields and the heading vector fields. Once the ground truth data is generated for each instance of the sensor data (e.g., for each image where the sensor data includes image data), the machine learning model(s) may be trained using the ground truth data. For example, the machine learning model(s) may generate output(s), and the output(s) may be compared using the loss function(s) to the ground truth data corresponding to the respective instance of the sensor data. As such, feedback from the loss function(s) 130 may be used to “update” parameters (e.g., weights and biases) of the machine learning model(s) 104 in view of the ground truth data 122 until the machine learning model(s) 104 converges to an acceptable or desirable accuracy. Using the process 100, the machine learning model(s) 104 may be trained to accurately predict the output(s) (and/or associated classifications) from the sensor data using the loss function(s) and the ground truth data. In some examples, different loss functions may be used to train the machine learning model(s) to predict different outputs. For example, a first loss function 130 may be used for comparing the heat map(s) and and a second loss function may be used for comparing the intensity maps 128 and the intensity maps 114. As such, in non-limiting embodiments, one or more of the output channels be trained using a different loss function 130 than another of the output channels).
Regarding claim 3, Pham discloses the method of claim 1, further comprising causing, based at least on the one or more locations associated with the one or more components, a machine to perform one or more operations (see claim 1, also page 1, paragraph, [0006] in contrast to conventional systems, such as those described above, the current system may use live perception of the vehicle to detect the intersection pose and generate paths for navigating the intersection. Key points (e.g., center points and/or end points) of line segments corresponding to features of an intersection such as lanes, crosswalks, intersection entry or exit lines, bike paths, etc. may be leveraged to generate potential paths for a vehicle to navigate an intersection. For example, machine learning algorithm(s) such as deep neural networks (DNNs) may be trained to compute information corresponding to an intersection—such as key points, heading directions, widths of lanes, number of lanes, etc. and this information may be used to connect together center key points (e.g., key points corresponding to centers of line segments) to generate paths and/or trajectories for the vehicle to effectively and accurately navigate the intersection. As such, semantic information associated with the predicted key points such as directionality, heading, width, and/or classification information corresponding to segments of the intersection—may be computed and leveraged in order to gain an understanding of the intersection pose. For example, the outputs of the DNN may be used to directly or indirectly (e.g., via decoding) determine: a location of each lane, bike path, cross-walk, and/or the like; a number of lanes associated with the intersection; a geometry of the lanes, bike paths, crosswalks, and/or the like; a direction of travel (or heading direction) corresponding to each lane; and/or other intersection structure information).
Regarding claim 4, Pham discloses the method of claim 1, wherein the one or more locations associated with the one or more components corresponds with at least one of one or more start points or one or more ends points associated with the one or more components (see claim 1, also page 1, paragraph, [0006] in contrast to conventional systems, such as those described above, the current system may use live perception of the vehicle to detect the intersection pose and generate paths for navigating the intersection. Key points (e.g., center points and/or end points) of line segments corresponding to features of an intersection such as lanes, crosswalks, intersection entry or exit lines, bike paths, etc. may be leveraged to generate potential paths for a vehicle to navigate an intersection. For example, machine learning algorithm(s) such as deep neural networks (DNNs) may be trained to compute information corresponding to an intersection such as key points, heading directions, widths of lanes, number of lanes, etc. and this information may be used to connect together center key points (e.g., key points corresponding to centers of line segments) to generate paths and/or trajectories for the vehicle to effectively and accurately navigate the intersection. As such, semantic information associated with the predicted key points such as directionality, heading, width, and/or classification information corresponding to segments of the intersection may be computed and leveraged in order to gain an understanding of the intersection pose. For example, the outputs of the DNN may be used to directly or indirectly (e.g., via decoding) determine: a location of each lane, bike path, cross-walk, and/or the like; a number of lanes associated with the intersection; a geometry of the lanes, bike paths, crosswalks, and/or the like; a direction of travel (or heading direction) corresponding to each lane; and/or other intersection structure information).
Regarding claim 9, Pham discloses the method of claim 1, wherein the input data is an intensity image generated based at least on LiDAR data, the intensity image representing the dashed line associated with the drivable surface from a top-down perspective ( see claim 1, also page 19, paragraph, [0175] LIDAR technologies, such as 3D flash LIDAR, may also be used. 3D Flash LIDAR uses a flash of a laser as a transmission source, to illuminate vehicle surroundings up to approximately 200 m. A flash LIDAR unit includes a receptor, which records the laser pulse transit time and the reflected light on each pixel, which in turn corresponds to the range from the vehicle to the objects. Flash LIDAR may allow for highly accurate and distortion-free images of the surroundings to be generated with every laser flash. In some examples, four flash LIDAR sensors may be deployed, one at each side of the vehicle 800. Available 3D flash LIDAR systems include a solid-state 3D staring array LIDAR camera with no moving parts other than a fan (e.g., a non-scanning LIDAR device). The flash LIDAR device may use a 5-nanosecond class I (eye-safe) laser pulse per frame and may capture the reflected laser light in the form of 3D range point clouds and co-registered intensity data. By using flash LIDAR, and because flash LIDAR is a solid-state device with no moving parts, the LIDAR sensor(s) 864 may be less susceptible to motion blur, vibration, and/or shock).
Regarding claim 11, Pham discloses the system of claim 10, wherein the one or more processors are further to determine, based at least on the representation, a relationship between at least a first portion of the feature and a second portion of the feature, wherein the determining the one or more locations associated with the one or more components is further based at least on the relationship (see claim 1, also see abstract, in various examples, live perception from sensors of a vehicle may be leveraged to generate potential paths for the vehicle to navigate an intersection in real-time or near real-time. For example, a deep neural network (DNN) may be trained to compute various outputs such as heat maps corresponding to key points associated with the intersection, vector fields corresponding to directionality, heading, and offsets with respect to lanes, intensity maps corresponding to widths of lanes, and/or classifications corresponding to line segments of the intersection. The outputs may be decoded and/or otherwise post-processed to reconstruct an intersection or key points corresponding thereto and to determine proposed or potential paths for navigating the vehicle through the intersection.
Also, page 7, paragraphs, [0055-0057] in some non-limiting embodiments, offset vectors may be determined and encoded to generate the offset vector field 126C. The offset 126 vector fields may be encoded by assigning, for each pixel in an offset vector field, a vector pointing to the closest ground truth key point pixel location. In this way, smaller encoding channels may be used even when the 2D encoding channels have a different spatial resolution than the input image (e.g., a down-sampled image). This allows the machine learning model(s) 104 to train and predict intersection structure and pose in a computationally less expensive manner because the smaller encoding channels may be used without losing information due to down-sampling of images during processing by the machine learning model(s) 104. In some examples, ground truth data 122 for a number of features (e.g., lanes) per classification(s) 118B may be encoded directly using a simple count. The machine learning model(s) 104 may then be trained to predict the number of features per classification(s) 118B directly. In some embodiments, the intensity map(s) 128 may be implemented to encode lane widths—as determined from the lane label(s) 118A corresponding to segments of the lane(s). For example, once the lane widths are determined, the lane width for the lane segment corresponding to each key point may be encoded by assigning an intensity value equal to the lane width (e.g., in image-space) normalized by image width (e.g., also in image-space) to the key point. In some examples, the same intensity value may be assigned to other pixels within a defined radius of the associated key point, similar to as described herein with respect to the direction vector fields 126A and the heading vector fields 126B. Once the ground truth data 122 is generated for each instance of the sensor data 102 (e.g., for each image where the sensor data 102 includes image data), the machine learning model(s) 104 may be trained using the ground truth data 122. For example, the machine learning model(s) 104 may generate output(s), and the output(s) may be compared using the loss function(s) to the ground truth data corresponding to the respective instance of the sensor data 102. As such, feedback from the loss function(s) 130 may be used to update parameters (e.g., weights and biases) of the machine learning model(s) 104 in view of the ground truth data 122 until the machine learning model(s) 104 converges to an acceptable or desirable accuracy. Using the process 100, the machine learning model(s) 104 may be trained to accurately predict the output(s) 106 (and/or associated classifications) from the sensor data 102 using the loss function(s) 130 and the ground truth data 122. In some examples, different loss functions 130 may be used to train the machine learning model(s) 104 to predict different outputs 106. For example, a first loss function 130 may be used for comparing the heat map(s) 108 and 124 and a second loss function 130 may be used for comparing the intensity maps 128 and the intensity maps 114. As such, in non-limiting embodiments, one or more of the output channels be trained using a different loss function 130 than another of the output channels).
Regarding claim 12, Pham discloses the system of claim 10, wherein the input data is an intensity image generated based at least on LiDAR data, the intensity image representing the feature associated with the drivable surface from a top-down perspective (see claim 1, also page 19, paragraph, [0175] in some examples, LIDAR technologies, such as 3D flash LIDAR, may also be used. 3D Flash LIDAR uses a flash of a laser as a transmission source, to illuminate vehicle surroundings up to approximately 200 m. A flash LIDAR unit includes a receptor, which records the laser pulse transit time and the reflected light on each pixel, which in turn corresponds to the range from the vehicle to the objects. Flash LIDAR may allow for highly accurate and distortion-free images of the surroundings to be generated with every laser flash. In some examples, four flash LIDAR sensors may be deployed, one at each side of the vehicle 800. Available 3D flash LIDAR systems include a solid-state 3D staring array LIDAR camera with no moving parts other than a fan (e.g., a non-scanning LIDAR device). The flash LIDAR device may use a 5-nanosecond class I (eye-safe) laser pulse per frame and may capture the reflected laser light in the form of 3D range point clouds and co-registered intensity data. By using flash LIDAR, and because flash LIDAR is a solid-state device with no moving parts, the LIDAR sensor(s) 864 may be less susceptible to motion blur, vibration, and/or shock).
With regard to claim 10 the arguments analogous to those presented above for claims 1, 2, 3, 4, 9, 11 and 12 , are respectively applicable to claim 10.
Allowable Subject Matter
Claims 5-8 and 13-14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Seyed Azarian whose telephone number is (571) 272-7443. The examiner can normally be reached on Monday through Thursday from 6:00 a.m. to 7:30 p.m.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Andrew Moyer, can be reached at (571) 272-9523. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application information Retrieval (PAIR) system. Status information for published application may be obtained from either Private PAIR or Public PAIR.
Status information about the PAIR system, see http:// pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/SEYED H AZARIAN/Primary Examiner, Art Unit 2667
May 15, 2026
Prosecution Insights