DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on December 21, 2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 4 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Relative Terminology:
The term “significantly differ” in claim 4 is a relative term which renders the claim indefinite. The term “significantly differ” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. For the sake of examination, the examiner will interpret “significantly differ” as differ.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 7-13 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over J. L. Apellániz, M. García, N. Aranjuelo, J. Barandiarán and M. Nieto, "LiDAR-based curb detection for ground truth annotation in automated driving validation," 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), Bilbao, Spain, 2023, pp. 5054-5059, doi: 10.1109/ITSC57777.2023.10422558. (hereinafter Apellániz), and further in view of J. B. Martirena, M. N. Doncel, A. C. Vidal, O. O. Madurga, J. F. Esnal and M. G. Romay, "Automated Annotation of Lane Markings Using LIDAR and Odometry," in IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 4, pp. 3115-3125, April 2022, doi: 10.1109/TITS.2020.3031921. (hereinafter Martirena).
Regarding independent claim 1, Apellániz discloses A computer-implemented method for generating an annotated training data set for training a perception algorithm of an automated driving system (ADS) of a vehicle (abstract, “This paper presents a method for detecting 3D curbs in a sequence of point clouds captured from a LiDAR sensor, which consists of two main steps. First, our approach detects the curbs at each scan using a segmentation deep neural network. Then, a sequence-level processing step estimates the 3D curbs in the reconstructed point cloud using the odometry of the vehicle. From these 3D points of the curb, we obtain polylines structured following ASAM OpenLABEL standard. These detections can be used as pre-annotations in labelling pipelines to efficiently generate curb-related ground truth data. We validate our approach through an experiment in which different human annotators were required to annotate curbs in a group of LiDAR-based sequences with and without our automatically generated pre-annotations.” Page 5054, left column, “Given the information LiDARs and cameras provide about an ego-vehicle environment, these sensors are part of many Automated Driving (AD) systems”), the method comprising:
obtaining a sequence of frames captured by one or more Light Detection and Ranging (LiDAR) sensors (abstract, “This paper presents a method for detecting 3D curbs in a sequence of point clouds captured from a LiDAR sensor, which consists of two main steps.”);
predicting, by using a road reference object (RRO) prediction neural network, an RRO position data set for each of at least a sub-set of the frames, wherein each RRO position data set comprises one or more RRO position data sub-sets for one or more RROs, respectively, wherein each RRO position data sub-set is related to spatial information of one RRO found in the frames (page 5055, right column, “We propose a semantic segmentation DNN to estimate 2D curbs. This network takes the M-channel BEV maps and infers a pixel-wise 2D mask, where each pixel is assigned a class label, in our case, “curb” and “non-curb”. ”);
wherein the global RRO position data set comprises one or more global RRO position data sub-sets, wherein each of the global RRO position data sub-sets has corresponding RRO position data sub-sets in at least two of the frames (page 5056, right column, “In this step, all the curbs' 3D points of each scan (obtained in the previous step), together with the input point clouds, are considered. Applying the odometry information, a cumulative reconstruction of all of them is made to obtain a curb point cloud of the entire sequence.”); and
forming the annotated training data set based on the sequence and the global RRO position data set (abstract, “These detections can be used as pre-annotations in labelling pipelines to efficiently generate curb-related ground truth data. ”).
Apellániz fails to explicitly disclose as further recited. However, Martirena discloses matching the one or more RRO position data sub-sets of one frame with the one or more RRO position data sub-sets of at least one other frame to populate a global RRO position data set (page 2118, left column, “The accumulated point cloud composed of the registered scans is progressively processed by blocks following the trajectory of the ego-vehicle. At each block, candidate lane marking points are detected and matched with those previously found. ”),
Apellániz is directed toward, “a method for detecting 3D curbs in a sequence of point clouds captured from a LiDAR sensor, which consists of two main steps (abstract).” Martirena is directed toward, “we present an efficient and robust method for the offline annotation of lane markings, using low-density LIDAR point clouds and odometry information (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Apellániz and Martirena are directed toward similar methods of endeavor of analyzing LiDAR data for vehicle driving. Further, one of ordinary skill in the art before the effective filing date of ht eclimaed invention would easily be aware when detecting objects across multiple vehicle image frames, one would want to ensure similar objects are tracke across the frames. Said differently, when driving one would want images of the same pedestrian/curb/sign to be recognized as the same, as opposed to detecting each frame as a new instance. One could easily see how inaccurate detection could lead to accidents. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Martirena in order to ensure the same objects across images are determined as being one object in different locations.
Regarding dependent claim 2, the rejection of claim 1 is incorporated herein. Additionally, Martirena in the combination further discloses wherein a matching criteria is that the one or more RRO position data sub-sets of one frame and the one or more RRO position data sub-sets of at least one other frame is placed within a minimum distance threshold (page 3118, left column, “At each block, candidate lane marking points are detected and matched with those previously found. Once the trajectory is completed, lane markings are classified as solid or dashed. ”).
One of ordinary skill in the art before the effective filing date of the claimed invention would be easily aware distance measurements can be used to determine if objects across frames are the same instance, or a new one. Tracking of these objects accurately allows for more accurate autonomous driving decision making; thus it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Martirena
Regarding dependent claim 3, the rejection of claim 1 is incorporated herein. Additionally, Martirena in the combination further discloses wherein the step of matching further comprises
for each of the one or more RRO position data sub-sets of the one frame, for each of the one or more RRO position data sub-sets of the at least one other frame, determining a distance between the RRO position data sub-set of the one frame and the RRO position data sub-set of the at least one other frame (page 3119, left column, “Each point is connected with the closest point of a previously detected lane marking if these criteria are met: If the previous marking has only one point: Distance between the current point and the point of the marking is smaller than tral”); and
in case the distance is below the minimum distance threshold, populating the RRO position data sub-set of the one frame to the global RRO position data set (page 3119, left column, “Each point is connected with the closest point of a previously detected lane marking if these criteria are met: If the previous marking has only one point: Distance between the current point and the point of the marking is smaller than tral”).
One of ordinary skill in the art before the effective filing date of the claimed invention would be easily aware distance measurements can be used to determine if objects across frames are the same instance, or a new one. Further, based on the closeness or farness of the distance value, the object would either be or not be the same object; said differently if the objects are too far apart to be the same object, they must be classified as different objects. Tracking of these objects accurately allows for more accurate autonomous driving decision making; thus it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Martirena
Regarding dependent claim 4, the rejection of claim 1 is incorporated herein. Additionally, Apellániz in the combination further discloses further comprising:
removing outliers from the global RRO position data set, wherein the outliers are global RRO position data sub-sets of the global RRO position data set that significantly differ from a rest of the global RRO position data sub-sets of the global RRO position data set (page 5056, right column, “At this point, we first group the detections in different curbs so that in later stages, we can manipulate them separately, for example, by loading them effectively in the labelling tools or deleting those sections that correspond to false detections. To carry out this separation, we perform a clustering stage using the DBSCAN algorithm [21].”… “The last post-processing step is the simplification. In this step, we reduce the number of points of the detected curb skeletons by means of the Ramer-Douglas-Peucker algorithm [23], [24]. This algorithm uses a given distance tolerance to determine which points on a line are to be eliminated or retained.”).
Regarding dependent claim 5, the rejection of claim 1 is incorporated herein. Additionally, Apellániz in the combination further discloses further comprising:
aggregating the global RRO position data set by adjusting the global RRO position data sub-sets placed inside a defined window (page 5055, right column, “3) clustering, skeletonization and simplification of the 3D curbs to generate final polylines in the standardized annotation file.” Page 5056, right column, “To obtain the final polyline curb representation, we use the skeletonization algorithm [22]. By applying this algorithm, specially designed for the skeletonization of a point cloud obtained from a LiDAR, we obtain the linear traces we expect to represent the curbs. The last post-processing step is the simplification. In this step, we reduce the number of points of the detected curb skeletons by means of the Ramer-Douglas-Peucker algorithm [23], [24]. This algorithm uses a given distance tolerance to determine which points on a line are to be eliminated or retained.” The skeletonization and simplification processing is applied to the point cloud within the window of the curb region).
Regarding dependent claim 7, the rejection of claim 1 is incorporated herein. Additionally, Apellániz in the combination further discloses wherein each of the frames is a 100 to 360 degree scan (abstract, “ This paper presents a method for detecting 3D curbs in a sequence of point clouds captured from a LiDAR sensor, which consists of two main steps.” The LiDAR is capturing images outside the vehicle, which can range in terms of field of view; page 5054, left column, “their near environment and make decisions accordingly. Cameras provide rich semantic information about the scene but lose depth information in their projective process, making them less aware of 3D structures than Light Detection and Ranging (LiDAR) sensors. LiDAR sensors are unaffected by light conditions [1], and due to the three-dimensional nature of the point clouds generated by the sensor, they are particularly useful for the precise localization of obstacles in a vehicle’s surroundings. Given the information LiDARs and cameras provide about an ego-vehicle environment, these sensors are part of many Automated Driving (AD) systems.”).
Regarding dependent claim 8, the rejection of claim 1 is incorporated herein. Additionally, Apellániz in the combination further discloses wherein the sequence of frames comprise detections made over a period of time (page 5056, right column, “In this step, all the curbs' 3D points of each scan (obtained in the previous step), together with the input point clouds, are considered. Applying the odometry information, a cumulative reconstruction of all of them is made to obtain a curb point cloud of the entire sequence.” Each scan is read as being performed over time).
Regarding dependent claim 9, the rejection of claim 1 is incorporated herein. Additionally, Apellániz in the combination fail to explicitly disclose wherein the sequence of frames comprise detections made by more than one LiDAR sensor over a range of positions in a three-dimensional space (page 5054, left column, “ Given the information LiDARs and cameras provide about an ego-vehicle environment, these sensors are part of many Automated Driving (AD) systems.” Multiple cameras is read as obtaining multiple ranges of positions in that the cameras are not placed in exactly the same position).
Regarding dependent claim 10, the rejection of claim 1 is incorporated herein. Additionally, Martirena in the combination further discloses wherein the RRO position data sub-sets comprises lane marker positions (abstract, “we present an efficient and robust method for the offline annotation of lane markings, using low-density LIDAR point clouds and odometry information.”… “The lane markings are tracked block wise, and their width is estimated and classified as either solid or dashed. The results are lists of connected 3D points that represent the different lane markings.”). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention that lane markers are objects often used to aid in accurate autonomous driving. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to ensure road markers can be detected for safe autonomous driving.
Regarding dependent claim 11, the rejection of claim 1 is incorporated herein. Additionally, Apellániz in the combination further discloses wherein the RRO position data set comprises position data sets of median barriers (abstract, “ First, our approach detects the curbs at each scan using a segmentation deep neural network. Then, a sequence-level processing step estimates the 3D curbs in the reconstructed point cloud using the odometry of the vehicle. From these 3D points of the curb, we obtain polylines structured following ASAM OpenLABEL standard. These detections can be used as pre-annotations in labelling pipelines to efficiently generate curb-related ground truth data.” A median is read as “curb-related”).
Regarding dependent claim 12, the rejection of claim 1 is incorporated herein. Additionally, Apellániz in the combination further discloses A non-transitory computer readable storage medium storing instructions which, when executed by a computing device, causes the computing device to carry out the method according to claim 1 (page 5054, left column, “ Given the information LiDARs and cameras provide about an ego-vehicle environment, these sensors are part of many Automated Driving (AD) systems. They are often combined with the latest advances in Artificial Intelligence (AI).” AI is well known to be implemented on a computer, through programming the network itself; see claim 1 analysis).
Regarding independent claim 13, the rejection of independent claim 1 applies directly. Additionally, Apellániz discloses An apparatus for generating an annotated training data set for training a perception algorithm of an automated driving system (ADS) of a vehicle (abstract, “This paper presents a method for detecting 3D curbs in a sequence of point clouds captured from a LiDAR sensor, which consists of two main steps. First, our approach detects the curbs at each scan using a segmentation deep neural network. Then, a sequence-level processing step estimates the 3D curbs in the reconstructed point cloud using the odometry of the vehicle. From these 3D points of the curb, we obtain polylines structured following ASAM OpenLABEL standard. These detections can be used as pre-annotations in labelling pipelines to efficiently generate curb-related ground truth data. We validate our approach through an experiment in which different human annotators were required to annotate curbs in a group of LiDAR-based sequences with and without our automatically generated pre-annotations.” Page 5054, left column, “Given the information LiDARs and cameras provide about an ego-vehicle environment, these sensors are part of many Automated Driving (AD) systems”), the apparatus comprising a control circuitry configured to:
obtain a sequence of frames captured by at least one LiDAR sensor (see claim 1 analysis),
predict, by using a road reference object (RRO) prediction neural network, an RRO position data set for each of at least a sub-set of the frames, wherein each RRO position data set comprises one or more RRO position data sub-sets for one or more RROs, respectively, wherein each RRO position data sub-set is related to spatial information of one RRO found in the frames (see claim 1 analysis),
match the one or more RRO position data sub-sets of one frame with one or more RRO position data sub-sets of at least one other frame to populate a global RRO position data set, wherein the global RRO position data set comprises one or more global RRO position data sub-sets, wherein each of the global RRO position data sub-sets has corresponding RRO position data sub-sets in at least two of the frames (see claim 1 analysis), and
form the annotated training data set based on the sequence and the global RRO position data set (see claim 1 analysis).
Regarding dependent claim 15, the rejection of claim 13 is incorporated herein. Additionally, Apellániz in the combination further discloses A vehicle (collected from a LiDAR sensor installed on a prototype vehicle) comprising:
an automated driving system (ADS) comprising a perception algorithm (page 5054, left column, “Given the information LiDARs and cameras provide about an ego-vehicle environment, these sensors are part of many Automated Driving (AD) systems.””),
at least one Light Detection and Ranging (LiDAR) sensor (page 5058, left column, “collected from a LiDAR sensor installed on a prototype vehicle” page 5054, left column, “Automated vehicles rely on different sensors to understand their near environment and make decisions accordingly. Cameras provide rich semantic information about the scene but lose depth information in their projective process, making them less aware of 3D structures than Light Detection and Ranging (LiDAR) sensors. LiDAR sensors are unaffected by light conditions [1], and due to the three-dimensional nature of the point clouds generated by the sensor, they are particularly useful for the precise localization of obstacles in a vehicle's surroundings. Given the information LiDARs and cameras provide about an ego-vehicle environment, these sensors are part of many Automated Driving (AD) systems.”), and
an apparatus for generating an annotated training data set for training the perception algorithm of the ADS according to claim 13 (see claim 13).
Allowable Subject Matter
Claim 6 and 14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: the closest prior arts of record teach methods of generating training data to be used for automated driving systems. However, none of them alone or in any combination teaches determining weights associated with lane marking tracking score data sets, then adjusting an aggregated global position data set based on the determined weights.
The closest prior art being previously cited Apellániz discloses, “a method for detecting 3D curbs in a sequence of point clouds captured from a LiDAR sensor, which consists of two main steps. First, our approach detects the curbs at each scan using a segmentation deep neural network. Then, a sequence-level processing step estimates the 3D curbs in the reconstructed point cloud using the odometry of the vehicle. From these 3D points of the curb, we obtain polylines structured following ASAM OpenLABEL standard. These detections can be used as pre-annotations in labelling pipelines to efficiently generate curb-related ground truth data (abstract).” Apellániz further aggregates data at page 5056, right column, “In this step, all the curbs' 3D points of each scan (obtained in the previous step), together with the input point clouds, are considered. Applying the odometry information, a cumulative reconstruction of all of them is made to obtain a curb point cloud of the entire sequence.”
However, Apellániz fails to disclose determining weights associated with lane marking tracking score data sets, then adjusting an aggregated global position data set based on the determined weights.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
U.S. Publication No. 2022/0126864 to Moustafa et al. discloses, “Sensor data is received from a plurality of sensors, where the plurality of sensors includes a first set of sensors and a second set of sensors, and at least a portion of the plurality of sensors are coupled to a vehicle. Control of the vehicle is automated based on at least a portion of the sensor data generated by the first set of sensors (abstract).”
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Courtney J. Windsor whose telephone number is (571)272-3956. The examiner can normally be reached Monday - Friday 8:00 - 4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/COURTNEY JOAN NELSON/Primary Examiner, Art Unit 2661
Prosecution Insights