DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This action is in response to amendments and remarks filed on 03/27/2026. Claim(s) 1-2, 4, 7-9, 12, 18-19, and 21-22 have been amended. Claim(s) 3 and 10-11 have been previously cancelled. Claim(s) 1-2, 4-9, and 12-23 are pending examination. Rejection to claim(s) 22 over the 35 USC 112(b) rejection has been withdrawn in light of the instant amendments. This action is made final.
Response to Arguments
Applicant presents the following argument(s) regarding the previous office action:
Applicant asserts that the 35 USC 103 rejections of claims 1-20 is improper. Applicant asserts that the newly added limitations to the independent claims 1, 9, and 18 overcome the 35 USC 103 rejection. Accordingly the independent claims would be allowable. Any dependent claims rejected under 35 USC 103 would also be allowable.
Applicant’s arguments with respect to claim(s) 1-2, 4-9, and 12-23 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Regarding applicant’s argument A, the examiner finds it moot. Applicant’s argument centers on the newly added limitations to independent claims 1, that recite, “determining, based at least on connecting the vertex locations, a first bounding shape that indicates an area of the intensity image that depicts the road marking, the first bounding shape being oriented orthogonal with respect to a direction of travel of the surface;” and “the second bounding shape also being oriented orthogonal with respect to the direction of travel of the surface.” Applicant has added similar language in claims 9 and 18 as well. However, only one explanation will be given. Applicant asserts that the cited prior art of Khadem and Bojarski fail to teach this limitation. After further search and consideration the examiner would rely on newly cited portions of Kumar to teach this limitation. Looking at Kumar it was previously used to reject claim 4. The examiner relied on Fig. 4 and [0062]/[0068] to teach that a bounding shape was oriented in the direction of travel. Further citations of Kumar show that a bounding shape can be oriented orthogonal to the direction of travel. [0071]-[0072] of Kumar show the system determining that the bounding shape of a crosswalk is orthogonal and that if the shape is “parallel” to the direction of travel it can disregard the shape as some kind of lane line. The teachings show that the vehicle determines the angle of travel with relation to the detected object and only cares if the angle is within the perpendicular axis to the vehicle. This teaching would correct the short comings of Khadem and Bojarski and overcome the new limitations. Therefore independent claims 1, 9, and 18 would remain rejected under 35 USC 103. Dependent claims would remain rejected at least due to their dependence on rejected subject matter. Please see the section below titled, “Claim Rejections – 35 USC 103,” for further mapping and explanation.
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 1-2, 4-9, and 12-23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Khadem (US PG Pub 2024/0125899) in view of Bojarski (US PG Pub 2020/0324795) and Kumar (US PG Pub 2021/0097308).
Regarding claim 1, Regarding claim 1, Khadem teaches a method comprising: obtaining LiDAR data obtained using one or more LiDAR sensors of a first machine, ([0028] teaches the use of LiDAR sensors to obtain data from a road; [0024]-[0025] teaches the LiDAR sensor on a vehicle) the LiDAR data representative of at least one or more intensity values associated with one or more points located on a surface within an environment; ([0027]-[0028] and [0055]-[0057] teach the LiDAR data including intensity values of one or more points within and environment. At least [0050] teaches that this data is representative of the surface of an environment)
generating, based at least on the one or more intensity values represented by the LiDAR data, image data representative of an intensity image corresponding to at least a portion of the surface of environment; ([0028], [0031]-[0032], Fig. 3, [0049]-[0056], and [0078] teach the system as able to generate image data, including 2D top down images, based on an input of LiDAR information received from a series of LiDAR sensors and the data’s intensity value. At least [0050] teaches that this data is representative of the surface of an environment)
generating, by one or more machine learning models and based at least on processing the image data, ([0031]-[0032] teaches the system using a machine learning model to output a segmented image based on input LiDAR data.) output data ([0016], [0058]-[0060] teach the use of bounding shapes that indicate an area for which a road marking is located. These markings are output data that represent a location in the world.)
determining, based at least on connecting the vertex locations, a first bounding shape that indicates an area of the intensity image that depicts the road marking, ([0016], [0058]-[0060] teach the use of bounding shapes that indicate an area for which a road marking is located. These bounding shapes are defined as labeled segments with x,y locations for which they exist in the real world.
determining that the ([0060] teaches that the bounding box encapsulates a specific are and is labeled using class labels. [0058] teaches that these class labels have locations in the (x, y) realm associated with the bounding box these locations represent points in the environment. The labels indicating the bounding shapes are output by the machine learned models.)
causing, based at least on the first bounding shape(Fig. 1, item 134 and [0029] teach the system indicating the elements determined on a map system; Fig. 3, item 126; and [0055]-[0056] and [0060] teach the system indicating on a map a road marking within an environment) wherein the map is used by one or more second machine to navigate within the environment. ([0075] teaches a vehicle using the generated map to navigate in the environment. [0078] teaches this data can be shared between vehicles so a second vehicle can use the map to navigate)
Khadem does not teach representing vertex locations and vertex locations that the define bounding shapes; the first bounding shape being oriented orthogonal with respect to a direction of travel of the surface; determining, based at least on connecting the point locations, a second bounding shape indicating an area of the map that corresponds to the intensity image the second bounding shape also being oriented orthogonal with respect to the direction of travel of the surface; and the second bounding shape.
However, Bojarski teaches “[output data] representing vertex locations” ([0031] teaches using LiDAR data to output information relating to pixel locations in 2D space, these locations can serve as vertices for a bounding shape) “the vertex locations that the define” ([0031] teaches using LiDAR data to output information relating to pixel locations in 2D space, these locations can serve as vertices for a bounding shape) “based at least on connecting the point locations, a second bounding shape indicating an area of the map that corresponds to the intensity image;” ([0029], [0046] teaches generation of a new bounding shape in order to encompass all points of previously identified ground truth data, this includes the bounding shapes and vertices of previously identified information) and “the second bounding shape.” ([0046] teaches the generation of a second bounding shape)
It would have been prima facie obvious to one of ordinary skill in the art, before the effective filing date, to incorporate the teachings of Khadem with Bojarski; and have a reasonable expectation of success. Both relate to systems that can capture environmental data with at least LiDAR sensors and use some form of machine learning to process this information. As Bojarski teaches in [0005]-[0006] the use of DNNs to label and determine locations of road surfaces based on images present a way to speed up image processing. The DNNs are reliable and less costly. The labelling allows for the determination of vertexes for bounding shapes as the system can determine that these bounding shapes encompass a specific area in the environment, this includes road markings.
The combination of Khadem and Bojarski does not teach the first bounding shape being oriented orthogonal with respect to a direction of travel of the surface; and the second bounding shape also being oriented orthogonal with respect to the direction of travel of the surface.
However, Kumar teaches “the first bounding shape being oriented orthogonal with respect to a direction of travel of the surface;” and “the second bounding shape also being oriented orthogonal with respect to the direction of travel of the surface.” (Fig. 4 and [0062]/[0068] teach that a bounding shape was oriented in the direction of travel; Further citating [0071]-[0072] teach a bounding shape can be oriented orthogonal to the direction of travel. The system determines that the bounding shape of a crosswalk is orthogonal. The teachings show that the vehicle determines the angle of travel with relation to the detected object and only cares if the angle is within the perpendicular axis to the vehicle.)
It would have been prima facie obvious to one of ordinary skill in the art, before the effective filing date, to incorporate the teachings of Khadem and Bojarski with Kumar; and have a reasonable expectation of success. All relate to perception systems for vehicles as they travel in a road network. The systems can be used to sense road marking elements and provide bounding shapes around the markings. As taught in [0068] certain road marking elements are more important to a vehicle as it travels in a road network, this is based on the direction that the vehicle is traveling. The process of determining the direction of the bounding shape allows the vehicle system to disregard elements that are not needed for the vehicle to process. This system prevents too much processing power from going into measuring and mapping elements that are parallel to vehicle travel, i.e. lane lines. The vehicle’s processor saves vital time and energy by ignoring these elements.
Claims 9 and 18 are substantially similar and would be rejected based on the same rationale as recited above.
Regarding claim 2, Khadem teaches the method of claim 1 further comprising: determining, based at least on the first bounding shape, the location associated with the road marking within the environment. ([0059]-[0060] teach the system having segmented bounding boxes in an environment indicating a road marking type and location.)
Regarding claim 4, Khadem teaches the method of claim 1 wherein:
the point locations are defined using second coordinates associated with the map. ([0092] teaches the system aligning the 3d voxel space elements with a global 3d coordinate system)
Khadem does not teach the vertex locations are defined using first coordinates associated with the intensity image.
However, Bojarski teaches “the vertex locations are defined using first coordinates associated with the intensity image.” ([0031] teaches the system determining the coordinate locations of the bounding shape vertexes in relation to ground truth data determined from an output 2d image)
It would have been prima facie obvious to one of ordinary skill in the art, before the effective filing date, to incorporate the teachings of Khadem with Bojarski; and have a reasonable expectation of success. Both relate to systems that can capture environmental data with at least LiDAR sensors and use some form of machine learning to process this information. As Bojarski teaches in [0005]-[0006] the use of DNNs to label and determine locations of road surfaces based on images present a way to speed up image processing. The DNNs are reliable and less costly. The labelling allows for the determination of vertexes for bounding shapes as the system can determine that these bounding shapes encompass a specific area in the environment, this includes road markings.
Claim 12 is substantially similar and would be rejected based on the same rationale as recited above.
Regarding claim 5, Khadem teaches the method of claim 1, further comprising: determining, using the one or more machine learning models and based at least on the image data, a classification associated with the road marking; ([0050] and [0058] teach the system determining a classification to label a segment of the environment) and
causing the map to indicate the classification associated with the road marking. ([0029] teaches the system may cause a map to indicate a map element by name, i.e. classification; Fig. 3, item 126; and [0055]-[0056] and [0060] teach the system indicating on a map a road marking within an environment)
Claims 13 and 19 are substantially similar and would be rejected based on the same rationale as recited above.
Regarding claim 6, Khadem teaches the method of claim 1, wherein the intensity image comprises at least one of: a top-down intensity image corresponding to the at least the portion of the surface of the environment; ([0028], [0031], [0049], and at least [0055] teach the system determining a top-down view of the environment based on the determined LiDAR data) or
a top-down intensity image indicating one or more intensities associated with one or more second points corresponding to the at least the portion of the surface of the environment. ([0055]-[0056] teaches the system may indicate pixel intensity values in a top-down image representation of the environment)
Regarding claim 7, Khadem teaches the method of claim 1, further comprising: determining a location associated with the first machine that obtained the LiDAR data using the one or more LiDAR sensors, ([0051] teaches determining a vehicle location based on sensed data, including from a LiDAR; [0071] teaches the system using a SLAM or CLAMS method to determine its location in an environment, this can be based on at least the LiDAR or other received sensor information)
wherein the causing the map to indicate the location associated with the road marking within the environment is further based at least on the location associated with the first machine. ([0026] and [0051] teach the system associating the location of the vehicle to the sensed voxel data from the LiDAR sensors in the environment, wherein the sensed data will be located in the environment based on the associated vehicle location and heading)
Regarding claim 8, Khadem teaches the method of claim 1, further comprising: generating, based at least on the LiDAR data and motion data representing a motion of the first machine when obtaining the LiDAR data, point cloud data representing at least the one or more intensity values associated with the one or more points, ([0040] and [0042] teach the system generating a point cloud based at least on the sensed LiDAR data as well as associated vehicle data including velocity, which is a type of motion data)
wherein the generating the image data is based at least on the point cloud data. (Fig. 3 and at least [0049] teach the system generating the image based on the received LiDAR data, which is taught in [0040]/[0042] to be a 3D point cloud)
Regarding claim 14, Khadem teaches the system of claim 9, wherein the top-down image indicates one or more intensity values associated with the one or more points within at least the portion of the environment. ([0055]-[0056] teaches the system may indicate pixel intensity values in a top-down image representation of the environment)
Regarding claim 15, Khadem teaches the system of claim 9, wherein: the sensor data obtained using the one or more sensors comprises LiDAR data obtained using one or more LiDAR sensors; ([0024] teaches the system using a series of LiDAR sensors to obtain sensed data in an environment) and
the generation of the image data representative of the top-down image corresponding to the at least the portion of the environment is based at least on a point cloud associated with the LiDAR data, (Fig. 3 and at least [0049] teach the system generating the image based on the received LiDAR data, which is taught in [0040]/[0042] to be a 3D point cloud) the point cloud indicating at least the one or more intensity values associated with the one or more points. ([0042] teaches the 3D point cloud indicating the intensity information of the LiDAR data)
Regarding claim 16, Khadem teaches the system of claim 9, wherein the one or more processors are further to: determine a location associated with the first machine that obtained the sensor data using the one or more sensors, ([0051] teaches determining a vehicle location based on sensed data, including from a LiDAR; [0071] teaches the system using a SLAM or CLAMS method to determine its location in an environment, this can be based on at least the LiDAR or other received sensor information)
wherein the location of the road marking is encoded into the map data further based at least on the location associated with the first machine. ([0026] and [0051] teach the system associating the location of the vehicle to the sensed voxel data from the LiDAR sensors in the environment, wherein the sensed data will be located in the environment based on the associated vehicle location and heading)
Regarding claim 17, Khadem teaches the system of claim 9, wherein the system is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system implementing one or more large language models; a system for performing conversational AI operations; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computers. ([0021] teaches the system relating to an autonomous/semi-autonomous vehicle control and perception system. [0077]/[0080] teach parts of the system being stored remotely and accessed via wireless networks, i.e. a system in a data center, partially using cloud computers, an edge device. [0021] teaches the system using simulated data.)
Claim 20 is substantially similar and would be rejected based on the same rationale as recited above.
Regarding claim 21, Khadem teaches the processor of claim 18, wherein the top-down image includes a first color associated with the road markings and a second color associated with the surface based at least on the one or more intensity values. ([0044] teaches that the vehicle’s as they pass an area, can determine the color of various elements on the road and each element would be properly reflected in the intensity of the intensity image. [0078] further teaches that the system can determine the colors of road elements and store them with the aggregated data.)
Regarding claim 22, Khadem teaches the method of claim 1, wherein the causing, the map to indicate the location associated with the road marking within the environment comprises at least converting the area of the map that is indicated by ([0097]-[0098] teach the method of determining that the system will update the map data of a map database if it is determined that there is a mismatch in the data collected. This map update is done by matching map data.)
Khadem does not teach the second bounding shape.
However, Bojarski teaches “the second bounding shape.” ([0046] teaches the generation of a second bounding shape. [0157] teaches the system can update map data based on the detected map data)
It would have been prima facie obvious to one of ordinary skill in the art, before the effective filing date, to incorporate the teachings of Khadem with Bojarski; and have a reasonable expectation of success. Both relate to systems that can capture environmental data with at least LiDAR sensors and use some form of machine learning to process this information. As Bojarski teaches in [0005]-[0006] the use of DNNs to label and determine locations of road surfaces based on images present a way to speed up image processing. The DNNs are reliable and less costly. The labelling allows for the determination of vertexes for bounding shapes as the system can determine that these bounding shapes encompass a specific area in the environment, this includes road markings.
Regarding claim 23, Khadem teaches the method of claim 1.
Khadem does not teach wherein the one or more machine learning models are trained to determine the vertex locations that define the first bounding shape.
However, Bojarski teaches “wherein the one or more machine learning models are trained to determine the vertex locations that define the first bounding shape.” ([0030]-[0031] teaches, “the DNN 116 may be trained to output the bounding shape coordinates as pixel locations of two or more vertices of the bounding shape in 2D image-space.” This would teach the system as intended by the claim)
It would have been prima facie obvious to one of ordinary skill in the art, before the effective filing date, to incorporate the teachings of Khadem with Bojarski; and have a reasonable expectation of success. Both relate to systems that can capture environmental data with at least LiDAR sensors and use some form of machine learning to process this information. As Bojarski teaches in [0005]-[0006] the use of DNNs to label and determine locations of road surfaces based on images present a way to speed up image processing. The DNNs are reliable and less costly. The labelling allows for the determination of vertexes for bounding shapes as the system can determine that these bounding shapes encompass a specific area in the environment, this includes road markings.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS STRYKER whose telephone number is (571)272-4659. The examiner can normally be reached Monday-Friday 7:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Christian Chace can be reached at (571) 272-4190. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.S./Examiner, Art Unit 3665 /CHRISTIAN CHACE/Supervisory Patent Examiner, Art Unit 3665
Prosecution Insights