DETAILED ACTION
Response to Amendment
This Action is responsive to Applicant’s response filed on 07/29/2026. All claims are still pending
in the present application. This Action is made FINAL.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 07/30/2026 is/are compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Arguments
During prosecution, claim scope not solely on the basis of claim language, but also on giving claims their broadest reasonable construction in light of the specification as it would be interpreted by one of ordinary skill in the art. In re Am. Acad. of Sci. Tech. Ctr., 367 F.3d 1359, 1364 (Fed. Cir. 2004). See also Superguide Corp. v. DirecTV Enterprises, Inc., 358 F.3d 870, 875 (Fed. Cir. 2004) (“Though understanding the claim language may be aided by explanations contained in the written description, it is important not to import into a claim limitations that are not part of the claim.”). Additionally, “[t]hough understanding the claim language may be aided by the explanations contained in the written description, it is important not to import into a claim limitations that are not a part of the claim. For example, a particular embodiment appearing in the written description may not be read into a claim when the claim language is broader than the embodiment.” Superguide Corp. v. DirecTV Enterprises, Inc., 358 F.3d 870, 875 (Fed. Cir. 2004).
With respect to the 35 U.S.C. 101 rejection, in view of Applicant’s amendment to claim(s) 16-20, and upon reconsideration of the claim(s) as a whole, the rejection of claim(s) 16-20 under 35 U.S.C. 101 is withdrawn.
Applicant argues that Zhu fails to teach or suggest “processing, for each frame of the sequence of frames, the first-type sensor data measured by the one or more first-type sensors to generate one or more segmentation masks for the frame” because Zhu's pixel dilation operation is based on LiDAR point-cloud data rather than the two-dimensional image data identified by the Examiner as the first-type sensor data.
The Examiner respectfully disagrees. As set forth in the rejection, Zhu teaches receiving both image data from one or more cameras and three-dimensional LiDAR point-cloud data. Zhu further teaches, for each image frame, projecting the corresponding 3D point cloud onto the 2D image plane to produce mapped image/point-cloud data (Zhu, Paragraph [0040]). Zhu further teaches mapping the remaining road points in the RPS to corresponding pixels in the related image, thereby producing a mapped RPS and corresponding image pixels (Zhu, Paragraph [0043]). Importantly, Zhu's subsequent processing is not limited to processing the LiDAR point-cloud data in isolation. Zhu expressly teaches that the pixel dilation and post-processing module receives “the mapped RPS and corresponding image pixels” and performs pixel-domain processing to produce a “dense pixel-level road surface representation” (Zhu, Paragraphs [0044] – [0045]). Zhu further processes that dense road-surface representation by applying Gaussian smoothing and identifying connected components of the road surface area represented in the camera image frames (Zhu, Paragraph [0046]). Accordingly, the rejection does not rely on the sparse LiDAR points themselves as constituting the claimed segmentation mask. Rather, Zhu's mapped road points and corresponding image pixels are processed in the image/pixel domain to generate a dense pixel-level representation identifying the road-surface region, which reasonably corresponds to the claimed segmentation mask. Furthermore, Claim 1 does not require that the segmentation mask be generated exclusively from the first-type sensor data or without contribution from the second-type sensor data. Therefore, Applicant's characterization of Zhu as merely processing LiDAR point-cloud data does not address the entirety of Zhu's disclosed multimodal image/point-cloud processing, and Applicant's argument is not persuasive.
As a result, the argued features are written such that they read upon the cited references. Therefore, the previous rejection still applies.
Applicant further argues that Zhu fails to teach or suggest “generating a road mask representing road surface information based at least on the one or more segmentation masks generated for the frame” because Zhu's road-surface map is based on dilated LiDAR points rather than segmentation masks generated by processing the first-type sensor data.
The Examiner respectfully disagrees. As discussed above, Zhu's dense pixel-level road surface representation reasonably corresponds to the claimed segmentation mask. Zhu expressly teaches that the secondary non-road surface filtering and mapping module maps the remaining road points in the road points set (RPS) to corresponding pixels in the related image, thereby producing a mapped RPS and corresponding image pixels (Zhu, Paragraph [0043]). Zhu further teaches that the pixel dilation and post-processing module processes the mapped RPS and corresponding image pixels to generate a dense pixel-level road surface representation (Zhu, Paragraphs [0044] – [0045]). Accordingly, Zhu's road surface representation is generated through processing involving both the mapped LiDAR road points and their corresponding image pixels, rather than processing LiDAR point-cloud data in isolation. Furthermore, the Examiner finds that Zhu, as illustrated in Figure 8, processing block 1040, and described in Paragraph [0048], expressly discloses performing a secondary filtering and mapping operation on the image data and point cloud data, the processing block must perform some type of filtration using mask data to remove unwanted point cloud data and road data points. Therefore, Zhu indirectly uses road mask information for the filtering and mapping operation. As further described in Paragraph [0042], the secondary non-road surface filtering and mapping module performs additional filtering of the initially filtered road points set to remove non-road surface points. Thus, Zhu teaches filtering and mapping operations that distinguish road surface information from non-road surface information and establish correspondence between the remaining road points and the associated image pixels.
Zhu further teaches that the dense road surface representation produced by the pixel dilation and post-processing module is subsequently provided to the pixel smoothing module (Zhu, Paragraphs [0045] - [0046]). Specifically, Zhu teaches that “the pixel smoothing module 235 of the example embodiment can perform the processing operations corresponding to the first assumption (assumption 1) set forth above by applying a Gaussian smoothing filter to the dense road surface representation obtained from the pixel dilation and post-processing module 230” (Zhu, Paragraph [0046]). A person of ordinary skill in the art would understand that Gaussian smoothing employs a filtering kernel, sometimes referred to as a filtering mask, to process pixel information. Moreover, the pixel smoothing module applies Gaussian smoothing to that dense road-surface representation and identifies connected components of the road surface area represented in the camera image frames, retaining the appropriate contiguous road-surface area (Zhu, Paragraph [0046]). Zhu further teaches identifies the resulting output as a “drivable road surface map or representation” and teaches that the drivable road surface map distinguishes drivable areas of a roadway from non-drivable areas (Zhu, Paragraphs [0046] – [0047]). Zhu further expressly describes the processing sequence as performing pixel dilation and post-processing on the mapped RPS and corresponding image pixels, followed by pixel smoothing of the dense road-surface representation “to produce a drivable road surface map or representation” (Zhu, Paragraph [0048]).
Thus, Zhu discloses a processing sequence in which the dense pixel-level road surface representation corresponding to the claimed segmentation mask is further processed to produce the drivable road surface map corresponding to the claimed road mask. Accordingly, the road mask is generated “based at least on” the segmentation mask representation as required by claim 1. Furthermore, the fact that LiDAR derived information also contributes to the generation of Zhu's road surface representation does not establish otherwise. Claim 1 requires the road mask to be generated “based at least on” the segmentation masks; the claim does not require that the road mask be generated exclusively from the segmentation masks or independently of other sensor information. Therefore, Applicant's argument is not persuasive. As a result, the argued features are written such that they read upon the cited references. Therefore, the previous rejection still applies.
Additionally, the Examiner notes that Zhu's disclosure is not limited to generating a drivable road surface representation. Zhu further teaches filtering and mapping operations involving LiDAR point-cloud data and corresponding image pixels (Paragraphs [0042] – [0043] and Paragraph [0048]). These teachings are also relevant to the relationship between the image/LiDAR correspondence, the identified road surface information, and the filtering of point-cloud data. The Examiner further addresses the claimed filtering operation in view of the combined teachings of Zhu and Holzel below.
Applicant further argues that Zhu fails to teach or suggest “filtering, using the respective correlations and the road mask, the second-type sensor data to remove a portion of the second-type sensor data that is irrelevant to the unlabeled objects captured in the sequence of frames.” In particular, Applicant argues that Zhu's RANSAC and DBSCAN filtering operations do not use the subsequently generated drivable road-surface map.
Applicant's argument is not persuasive because it considers Zhu in isolation and does not address the rejection as actually made, which relies on the combined teachings of Zhu and Holzel for the claimed filtering and clustering operations. The Office Action does not rely solely on Zhu's RANSAC or DBSCAN operations to establish the claimed filtering operation. Rather, the Examiner expressly relied on Holzel in the Office Action for the claimed filtering operation. In particular, Holzel teaches preprocessing LiDAR point-cloud data for ground removal, wherein the ground is identified and removed from the LiDAR data stream “as a pre-filter process, in order to reduce clutter” (Holzel, Paragraph [0057]). Holzel further teaches that the filtering is not limited to ground removal, but may remove other objects identified as not important for subsequent processing (Holzel, Paragraph [0057]). After the ground has been removed from the point cloud, Holzel performs clustering on the remaining point-cloud data (Holzel, Paragraphs [0058] - [0059]). The Examiner expressly relied on these teachings in the Office Action for filtering irrelevant second-type sensor data and clustering the remaining second-type sensor data. The rejection further relies upon the sensor correlation teachings of the references. Zhu teaches correlating image data with LiDAR point-cloud data, including projecting the 3D point cloud onto the 2D image plane to produce mapped image/point-cloud data (Zhu, Paragraph [0040]). Holzel similarly teaches camera/LiDAR sensor fusion, including transforming a 3D LiDAR point to a corresponding 2D pixel in the camera image frame (Holzel, Paragraph [0069]).
Accordingly, the rejection does not require Zhu's RANSAC or DBSCAN operation, standing alone, to use Zhu's subsequently generated road surface map. Rather, the rejection is based on the combined teachings of Zhu and Holzel. Zhu provides correlated image/LiDAR road-surface information and a road-surface map identifying the road region, while Holzel provides the teaching of removing ground and other irrelevant portions of LiDAR point-cloud data prior to clustering the remaining point-cloud data. Therefore, it would have been obvious to one of ordinary skill in the art to apply Holzel's teaching of removing ground and other irrelevant LiDAR data prior to clustering to Zhu's correlated image/LiDAR road surface representation, thereby using the identified road region and the correspondence between the image and LiDAR data to identify and remove corresponding road/ground portions of the LiDAR data prior to clustering. Such a modification would predictably reduce irrelevant ground/clutter information in the point cloud prior to object clustering, consistent with Holzel's express teaching that its ground-removal preprocessing is performed “in order to reduce clutter.” This modification is also consistent with the rationale expressly provided in the Office Action for combining Zhu and Holzel, namely, “to improve the accuracy and robustness of the drivable road surface representation by removing irrelevant data and enhancing object delineation in the point cloud.”
Thus, even assuming Applicant is correct that Zhu's RANSAC and DBSCAN operations, considered by themselves, precede the generation of Zhu's final drivable road-surface map, that observation does not address or overcome the rejection based on the combined teachings of Zhu and Holzel. Applicant has not substantively addressed the Examiner's reliance on Holzel in the Office Action for removing ground and other irrelevant LiDAR data prior to clustering, or explained why the proposed combination of Zhu and Holzel would not have been obvious to one of ordinary skill in the art. Accordingly, Applicant's argument does not overcome the rejection. As a result, the argued features are written such that they read upon the cited references. Therefore, the previous rejection still applies.
Office Action Summary
Claim(s) 1-3, 10-12, and 16-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al (US 2019/0164018 A1) in view of Holzel et al (US 2022/0269900 A1).
Claim(s) 4-5, 9, 13-14, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al (US 2019/0164018 A1) in view of Holzel et al (US 2022/0269900 A1), further in view of Huang et al (US 2025/0054286 A1).
Claim(s) 6-8 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al (US 2019/0164018 A1) in view of Holzel et al (US 2022/0269900 A1) and Huang et al (US 2025/0054286 A1), further in view of Carvalho et al (US 2023/0053785 A1).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-3, 10-12, and 16-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al (US 2019/0164018 A1) in view of Holzel et al (US 2022/0269900 A1).
Regarding claim(s) 1, 10, and 16, Zhu teaches one or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform respective operations (Figure 1; and Paragraph [0058]), the respective operations comprising:
receiving sensor data capturing unlabeled objects in one or more scenes in a sequence of frames, wherein the sensor data comprises first-type sensor data measured by one or more first-type sensors and second-type sensor data measured by a second-type sensor (Figure 2; Paragraph [0026]: “The cameras may be still image cameras or motion video cameras […] the LIDAR device can be a 64-line Velodyne™ LIDAR […]”; and Paragraph [0039]: “The images or image data and timing data from the cameras can be obtained by the data collector 210 and processed for time and location alignment with the point cloud data received from the LIDAR”);
processing, for each frame of the sequence of frames, the first-type sensor data measured by the one or more first-type sensors to generate one or more segmentation masks for the frame (Figure 2; Figure 3; Paragraph [0040]: “the point cloud to image projection module 215 can project the 3D point cloud data onto the 2D image data to produce mapped image/point cloud data […] In particular, for each image frame, the point cloud to image projection module 215 can map the corresponding 3D point cloud, which contains width, height, and depth information, to the 2D image plane through a coordinate transformation”; Paragraph [0043]: “Given the remaining points in the RPS that can be considered likely points on the road surface, the secondary non-road surface filtering and mapping module 225 can map the remaining points in the RPS to the corresponding pixels in the related image. As a result, the secondary non-road surface filtering and mapping module 225 can produce a mapped RPS and corresponding image pixels”; Paragraph [0044]: “To produce a dense pixel-level road surface representation from the sparse point cloud points, the pixel dilation and post-processing module 230 can first perform a dilation operation, which adds every adjacent pixel corresponding to a current point cloud point (e.g., within 10 pixels in the x direction) in the RPS to the RPS. As a result, the adjacent pixels corresponding to point cloud points in the RPS blend together in a more coherent representation of the drivable road surface”; and Paragraph [0045]: “All pixels positioned straight below drivable pixels (i.e., drivable pixels are those included in the drivable road surface representation as described above) are drivable pixels. A pixel positioned below a drivable pixel means a pixel positioned between the drivable pixel and the vehicle or camera position; and 2) Pixels in areas around non-drivable pixels are non-drivable pixels”);
generating a road mask representing road surface information based at least on the one or more segmentation masks generated for the frame (Figure 3; Figure 8; Paragraph [0047]: “the multimodal drivable road surface detection module 200 of the example embodiments can produce a drivable road surface map that can be used to distinguish drivable areas of a roadway from non-drivable areas”; and Paragraph [0048]: “perform pixel dilation and post-processing operations on the mapped road points set (RPS) and corresponding image pixels (processing block 1050); and perform a pixel smoothing operation on the dense road surface representation to produce a drivable road surface map or representation (processing block 1060)”);
generating, for each frame in the sequence of frames, a correlation between the first-type sensor data and the second-type sensor data (Figure 3; Figure 8; and Paragraph [0040]: “the point cloud to image projection module 215 can project the 3D point cloud data onto the 2D image data to produce mapped image/point cloud data. In another example embodiment, the image data and the point cloud data can be transformed to a common coordinate space—the image data in a 2D image space and the point cloud data in a 3D space”); and
filtering, using the respective correlations and the road mask, the second-type sensor data to remove a portion of the second-type sensor data that is irrelevant to the unlabeled objects captured in the sequence of frames (Paragraph [0040]; Paragraph [0041]: “In order to determine the points in the point cloud data that correspond to the road surface or to remove non-road surface points, the primary non-road surface filtering module 220 can perform a primary […] filtering operation can be performed using a well-known Random Sample Consensus (RANSAC) process on the raw LIDAR point cloud data”; Paragraph [0042]: “To further distill and secondarily filter the LIDAR points in the initially filtered RPS, the secondary non-road surface filtering and mapping module 225 can apply a secondary filtering operation. In an example embodiment, a well-known DBSCAN process can be used”; Paragraph [0045]: “[…] removes adjacent non-road surface pixels from the RPS. These post processing operations can ensure that the pixel dilation and post-processing module 230 can produce a maximum set of drivable points in the RPS while minimizing the false positives […]”; and Paragraph [0046] – Paragraph [0048]).
Zhu fails to teach clustering the remaining second-type sensor data in the sequence of frames to classify one or more unlabeled objects captured in the sensor data.
However, Holzel teaches generating, for each frame in the sequence of frames, a correlation between the first-type sensor data and the second-type sensor data (Paragraph [0069]: “for the camera based fusion system, the rigid body transformation may be followed by an additional step of intrinsic transformation to convert a 3D point (x, y, z) to 2D pixel (u, v) in the image frame of the camera system. In this way, the corresponding pixels in the camera frame are also grouped together. Either a 2D bounding box or a polygon is constructed enclosing the projected points”);
filtering, using the respective correlations and the road mask, the second-type sensor data to remove a portion of the second-type sensor data that is irrelevant to the unlabeled objects captured in the sequence of frames (Figure 2; Figure 3; Paragraph [0043]: “[…] ground is first removed from the scene captured by one of the sensors 120 to reduce clutter […] the texture of the street 203 is removed from the LIDAR data stream”; and Paragraph [0057]: “In step 320, the data stream is pre-processed for ground removal […] The ground may then be removed from the data stream as a pre-filter process, in order to reduce clutter […] not only the ground may be removed from the data stream, but also other objects identified as not important for processing, such as a ceiling in a tunnel or in a parking garage”); and
clustering the remaining second-type sensor data in the sequence of frames to classify one or more unlabeled objects captured in the sensor data (Figure 3; Paragraph [0012]: “[…] removing a ground of an environmental scene within the first data stream, perform clustering on the ground-removed point cloud, and based on the clustered point cloud, create one or more features representing one or more region of interests”; Paragraph [0058]: “After the ground has been removed from the point cloud, clustering is performed in step S330. While there exist different approaches of clustering in the field, in some examples, Voxel Adjoint Clustering (VAC) is employed”; and Paragraph [0059]: “With the clustering of step 330, features within the point cloud objects are available and may be further processed in step 340. In particular, additional features may be created representing regions of interest”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhu's correlated image/LiDAR road surface processing in view of Holzel's teaching of removing ground and other irrelevant LiDAR data prior to clustering, such that the road region identified by Zhu and the correspondence between the image and LiDAR data are used to identify and remove corresponding road/ground portions of the LiDAR data prior to clustering. Such a modification would have predictably reduced irrelevant ground/clutter information in the point cloud prior to object clustering, thereby enhancing object delineation in the remaining point-cloud data, consistent with Holzel's express teaching that ground removal is performed “in order to reduce clutter.” This motivation for the combination of Zhu and Holzel is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim(s) 2, 11, and 17, Zhu as modified by Holzel, where Zhu teaches the one or more computer-readable storage media (Figure 1; and Paragraph [0058]) of claim 16, wherein the one or more first-type sensors comprise one or more cameras located on a vehicle, and the first-type sensor data comprise a respective sequence of two-dimensional image frames captured by a respective camera of the one or more cameras (Figure 1; and Paragraph [0015]: “a camera installed in the vehicle 105, as one of the devices of vehicle subsystems 140, can generate image and timing data that can be received by the control system 150. The control system 150 and the multimodal drivable road surface detection module 200 executing thereon can receive this image and timing data input”); and
wherein the second-type sensor comprises a LiDAR sensor, and the second-type sensor data comprise a sequence of three-dimensional point clouds (Figure 1; and Paragraph [0015]: “the multimodal drivable road surface detection module 200 can receive the image data, receive 3D LIDAR point cloud data, process the input LIDAR point clouds, project the 3D point clouds onto the two-dimensional (2D) image data, process the input vehicle metric data (e.g., vehicle position, speed, heading etc.), perform drivable road surface detection using the multimodal sensor data”).
Regarding claim(s) 3, 12, and 18, Zhu as modified by Holzel, where Zhu teaches the one or more computer-readable storage media (Figure 1; and Paragraph [0058]) of claim 17, wherein processing, for each frame of the sequence of frames, the first-type sensor data measured by the one or more first-type sensors to generate the one or more segmentation masks for the frame, comprises:
for each camera of the one or more cameras, processing a two-dimensional image captured by the camera for the frame to generate a respective detection result, the respective detection result indicating pixels in the two-dimensional image that represent a road surface for the frame (Figure 2; Figure 3; Paragraph [0043]: “the secondary non-road surface filtering and mapping module 225 can produce a mapped RPS and corresponding image pixels”; Paragraph [0044]: “To produce a dense pixel-level road surface representation from the sparse point cloud points, the pixel dilation and post-processing module 230 can first perform a dilation operation, which adds every adjacent pixel corresponding to a current point cloud point (e.g., within 10 pixels in the x direction) in the RPS to the RPS. As a result, the adjacent pixels corresponding to point cloud points in the RPS blend together in a more coherent representation of the drivable road surface”; and Paragraph [0045]: “All pixels positioned straight below drivable pixels (i.e., drivable pixels are those included in the drivable road surface representation as described above) are drivable pixels. A pixel positioned below a drivable pixel means a pixel positioned between the drivable pixel and the vehicle or camera position; and 2) Pixels in areas around non-drivable pixels are non-drivable pixels”); and
transforming the respective detection results into one or more segmentation masks for the frame, comprising:
transforming, for each of the respective detection results, a respective free space detection contour (read as “road surface representation”) into a corresponding segmentation mask for the frame (Figure 5; Paragraph [0043]: “the primary and secondary filtering operations […] removed the non-road surface points and corresponding image pixels from the sample image”; Paragraph [0045]: “[…] removes adjacent non-road surface pixels from the RPS. These post processing operations can ensure that the pixel dilation and post-processing module 230 can produce a maximum set of drivable points in the RPS while minimizing the false positives […]”; and Paragraph [0044]: “To produce a dense pixel-level road surface representation from the sparse point cloud points, the pixel dilation and post-processing module 230 can first perform a dilation operation, which adds every adjacent pixel corresponding to a current point cloud point (e.g., within 10 pixels in the x direction) in the RPS to the RPS. As a result, the adjacent pixels corresponding to point cloud points in the RPS blend together in a more coherent representation of the drivable road surface”).
Claim(s) 4-5, 9, 13-14, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al (US 2019/0164018 A1) in view of Holzel et al (US 2022/0269900 A1), further in view of Huang et al (US 2025/0054286 A1).
Regarding claim(s) 4, 13, and 19, Zhu as modified by Holzel, where Zhu teaches the one or more computer-readable storage media (Figure 1; and Paragraph [0058]) of claim 18, wherein the road mask comprises (Figure 3; Figure 8; Paragraph [0047]: “the multimodal drivable road surface detection module 200 of the example embodiments can produce a drivable road surface map that can be used to distinguish drivable areas of a roadway from non-drivable areas”; and Paragraph [0048]: “perform pixel dilation and post-processing operations on the mapped road points set (RPS) and corresponding image pixels (processing block 1050); and perform a pixel smoothing operation on the dense road surface representation to produce a drivable road surface map or representation (processing block 1060)”).
Zhu and Holzel fails to teach wherein the road mask comprises a bird’s eye view (BEV) road mask represented in a BEV coordinate frame, and wherein generating a road mask representing road surface information based at least on the one or more segmentation masks generated for the frame, comprises: for each frame of the sequence of frames, fusing the one or more segmentation masks for the frame to generate the BEV road mask for the frame.
However, Huang teaches wherein the road mask comprises a bird’s eye view (BEV) road mask represented in a BEV coordinate frame, and wherein generating a road mask representing road surface information based at least on the one or more segmentation masks generated for the frame (Figure 5; Paragraph [0007]: “determining a three-dimensional (3D) feature map of the one or more images based on a feature extraction algorithm; detecting objects in the one or more images by performing a multi-level refinement on the 2D feature map and the 3D feature map […] and performing a bird's eye view (BEV) segmentation using the 3D feature map”; and Paragraph [0038]: “an example of a bird's eye view (BEV) of a scene. In this view, environment around a vehicle may be seen in a top-down manner […]”), comprises:
for each frame of the sequence of frames, fusing the one or more segmentation masks for the frame to generate the BEV road mask for the frame (Figure 3; Figure 17; Paragraph [0036]: “image segmentation performed on a frame of a street view. For each pixel in the image, a class attribute(s) may be attached such as road surface, vegetation, lane marking, pedestrian, etc.”; Paragraph [0007]: “performing a bird's eye view (BEV) segmentation using the 3D feature map”; Paragraph [0090]: “a 2-dimensional (2D) feature extraction; performing (1704), a 3-dimensional (3D) feature extraction on the images; detecting (1706) objects in the images by fusing detection results from the 2D feature extraction and the 3D feature extraction”; and Paragraph [0040]: “BEV offers the opportunity to perform multi-sensor fusion, providing more robust data. BEV also allows easier temporal aggregation using sequential video inputs”).
Zhu discloses generating a drivable road surface map or representation based on image data and three-dimensional (3D) point cloud data, which corresponds to a road mask representing drivable regions. Huang discloses representing environmental data in a bird’s-eye-view (BEV) coordinate frame, as well as fusing information from multiple sources and across frames, and performing temporal aggregation using sequential inputs. Accordingly, Huang teaches generating segmentation outputs in a BEV coordinate frame and combining such outputs across frames to produce a unified BEV representation.
Therefore, it would have been obvious to one of ordinary skill in the art to combine Zhu, Holzel and Huang before the effective filing date of the claimed invention. The motivation for this combination of references would have been to fuse detection results from the 2D feature extraction and the 3D feature extraction and to perform temporal aggregation using sequential video inputs, thereby enabling a more comprehensive and robust representation of the environment. This motivation for the combination of 1st ref and 2nd ref is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim(s) 5, 14, and 20, Zhu as modified by Holzel and Huang, where Huang teaches the one or more computer-readable storage media (Figure 10; and Paragraph [0063]) of claim 19, wherein the operations further comprise:
where Huang teaches refining the BEV road mask Figure 5; Paragraph [0007]: “determining a three-dimensional (3D) feature map of the one or more images based on a feature extraction algorithm; detecting objects in the one or more images by performing a multi-level refinement on the 2D feature map and the 3D feature map […] and performing a bird's eye view (BEV) segmentation using the 3D feature map”; Paragraph [0038]: “an example of a bird's eye view (BEV) of a scene. In this view, environment around a vehicle may be seen in a top-down manner […]”; and Paragraph [0067]: “The 3D features may be refined using a refinement algorithm (stage 1410), which may be applied iteratively. Examples of such algorithms include Conv3D and Conv2D.”) by where Zhu teaches performing one or more morphological operations or one or more filtering operations, wherein the one or more filtering operations comprise using a bilateral filter, a median filter, or a Gaussian filter (Paragraph [0048]: “[…] perform a pixel smoothing operation on the dense road surface representation to produce a drivable road surface map or representation (processing block 1060)”; Paragraph [0041]: “filtering operation can be performed using a well-known Random Sample Consensus (RANSAC) process on the raw LIDAR point cloud data”; Paragraph [0042]: “To further distill and secondarily filter the LIDAR points in the initially filtered RPS, the secondary non-road surface filtering and mapping module 225 can apply a secondary filtering operation […] a well-known DBSCAN process can be used”; and Paragraph [0046]: “The pixel smoothing module 235 […] perform the processing operations corresponding to the first assumption (assumption 1) set forth above by applying a Gaussian smoothing filter to the dense road surface representation obtained from the pixel dilation and post-processing module 230”).
Regarding claim(s) 9, Zhu as modified by Holzel teaches the method of claim 3, where Zhu teaches wherein generating, for each frame in the sequence of frames, the correlation between the first-type sensor data and the second-type sensor data comprises:
matching the projected points with pixels in the two-dimensional images that represent the road surface (Figure 3; Figure 5; Paragraph [0043]: “the secondary non-road surface filtering and mapping module 225 can produce a mapped RPS and corresponding image pixels. As shown in FIG. 3, the mapped RPS can be provided as an output from the secondary non-road surface filtering and mapping module 225. An example of a sample image corresponding to the mapped RPS is shown in FIG. 5”; and Paragraph [0044]: “The pixel dilation and post-processing module 230 can receive the mapped RPS and corresponding image pixels produced by secondary non-road surface filtering and mapping module 225”).
Zhu and Holzel fails to teach projecting each point of the three-dimensional point cloud for the frame into a bird’s eye view (BEV) coordinate frame. However, Huang teaches to projecting each point of the three-dimensional point cloud for the frame into a bird’s eye view (BEV) coordinate frame (Figure 5; Paragraph [0007]: “determining a three-dimensional (3D) feature map of the one or more images based on a feature extraction algorithm; detecting objects in the one or more images by performing a multi-level refinement on the 2D feature map and the 3D feature map […] and performing a bird's eye view (BEV) segmentation using the 3D feature map”; and Paragraph [0038]: “an example of a bird's eye view (BEV) of a scene. In this view, environment around a vehicle may be seen in a top-down manner […]”).
Therefore, it would have been obvious to one of ordinary skill in the art to combine Zhu, Holzel, and Huang before the effective filing date of the claimed invention. The motivation for this combination of references would have been to provide a unified top-down spatial representation that facilitates more accurate and efficient association between point cloud data and road surface pixels. Accordingly, projecting each point of the 3D point cloud into a BEV coordinate frame and matching the projected points with pixels representing the road surface. This motivation for the combination of Zhu, Holzel, and Huang is/are supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Claim(s) 6-8 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al (US 2019/0164018 A1) in view of Holzel et al (US 2022/0269900 A1) and Huang et al (US 2025/0054286 A1), further in view of Carvalho et al (US 2023/0053785 A1).
Regarding claim(s) 6 and 15, Zhu as modified by Holzel and Huang, where Huang teaches the method of claim 4, further comprising:
generating a dense BEV road mask by aggregating a plurality of BEV road masks generated for respective frames (Figure 3; Figure 17; Paragraph [0036]: “image segmentation performed on a frame of a street view. For each pixel in the image, a class attribute(s) may be attached such as road surface, vegetation, lane marking, pedestrian, etc.”; Paragraph [0007]: “performing a bird's eye view (BEV) segmentation using the 3D feature map”; Paragraph [0090]: “a 2-dimensional (2D) feature extraction; performing (1704), a 3-dimensional (3D) feature extraction on the images; detecting (1706) objects in the images by fusing detection results from the 2D feature extraction and the 3D feature extraction”; and Paragraph [0040]: “BEV offers the opportunity to perform multi-sensor fusion, providing more robust data. BEV also allows easier temporal aggregation using sequential video inputs”).
Zhu, Holzel, and Huang fails to teach selecting a frame in the sequence of frames as a reference frame, converting one or more BEV road masks of the plurality of BEV road masks to the reference frame; and stacking the converted one or more BEV road masks to generate the dense BEV road mask.
However, Carvalho teaches to selecting a frame in the sequence of frames as a reference frame (read as “BEV vector space”), converting one or more BEV road masks of the plurality of BEV road masks to the reference frame (Figure 5; Abstract: “The features are projected into a vector space associated with a birds-eye view based on the machine learning model. The projected features are aggregated with other projected features associated with prior images”; Paragraph [0031]: “objects positioned about the vehicle are projected into this birds-eye view vector space effectuating the accurate identification of certain types of objects”; and Paragraph [0104]: “the system projects features determined from the images into a birds-eye view. For example, a transformer network included in a birds-eye view model may project the features into a consistent vector space.”); and
stacking (read as “aggregation”) the converted one or more BEV road masks to generate the dense BEV road mask (Figure 5; Paragraph [0037]: “[…] the feature queue may be spatially indexed such that information is aggregated over the previous threshold distance […]”; and Paragraph [0105]: “the system aggregates spatially indexed features […] a video module included in the birds-eye view network may be used to aggregate information which is determined within a threshold distance from the vehicle’s location”).
Zhu teaches generating a road surface representation (read as “a road mask”) for individual frames based on sensor data. Huang teaches generating bird’s-eye-view (BEV) representations and processing multiple frames, including stacking temporal BEV feature maps and generating BEV segmentation outputs for respective frames. Carvalho teaches projecting features from respective frames into a common vector space associated with a bird’s-eye view and aggregating the projected features with other projected features associated with prior images, thereby effectively converting data from multiple frames into a common BEV coordinate frame and combining such data to form a more complete representation. This projection into a common BEV vector space inherently defines a reference frame, and the aggregation of projected features across prior images corresponds to stacking data from multiple frames to generate a dense representation.
Therefore, it would have been obvious to one of ordinary skill in the art to combine Zhu, Holzel, Huang, and Carvalho before the effective filing date of the claimed invention. The motivation for this combination of references would have been to improve completeness and robustness of the road surface representation by incorporating information from multiple temporal observations. Additionally, aggregate projected features associated with prior images after projecting such features into a vector space associated with a birds-eye view, thereby enabling a more comprehensive and spatially consistent representation of the environment. This motivation for the combination of Zhu, Holzel, Huang, and Carvalho is/are supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim(s) 7, Zhu as modified by Holzel, Huang, and Carvalho teaches the method of claim 6, where Carvalho teaches wherein the reference frame (read as “BEV vector space”) is a frame in the median position in the sequence of frames (Figure 5; Paragraph [0104]: “the system projects features determined from the images into a birds-eye view. For example, a transformer network included in a birds-eye view model may project the features into a consistent vector space”; Paragraph [0037]: “[…] the feature queue may be spatially indexed such that information is aggregated over the previous threshold distance […]”; and Paragraph [0105]: “the system aggregates spatially indexed features […] a video module included in the birds-eye view network may be used to aggregate information which is determined within a threshold distance from the vehicle’s location”, Examiner’s Note: A person of ordinary skill in the art would recognize that a reference frame may be selected from any of the frames in the sequence (e.g., first, last, or intermediate frames), and selecting a median frame would have predictably balanced temporal information across the sequence and reduced bias toward earlier or later frames).
Regarding claim(s) 8, Zhu as modified by Holzel, Huang, and Carvalho teaches the method of claim 6, where Huang teaches wherein the one or more BEV road masks of the plurality of BEV road masks (Figure 3; Figure 17; Paragraph [0036]: “image segmentation performed on a frame of a street view. For each pixel in the image, a class attribute(s) may be attached such as road surface, vegetation, lane marking, pedestrian, etc.”; Paragraph [0007]: “performing a bird's eye view (BEV) segmentation using the 3D feature map”; Paragraph [0090]: “a 2-dimensional (2D) feature extraction; performing (1704), a 3-dimensional (3D) feature extraction on the images; detecting (1706) objects in the images by fusing detection results from the 2D feature extraction and the 3D feature extraction”; and Paragraph [0040]: “BEV offers the opportunity to perform multi-sensor fusion, providing more robust data. BEV also allows easier temporal aggregation using sequential video inputs”) are where Carvalho teaches selected at an interval in the sequence of frames (Figure 5; Paragraph [0104]: “the system projects features determined from the images into a birds-eye view. For example, a transformer network included in a birds-eye view model may project the features into a consistent vector space”; Paragraph [0037]: “[…] the feature queue may be spatially indexed such that information is aggregated over the previous threshold distance […]”; and Paragraph [0105]: “the system aggregates spatially indexed features […] a video module included in the birds-eye view network may be used to aggregate information which is determined within a threshold distance from the vehicle’s location”, Examiner’s Note: A person of ordinary skill in the art would have recognized that selecting BEV road masks at intervals from a sequence of frames would reduce computational burden and redundancy, as consecutive frames often contain overlapping information. Additionally, such interval-based selection would improve robustness by reducing noise and increasing temporal diversity, thereby yielding predictable improvements in efficiency and performance).
Relevant Prior Art Directed to State of Art
Cronvall et al (US 2025/0292596 A1) are relevant prior art not applied in the rejection(s) above. Cronvall discloses an apparatus for object detection, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: extract, using an encoder, a plurality of features from one or more images of an environment of the apparatus; determine, based on the plurality of features, a first detection of one or more objects and three-dimensional (3D) coordinates for the one or more objects; back-project the 3D coordinates of the one or more objects onto the one or more images; determine one or more regions of at least one first image of the one or more images based on the back-projection of the 3D coordinates of the one or more objects; and determine, based on the one or more regions of the at least one first image, a second detection of the one or more objects.
Sheu et al (US 2022/0172495 A1) are relevant prior art not applied in the rejection(s) above. Sheu discloses a computer-implemented method for training an instance segmentation model using sensor data having different dimensionalities, the method comprising: capturing first sensor data having a first dimensionality and second sensor data having a second dimensionality; assigning a first set of labels to the first sensor data to obtain labeled first sensor data and a second set of labels to the second sensor data to obtain labeled second sensor data; projecting the first sensor data onto the second sensor data to obtain training data; providing the training data as input to the instance segmentation model; and training the instance segmentation model using the training data.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONGBONG NAH whose telephone number is (571)272-1361. The examiner can normally be reached M - F: 7:30am - 4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Edward Urban can be reached on 571-272-7899. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JONGBONG NAH/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674
Prosecution Insights