DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-20 are pending for examination in the application filed 11/10/2025. Claims 1, 3, 5, 8-9, 11, and 13 have been amended. Claims 16-20 are new.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 11/10/2025 has been entered.
Priority
Acknowledgement is made of Applicant’s claim for foreign priority under 35
U.S.C. 119 (a)-(d). The certified copy has been filed in parent applications KR10-2022-0013290 filed on 01/28/2022.
Response to Arguments and Amendments
Applicant’s arguments with respect to independent claims 1, 8, and 9 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument, as facilitated by the newly added amendments.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-6, 8-14, and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Douillard (US10509947B1) in view of Holtz (US20190248487A1).
Regarding claim 1, Douillard teaches an object classification method for a LiDAR system ([col. 2 ln. 36-41] This disclosure describes methods, apparatuses, and systems for converting multi-dimensional data for image analysis. In some examples, the multi-dimensional data may include data captured by a LIDAR system for use in conjunction with a perception system for an autonomous vehicle. [col. 3 ln. 48-52] As mentioned above, after converting the three-dimensional LIDAR data to two-dimensional data, the operations can include inputting the two-dimensional data into a convolutional neural network (CNN) for segmentation and classification), the object classification method comprising:
projecting a three-dimensional point cloud acquired from an object by a LiDAR sensor into a two-dimensional image ([col. 2 ln. 56-65] As mentioned above, the three-dimensional LIDAR data can include a three dimensional map or point cloud which may be represented as a plurality of vectors emanating from a light emitter and terminating at an object or surface. To convert the three-dimensional LIDAR data to two-dimensional data, an example method can include mapping the LIDAR data to a three-dimensional projection shape and converting the projection shape to a two-dimensional plane, while subsequently performing segmentation and/or classification on the two-dimensional data) and extracting two-dimensional image-based feature information comprising shape information of the object ([col. 6 ln. 26-34] At operation 124, the process may include performing segmentation and/or classification on the multi-channel two-dimensional image. An example 126 illustrates an output of one such segmentation operation, including segmentation information 128 associated with an object. In some instances, the segmentation information 128 may include a segmentation identification (ID) associated with each pixel or LIDAR data point, for example, with a particular segmentation ID defining a particular object);
and determining a type of the object by processing the two-dimensional image-based feature information based on a convolutional neural network ([col. 3 ln. 48-52] As mentioned above, after converting the three-dimensional LIDAR data to two-dimensional data, the operations can include inputting the two-dimensional data into a convolutional neural network (CNN) for segmentation and classification. [col. 6 ln. 39-43] In some instances, after segmentation information has been generated, the segmentation information can be applied to three-dimensional data to isolate or segment one or more objects for classification on a per object basis),
wherein projecting the three-dimensional point cloud and extracting the two- dimensional image-based feature information comprises generating a grid map based on the two-dimensional image ([col 5. ln 62 - col. 6 ln. 3] At operation 112, the process can include converting the projection shape into a multi-channel two-dimensional image. In an example 114, the projection shape is converted into a plurality of two-dimensional arrays 116, 118, 120, and 122. In some instances, the two-dimensional arrays 116, 118, 120, and 122 may be considered to be individual “images”, with each image corresponding to an individual dimension of the LIDAR data stored in the cell 110 of the projection shape).
Douillard does not teach wherein each cell of the grid map includes a maximum depth, a minimum depth, and a number of corresponding LiDAR points as the two-dimensional image- based feature information.
Holtz, in the same field of endeavor of LiDAR image processing, teaches wherein each cell of the grid map includes a maximum depth, a minimum depth, and a number of corresponding LiDAR points as the two-dimensional image- based feature information ([0080] Once the particular cell is identified, the height value for the particular point is added to that cell. Specifically, in some embodiments, this process of adding height data to a cell may include, at step 706b, updating the height statistics for the cell such as, the number of data points collected, the mean height value, the median height value, minimum height value, maximum height value, the sum of the squared differences in height values, or any other type of statistic based on the aggregation of data points for a particular cell. [0078] a UAV 100 may be equipped with downward facing range sensors such as LIDAR to continually scan the ground below the UAV to collect height data).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Douillard with the teachings of Holtz to have each cell of the grid map include maximum depth, minimum depth, and a number of LiDAR points because ([Holtz 0059] each of the multiple cells representing a particular portion of a surface in the physical environment) and ([Holtz 0063] Associating characteristic data (e.g., height statistics) with portions of the ground map (i.e., cells) may simplify the calculations needed to later select an appropriate area on the surface to land thereby reducing required computing resources to store and compute characteristic data and reducing latency when selecting an appropriate landing area).
Regarding claim 2, Douillard and Holtz teach the method of claim 1. Douillard further teaches wherein extracting the two- dimensional image-based feature information comprises: extracting a two-dimensional image (rendering plane) of a yz plane by projecting the three-dimensional point cloud in an x-axis (normal) direction ([col. 4 ln. 5-10] the three-dimensional data can be converted to two-dimensional data by projecting the three-dimensional data onto a projection plane (also referred to as a rendering plane), which may include adapting or positioning a rendering perspective (e.g., the rendering plane) relative to the object. [col. 19. ln. 16-23] FIG. 10D is a plane view 1032 of an example of projecting three-dimensional segmented data onto a rendering plane. For example, the rendering plane 1010 is illustrated as plane with a normal vector 1042 in a direction of the point 1014. In 1034, the LIDAR data 1006′ can be orthographically or geometrically projected onto the rendering plane 1010 such that the LIDAR data 1006′ projected as projected data 1036 has a horizontal extent 1038. [col. 19 ln. 35-38] Further, it may be understood in the context of this disclosure that a vertical extent may be maximized or optimized (e.g., with respect to an (x, z) or (y, z) representation of the data));
extracting a two-dimensional image (rendering plane) of a zx plane by projecting the three-dimensional point cloud in a y-axis (normal) direction ([col. 19. ln. 16-23] FIG. 10D is a plane view 1032 of an example of projecting three-dimensional segmented data onto a rendering plane. For example, the rendering plane 1010 is illustrated as plane with a normal vector 1042 in a direction of the point 1014. In 1034, the LIDAR data 1006′ can be orthographically or geometrically projected onto the rendering plane 1010 such that the LIDAR data 1006′ projected as projected data 1036 has a horizontal extent 1038. [col. 19 ln. 35-38] Further, it may be understood in the context of this disclosure that a vertical extent may be maximized or optimized (e.g., with respect to an (x, z) or (y, z) representation of the data));
and extracting a two-dimensional image (rendering plane) of an xy plane by projecting the three-dimensional point cloud in a z-axis (normal) direction ([col. 19. ln. 16-23] FIG. 10D is a plane view 1032 of an example of projecting three-dimensional segmented data onto a rendering plane. For example, the rendering plane 1010 is illustrated as plane with a normal vector 1042 in a direction of the point 1014. In 1034, the LIDAR data 1006′ can be orthographically or geometrically projected onto the rendering plane 1010 such that the LIDAR data 1006′ projected as projected data 1036 has a horizontal extent 1038. [col. 18 ln. 66 - col. 19 ln. 4] FIG. 10B is a plane view 1020 of a simplified representation of the graphical representation illustrated in FIG. 10A. As illustrated, FIG. 10B represents a two-dimensional view (e.g., viewing the (x, y) axes) of FIG. 10A. The simplified data 1006′ represents simplified data compared to the LIDAR data 1006 illustrated in FIG. 10A).
Regarding claim 3, Douillard and Holtz teach the method of claim 2. Douillard further teaches wherein extracting the two- dimensional image-based feature information comprises: setting a grid in each of the two-dimensional image of the yz plane, the two-dimensional image of the zx plane, and the two-dimensional image of the xy plane ([col. 18 ln. 66 - col. 19 ln. 4] FIG. 10B is a plane view 1020 of a simplified representation of the graphical representation illustrated in FIG. 10A. As illustrated, FIG. 10B represents a two-dimensional view (e.g., viewing the (x, y) axes) of FIG. 10A. The simplified data 1006′ represents simplified data compared to the LIDAR data 1006 illustrated in FIG. 10A. [col. 19 ln. 35-38] Further, it may be understood in the context of this disclosure that a vertical extent may be maximized or optimized (e.g., with respect to an (x, z) or (y, z) representation of the data. [col. 22 ln. 8-19] At operation 1310, the process can include projecting the data associated with the three-dimensional object onto the rendering plane to generate a two dimensional representation of the object. In some instances, the operation 1310 may include determining a projection type for projecting the data onto the rendering plane. For example, a projection type may utilize perspective geometry with the virtual LIDAR sensor as the focus of the perspective. By way of another example, a projection type may utilize orthogonal geometry for projecting data to the projection plane. In some instances, the rendering plane may include any number of cells, corresponding to a resolution of the rendering plane);
and storing information on a physical quantity for computing the shape information of the object in a grid cell of the grid ([col. 16 ln. 12-16] FIG. 6D illustrates an example of projection types for use in converting multi-dimensional data for image analysis. In some instances, a cell 622 of the spherical projection shape may be associated with three dimensional information, as described herein. [col. 16 ln. 52-63] FIG. 7B illustrates an example 714 of determining a projection shape and projecting the three-dimensional data onto the projection shape. The example 714 illustrates a partial spherical projection shape 716 having at least one cell 718 through which the vector 708 passes. As described above, the vector 708 may be associated with captured data such as (x, y, z) coordinates of the point P 710 on the building 706. Further, the vector 708 maybe associated with the surface normal vector 712 associated with the point P 710. The information associated with the vector 708 may be stored in the cell 718 and further processed for converting, as described herein).
Regarding claim 4, Douillard and Holtz teach the method of claim 3. Douillard further teaches wherein setting the grid in each of the two-dimensional image of the yz plane, the two-dimensional image of the zx plane, and the two-dimensional image of the xy plane comprises setting a grid ([col. 22 ln. 8-19] At operation 1310, the process can include projecting the data associated with the three-dimensional object onto the rendering plane to generate a two dimensional representation of the object. In some instances, the operation 1310 may include determining a projection type for projecting the data onto the rendering plane. For example, a projection type may utilize perspective geometry with the virtual LIDAR sensor as the focus of the perspective. By way of another example, a projection type may utilize orthogonal geometry for projecting data to the projection plane. In some instances, the rendering plane may include any number of cells, corresponding to a resolution of the rendering plane).
Douillard does not teach setting a grid of a same NxM dimension in each of the two-dimensional images.
Holtz teaches setting a grid of a same NxM dimension in each of the two-dimensional images ([0060] As shown in FIG. 5A, a landing system 150 associated with the UAV 100 may maintain and continually update a two-dimensional (2D) ground map 510 comprising multiple cells. In the example depicted in FIG. 5A, the cells of the ground map 510 are rectangular in shape and are arranged in a rectangular 2D grid that is M cells wide by M cells long…an alternative ground map may be M1 cells wide by M2 cells long where M1 does not equal M2. [0061] In some embodiments, steps 402 and 404 are performed continually as the UAV 100 is in flight over the physical environment 502. In other words, as new sensor data is received while the UAV is in flight, the characteristic data associated with each cell in the ground map 510 is continually updated to reflect the characteristics of the portion of the surface of the physical environment 502 associated with that cell).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Douillard with the teachings of Holtz to set a grid of NxM dimensions because “each of the multiple cells [is] representing a particular portion of a surface in the physical environment” and “associating characteristic data (e.g., height statistics) with portions of the ground map (i.e., cells) may simplify the calculations needed to later select an appropriate area on the surface to land thereby reducing required computing resources to store and compute characteristic data and reducing latency when selecting an appropriate landing area” [Holtz 0059 and 0063].
Regarding claim 5, Douillard and Holtz teach the method of claim 3. Holtz teaches wherein storing the information on the physical quantity for computing the shape information of the object in the grid cell of the grid comprises: for points in each grid cell, determining vertical distances from the point to a projection plane and storing a value of a largest vertical distance in each grid cell to create a first specific information map; storing a value of a smallest vertical distance in each grid cell to create a second specific information map; and storing the number of the points comprised in each grid cell to create a third feature information map ([0080] Once the particular cell is identified, the height value for the particular point is added to that cell. Specifically, in some embodiments, this process of adding height data to a cell may include, at step 706b, updating the height statistics for the cell such as, the number of data points collected, the mean height value, the median height value, minimum height value, maximum height value, the sum of the squared differences in height values, or any other type of statistic based on the aggregation of data points for a particular cell. [0073] As previously discussed, characteristic data such as height statistics can be generated and maintained for multiple cells in a continually updated ground map based on sensor data received from one or more sensors, for example, as described with respect to FIGS. 5A-5B. [0094] For example, characteristic data associated with height values and/or semantic information may be added to the same ground map or to separate overlapping ground maps).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Douillard with the teachings of Holtz to create specific information maps of include largest height, smallest height, and a number of LiDAR points because ([Holtz 0059] each of the multiple cells representing a particular portion of a surface in the physical environment) and ([Holtz 0063] Associating characteristic data (e.g., height statistics) with portions of the ground map (i.e., cells) may simplify the calculations needed to later select an appropriate area on the surface to land thereby reducing required computing resources to store and compute characteristic data and reducing latency when selecting an appropriate landing area).
Regarding claim 6, Douillard and Holtz teach the method of claim 1. Douillard further teaches wherein determining the type of the object by processing the two-dimensional image-based feature information based on the convolutional neural network ([col. 3 ln. 48-52] As mentioned above, after converting the three-dimensional LIDAR data to two-dimensional data, the operations can include inputting the two-dimensional data into a convolutional neural network (CNN) for segmentation and classification) comprises determining the type of the object as a passenger vehicle, a commercial vehicle, a road boundary, a pedestrian, or a two-wheeled vehicle ([col. 11 ln. 5-10] The classification module 316 may include functionality to receive segmented data and to identify a type of object represented by the data. For example, the classification module 316 may classify one or more objects, including but not limited to cars, buildings, pedestrians, bicycles, trees, free space, occupied space, street signs, lane markings, etc.).
Regarding claim 8, Douillard teaches a non-transitory computer-readable storage medium storing a program ([col. 26 ln. 60-67] For example, the present disclosure can be provided as a computer program product, as outlined above. In this environment, the embodiments can include a machine-readable medium having instructions stored on it. The instructions can be used to program any processor or processors (or other electronic devices) to perform a process or method according to the present exemplary embodiments) for executing
an object classification method for a LiDAR system ([col. 2 ln. 36-41] This disclosure describes methods, apparatuses, and systems for converting multi-dimensional data for image analysis. In some examples, the multi-dimensional data may include data captured by a LIDAR system for use in conjunction with a perception system for an autonomous vehicle. [col. 3 ln. 48-52] As mentioned above, after converting the three-dimensional LIDAR data to two-dimensional data, the operations can include inputting the two-dimensional data into a convolutional neural network (CNN) for segmentation and classification), the program comprising instructions for:
projecting a three-dimensional point cloud acquired from an object by a LiDAR sensor into a two-dimensional image ([col. 2 ln. 56-65] As mentioned above, the three-dimensional LIDAR data can include a three dimensional map or point cloud which may be represented as a plurality of vectors emanating from a light emitter and terminating at an object or surface. To convert the three-dimensional LIDAR data to two-dimensional data, an example method can include mapping the LIDAR data to a three-dimensional projection shape and converting the projection shape to a two-dimensional plane, while subsequently performing segmentation and/or classification on the two-dimensional data) and extracting two-dimensional image-based feature information comprising shape information of the object ([col. 6 ln. 26-34] At operation 124, the process may include performing segmentation and/or classification on the multi-channel two-dimensional image. An example 126 illustrates an output of one such segmentation operation, including segmentation information 128 associated with an object. In some instances, the segmentation information 128 may include a segmentation identification (ID) associated with each pixel or LIDAR data point, for example, with a particular segmentation ID defining a particular object);
determining a type of the object by processing the two-dimensional image-based feature information based on a convolutional neural network ([col. 3 ln. 48-52] As mentioned above, after converting the three-dimensional LIDAR data to two-dimensional data, the operations can include inputting the two-dimensional data into a convolutional neural network (CNN) for segmentation and classification),
wherein projecting the three-dimensional point cloud and extracting the two- dimensional image-based feature information comprises generating a grid map based on the two-dimensional image ([col 5. ln 62 - col. 6 ln. 3] At operation 112, the process can include converting the projection shape into a multi-channel two-dimensional image. In an example 114, the projection shape is converted into a plurality of two-dimensional arrays 116, 118, 120, and 122. In some instances, the two-dimensional arrays 116, 118, 120, and 122 may be considered to be individual “images”, with each image corresponding to an individual dimension of the LIDAR data stored in the cell 110 of the projection shape).
Douillard does not teach wherein each cell of the grid map includes a maximum depth, a minimum depth, and a number of corresponding LiDAR points as the two-dimensional image- based feature information.
Holtz, in the same field of endeavor of LiDAR image processing, teaches wherein each cell of the grid map includes a maximum depth, a minimum depth, and a number of corresponding LiDAR points as the two-dimensional image- based feature information ([0080] Once the particular cell is identified, the height value for the particular point is added to that cell. Specifically, in some embodiments, this process of adding height data to a cell may include, at step 706b, updating the height statistics for the cell such as, the number of data points collected, the mean height value, the median height value, minimum height value, maximum height value, the sum of the squared differences in height values, or any other type of statistic based on the aggregation of data points for a particular cell. [0078] a UAV 100 may be equipped with downward facing range sensors such as LIDAR to continually scan the ground below the UAV to collect height data).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the medium of Douillard with the teachings of Holtz to have each cell of the grid map include maximum depth, minimum depth, and a number of LiDAR points because ([Holtz 0059] each of the multiple cells representing a particular portion of a surface in the physical environment) and ([Holtz 0063] Associating characteristic data (e.g., height statistics) with portions of the ground map (i.e., cells) may simplify the calculations needed to later select an appropriate area on the surface to land thereby reducing required computing resources to store and compute characteristic data and reducing latency when selecting an appropriate landing area).
Regarding claim 9, Douillard teaches a LiDAR system ([col. 2 ln. 36-41] This disclosure describes methods, apparatuses, and systems for converting multi-dimensional data for image analysis. In some examples, the multi-dimensional data may include data captured by a LIDAR system for use in conjunction with a perception system for an autonomous vehicle) comprising: a LiDAR sensor ([col. 5 ln. 15-20] At operation 102, the process can include receiving at least one three-dimensional LIDAR dataset. In some instances, the operation 102 may include receiving a plurality of LIDAR datasets from a plurality of LIDAR sensors operated in connection with a perception system of an autonomous vehicle); and a LiDAR signal processing device ([col. 8 ln. 56-67] FIG. 3 illustrates an example architecture for implementing the data conversion for image analysis…For example, the computer system(s) 302 may include a LIDAR module 304, a camera module 306, a Radar module 308, a projection shape module 310, a dimension conversion module 312, a segmentation module 314, a classification module 316, an object isolation module 318, a rendering perspective module 320, and a trajectory module 322) configured to:
project a three-dimensional point cloud acquired from an object by the LiDAR sensor into a two-dimensional image ([col. 2 ln. 56-65] As mentioned above, the three-dimensional LIDAR data can include a three dimensional map or point cloud which may be represented as a plurality of vectors emanating from a light emitter and terminating at an object or surface. To convert the three-dimensional LIDAR data to two-dimensional data, an example method can include mapping the LIDAR data to a three-dimensional projection shape and converting the projection shape to a two-dimensional plane, while subsequently performing segmentation and/or classification on the two-dimensional data) and extracting two-dimensional image-based feature information comprising shape information of the object ([col. 6 ln. 26-34] At operation 124, the process may include performing segmentation and/or classification on the multi-channel two-dimensional image. An example 126 illustrates an output of one such segmentation operation, including segmentation information 128 associated with an object. In some instances, the segmentation information 128 may include a segmentation identification (ID) associated with each pixel or LIDAR data point, for example, with a particular segmentation ID defining a particular object);
generate a grid map based on the two-dimensional image ([col 5. ln 62 - col. 6 ln. 3] At operation 112, the process can include converting the projection shape into a multi-channel two-dimensional image. In an example 114, the projection shape is converted into a plurality of two-dimensional arrays 116, 118, 120, and 122. In some instances, the two-dimensional arrays 116, 118, 120, and 122 may be considered to be individual “images”, with each image corresponding to an individual dimension of the LIDAR data stored in the cell 110 of the projection shape);
and determine a type of the object by processing the two-dimensional image-based feature information based on a convolutional neural network ([col. 3 ln. 48-52] As mentioned above, after converting the three-dimensional LIDAR data to two-dimensional data, the operations can include inputting the two-dimensional data into a convolutional neural network (CNN) for segmentation and classification).
Douillard does not teach wherein each cell of the grid map includes a maximum depth, a minimum depth, and a number of corresponding LiDAR points as the two-dimensional image- based feature information.
Holtz, in the same field of endeavor of LiDAR image processing, teaches wherein each cell of the grid map includes a maximum depth, a minimum depth, and a number of corresponding LiDAR points as the two-dimensional image- based feature information ([0080] Once the particular cell is identified, the height value for the particular point is added to that cell. Specifically, in some embodiments, this process of adding height data to a cell may include, at step 706b, updating the height statistics for the cell such as, the number of data points collected, the mean height value, the median height value, minimum height value, maximum height value, the sum of the squared differences in height values, or any other type of statistic based on the aggregation of data points for a particular cell. [0078] a UAV 100 may be equipped with downward facing range sensors such as LIDAR to continually scan the ground below the UAV to collect height data).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the system of Douillard with the teachings of Holtz to have each cell of the grid map include maximum depth, minimum depth, and a number of LiDAR points because ([Holtz 0059] each of the multiple cells representing a particular portion of a surface in the physical environment) and ([Holtz 0063] Associating characteristic data (e.g., height statistics) with portions of the ground map (i.e., cells) may simplify the calculations needed to later select an appropriate area on the surface to land thereby reducing required computing resources to store and compute characteristic data and reducing latency when selecting an appropriate landing area).
Regarding claim 10, Douillard and Holtz teach the system of claim 9. Douillard further teaches wherein the LiDAR signal processing device is configured to: extract a two-dimensional image (rendering plane) of a yz plane by projecting the three-dimensional point cloud in an x-axis (normal) direction ([col. 4 ln. 5-10] the three-dimensional data can be converted to two-dimensional data by projecting the three-dimensional data onto a projection plane (also referred to as a rendering plane), which may include adapting or positioning a rendering perspective (e.g., the rendering plane) relative to the object. [col. 19. ln. 16-23] FIG. 10D is a plane view 1032 of an example of projecting three-dimensional segmented data onto a rendering plane. For example, the rendering plane 1010 is illustrated as plane with a normal vector 1042 in a direction of the point 1014. In 1034, the LIDAR data 1006′ can be orthographically or geometrically projected onto the rendering plane 1010 such that the LIDAR data 1006′ projected as projected data 1036 has a horizontal extent 1038. [col. 19 ln. 35-38] Further, it may be understood in the context of this disclosure that a vertical extent may be maximized or optimized (e.g., with respect to an (x, z) or (y, z) representation of the data));
extract a two-dimensional image (rendering plane) of a zx plane by projecting the three-dimensional point cloud in a y-axis (normal) direction ([col. 19. ln. 16-23] FIG. 10D is a plane view 1032 of an example of projecting three-dimensional segmented data onto a rendering plane. For example, the rendering plane 1010 is illustrated as plane with a normal vector 1042 in a direction of the point 1014. In 1034, the LIDAR data 1006′ can be orthographically or geometrically projected onto the rendering plane 1010 such that the LIDAR data 1006′ projected as projected data 1036 has a horizontal extent 1038. [col. 19 ln. 35-38] Further, it may be understood in the context of this disclosure that a vertical extent may be maximized or optimized (e.g., with respect to an (x, z) or (y, z) representation of the data));
and extract a two-dimensional image (rendering plane) of an xy plane by projecting the three-dimensional point cloud in a z-axis (normal) direction ([col. 19. ln. 16-23] FIG. 10D is a plane view 1032 of an example of projecting three-dimensional segmented data onto a rendering plane. For example, the rendering plane 1010 is illustrated as plane with a normal vector 1042 in a direction of the point 1014. In 1034, the LIDAR data 1006′ can be orthographically or geometrically projected onto the rendering plane 1010 such that the LIDAR data 1006′ projected as projected data 1036 has a horizontal extent 1038. [col. 18 ln. 66 - col. 19 ln. 4] FIG. 10B is a plane view 1020 of a simplified representation of the graphical representation illustrated in FIG. 10A. As illustrated, FIG. 10B represents a two-dimensional view (e.g., viewing the (x, y) axes) of FIG. 10A. The simplified data 1006′ represents simplified data compared to the LIDAR data 1006 illustrated in FIG. 10A).
Regarding claim 11, Douillard and Holtz teach the system of claim 10. Douillard further teaches wherein the LiDAR signal processing device is configured to: set a grid in each of the two-dimensional image of the yz plane, the two-dimensional image of the zx plane, and the two-dimensional image of the xy plane ([col. 18 ln. 66 - col. 19 ln. 4] FIG. 10B is a plane view 1020 of a simplified representation of the graphical representation illustrated in FIG. 10A. As illustrated, FIG. 10B represents a two-dimensional view (e.g., viewing the (x, y) axes) of FIG. 10A. The simplified data 1006′ represents simplified data compared to the LIDAR data 1006 illustrated in FIG. 10A. [col. 19 ln. 35-38] Further, it may be understood in the context of this disclosure that a vertical extent may be maximized or optimized (e.g., with respect to an (x, z) or (y, z) representation of the data. [col. 22 ln. 8-19] At operation 1310, the process can include projecting the data associated with the three-dimensional object onto the rendering plane to generate a two dimensional representation of the object. In some instances, the operation 1310 may include determining a projection type for projecting the data onto the rendering plane. For example, a projection type may utilize perspective geometry with the virtual LIDAR sensor as the focus of the perspective. By way of another example, a projection type may utilize orthogonal geometry for projecting data to the projection plane. In some instances, the rendering plane may include any number of cells, corresponding to a resolution of the rendering plane);
and store information on a physical quantity for computing the shape information of the object in a grid cell of the grid ([col. 16 ln. 12-16] FIG. 6D illustrates an example of projection types for use in converting multi-dimensional data for image analysis. In some instances, a cell 622 of the spherical projection shape may be associated with three dimensional information, as described herein. [col. 16 ln. 52-63] FIG. 7B illustrates an example 714 of determining a projection shape and projecting the three-dimensional data onto the projection shape. The example 714 illustrates a partial spherical projection shape 716 having at least one cell 718 through which the vector 708 passes. As described above, the vector 708 may be associated with captured data such as (x, y, z) coordinates of the point P 710 on the building 706. Further, the vector 708 maybe associated with the surface normal vector 712 associated with the point P 710. The information associated with the vector 708 may be stored in the cell 718 and further processed for converting, as described herein).
Regarding claim 12, Douillard and Holtz teach the system of claim 11. Douillard does not teach wherein each of the two-dimensional images has a grid of the same NxM dimension.
Holtz teaches wherein each of the two-dimensional images has a grid of the same NxM dimension. ([0060] As shown in FIG. 5A, a landing system 150 associated with the UAV 100 may maintain and continually update a two-dimensional (2D) ground map 510 comprising multiple cells. In the example depicted in FIG. 5A, the cells of the ground map 510 are rectangular in shape and are arranged in a rectangular 2D grid that is M cells wide by M cells long…an alternative ground map may be M1 cells wide by M2 cells long where M1 does not equal M2. [0061] In some embodiments, steps 402 and 404 are performed continually as the UAV 100 is in flight over the physical environment 502. In other words, as new sensor data is received while the UAV is in flight, the characteristic data associated with each cell in the ground map 510 is continually updated to reflect the characteristics of the portion of the surface of the physical environment 502 associated with that cell).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the system of Douillard with the teachings of Holtz to set a grid of NxM dimensions because “each of the multiple cells [is] representing a particular portion of a surface in the physical environment” and “associating characteristic data (e.g., height statistics) with portions of the ground map (i.e., cells) may simplify the calculations needed to later select an appropriate area on the surface to land thereby reducing required computing resources to store and compute characteristic data and reducing latency when selecting an appropriate landing area” [Holtz 0059 and 0063].
Regarding claim 13, Douillard and Holtz teach the system of claim 11. Holtz teaches wherein the LIDAR signal processing device is configured to: for points in each grid cell, determining vertical distances from the point to a projection plane and storing a value of a largest vertical distance in each grid cell to create a first specific information map; storing a value of a smallest vertical distance in each grid cell to create a second specific information map; and storing the number of the points comprised in each grid cell to create a third feature information map ([0080] Once the particular cell is identified, the height value for the particular point is added to that cell. Specifically, in some embodiments, this process of adding height data to a cell may include, at step 706b, updating the height statistics for the cell such as, the number of data points collected, the mean height value, the median height value, minimum height value, maximum height value, the sum of the squared differences in height values, or any other type of statistic based on the aggregation of data points for a particular cell. [0073] As previously discussed, characteristic data such as height statistics can be generated and maintained for multiple cells in a continually updated ground map based on sensor data received from one or more sensors, for example, as described with respect to FIGS. 5A-5B. [0094] For example, characteristic data associated with height values and/or semantic information may be added to the same ground map or to separate overlapping ground maps).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the system of Douillard with the teachings of Holtz to create specific information maps of include largest depth, smallest depth, and a number of LiDAR points because ([Holtz 0059] each of the multiple cells representing a particular portion of a surface in the physical environment) and ([Holtz 0063] Associating characteristic data (e.g., height statistics) with portions of the ground map (i.e., cells) may simplify the calculations needed to later select an appropriate area on the surface to land thereby reducing required computing resources to store and compute characteristic data and reducing latency when selecting an appropriate landing area).
Regarding claim 14, Douillard and Holtz teach the system of claim 9. Douillard further teaches wherein the type of the object comprises a passenger vehicle, a commercial vehicle, a road boundary, a pedestrian, or a two-wheeled vehicle ([col. 11 ln. 5-10] The classification module 316 may include functionality to receive segmented data and to identify a type of object represented by the data. For example, the classification module 316 may classify one or more objects, including but not limited to cars, buildings, pedestrians, bicycles, trees, free space, occupied space, street signs, lane markings, etc.).
Regarding claim 16, Douillard teaches the medium of claim 8. Douillard further teaches wherein the instructions for extracting the two- dimensional image-based feature information comprises instructions for: extracting a two-dimensional image (rendering plane) of a yz plane by projecting the three-dimensional point cloud in an x-axis (normal) direction ([col. 4 ln. 5-10] the three-dimensional data can be converted to two-dimensional data by projecting the three-dimensional data onto a projection plane (also referred to as a rendering plane), which may include adapting or positioning a rendering perspective (e.g., the rendering plane) relative to the object. [col. 19. ln. 16-23] FIG. 10D is a plane view 1032 of an example of projecting three-dimensional segmented data onto a rendering plane. For example, the rendering plane 1010 is illustrated as plane with a normal vector 1042 in a direction of the point 1014. In 1034, the LIDAR data 1006′ can be orthographically or geometrically projected onto the rendering plane 1010 such that the LIDAR data 1006′ projected as projected data 1036 has a horizontal extent 1038. [col. 19 ln. 35-38] Further, it may be understood in the context of this disclosure that a vertical extent may be maximized or optimized (e.g., with respect to an (x, z) or (y, z) representation of the data));
extracting a two-dimensional image (rendering plane) of a zx plane by projecting the three-dimensional point cloud in a y-axis (normal) direction ([col. 19. ln. 16-23] FIG. 10D is a plane view 1032 of an example of projecting three-dimensional segmented data onto a rendering plane. For example, the rendering plane 1010 is illustrated as plane with a normal vector 1042 in a direction of the point 1014. In 1034, the LIDAR data 1006′ can be orthographically or geometrically projected onto the rendering plane 1010 such that the LIDAR data 1006′ projected as projected data 1036 has a horizontal extent 1038. [col. 19 ln. 35-38] Further, it may be understood in the context of this disclosure that a vertical extent may be maximized or optimized (e.g., with respect to an (x, z) or (y, z) representation of the data));
and extracting a two-dimensional image (rendering plane) of an xy plane by projecting the three-dimensional point cloud in a z-axis (normal) direction ([col. 19. ln. 16-23] FIG. 10D is a plane view 1032 of an example of projecting three-dimensional segmented data onto a rendering plane. For example, the rendering plane 1010 is illustrated as plane with a normal vector 1042 in a direction of the point 1014. In 1034, the LIDAR data 1006′ can be orthographically or geometrically projected onto the rendering plane 1010 such that the LIDAR data 1006′ projected as projected data 1036 has a horizontal extent 1038. [col. 18 ln. 66 - col. 19 ln. 4] FIG. 10B is a plane view 1020 of a simplified representation of the graphical representation illustrated in FIG. 10A. As illustrated, FIG. 10B represents a two-dimensional view (e.g., viewing the (x, y) axes) of FIG. 10A. The simplified data 1006′ represents simplified data compared to the LIDAR data 1006 illustrated in FIG. 10A).
Regarding claim 17, Douillard teaches the medium of claim 16. Douillard further teaches wherein the instructions for extracting the two- dimensional image-based feature information comprises instructions for: extracting the two-dimensional image-based feature information comprises: setting a grid in each of the two-dimensional image of the yz plane, the two-dimensional image of the zx plane, and the two-dimensional image of the xy plane ([col. 18 ln. 66 - col. 19 ln. 4] FIG. 10B is a plane view 1020 of a simplified representation of the graphical representation illustrated in FIG. 10A. As illustrated, FIG. 10B represents a two-dimensional view (e.g., viewing the (x, y) axes) of FIG. 10A. The simplified data 1006′ represents simplified data compared to the LIDAR data 1006 illustrated in FIG. 10A. [col. 19 ln. 35-38] Further, it may be understood in the context of this disclosure that a vertical extent may be maximized or optimized (e.g., with respect to an (x, z) or (y, z) representation of the data. [col. 22 ln. 8-19] At operation 1310, the process can include projecting the data associated with the three-dimensional object onto the rendering plane to generate a two dimensional representation of the object. In some instances, the operation 1310 may include determining a projection type for projecting the data onto the rendering plane. For example, a projection type may utilize perspective geometry with the virtual LIDAR sensor as the focus of the perspective. By way of another example, a projection type may utilize orthogonal geometry for projecting data to the projection plane. In some instances, the rendering plane may include any number of cells, corresponding to a resolution of the rendering plane);
and storing information on a physical quantity required for computing the shape information of the object in a grid cell of the grid ([col. 16 ln. 12-16] FIG. 6D illustrates an example of projection types for use in converting multi-dimensional data for image analysis. In some instances, a cell 622 of the spherical projection shape may be associated with three dimensional information, as described herein. [col. 16 ln. 52-63] FIG. 7B illustrates an example 714 of determining a projection shape and projecting the three-dimensional data onto the projection shape. The example 714 illustrates a partial spherical projection shape 716 having at least one cell 718 through which the vector 708 passes. As described above, the vector 708 may be associated with captured data such as (x, y, z) coordinates of the point P 710 on the building 706. Further, the vector 708 maybe associated with the surface normal vector 712 associated with the point P 710. The information associated with the vector 708 may be stored in the cell 718 and further processed for converting, as described herein).
Regarding claim 18, Douillard teaches the medium of claim 17. Douillard further teaches wherein the instructions for setting the grid in each of the two-dimensional image of the yz plane, the two-dimensional image of the zx plane, and the two-dimensional image of the xy plane comprises instructions for setting a grid ([col. 22 ln. 8-19] At operation 1310, the process can include projecting the data associated with the three-dimensional object onto the rendering plane to generate a two dimensional representation of the object. In some instances, the operation 1310 may include determining a projection type for projecting the data onto the rendering plane. For example, a projection type may utilize perspective geometry with the virtual LIDAR sensor as the focus of the perspective. By way of another example, a projection type may utilize orthogonal geometry for projecting data to the projection plane. In some instances, the rendering plane may include any number of cells, corresponding to a resolution of the rendering plane).
Douillard does not teach setting a grid of a same NxM dimension in each of the two-dimensional images.
Holtz teaches setting a grid of a same NxM dimension in each of the two-dimensional images ([0060] As shown in FIG. 5A, a landing system 150 associated with the UAV 100 may maintain and continually update a two-dimensional (2D) ground map 510 comprising multiple cells. In the example depicted in FIG. 5A, the cells of the ground map 510 are rectangular in shape and are arranged in a rectangular 2D grid that is M cells wide by M cells long…an alternative ground map may be M1 cells wide by M2 cells long where M1 does not equal M2. [0061] In some embodiments, steps 402 and 404 are performed continually as the UAV 100 is in flight over the physical environment 502. In other words, as new sensor data is received while the UAV is in flight, the characteristic data associated with each cell in the ground map 510 is continually updated to reflect the characteristics of the portion of the surface of the physical environment 502 associated with that cell).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the medium of Douillard with the teachings of Holtz to set a grid of NxM dimensions because ([Holtz 0059] each of the multiple cells representing a particular portion of a surface in the physical environment) and ([Holtz 0063] Associating characteristic data (e.g., height statistics) with portions of the ground map (i.e., cells) may simplify the calculations needed to later select an appropriate area on the surface to land thereby reducing required computing resources to store and compute characteristic data and reducing latency when selecting an appropriate landing area).
Regarding claim 19, Douillard and Holtz teach the medium of claim 18. Holtz teaches wherein the instructions for storing the information on the physical quantity required for computing the shape information of the object in the grid cell of the grid comprises instructions for: for points in each grid cell, determining vertical distances from the point to a projection plane and storing a value of a largest vertical distance in each grid cell to create a first specific information map; storing a value of a smallest vertical distance in each grid cell to create a second specific information map; and storing the number of the points comprised in each grid cell to create a third feature information map ([0080] Once the particular cell is identified, the height value for the particular point is added to that cell. Specifically, in some embodiments, this process of adding height data to a cell may include, at step 706b, updating the height statistics for the cell such as, the number of data points collected, the mean height value, the median height value, minimum height value, maximum height value, the sum of the squared differences in height values, or any other type of statistic based on the aggregation of data points for a particular cell. [0073] As previously discussed, characteristic data such as height statistics can be generated and maintained for multiple cells in a continually updated ground map based on sensor data received from one or more sensors, for example, as described with respect to FIGS. 5A-5B. [0094] For example, characteristic data associated with height values and/or semantic information may be added to the same ground map or to separate overlapping ground maps).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the medium of Douillard with the teachings of Holtz to create specific information maps of include largest depth, smallest depth, and a number of LiDAR points because ([Holtz 0059] each of the multiple cells representing a particular portion of a surface in the physical environment) and ([Holtz 0063] Associating characteristic data (e.g., height statistics) with portions of the ground map (i.e., cells) may simplify the calculations needed to later select an appropriate area on the surface to land thereby reducing required computing resources to store and compute characteristic data and reducing latency when selecting an appropriate landing area).
Regarding claim 20, Douillard teaches the medium of claim 8. Douillard further teaches wherein instructions for determining the type of the object by processing the two-dimensional image-based feature information based on the convolutional neural network ([col. 3 ln. 48-52] As mentioned above, after converting the three-dimensional LIDAR data to two-dimensional data, the operations can include inputting the two-dimensional data into a convolutional neural network (CNN) for segmentation and classification) comprises instructions for determining the type of the object as a passenger vehicle, a commercial vehicle, a road boundary, a pedestrian, or a two-wheeled vehicle ([col. 11 ln. 5-10] The classification module 316 may include functionality to receive segmented data and to identify a type of object represented by the data. For example, the classification module 316 may classify one or more objects, including but not limited to cars, buildings, pedestrians, bicycles, trees, free space, occupied space, street signs, lane markings, etc.).
Claims 7 and 15 rejected under 35 U.S.C. 103 as being unpatentable over Douillard in view of Holtz and Zhao (US20220405545A1).
Regarding claim 7, Douillard and Holtz teach the method of claim 1. Zhao, in the same field of endeavor of LiDAR analysis using convolutional neural networks, teaches wherein the convolutional neural network ([0197] In at least one embodiment, a CNN for facial recognition and vehicle owner identification may use data from camera sensors to identify presence of an authorized driver and/or owner of vehicle 1000) comprises a depth-wise separable convolution block ([0068] In at least one embodiment, fusion combines a depthwise separable convolution sub-block with a pointwise convolution sub-block by letting this pointwise convolution sub-block fetch depthwise separable convolution sub-block output via processor memory).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Douillard with the teachings of Zhao to use a depth-wise separable convolution block in the CNN because “this approach avoids each fused sub-block from needing to repeatedly fetch filters of both depthwise separable convolution and pointwise convolution layers, which if allowed to occur would increase pressure on L2 cache and main memory, as well as increase memory pressure caused by loading runtime instructions” [Zhao 0069].
Regarding claim 15, Douillard and Holtz teach the system of claim 9. Zhao, in the same field of endeavor of LiDAR analysis using convolutional neural networks, teaches wherein the convolutional neural network ([0197] In at least one embodiment, a CNN for facial recognition and vehicle owner identification may use data from camera sensors to identify presence of an authorized driver and/or owner of vehicle 1000) comprises a depth-wise separable convolution block ([0068] In at least one embodiment, fusion combines a depthwise separable convolution sub-block with a pointwise convolution sub-block by letting this pointwise convolution sub-block fetch depthwise separable convolution sub-block output via processor memory).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the system of Douillard with the teachings of Zhao to use a depth-wise separable convolution block in the CNN because “this approach avoids each fused sub-block from needing to repeatedly fetch filters of both depthwise separable convolution and pointwise convolution layers, which if allowed to occur would increase pressure on L2 cache and main memory, as well as increase memory pressure caused by loading runtime instructions” [Zhao 0069].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jacqueline R Zak whose telephone number is (571)272-4077. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACQUELINE R ZAK/Examiner, Art Unit 2666
/EMILY C TERRELL/Supervisory Patent Examiner, Art Unit 2666