DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see page 8, filed 2/27/2026, with respect to the specification have been fully considered and are persuasive. The objection of specification has been withdrawn.
Applicant’s arguments, see page 8, filed 2/27/2026, with respect to claim objections have been fully considered and are persuasive. The objections of the claims has been withdrawn.
Applicant’s arguments with respect to claim(s) 22-37 have been considered but are moot because the new ground of rejection does not rely on all references applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. The remarks state that the newly amended features of the claims were not disclosed by the prior references:
“wherein the normal vector is calculated based on at least one plane defined by at least three points selected from the target point and the near-points, generating a geometrically enhanced intensity value of the target point based on the intensity value of the target point and the estimated angle of the target point, and changing the intensity value of the target point to the geometrically enhanced intensity value of the target point.” The reference of Chen is used to cure the deficiencies of the previously applied references and will be explained why below.
The reference of Chen performs the feature of calculating a normal vector based on an initial pixel and adjacent pixels to the initial, or target, pixel. This is taught in ¶ [93]-[96]. The invention further generates a new intensity value that is enhanced by the angle estimated that is found associated with a point in the point data. The system takes an intensity associated with a target point and an angle estimated to calculate another intensity value. This process updates or modifies an intensity value of an initial, or target, point based on the angle estimated at that particular point, which is taught in ¶ [106]-[109]. This reference, in combination with the previously applied references, performs the features of the contended claim limitations.
Thus, based on the above, the features of the claims are disclosed below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 22-37 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nikic (US Pat 9245170) in view of Manivasagam (US Pub 2020/0160598) and Chen (US Pub 2020/0160598).
Please amend Claims 1, 4, 6, 7, 11, 14, and 16-21 as follows.
Re claim 1: (Currently amended) A method for processing point data obtained from a light detection and ranging (LIDAR) device, comprising:
obtaining point cloud data based on a detection signal generated by the LIDAR device (e.g. the invention discloses obtaining point cloud data in order to identify different points within a scene in relationship with a point by detecting light that generates a signal detected, which is taught in col. 3, ll. 29-54 and col. 4, ll. 4-20.); and
(17) After the relationships between points have been identified, the point cloud may then be processed to identify particular objects or a number of objects. As used herein, “number of” when used with reference to an item means one or more items. For example, the number of objects is one or more objects.
(18) In other words, currently used systems may identify a point in a scene, but the relationship of other points in the scene to the identified point may be unknown. All of the image data for the point cloud are processed to identify other points that may be around the point of interest. This processing occurs each time one or more points of interest are identified. The illustrative embodiments recognize and take into account that this process takes more time than desired to process a point cloud.
(19) Thus, the illustrative embodiments provide a method and apparatus for processing image data. In particular, the illustrative embodiments provide a method and apparatus for processing image data for a point cloud. One or more illustrative embodiments provide an image processing system comprising a data repository and an image processor. The data repository is configured to restore data. The image processor is configured to place the image data into a three-dimensional mesh. The image processor also identifies vectors of the image data in the three-dimensional mesh and identifies a number of clusters in the vectors of the image data in the three-dimensional mesh.
(22) In these illustrative examples, unmanned aerial vehicle 102 generates image data about objects 104 or a number of objects 104 using imaging system 108. In particular, imaging system 108 may be, for example, a light detection and ranging (LIDAR) system. Of course, imaging system 108 may be implemented using another suitable type of imaging system. In this illustrative example, imaging system 108 generates image data for a point cloud of objects 104 or the number of objects 104 in scene 106.
(23) As depicted, the number of objects 104 in scene 106 include house 110, trees 112, road 114, pathway 115, light poles 116, and vehicle 118. In these illustrative examples, the image data generated by imaging system 108 may take the form of image data for a point cloud of scene 106. Each point in the image data may include information about the light detected by imaging system 108 defined by three-dimensional coordinates of the point.
wherein the point cloud data comprises a plurality of point data corresponding to a plurality of points detected by the LIDAR device (e.g. the system discloses generating an image for a point cloud of objects detected, which is taught in col. 4, ll. 4-20 above.);
wherein each of the plurality of point data comprises distance information and an intensity value of a respective point of the plurality of points (e.g. the system discloses point data to be associated with distance information as well as intensity information within the metadata. This is taught in col. 6, ll. 36-42, col. 6, ll. 65-col. 7, ll. 35.); and
(42) In this illustrative example, image data 206 is in first coordinate system 300. Three-dimensional mesh 222 may use both first coordinate system 300 and second coordinate system 302. Points 304 in point cloud 212 for image data 206 are described using first coordinate system 300 and have locations based on first coordinate system 300 in three-dimensional mesh 222.
(46) Metadata 312 may be associated with locations 306 in which points 304 are present. In some cases, metadata 312 may be associated directly with points 304. In these illustrative examples, metadata 312 may include at least one of identifier 316 and attributes 317.
(47) Identifier 316 is an identifier of a cell in cells 310 within three-dimensional mesh 222.
(48) In these illustrative examples, attributes 317 is information about the cell based on points 304 located in the cell. Attributes 317 in metadata 312 may include implicit geometry data 318 based on points 304. In these illustrative examples, implicit geometry data 318 is an implicit geometry representation of points relative to each cell in three-dimensional mesh 222. Implicit geometry data 318 may be values that are based on the relationship of points located within a cell or group of cells in cells 310. In this illustrative example, a group of cells is two or more cells.
(49) Illustrative examples of implicit geometry data in attributes 317 include population 319, distance 320, and validity 321. Population 319 is a value identifying the number of points 304 located in a cell. Distance 320 is the shortest distance from a point to the center of a cell in cells 310 in three-dimensional mesh 222. Validity 321 is the collective effect of neighboring points in points 304 on a cell in cells 310 weighted by a probability distribution function to the center of the points.
(50) In these illustrative examples, implicit geometry data 318 may be selected as attributes 317 that may be useful in identifying clusters 226 and classes 230. In other words, implicit geometry data 318 may be selected for use in forming clusters. These clusters may be used as training data for future classification calculations in some illustrative examples.
(51) Attributes 317 also may include other information 322. Examples of other information 322 include at least one of intensity, wavelength, and other suitable information about a point or points in a cell. These attributes may identify information about each point individually in a cell or all of the points collectively in the cell.
However, Nikic fails to specifically teach the features of modifying an intensity value of a target point based on point data or near-points adjacent to the target point, wherein the target point is one of the plurality of points, wherein modifying the intensity value of the target point comprises: estimating an angle of the target point between a first direction and a second direction,
wherein the first direction is defined by an origin of the LIDAR device and the target point, wherein the second direction is defined based on a normal vector at the target point.
However, this is well known in the art as evidenced by Manivasagam. Similar to the primary reference, Manivasagam discloses using a machine learning system with LIDAR data (same field of endeavor or reasonably pertinent to the problem).
Manivasagam discloses modifying an intensity value of a target point based on point data or near-points adjacent to the target point, wherein the target point is one of the plurality of points (e.g. the system discloses generating an intensity of a point in the point cloud by averaging the intensity values of points located within a radius of the point, which is taught in ¶ [115]. The particular point with the generated intensity can be considered as a target point. The generating of an intensity by other points, which is a form of modifying an intensity of a particular point by the points within a radius, is considered as a target point.),
[0115] In some implementations, the computing system can also generate intensity data for each point in the initial three-dimensional point cloud or the adjusted three-dimensional point cloud. For example, for each of such points, the computing system can determine a respective intensity value based at least in part on intensity data included in the three-dimensional map for locations within a radius of a respective location associated with such point in either the initial three-dimensional point cloud or the adjusted three-dimensional point cloud. For example, the average intensity in this local radius can be assigned to the point.
wherein modifying the intensity value of the target point comprises: estimating an angle of the target point between a first direction and a second direction, wherein the first direction is defined by an origin of the LIDAR device and the target point, wherein the second direction is defined based on a normal vector at the target point (e.g. each LIDAR point within a point cloud is described using a sensor angle and laser pitch. The sensor angle is direction using the origin of the sensor and the LIDAR point, which can be considered as the target point. The LIDAR point can be further described with the laser pitch that is an angle based on the normal line or vector at the LIDAR point. This is taught in ¶ [92]-[98].).
[0092] To account and correct the aforementioned limitations in raycasted LiDAR 308, the illustrated process can include application of machine learning to bridge the gap between simulated and real-world LiDAR data. The main architecture is a geometry network 314 that aims at improving the geometry of the simulated point cloud 308.
[0093] The geometry network 314 aims at improving the initial LiDAR point cloud 308 produced from raycasting to be perceptually similar to real LiDAR sensor data.
[0094] In some implementations, the neural network's input is the initial LiDAR point cloud 308 in sensor polar coordinates. Each LiDAR point can be described with (ϕ, θ, d) where ϕ is the sensor angle (yaw), θ is the laser pitch, and d is the depth value of the returned LiDAR point.
[0095] The output 316 of the network 314 is the adjusted depth per point. This output representation ensures the resulting point cloud is physically feasible (e.g., rays do not intersect, rays do not have multiple returns, no impossible geometries created, etc.). In one example, the geometry network 314 can include a parametric continuous convolution network as its backbone architecture. This is a powerful deep learning architecture that works directly on unstructured point cloud data without voxelization or rasterization, thereby maintaining high fidelity of the input geometry. Moreover, similar to CNNs for images, continuous convolutions capture the contextual relationship between a point and its neighbors in a bottom-up manner.
[0096] As one particular example, the geometry network 314 can include four layers of continuous fusion, a memory-efficient variant of the continuous convolution layer with residual connections between each adjacent layer. The k-nearest neighbors per point p.sub.i can be computed based on a Mahalanobis distance d.sub.i=√{square root over ((p−p.sub.i).sup.TS.sup.−1(p−p.sub.i))} where p=(ϕ, θ, d/d.sub.max) is polar coordinate with a normalized depth and S is diagonal reweighting matrix.
[0097] In some implementations, the adjusted geometry 316 can be combined with the intensity data 310 to generate a set of simulated LiDAR 318 which reflects both the adjusted depths and the determined intensity. For example, the respective intensity data can be assigned to each point in the adjusted geometry 316 to generate the simulated LiDAR data 318.
[0098] In some implementations, the intensity data 310 can be updated based on the adjusted depth(s). For example, for each point that has had its depth adjusted, the computing system can determine a new set of intensity data (e.g., using the metadata 312) based on its adjusted depth. For example, the neighborhood analysis described above can be performed given the point's new depth. In other implementations, the intensity data 310 is not computed at all until after the adjusted geometry 316 has been determined. Thus, the intensity data 310 can be computed at the time of ray casting 306 and left unmodified; computed at the time of ray casting 306 and then modified to account for the adjusted depths; or computed only for the adjusted geometry 316 following adjustment of the depths.
Therefore, in view of Manivasagam, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of modifying an intensity value of a target point based on point data or near-points adjacent to the target point, wherein the target point is one of the plurality of points, wherein modifying the intensity value of the target point comprises: estimating an angle of the target point between a first direction and a second direction, wherein the first direction is defined by an origin of the LIDAR device and the target point, wherein the second direction is defined based on a normal vector at the target point, incorporated in the device of Nikic, in order to utilize a machine learned geometry model to modify geometry of points clouds generated for training of autonomous vehicle system, which can improve safety, efficiency and performance of autonomous systems (as stated in Manivasagam ¶ [31] and [45]).
However, the combination of references above fails to specifically teach the features of wherein the normal vector is calculated based on at least one plane defined by at least three points selected from the target point and the near- points, generating a geometrically enhanced intensity value of the target point based on the intensity value of the target point and the estimated angle of the target point, and changing the intensity value of the target point to the geometrically enhanced intensity value of the target point.
However, this is well known in the art as evidenced by Chen. Similar to the primary reference, Chen discloses computing the angle of incidence of a LIDAR signal (same field of endeavor or reasonably pertinent to the problem).
Chen discloses wherein the normal vector is calculated based on at least one plane defined by at least three points selected from the target point and the near- points (e.g. the invention discloses calculating normal vectors of pixels based on pixels adjacent to a pixel for which the normal is determined. The pixel for which the normal is being determined is the target pixel and the adjacent pixels are two or more. This is explained in ¶ [93]-[96].),
[0093] At block 304, the method 300 may include, determining a first surface normal for at least a first data point of the first plurality of data points. The computing device may calculate the normal vectors of pixels that correspond to the same surface in the environment for object recognition purposes, which may assist the vehicle in autonomous navigation. The computing device may determine the surface normal based on the pixels adjacent (or otherwise nearby) to the pixel for which the normal is being determined.
[0094] In some examples, block 304 may be repeated for a plurality of the data points. In these examples, several data points may be associated with one surface. In some examples, the entire surface may have the same surface normal, such as a flat wall. In other examples, various portions of the surface may have difference surface normals, such as a curved surface.
[0095] In order to improve object recognition and ensure safe navigation, the computing device may determine normal vectors of surfaces as represented in the range image in real time as the range image is developed and processed. As such, the computing device may be configured to determine normal vectors within lidar data received in point clouds efficiently with relatively few CPU cycles, and may provide the determined normal with minimal latency to various systems of the vehicle.
[0096] In one example embodiment, the computing device may determine vector normal of sets of pixels by computing gradients across and down the image and smoothing the gradients using integral images. As a result, the computing device may compute normalized cross products of the gradients to produce the vector normal of various surfaces as indicated by sets of pixels. For example, the computing device may compute two vectors which are tangential to the local surface at the center of a set of pixels. The two vectors may be based on neighboring or nearby pixels. In some examples, the two vectors may be perpendicular to each other. From the two tangential vectors, the computing device may compute the vector normal using the cross product. For example, the computing device may compute a surface normal between either a left and a right neighboring pixel and between either an upper and lowering neighboring pixel, to have two perpendicular vectors.
generating a geometrically enhanced intensity value of the target point based on the intensity value of the target point and the estimated angle of the target point (e.g. generating a different intensity occurs based on an initial intensity and a cosine of an incidence angle. The data points associated with a normal vector associated with the incidence angle have their intensity values adjusted, which is taught in ¶ [106]-[109].), and
[0106] At block 308, the method 300 may include, adjusting the intensity of the first data point based on the first angle of incidence to create a first adjusted intensity for the first data point. Each respective data point may have an associated intensity (e.g., strength) of the received laser signal. The intensity may correspond to the amount of laser light that reflects back from the surface to the laser unit. When the surface is not normal (e.g., perpendicular to) the laser source, the intensity is reduced. Therefore, the received intensity may be adjusted to determine a true reflectivity of the object.
[0107] In intensity may be adjusted based on the cosine of the angle of incidence. Based on Lambert's law, a diffuse radiator (i.e., an object reflecting laser light) reflects light with an intensity that is proportional to the cosine of the angle of incidence. Thus, for surfaces that are Lambertian (i.e., those that reflect light according to Lambert's law), the cosine of the angle of incidence may be used to adjust the intensity of the received laser reflections. To adjust the intensity, the measured (or sensed) intensity may be divided by the cosine of the angle of incidence. Because the value of cosine ranges from zero to one (as the angle change from ±90 degrees to 0 degrees), dividing the intensity by cosine will always either increase the value of the intensity or keep it the same. Therefore, by dividing by cosine, the adjusted intensity of a given point will be increased compared to the measured intensity when the angle of incidence is not 0 degrees and the adjusted intensity will be the same as the measured intensity when the angle of incidence is 0 degrees.
[0108] In some examples, the system may also apply an correction based on a distance from the object causing reflections. Because the intensity of light drops proportionally to the square of distance, further objects will produce relatively less intense reflections. In these examples, reflections may be received having a “raw” intensity. The “raw” intensity may be adjusted based on a distance to the reflecting object to produce a “range-adjusted” intensity. Some or all of steps 304-308 may be performed on the “range-adjusted” intensity to determine the adjusted intensity. In these examples, two adjustments may be applied, one for range and one for the angle of incidence. In some other examples, the distance adjustment may be performed after blocks 304-308, therefore the angle-adjusted intensity may be further adjusted based on the range.
[0109] Steps 304-308, may be repeated for a plurality of data points within a set of received laser points. In some examples, a set of points may be determined to have the same surface normal. In these examples, all points associated with the surface may have their intensity adjusted at the same time, as the surface normal is the same. In some other examples, steps 304-308 may be repeated based on the system determining a likelihood that the points are associated with the same object. In yet other examples, steps 304-308 may be repeated for a majority of the points of the set of data. Additionally in some examples, the data points that have the same surface normal may be grouped based on the surface being associated with a single object. In some further examples, the method may also determine a reflectivity of the surface of the object that comprises the data points that have the same surface normal.
changing the intensity value of the target point to the geometrically enhanced intensity value of the target point (e.g. with the adjustment of the intensity values, the adjusted data point has its intensity value changed based on the point being associated with a normal and an incidence angle. This is taught in ¶ [106]-[109] above.).
Therefore, in view of Chen, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the normal vector is calculated based on at least one plane defined by at least three points selected from the target point and the near- points, generating a geometrically enhanced intensity value of the target point based on the intensity value of the target point and the estimated angle of the target point, and changing the intensity value of the target point to the geometrically enhanced intensity value of the target point, incorporated in the device of Nikic, as modified by Manivasagam, in order to use a pixel neighborhood to detect pixels to use in adjusting the intensity value of a particular pixel and calculating normal vectors, which can reduce the computational resources in identifying neighboring pixels for the intensity operation (as stated in Chen ¶ [103]).
Re claim 23: (Original) However, Nikic fails to specifically teach the features of the method of claim 22, wherein the distance information reflects a distance between the LiDAR device and the respective point.
However, this is well known in the art as evidenced by Manivasagam. Similar to the primary reference, Manivasagam discloses using a machine learning system with LIDAR data (same field of endeavor or reasonably pertinent to the problem).
Manivasagam discloses wherein the distance information reflects a distance between the LiDAR device and the respective point (e.g. location information of a point in the point cloud can be calculated based on the distance measured from the actual sensor and reflection back to the sensor providing an intensity, which is taught in ¶ [83] and [106].).
[0083] The illustrated process focuses on simulating a scanning LiDAR system. One example system that can be simulated has 64 emitter-detector pairs vertically arranged, each of which uses light pulses to measure distance. The basic concept is that each emitter emits a light pulse which travels until it hits a target, and a portion of the light energy is reflected back and received by the detector. Distance is measured by calculating the time of travel and material reflectance is measured through the intensity of the returned pulse. The entire optical assembly rotates on a base to provide a 360-degree azimuth field of view at around 10 Hz with each full “sweep” providing approximately 70 k returns.
[0106] The computing system can associate the plurality of sets of real-world LiDAR data to a common coordinate system to generate an aggregate LiDAR point cloud. For example, each set of LiDAR data can be transitioned from respective vehicle coordinate system to a common coordinate system based on a respective pose (e.g., location and orientation) of the vehicle at the time of data collection.
Therefore, in view of Manivasagam, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the distance information reflects a distance between the LiDAR device and the respective point, incorporated in the device of Nikic, in order to utilize a machine learned geometry model to modify geometry of points clouds generated for training of autonomous vehicle system, which can improve safety, efficiency and performance of autonomous systems (as stated in Manivasagam ¶ [31] and [45]).
Re claim 24: (Original) However, Nikic fails to specifically teach the features of the method of claim 22, wherein the distance information for the respective point is generated based on a detection time point of the detection signal and a light emission time point of the LiDAR device.
However, this is well known in the art as evidenced by Manivasagam. Similar to the primary reference, Manivasagam discloses using a machine learning system with LIDAR data (same field of endeavor or reasonably pertinent to the problem).
Manivasagam discloses wherein the distance information for the respective point is generated based on a detection time point of the detection signal and a light emission time point of the LiDAR device (e.g. points detected with the LIDAR system are generated based on the measured time and reflection intensity of a detection time point of the detected signal and the emission of the light from the LIDAR device, which is taught in ¶ [83] above.).
Therefore, in view of Manivasagam, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the distance information for the respective point is generated based on a detection time point of the detection signal and a light emission time point of the LiDAR device, incorporated in the device of Nikic, in order to utilize a machine learned geometry model to modify geometry of points clouds generated for training of autonomous vehicle system, which can improve safety, efficiency and performance of autonomous systems (as stated in Manivasagam ¶ [31] and [45]).
Re claim 25: (Currently amended) Nikic discloses the method of claim 22, wherein the intensity value of the respective point is obtained based on a characteristic of the detection signal (e.g. the detected signal is represented in a 3D coordinate system that is evaluated, which is characteristic taught in col. 4, ll. 4-20 above.), and
wherein the characteristic of the detection signal includes at least one of a pulse width of the detection signal, a rising edge of the detection signal, a falling edge of the detection signal or a pulse area of the detection signal (e.g. the attributes of the metadata includes a wavelength about a point or points in a cell along with the intensity. These are used to provide characteristic information about a point or points in a cell, which is taught in col. 7, ll. 30-35 above.).
Re claim 26: (Original) Nikic discloses the method of claim 22, wherein the detection signal is generated by detecting at least a portion of laser scattered at the respective point when the laser emitted from the LiDAR device reaches the respective point (e.g. using a LADAR system provides the detection of a laser detected through backscattering imaging, which is taught in col. 4, ll. 40-55 above.).
Re claim 27: (Currently amended) Nikic discloses the method of claim 22, wherein the geometrically enhanced intensity value of the target point reflects a geometrical shape formed by the target point and the near-points (e.g. the detection points detected can be a shape of an object, such as a tree or a pole. The points can be associated with this class, which is taught in col. 6, ll. 15-26 above.).
Re claim 28: (Original) However, Nikic fails to specifically teach the features of the method of claim 22, wherein the intensity value of the respective point depends on an intrinsic property of the respective point and a distance between the LiDAR device and the respective point.
However, this is well known in the art as evidenced by Manivasagam. Similar to the primary reference, Manivasagam discloses using a machine learning system with LIDAR data (same field of endeavor or reasonably pertinent to the problem).
Manivasagam discloses wherein the intensity value of the respective point depends on an intrinsic property of the respective point and a distance between the LiDAR device and the respective point (e.g. the reflection parameter depends on the reflectivity of the detection point and the distance between the LIDAR device and detection point measured by the time of the reflected signal being detected, which is taught in ¶ [83] above, [28], [97] and [98].).
[0028] In addition, geometry is only part of the equation. LiDAR point clouds contain intensity returns, which are typically exploited in applications such as lane detection, semantic segmentation and construction detection, as the reflectivity of some materials is very informative. Intensity returns are very difficult to simulate as they depend on many factors including incidence angle, material reflectivity, laser bias, and atmospheric transmittance, as well as black box normalization procedures that are done by the LiDAR provider.
[0097] In some implementations, the adjusted geometry 316 can be combined with the intensity data 310 to generate a set of simulated LiDAR 318 which reflects both the adjusted depths and the determined intensity. For example, the respective intensity data can be assigned to each point in the adjusted geometry 316 to generate the simulated LiDAR data 318.
[0098] In some implementations, the intensity data 310 can be updated based on the adjusted depth(s). For example, for each point that has had its depth adjusted, the computing system can determine a new set of intensity data (e.g., using the metadata 312) based on its adjusted depth. For example, the neighborhood analysis described above can be performed given the point's new depth. In other implementations, the intensity data 310 is not computed at all until after the adjusted geometry 316 has been determined. Thus, the intensity data 310 can be computed at the time of ray casting 306 and left unmodified; computed at the time of ray casting 306 and then modified to account for the adjusted depths; or computed only for the adjusted geometry 316 following adjustment of the depths.
Therefore, in view of Manivasagam, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the intensity value of the respective point depends on an intrinsic property of the respective point and a distance between the LiDAR device and the respective point, incorporated in the device of Nikic, in order to utilize a machine learned geometry model to modify geometry of points clouds generated for training of autonomous vehicle system, which can improve safety, efficiency and performance of autonomous systems (as stated in Manivasagam ¶ [31] and [45]).
Re claim 29: (Original) However, Nikic fails to specifically teach the features of the method of claim 22, wherein the combination of the intensity value of the target point and the estimated angle of the target point is performed such that a numerical range of the geometrically enhanced intensity value of the target point is equal to a numerical range of the intensity value of the target point.
However, this is well known in the art as evidenced by Manivasagam. Similar to the primary reference, Manivasagam discloses using a machine learning system with LIDAR data (same field of endeavor or reasonably pertinent to the problem).
Manivasagam discloses wherein the combination of the intensity value of the target point and the estimated angle of the target point is performed such that a numerical range of the geometrically enhanced intensity value of the target point is equal to a numerical range of the intensity value of the target point (e.g. the adjusted geometry 316 contains the modification or depth adjustment. The adjusted depth allows for adjusting the intensity data based on the adjusted depth, which is taught in ¶ [92]-[98] above. Based on these adjustments, the overall intensity data is within the same numerical range as the depth adjustment in the adjusted geometry since the combination of variables is equated to the overall determined intensity.).
Therefore, in view of Manivasagam, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the combination of the intensity value of the target point and the estimated angle of the target point is performed such that a numerical range of the geometrically enhanced intensity value of the target point is equal to a numerical range of the intensity value of the target point, incorporated in the device of Nikic, in order to utilize a machine learned geometry model to modify geometry of points clouds generated for training of autonomous vehicle system, which can improve safety, efficiency and performance of autonomous systems (as stated in Manivasagam ¶ [31] and [45]).
Re claim 30: (Currently amended) However, Nikic fails to specifically teach the features of the method of claim 22, wherein the intensity value of the target point and the estimated angle of the target point are normalized based on the same numerical range.
However, this is well known in the art as evidenced by Manivasagam. Similar to the primary reference, Manivasagam discloses using a machine learning system with LIDAR data (same field of endeavor or reasonably pertinent to the problem).
Manivasagam discloses wherein the intensity value of the target point and the estimated angle of the target point are normalized based on the same numerical range (e.g. adjusted depths within the adjusted geometry is used to update or adjust the intensity data, which is taught on ¶ [92]-[98] above and [90]. With the values of the adjusted depth being used to update the intensity data, this is considered as adjusting or normalizing these numbers based on the same numerical range.).
[0090] The intensity value of a point is influenced by many factors including incidence angle, range, and the beam bias. The computing system can employ nearest neighbors as the estimator for intensity. To be specific, for each returned ray, the computing system can conduct a nearest neighbor search within a small radius of the hitted surfel where reflectance of the local surface is assumed to be the same. Note that this assumption might not hold true along geometric boundaries or material boundary over the same object. The computing system can then assign the average intensity in this local radius as our target intensity value (e.g., shown as physics intensity 310 generated from metadata 312).
Therefore, in view of Manivasagam, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the intensity value of the target point and the estimated angle of the target point are normalized based on the same numerical range, incorporated in the device of Nikic, in order to utilize a machine learned geometry model to modify geometry of points clouds generated for training of autonomous vehicle system, which can improve safety, efficiency and performance of autonomous systems (as stated in Manivasagam ¶ [31] and [45]).
Re claim 31: (Original) However, Nikic fails to specifically teach the features of the method of claim 22, wherein the combination of the intensity value of the target point and the estimated angle of the target point is a linear combination of the intensity value of the target point and the estimated angle of the target point.
However, this is well known in the art as evidenced by Manivasagam. Similar to the primary reference, Manivasagam discloses using a machine learning system with LIDAR data (same field of endeavor or reasonably pertinent to the problem).
Manivasagam discloses wherein the combination of the intensity value of the target point and the estimated angle of the target point is a linear combination of the intensity value of the target point and the estimated angle of the target point (e.g. the adjusted geometry reflects adjusted depths that are calculated from 3D input that is associated with a Mahalanobis distance, which involves the product of a matrix with other variables. The result of this is combined linearly with an intensity and metadata in order to create simulated LIDAR data of a scene, which is taught in ¶ [95]-[98] above, [107] and [110]-[115].).
[0107] The computing system can convert the aggregate LiDAR point cloud to a surface element-based three-dimensional mesh. For example, the computing system can perform voxel-based downsampling and normal estimation to perform the conversion. In addition to the geometric information, sensory metadata (e.g., incidence angle, raw intensity, transmitted power level, range value, unique ID per beam, etc.) can be recorded for each surface element (e.g., to be used for intensity simulation).
[0110] At 506, the computing system can perform ray casting on the three-dimensional map according to the trajectory to generate an initial three-dimensional point cloud that includes a plurality of points. As one example, a graphics-based ray casting engine can be given the trajectory (e.g., in the form of a desired sensor 6-degrees of freedom pose and velocity). The engine can cast a set of ray casting rays from the simulated, virtual LiDAR system into the environment.
[0111] In some implementations, the computing system can account for the rotary motion of the virtual LiDAR system (also known as “rolling shutter effects”) by compensating for motion of the virtual system along the trajectory during the simulated LiDAR sweep. In particular, performing the ray casting can include determining, for each of a plurality of rays, a ray casting location and a ray casting direction based at least in part on the trajectory.
[0112] The computing system (e.g., the ray casting engine) can provide at least a respective depth for each of the plurality of points in the initial three-dimensional point cloud. As one example, performing the ray casting to generate the initial three-dimensional point cloud can include, for each of the plurality of rays: identifying a closest surface element in the three-dimensional map to the ray casting location and along the ray casting direction and generating one of the plurality of points with its respective depth based at least in part on a distance from the ray casting location to the closest surface element.
[0113] At 508, the computing system can process, using a machine-learned geometry network, the initial three-dimensional point cloud to predict a respective adjusted depth for one or more of the plurality of points. For example, the computing system can input the initial three-dimensional point cloud into the machine-learned geometry network and, in response, the machine-learned geometry network can provide the one or more adjusted depths for the one or more of the plurality of points as an output. In one example, the machine-learned geometry network can be a parametric continuous convolution neural network.
[0114] At 510, the computing system can generate an adjusted three-dimensional point cloud in which the one or more of the plurality of points have the respective adjusted depth predicted by the machine-learned geometry network. For example, the computing system can separately generate the adjusted three-dimensional point cloud based on an output of the geometry model or, in other implementations, the adjusted three-dimensional point cloud can be directly output by the geometry model.
[0115] In some implementations, the computing system can also generate intensity data for each point in the initial three-dimensional point cloud or the adjusted three-dimensional point cloud. For example, for each of such points, the computing system can determine a respective intensity value based at least in part on intensity data included in the three-dimensional map for locations within a radius of a respective location associated with such point in either the initial three-dimensional point cloud or the adjusted three-dimensional point cloud. For example, the average intensity in this local radius can be assigned to the point.
Therefore, in view of Manivasagam, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the combination of the intensity value of the target point and the estimated angle of the target point is a linear combination of the intensity value of the target point and the estimated angle of the target point, incorporated in the device of Nikic, in order to utilize a machine learned geometry model to modify geometry of points clouds generated for training of autonomous vehicle system, which can improve safety, efficiency and performance of autonomous systems (as stated in Manivasagam ¶ [31] and [45])..
Re claim 32: (Original) However, Nikic fails to specifically teach the features of the method of claim 22, wherein the combination of the intensity value of the target point and the estimated angle of the target point is performed by assigning a weight to each of the intensity value of the target point and the estimated angle of the target point.
However, this is well known in the art as evidenced by Chen. Similar to the primary reference, Chen discloses computing the angle of incidence of a LIDAR signal (same field of endeavor or reasonably pertinent to the problem).
Chen discloses wherein the combination of the intensity value of the target point and the estimated angle of the target point is performed by assigning a weight to each of the intensity value of the target point and the estimated angle of the target point (e.g. in order to update the target point intensity value, the intensity value of an initial point and angle are added and used to form the updated intensity value. These two initial values act as weights to the intensity value and estimated angle. The updating of the intensity value with the estimated angle is explained in ¶ [93]-[96] and [106]-[109] above.).
Therefore, in view of Chen, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the combination of the intensity value of the target point and the estimated angle of the target point is performed by assigning a weight to each of the intensity value of the target point and the estimated angle of the target point, incorporated in the device of Nikic, as modified by Manivasagam, in order to use a pixel neighborhood to detect pixels to use in adjusting the intensity value of a particular pixel and calculating normal vectors, which can reduce the computational resources in identifying neighboring pixels for the intensity operation (as stated in Chen ¶ [103]).
Re claim 33: (Currently amended) However, Nikic fails to specifically teach the features of the method of claim 32, wherein a weight for the intensity value of the target point and a weight for the estimated angle of the target point are determined such that a sum of the weight for the intensity value of the target point and the weight for the estimated angle of the target point is constant.
However, this is well known in the art as evidenced by Chen. Similar to the primary reference, Chen discloses computing the angle of incidence of a LIDAR signal (same field of endeavor or reasonably pertinent to the problem).
Chen discloses wherein a weight for the intensity value of the target point and a weight for the estimated angle of the target point are determined such that a sum of the weight for the intensity value of the target point and the weight for the estimated angle of the target point is constant (e.g. the weight of the previous intensity value and angle is used to update the intensity value and angle of the target or current point. The cosine of the angle is between 0 and 1 and allows for the intensity value to be the same or increased. With the intensity able to be the same or increased with the weight of the estimated angle being a certain number greater than zero or 1, this is considered as having a constant intensity value since it is either the same or increased based on a constant weight associated with the angle being 1 or less. This is taught in ¶ [93]-[96] and [106]-[109] above.).
Therefore, in view of Chen, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein a weight for the intensity value of the target point and a weight for the estimated angle of the target point are determined such that a sum of the weight for the intensity value of the target point and the weight for the estimated angle of the target point is constant, incorporated in the device of Nikic, as modified by Manivasagam, in order to use a pixel neighborhood to detect pixels to use in adjusting the intensity value of a particular pixel and calculating normal vectors, which can reduce the computational resources in identifying neighboring pixels for the intensity operation (as stated in Chen ¶ [103]).
Re claim 34: (Original) However, Nikic fails to specifically teach the features of the method of claim 32, wherein each of a weight for the intensity value of the target point and a weight for the estimated angle of the target point is determined based on a property information of a set of point data including a point data for the respective point.
However, this is well known in the art as evidenced by Chen. Similar to the primary reference, Chen discloses computing the angle of incidence of a LIDAR signal (same field of endeavor or reasonably pertinent to the problem).
Chen discloses wherein each of a weight for the intensity value of the target point and a weight for the estimated angle of the target point is determined based on a property information of a set of point data including a point data for the respective point (e.g. the values associated with the prior intensity values and the prior estimated angle are based on the angle of neighboring points and including the angle associated with the target point. The updated intensity values uses values of a prior intensity value that can be considered as weight as well the prior angle, which can also be considered as a weight. The update of the intensity value and angle is taught in ¶ [93]-[96] and [106]-[109] above.).
Therefore, in view of Chen, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein each of a weight for the intensity value of the target point and a weight for the estimated angle of the target point is determined based on a property information of a set of point data including a point data for the respective point, incorporated in the device of Nikic, as modified by Manivasagam, in order to use a pixel neighborhood to detect pixels to use in adjusting the intensity value of a particular pixel and calculating normal vectors, which can reduce the computational resources in identifying neighboring pixels for the intensity operation (as stated in Chen ¶ [103]).
Re claim 35: (Currently amended) Nikic discloses the method of claim 22, further comprising generating an image based on the point cloud data (e.g. the system generates image data based on the point cloud data, which is taught in col. 4, ll. 4-20 above.).
However, Nikic fails to specifically teach the features of wherein the image includes a plurality of pixel data corresponding to the plurality of point data, wherein a pixel coordinate of each of the plurality of pixel data is determined based on the distance information of each of the plurality of point data, and
wherein a pixel value of each of the plurality of pixel data is determined based on the geometrically enhanced intensity of each of the plurality of point data.
However, this is well known in the art as evidenced by Manivasagam. Similar to the primary reference, Manivasagam discloses using a machine learning system with LIDAR data (same field of endeavor or reasonably pertinent to the problem).
Manivasagam discloses wherein the image includes a plurality of pixel data corresponding to the plurality of point data, wherein a pixel coordinate of each of the plurality of pixel data is determined based on the distance information of each of the plurality of point data (e.g. the simulated LIDAR data represents pixel data that is output based on the output point cloud data that has been adjusted. The point cloud data has been adjusted based on the update of depth values, or distance information, from the geometry networks, which is taught in ¶ [92]-[98] above.), and
wherein a pixel value of each of the plurality of pixel data is determined based on the geometrically enhanced intensity of each of the plurality of point data (e.g. the output values of the simulated data is based on the updated intensity of the simulated data based on the intensity data, which is taught in ¶ [92]-[98] above.).
Therefore, in view of Manivasagam, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the image includes a plurality of pixel data corresponding to the plurality of point data, wherein a pixel coordinate of each of the plurality of pixel data is determined based on the distance information of each of the plurality of point data, and wherein a pixel value of each of the plurality of pixel data is determined based on the geometrically enhanced intensity of each of the plurality of point data, incorporated in the device of Nikic, in order to utilize a machine learned geometry model to modify geometry of points clouds generated for training of autonomous vehicle system, which can improve safety, efficiency and performance of autonomous systems (as stated in Manivasagam ¶ [31] and [45]).
Re claim 36: (Currently amended) However, Nikic fails to specifically teach the features of the method of claim 35, wherein generating the image comprises:
projecting the point data of the detection point to a pixel data, wherein a value of the pixel data corresponds to the geometrically enhanced intensity; and
generating the image including a plurality of pixel data by performing the projection for each of the plurality of point data for the plurality of points.
However, this is well known in the art as evidenced by Manivasagam. Similar to the primary reference, Manivasagam discloses using a machine learning system with LIDAR data (same field of endeavor or reasonably pertinent to the problem).
Manivasagam discloses wherein generating the image comprises:
projecting the point data of the detection point to a pixel data, wherein a value of the pixel data corresponds to the geometrically enhanced intensity (e.g. the system discloses generating simulating data that is a result of the enhanced geometric data combined with the adjusted depth information. Once the data is combined, the simulated data corresponds to the point cloud data, which is taught in ¶ [92]-[98] above.); and
generating the image including a plurality of pixel data by performing the projection for each of the plurality of point data for the plurality of points (e.g. the simulated data projects the point cloud data that contains an adjusted depth and intensity based on the adjusted depth that are combined to reflect the point cloud data. This is taught in ¶ [92]-[98] above.).
Therefore, in view of Manivasagam, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein generating the image comprises: projecting the point data of the detection point to a pixel data, wherein a value of the pixel data corresponds to the geometrically enhanced intensity; and generating the image including a plurality of pixel data by performing the projection for each of the plurality of point data for the plurality of points, incorporated in the device of Nikic, in order to utilize a machine learned geometry model to modify geometry of points clouds generated for training of autonomous vehicle system, which can improve safety, efficiency and performance of autonomous systems (as stated in Manivasagam ¶ [31] and [45]).
Re claim 37: (Currently amended) Nikic discloses a non-transitory computer-readable recording medium for storing instructions, when executed by one or more processors, configured to perform the method of claim 22 (e.g. a processor may implement a program that can be loaded from a computer readable medium, which is taught in col. 14, ll. 43-65.).
(119) Instructions for the operating system, applications, and/or programs may be located in storage devices 1416, which are in communication with processor unit 1404 through communications framework 1402. The processes of the different embodiments may be performed by processor unit 1404 using computer-implemented instructions, which may be located in a memory, such as memory 1406.
(120) These instructions are referred to as program code, computer usable program code, or computer readable program code that may be read and executed by a processor in processor unit 1404. The program code in the different embodiments may be embodied on different physical or computer readable storage media, such as memory 1406 or persistent storage 1408.
(121) Program code 1418 is located in a functional form on computer readable media 1420 that is selectively removable and may be loaded onto or transferred to data processing system 1400 for execution by processor unit 1404. Program code 1418 and computer readable media 1420 form computer program product 1422 in these illustrative examples. In one example, computer readable media 1420 may be computer readable storage media 1424 or computer readable signal media 1426.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Chung discloses identifying objects by a LIDAR device.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHAD S DICKERSON whose telephone number is (571)270-1351. The examiner can normally be reached Monday-Friday 10AM-6PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abderrahim Merouan can be reached at 571-270-5254. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHAD DICKERSON/ Primary Examiner, Art Unit 2682
Prosecution Insights