DETAILED ACTION
Claims 1-5, 8-12, and 15-18 have been presented for examination.
Claims 6, 7, 13, 14, 19 and 20 have been cancelled.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments regarding the 35 U.S.C. 101 rejection filed August 28, 2026, have been fully considered but they are not persuasive.
Applicant argue “the character of amended claim 1 as a whole (and similarly independent claims 8 and 15) is directed to a specific technological technique for modeling the physical energy distribution of a diverging LiDAR laser beam within a sensor simulation, delivering the resulting per-ray intensity values to the perception stack of an autonomous vehicle, and optimizing a machine learning model that thereafter detects and identifies objects from LiDAR data captured by a physical LiDAR sensor of a physical autonomous vehicle in operation”, which “is a technical solution to a technical problem.” However, the amended independent claims are directed to mathematical concepts and data processing, specifically, mathematical modeling of physical energy distribution, calculation intensity values per ray, and optimizing machine learning model. These operations constitute mental processes, mathematical algorithms, and data manipulation, which fall under the abstract idea of mathematical concepts. While the claim recites a final application involving a physical autonomous vehicle and a physical LiDAR sensor, the claim as a whole does not integrate the abstract simulation into a practical application. The generation of at least one virtual beam transmission and the training or optimization of the machine learning model occur entirely within a data-processing and simulation environment. Reciting that the resulting model is subsequently capable of being used with a real-world sensor during operation amounts to a mere field of use or an extra-solution activity. The steps of the claim do not actively control, transform, or interact with the physical Lidar sensor or the physical vehicle during the simulation and optimization process. Therefore, the claim fails to integrate the abstract simulation into a practical application that practically applies or improves a technological process. Therefore, the 101 rejection is MAINTAINED.
Applicant further argue that amended claim 1 “describe concrete operations on a sensor model and on autonomous vehicle software and hardware that go far beyond mental analysis or judgment. Considering these limitations, the claim cannot practically be performed in the human mind.” However, the limitations are all performed within a simulation environment, which are conducted through the use of mathematical algorithms and data manipulation through the use of a computer. Furthermore, the second to last limitation recites “send the one or more modified intensity parameters to a perception stack of an autonomous vehicle operating in the simulation environment” signifying that this is all performed in the simulation environment. The final limitation does not sufficiently integrate the abstract simulation into a practical application that practically applies or improves a technological process, as shown in Section “i)”. Therefore, the 101 rejection is MAINTAINED.
Applicant further argue that the “rotating LiDAR sensor model simulates a sensor capable of capturing a 360-degree field of view, so this per-ray tracing, intensity determination and radial weighting must be carried out for every ray of every beam transmission across the sweep, in a fixed geometric relationship to each beam's center. The human mind is not equipped to maintain a diverging beam's geometry, resolve which of a beam's constituent rays strikes which virtual object, and apply a distinct radial weight to each resulting return” However, the amended limitations are all performed in a simulated environment, such that the claim merely uses a computer to perform the abstract ideas of performing mathematical formulas, mental processes, or data manipulation. The amended limitations do not sufficiently integrate the abstract simulation into a practical application that practically applies or improves a technological process, as shown in Section “i)” Therefore, the 101 rejection is MAINTAINED.
Applicant further argue that "training the neural network in a first stage using the first training set" of Example 39 does not recite a judicial exception because, even though training a neural network may involve or rely upon mathematical concepts, the limitation does not set forth or describe any mathematical relationships, calculations, formulas, or equations. Id. at Page 3. The recited optimizing a performance of a machine learning model stands in the same posture, since claim 1 as amended sets forth no mathematical relationship, calculation, formula, or equation. Thus, the character of amended claim 1 as a whole (and similarly independent claims 8 and 15) is directed to a specific technological technique for simulating LiDAR beam divergence and for optimizing perception software that operates on data from a physical LiDAR sensor, not to a mental process. However, terms like “optimizing” or “simulating” require mathematical algorithms, cost functions, or iterative adjustments, even if explicit equations are omitted. Furthermore, data comparison and evaluating perception data are all cognitive tasks, such as visualization or pattern recognition. The amended limitations do not sufficiently integrate the abstract simulation into a practical application that practically applies or improves a technological process, as shown in Section “i)” Therefore, the 101 rejection is MAINTAINED.
Applicant further argue, that “claims, as amended, are directed toward a specific technical technique for modeling the radial energy distribution of a diverging LiDAR beam within a sensor model and for using the resulting values to optimize perception software that thereafter operates on data captured by a physical LiDAR sensor of a physical autonomous vehicle. This provides a concrete technical solution to technical problems in LiDAR sensor simulation and autonomous vehicle software development. The amendments add concrete limitations tied to specific software modules and to physical sensor hardware, and go well beyond generic computer implementation.” The amended limitations do not sufficiently integrate the abstract simulation into a practical application that practically applies or improves a technological process, as shown in Section “i)” Therefore, the 101 rejection is MAINTAINED.
Applicant further argue “the amended claims overcome these technical problems by reciting specific, technical steps that are inextricably tied to the operation of autonomous vehicle perception software and to a physical LiDAR sensor.” The amended limitations do not sufficiently integrate the abstract simulation into a practical application that practically applies or improves a technological process, as shown in Section “i)” Therefore, the 101 rejection is MAINTAINED.
Applicant further argue “the final limitation carries the claimed technique out of the simulation and into the vehicle: the optimized model must detect and identify objects from LiDAR data captured by a physical LiDAR sensor of a physical autonomous vehicle during operation of the physical autonomous vehicle, and the recited LiDAR sensor simulates that physical sensor.” The amended limitations do not sufficiently integrate the abstract simulation into a practical application that practically applies or improves a technological process, as shown in Section “i)” Therefore, the 101 rejection is MAINTAINED.
Applicant’s arguments regarding the 35 U.S.C 103 rejection filed August 28, 2026, with respect to the rejection(s) of claim(s) 1-5, 8-12, and 15-18 under U.S. Patent Application 2022/0262072 A1 in view of Lidar Essential Beam Model for Accurate Width Estimation of Thin Poles (Long, 2020) have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration and in view of Applicant’s amendments resulting in a change in scope, a new ground(s) of rejection is made in view of U.S. Patent Application 2022/0262072 A1 in view of EP 3637138 A1, further in view of Fan Xu, Yuanqing Wang, Xingyu Yang, Bingqing Zhang, Fenfang Li, Correction of linear-array lidar intensity data using an optimal beam shaping approach, Optics and Lasers in Engineering, Volume 83, 2016, Pages 90-98.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-5, 8-12, and 15-18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to judicial exception (i.e. abstract idea) without significantly more.
Step 1: Claims 1-5 are directed to a system, which is a machine, which is a statutory category of invention. Claims 8-12 are directed to a method, which is a process, which is a statutory category of invention. Claims 15-18 are directed to a non-transitory computer readable media which is an article of manufacture, which is a statutory category of invention. Therefore, claims 1-20 are directed to patent eligible categories of invention.
Step 2A, Prong 1: Claims 1, 8, and 15 recite the abstract idea of simulating beam divergence for Light Detection and Ranging (LiDAR) sensors used by autonomous vehicles, constituting an abstract idea based on Mathematical Concepts including mathematical formulas or equations as well as calculations or alternatively Mental Processes based on concepts performed in the human mind, or with the aid of pencil and paper. The limitations of “determine one or more intensity parameters associated with one or more virtual beam receptions by the LiDAR sensor, wherein the one or more virtual beam receptions correspond to one or more rays from the plurality of rays that are reflected from one or more virtual objects;” and “determine at least one object intensity parameter for each of the one or more virtual objects based on the one or more modified intensity parameters” cover mathematical concepts including utilizing an underlying algorithm or statistical model, in this case the beam divergence model, being used to determine parameters for the virtual beam and objects of the simulation. Alternatively, this limitation cover mental process including analyzing the virtual beam receptions and virtual objects of a simulation and determining intensity parameters based on that analysis. Additionally, the limitation of “adjust the one or more intensity parameters based on one or more transmission intensity weights corresponding to the one or more rays to yield one or more modified intensity parameters” covers mathematical concepts including utilizing an underlying algorithm or statistical model, in this case the beam divergence model, being used to determine adjusted parameters for the simulation. Alternatively, this limitation covers mental process including analyzing the transmission intensity weights of the virtual beam receptions and adjusting intensity parameters based on that analysis. Thus, the claims recite the abstract idea of a mental process performed in the human mind, or with the aid of pencil and paper. Additionally, the limitation of “wherein each transmission intensity weight of the one or more transmission intensity weights is based on a distance of a corresponding ray of the plurality of rays from the center of the at least one virtual beam transmission, and wherein the one or more transmission intensity weights simulate a decrease in energy due to beam divergence” covers mental processes including defining what the transmission intensity weights are based on. Additionally, the limitation of “optimize a performance of a machine learning model that utilizes LiDAR data to detect and identify objects, the machine learning model being optimized based on the one or more modified intensity parameters sent to the perception stack” covers mathematical concepts including analyzing parameters sent to the perception stack and performing optimization algorithms to optimize the performance of the machine learning model. Alternatively, this limitation covers mental processes including analyzing parameters sent to the perception stack and performing optimization algorithms to optimize the performance of the machine learning model, which these algorithms can be performed with the use of a pencil and paper.
Dependent claims 2-7, 9-14, and 16-20 further narrow the abstract ideas, identified in the independent claims.
Step 2A, Prong 2: The judicial exception is not integrated into a practical application. The limitation of “generate, within a simulation environment, at least one virtual beam transmission from a Light Detection and Ranging (LiDAR) sensor using a beam divergence model, wherein the at least one virtual beam transmission includes a plurality of rays distributed around a center of the at least one virtual beam transmission” in claims 1, 8, and 15, merely uses a computer device as a tool to perform the abstract idea. (MPEP 2106.05(f)) The generation of a virtual transmission from a LiDAR sensor are mere instructions to apply an exception as set forth in MPEP2106.05(f). Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a mental process) does not integrate a judicial exception into a practical application. (MPEP 2106.05(f)(2)) The background of the specification discloses that “a LiDAR sensor can be used to determine ranges (variable distance) of one or more targets by directing a laser to a surface of an entity (e.g., a person, an object, a structure, an animal, etc.) and measuring the time for light reflected from the surface to return to the LiDAR.” (See [0002]) Additionally, the limitations of “send the one or more modified intensity parameters to a perception stack of an autonomous vehicle operating in the simulation environment, wherein the one or more modified intensity parameters, when received by the perception stack, are configured to cause the perception stack to detect and classify the one or more virtual objects” and “wherein the machine learning model, after being optimized, is configured to detect and identify objects from LiDAR data captured by a physical LiDAR sensor of a physical autonomous vehicle during operation of the physical autonomous vehicle, the physical LiDAR sensor being simulated by the LiDAR sensor” in claims 1, 8, and 15 can be viewed as is insignificant extra-solution activity, specifically pertaining to mere data gathering/output necessary to perform the abstract idea (MPEP 2106.05(g)) and is not sufficient to integrate the judicial exception into a practical application. This is akin to selecting information, based on types of information and availability of information in geomatics engineering, for collection, analysis and display, which has been identified as extra solution activity. Therefore, the judicial exception is not integrated into a practical application.
Step 2B: Claims 1, 8, and 15 do not include additional elements that are sufficient to amount to significantly more than the judicial exception. The limitation of “generate, within a simulation environment, at least one virtual beam transmission from a Light Detection and Ranging (LiDAR) sensor using a beam divergence model, wherein the at least one virtual beam transmission includes a plurality of rays” in claims 1, 8, and 15, merely uses a computer device as a tool to perform the abstract idea. (MPEP 2106.05(f)) The generation of a virtual transmission from a LiDAR sensor are mere instructions to apply an exception as set forth in MPEP2106.05(f). Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a mental process) does not integrate a judicial exception into a practical application. (MPEP 2106.05(f)(2)) Additionally, the limitations of “send the one or more modified intensity parameters to a perception stack of an autonomous vehicle operating in the simulation environment, wherein the one or more modified intensity parameters, when received by the perception stack, are configured to cause the perception stack to detect and classify the one or more virtual objects” and “wherein the machine learning model, after being optimized, is configured to detect and identify objects from LiDAR data captured by a physical LiDAR sensor of a physical autonomous vehicle during operation of the physical autonomous vehicle, the physical LiDAR sensor being simulated by the LiDAR sensor” in claims 1, 8, and 15 can be viewed as is insignificant extra-solution activity, specifically pertaining to mere data gathering/output necessary to perform the abstract idea (MPEP 2106.05(g)) and is not sufficient to integrate the judicial exception into a practical application. This is akin to selecting information, based on types of information and availability of information in geomatics engineering, for collection, analysis and display, which has been identified as extra solution activity. Therefore, the claim as a whole does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements, when considered alone or in combination, do not amount to significantly more than the judicial exception. As stated in Section I.B. of the December 16, 2014 101 Examination Guidelines, “[t]o be patent-eligible, a claim that is directed to a judicial exception must include additional features to ensure that the claim describes a process or product that applies the exception in a meaningful way, such that it is more than a drafting effort designed to monopolize the exception.”
The dependent claims include the same abstract ideas recited as recited in the independent claims, and merely incorporate additional details that narrow the abstract ideas and fail to add significantly more to the claims.
Dependent claims 2, 9, and 16 are directed to further defining the intensity parameters corresponding to a virtual object and filtering a portion of the model based on the parameters corresponding to that object, which further narrows the abstract idea identified in the independent claim, which is directed to “Mental Processes” or alternatively “Mathematical Concepts.”
Dependent claims 3 and 10 are directed to further defining the model used in the simulation, which further narrows the abstract idea identified in the independent claim, which is directed to “Mental Processes” or alternatively “Mathematical Concepts.”
Dependent claims 4, 11, and 17 are directed to further defining the intensity parameters of the simulation, which further narrows the abstract idea identified in the independent claim, which is directed to “Mental Processes” or alternatively “Mathematical Concepts.”
Dependent claims 5, 12, and 18 are directed to further defining the intensity parameters of the simulation, which further narrows the abstract idea identified in the independent claim, which is directed to “Mental Processes” or alternatively “Mathematical Concepts.”
Accordingly, claims 1-5, 8-12, and 15-18 are rejected under 35 U.S.C 101 because the claimed invention is directed to a judicial exception (i.e. an abstract idea) without anything significantly more.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-5, 8-12, and 15-18 are rejected under 35 U.S.C 103 as being unpatentable over U.S. Patent Application 2022/0262072 A1, hereafter M in view of EP 3637138 A1, hereafter P, further in view of NPL: Fan Xu, Yuanqing Wang, Xingyu Yang, Bingqing Zhang, Fenfang Li, Correction of linear-array lidar intensity data using an optimal beam shaping approach, Optics and Lasers in Engineering, Volume 83, 2016, Pages 90-98, hereafter F.
Regarding Claim 1: M discloses a memory; and one or more processors coupled to the memory
M [0062] “The LiDAR synthesis computing system 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, one or more memory devices, flash memory devices, etc., and combinations thereof.”
the one or more processors being configured to:
generate, within a simulation environment, at least one virtual beam transmission from a Light Detection and Ranging (LiDAR) sensor;
M [0034] “Certain existing approaches to LiDAR simulation for autonomous driving focus on employing handcrafted 3D primitives (such as buildings, cars, trees, roads). Graphics engines have been utilized to ray cast the scene and create virtual LiDAR data.”
M [0049] “In some implementations, the computing system can account for the rotary motion of the virtual LiDAR system (also known as “rolling shutter effects”) by compensating for motion of the virtual system along the trajectory during the simulated LiDAR sweep. In particular, performing the ray casting can include determining, for each of a plurality of rays, a ray casting location and a ray casting direction based at least in part on the trajectory.”
M [0039] “In particular, aspects of the present disclosure are directed to systems and methods that use a machine-learned model to make an initial three-dimensional point cloud generated using a physics-based approach more realistic. In particular, the machine-learned model can learn to modify the geometry of point clouds (e.g., as exhibited by ray dropouts) generated through ray casting and/or other physics-based approaches to better match ground truth counterparts that were physically collected by LiDAR systems in the real world.”
determine one or more intensity parameters associated with one or more virtual beam receptions by the LiDAR sensor, wherein the one or more virtual beam receptions correspond to one or more rays from the plurality of rays that are reflected from one or more virtual objects;
M [0044] “The computing system can convert the aggregate LiDAR point cloud to a surface element-based three-dimensional mesh. For example, the computing system can perform voxel-based downsampling and normal estimation to perform the conversion. In addition to the geometric information, sensory metadata (e.g., incidence angle, raw intensity, transmitted power level, range value, unique ID per beam, etc.) can be recorded for each surface element (e.g., to be used for intensity simulation).”
M [0086] “The illustrated process focuses on simulating a scanning LiDAR system. One example system that can be simulated is the Velodyne HDL-64E which has 64 emitter-detector pairs vertically arranged, each of which uses light pulses to measure distance. The basic concept is that each emitter emits a light pulse which travels until it hits a target, and a portion of the light energy is reflected back and received by the detector. Distance is measured by calculating the time of travel and material reflectance is measured through the intensity of the returned pulse. The entire optical assembly rotates on a base to provide a 360-degree azimuth field of view at around 10 Hz with each full “sweep” providing approximately 70 k returns.”
adjust the one or more intensity parameters … to yield one or more modified intensity parameters;
M [0155] “At 614, the computing system can modify one or more values of one or more parameters of the machine-learned model based at least in part on the objective function. For example, the objective function can be backpropagated through the model and the values of the parameters can be updated based on a gradient of the objective function.”
and determine at least one object intensity parameter for each of the one or more virtual objects based on the one or more modified intensity parameters.
M [0044] “The computing system can convert the aggregate LiDAR point cloud to a surface element-based three-dimensional mesh. For example, the computing system can perform voxel-based downsampling and normal estimation to perform the conversion. In addition to the geometric information, sensory metadata (e.g., incidence angle, raw intensity, transmitted power level, range value, unique ID per beam, etc.) can be recorded for each surface element (e.g., to be used for intensity simulation).”
send the one or more modified intensity parameters to a perception stack of an autonomous vehicle operating in the simulation environment, wherein the one or more modified intensity parameters, when received by the perception stack, are configured to cause the perception stack to detect and classify the one or more virtual objects
M [0029] “The simulated LiDAR data can be used, for example, as simulated input for testing autonomous vehicle control systems. The systems and methods of the present disclosure improve both quantitatively and qualitatively the synthesized LiDAR data over solely physics-based rendering. The improved quality of the synthesized LiDAR point cloud demonstrates the potential of this LiDAR simulation approach and application to generating realistic sensor data, which will ultimately improve the safety an autonomous vehicle.”
and optimize a performance of a machine learning model that utilizes LiDAR data to detect and identify objects, the machine learning model being optimized based on the one or more modified intensity parameters sent to the perception stack, wherein the machine learning model, after being optimized, is configured to detect and identify objects from LiDAR data captured by a physical LiDAR sensor of a physical autonomous vehicle during operation of the physical autonomous vehicle, the physical LiDAR sensor being simulated by the LiDAR sensor.
M [0029] “The simulated LiDAR data can be used, for example, as simulated input for testing autonomous vehicle control systems. The systems and methods of the present disclosure improve both quantitatively and qualitatively the synthesized LiDAR data over solely physics-based rendering. The improved quality of the synthesized LiDAR point cloud demonstrates the potential of this LiDAR simulation approach and application to generating realistic sensor data, which will ultimately improve the safety an autonomous vehicle.”
M [0041] “Thus, in some implementations, to generate a three-dimensional map of an environment, the computing system can first obtain a plurality of sets of real-world LiDAR data physically collected by one or more LiDAR systems in the environment. For example, these sets of real-world LiDAR data can have been collected by autonomous vehicles and/or non-autonomous vehicles as they traveled through the environment.”
M [0054] “In such fashion, the systems and methods enable the efficient and accurate generation of synthetic—yet realistic—LiDAR data. The ability to generate such synthetic LiDAR data has a number of benefits, including, for example, the ability to test/train autonomous vehicle systems on the synthetic LiDAR data. In particular, LiDAR data can be synthesized for challenging edge cases which can enable more robust testing/training of autonomous vehicle systems, thereby leading to autonomous vehicles which demonstrate improved safety, efficiency, and/or other performance measures.”
M does not disclose using a beam divergence model, wherein the at least one virtual beam transmission includes a plurality of rays distributed around a center of the at least one virtual beam transmission, one or more transmission intensity weights corresponding to the one or more rays, wherein each transmission intensity weight of the one or more transmission intensity weights is based on a distance of a corresponding ray of the plurality of rays from the center of the at least one virtual beam transmission, and wherein the one or more transmission intensity weights simulate a decrease in energy due to beam divergence
However, P discloses a beam divergence model
P [0010] “A LIDAR beam is characterized by several parameters: e.g. the beam cross-section, the beam divergence, the beam emission pattern and the beam pulse pattern. A LIDAR beam commonly used in automotive has a rectangular or square cross-section within which a laser or multiple lasers have a emission pattern comprising horizontal or vertical rows/stripes or any other suitable pattern including meander shapes, circles, ellipses, spirals, a Gaussian configuration and the like. The beam may originate from a point source or several point sources defining a source area. From the point source or the source area the beam diverges with a certain divergence angle or divergence angles, which may be measured relative to the emission direction of the point source or source area, respectively. If the LIDAR beam is emitted by a point source, the beam has a certain initial source area some distance away from the actual point source. In order to be able to use the method for single laser and multi laser LIDAR beam simulation the beam is always simulated starting from a source area.”
wherein the at least one virtual beam transmission includes a plurality of rays distributed around a center of the at least one virtual beam transmission
P [0011] “In the first step a) the source area is discretized into separate origins. The source area may be discretized into 100,000 or less, preferably 10,000 or less and most preferably 4000 or less origins. Each origin can be the origin of one single ray of the LIDAR beam. Each ray has a certain emitting direction such that the LIDAR beam is simulated as a truncated pyramid of arbitrary cross-sectional shape like a frustum originating at the source area and diverging with a certain angle of divergence from the emission direction of the source area. The direction of each ray may be characterized by two divergence angles, θ and φ. The two divergence angles may each be within one of two perpendicular planes spanned by the emission direction and a respective radial direction perpendicular to the emission direction.” Examiner notes that the divergence angles represent the lateral expansion of the rays from the central axis as they travel forward, where θ is the angle the ray travels in the x-direction, whereas φ is the angle the ray travels in the y-direction.
M and P are analogous to the claimed invention because they both pertain to simulation of a LiDAR sensor.
It would have been obvious to one with ordinary skill in the art to combine the teachings of M and P because P runs high-quality, physics based simulations in accurately simulate a LiDAR beam, to produce data which is as close as possible to real data and simulates a LiDAR beam with a 2D tree canopy in a 2D grid using a probability
map to determine how the light propagates, whereby Lambertian or mirror reflection can be considered. (See P [0004-0005])
M and P do not explicitly disclose wherein one or more transmission intensity weights corresponding to the one or more rays, wherein each transmission intensity weight of the one or more transmission intensity weights is based on a distance of a corresponding ray of the plurality of rays from the center of the at least one virtual beam transmission
However, F discloses wherein one or more transmission intensity weights corresponding to the one or more rays, wherein each transmission intensity weight of the one or more transmission intensity weights is based on a distance of a corresponding ray of the plurality of rays from the center of the at least one virtual beam transmission.
F [Page 93: Section 3.2] “Assuming the incident laser is Gaussian, the normalized intensity distribution is as follows:
PNG
media_image1.png
26
97
media_image1.png
Greyscale
where I(r) is the intensity at a radial distance r from central axis. The shaped annular laser has a characteristic that the central energy of the incident Gaussian is transferred to the external edge of the annular beam while the external edge energy of the incident Gaussian is transferred to the center of the annular beam. Therefore, the intensity distribution of the annular beam is:
PNG
media_image2.png
36
262
media_image2.png
Greyscale
Examiner notes that given the equations above, as the radius increases, the intensity of the ray decreases.
M, P and F are all analogous to the claimed invention because they all pertain with analysis of a LiDAR beam.
It would have been obvious to one with ordinary skill in the art before the effective filing date to combine the teachings of F with M and P because F discloses an optimal beam shaping approach that analyzes the intensity distribution of a Gaussian beam and shapes the beam such that it improves the imaging quality of linear-array LiDAR by providing an oblate U-shaped intensity distribution, which yields approximately a 30% gain in detection area. (See F [Abstract])
Regarding Claim 2: M in view of P, further in view of F discloses the system of claim 1 wherein at least one virtual beam transmission further includes one or more additional rays reflected from a first virtual object other than the one or more virtual objects, and wherein the one or more processors are further configured to:
determine an object intensity parameter corresponding to the first virtual object based on one or more additional virtual beam receptions corresponding to the one or more additional rays
M [0086] “The illustrated process focuses on simulating a scanning LiDAR system. One example system that can be simulated is the Velodyne HDL-64E which has 64 emitter-detector pairs vertically arranged, each of which uses light pulses to measure distance. The basic concept is that each emitter emits a light pulse which travels until it hits a target, and a portion of the light energy is reflected back and received by the detector. Distance is measured by calculating the time of travel and material reflectance is measured through the intensity of the returned pulse. The entire optical assembly rotates on a base to provide a 360-degree azimuth field of view at around 10 Hz with each full “sweep” providing approximately 70 k returns.”
and determine that the object intensity parameter corresponding to the first virtual object fails to satisfy
M [0125] “The output of the model is a ray dropout probability that predicts, for each element in the array, if it returns or not (e.g., with some probability). In some implementations, to simulate LiDAR noise, the computing system can sample from the probability mask to generate the output LiDAR point cloud. Sampling of the probability mask instead of doing direct thresholding has the following benefits: (1) Raydrop can be learned with cross-entropy loss, meaning the estimated probabilities may not be well calibrated. Sampling helps mitigate this issue compared to thresholding. (2) Real lidar data is non-deterministic due to additional noises (atmospheric transmittance, sensor bias, etc.) that the proposed approach may not completely model.”
M [0029] “The simulated LiDAR data can be used, for example, as simulated input for testing autonomous vehicle control systems. The systems and methods of the present disclosure improve both quantitatively and qualitatively the synthesized LiDAR data over solely physics-based rendering. The improved quality of the synthesized LiDAR point cloud demonstrates the potential of this LiDAR simulation approach and application to generating realistic sensor data, which will ultimately improve the safety an autonomous vehicle.”
M doesn’t disclose a threshold intensity value.
However, P discloses a threshold intensity value.
P [0016] “If no (more) intersection of a current ray (originally emitted ray or scattered ray, respectively) occurs during ray-tracing in step c1) then according to step c3) for each of the rays, the next ray is ray-traced in step c1 ). Consequently, each of the selected rays is ray-traced and, if an intersection is found, scattered and, in turn, the resulting scattered ray is traced until no more intersection with an object in the scene occurs or optionally until the lower threshold for the energy of the ray or the maximal number of reflections is reached. At each intersection the reflected energy based on the current ray (originally emitted ray or scattered ray, respectively) is added to the signal of the LIDAR sensor.”
M and P are analogous to the claimed invention because they both pertain to simulation of a LiDAR sensor.
It would have been obvious to one with ordinary skill in the art to combine the teachings of M and P because P runs high-quality, physics based simulations in accurately simulate a LiDAR beam, to produce data which is as close as possible to real data and simulates a LiDAR beam with a 2D tree canopy in a 2D grid using a probability
map to determine how the light propagates, whereby Lambertian or mirror reflection can be considered. (See P [0004-0005])
Regarding Claim 3: M in view of P, further in view of F disclose the system of claim 1.
M does not disclose, wherein the beam divergence model corresponds to a gaussian beam divergence model.
However, P discloses wherein the beam divergence model corresponds to a gaussian beam divergence model.
P [0010] “A LIDAR beam is characterized by several parameters: e.g. the beam cross-section, the beam divergence, the beam emission pattern and the beam pulse pattern. A LIDAR beam commonly used in automotive has a rectangular or square cross-section within which a laser or multiple lasers have a emission pattern comprising horizontal or vertical rows/stripes or any other suitable pattern including meander shapes, circles, ellipses, spirals, a Gaussian configuration and the like. The beam may originate from a point source or several point sources defining a source area. From the point source or the source area the beam diverges with a certain divergence angle or divergence angles, which may be measured relative to the emission direction of the point source or source area, respectively. If the LIDAR beam is emitted by a point source, the beam has a certain initial source area some distance away from the actual point source. In order to be able to use the method for single laser and multi laser LIDAR beam simulation the beam is always simulated starting from a source area.”
M and P are analogous to the claimed invention because they both pertain to simulation of a LiDAR sensor.
It would have been obvious to one with ordinary skill in the art to combine the teachings of M and P because P runs high-quality, physics based simulations in accurately simulate a LiDAR beam, to produce data which is as close as possible to real data and simulates a LiDAR beam with a 2D tree canopy in a 2D grid using a probability
map to determine how the light propagates, whereby Lambertian or mirror reflection can be considered. (See P [0004-0005])
Regarding Claim 4: M in view of P, further in view of F discloses the system of claim 1, wherein the one or more intensity parameters are based on an angle of incidence between a corresponding ray from the one or more rays and a corresponding virtual object from the one or more virtual objects.
M [0096] “In addition to geometric information, the computing system can record sensory metadata 210 for each surfel to be used for intensity and ray drop simulation. This can include, among other information, the incidence angle, raw intensity, transmitted power level, range value as well as a unique ID per beam.”
Regarding Claim 5: M in view of P, further in view of F discloses the system of claim 1, wherein the one or more intensity parameters are based on one or more reflectivity parameters corresponding to the one or more virtual objects.
M [0086] “The illustrated process focuses on simulating a scanning LiDAR system. One example system that can be simulated is the Velodyne HDL-64E which has 64 emitter-detector pairs vertically arranged, each of which uses light pulses to measure distance. The basic concept is that each emitter emits a light pulse which travels until it hits a target, and a portion of the light energy is reflected back and received by the detector. Distance is measured by calculating the time of travel and material reflectance is measured through the intensity of the returned pulse.”
Regarding Claim 8: M discloses a method comprising:
generating, within a simulation environment, at least one virtual beam transmission from a Light Detection and Ranging (LiDAR) sensor;
M [0034] “Certain existing approaches to LiDAR simulation for autonomous driving focus on employing handcrafted 3D primitives (such as buildings, cars, trees, roads). Graphics engines have been utilized to ray cast the scene and create virtual LiDAR data.”
M [0049] “In some implementations, the computing system can account for the rotary motion of the virtual LiDAR system (also known as “rolling shutter effects”) by compensating for motion of the virtual system along the trajectory during the simulated LiDAR sweep. In particular, performing the ray casting can include determining, for each of a plurality of rays, a ray casting location and a ray casting direction based at least in part on the trajectory.”
M [0039] “In particular, aspects of the present disclosure are directed to systems and methods that use a machine-learned model to make an initial three-dimensional point cloud generated using a physics-based approach more realistic. In particular, the machine-learned model can learn to modify the geometry of point clouds (e.g., as exhibited by ray dropouts) generated through ray casting and/or other physics-based approaches to better match ground truth counterparts that were physically collected by LiDAR systems in the real world.”
determining one or more intensity parameters associated with one or more virtual beam receptions by the LiDAR sensor, wherein the one or more virtual beam receptions correspond to one or more rays from the plurality of rays that are reflected from one or more virtual objects;
M [0044] “The computing system can convert the aggregate LiDAR point cloud to a surface element-based three-dimensional mesh. For example, the computing system can perform voxel-based downsampling and normal estimation to perform the conversion. In addition to the geometric information, sensory metadata (e.g., incidence angle, raw intensity, transmitted power level, range value, unique ID per beam, etc.) can be recorded for each surface element (e.g., to be used for intensity simulation).”
M [0086] “The illustrated process focuses on simulating a scanning LiDAR system. One example system that can be simulated is the Velodyne HDL-64E which has 64 emitter-detector pairs vertically arranged, each of which uses light pulses to measure distance. The basic concept is that each emitter emits a light pulse which travels until it hits a target, and a portion of the light energy is reflected back and received by the detector. Distance is measured by calculating the time of travel and material reflectance is measured through the intensity of the returned pulse. The entire optical assembly rotates on a base to provide a 360-degree azimuth field of view at around 10 Hz with each full “sweep” providing approximately 70 k returns.”
adjusting the one or more intensity parameters … to yield one or more modified intensity parameters;
M [0155] “At 614, the computing system can modify one or more values of one or more parameters of the machine-learned model based at least in part on the objective function. For example, the objective function can be backpropagated through the model and the values of the parameters can be updated based on a gradient of the objective function.”
and determining at least one object intensity parameter for each of the one or more virtual objects based on the one or more modified intensity parameters.
M [0044] “The computing system can convert the aggregate LiDAR point cloud to a surface element-based three-dimensional mesh. For example, the computing system can perform voxel-based downsampling and normal estimation to perform the conversion. In addition to the geometric information, sensory metadata (e.g., incidence angle, raw intensity, transmitted power level, range value, unique ID per beam, etc.) can be recorded for each surface element (e.g., to be used for intensity simulation).”
sending the one or more modified intensity parameters to a perception stack of an autonomous vehicle operating in the simulation environment, wherein the one or more modified intensity parameters, when received by the perception stack, are configured to cause the perception stack to detect and classify the one or more virtual objects
M [0029] “The simulated LiDAR data can be used, for example, as simulated input for testing autonomous vehicle control systems. The systems and methods of the present disclosure improve both quantitatively and qualitatively the synthesized LiDAR data over solely physics-based rendering. The improved quality of the synthesized LiDAR point cloud demonstrates the potential of this LiDAR simulation approach and application to generating realistic sensor data, which will ultimately improve the safety an autonomous vehicle.”
and optimizing a performance of a machine learning model that utilizes LiDAR data to detect and identify objects, the machine learning model being optimized based on the one or more modified intensity parameters sent to the perception stack, wherein the machine learning model, after being optimized, is configured to detect and identify objects from LiDAR data captured by a physical LiDAR sensor of a physical autonomous vehicle during operation of the physical autonomous vehicle, the physical LiDAR sensor being simulated by the LiDAR sensor.
M [0029] “The simulated LiDAR data can be used, for example, as simulated input for testing autonomous vehicle control systems. The systems and methods of the present disclosure improve both quantitatively and qualitatively the synthesized LiDAR data over solely physics-based rendering. The improved quality of the synthesized LiDAR point cloud demonstrates the potential of this LiDAR simulation approach and application to generating realistic sensor data, which will ultimately improve the safety an autonomous vehicle.”
M [0041] “Thus, in some implementations, to generate a three-dimensional map of an environment, the computing system can first obtain a plurality of sets of real-world LiDAR data physically collected by one or more LiDAR systems in the environment. For example, these sets of real-world LiDAR data can have been collected by autonomous vehicles and/or non-autonomous vehicles as they traveled through the environment.”
M [0054] “In such fashion, the systems and methods enable the efficient and accurate generation of synthetic—yet realistic—LiDAR data. The ability to generate such synthetic LiDAR data has a number of benefits, including, for example, the ability to test/train autonomous vehicle systems on the synthetic LiDAR data. In particular, LiDAR data can be synthesized for challenging edge cases which can enable more robust testing/training of autonomous vehicle systems, thereby leading to autonomous vehicles which demonstrate improved safety, efficiency, and/or other performance measures.”
M does not disclose using a beam divergence model, wherein the at least one virtual beam transmission includes a plurality of rays distributed around a center of the at least one virtual beam transmission, one or more transmission intensity weights corresponding to the one or more rays, wherein each transmission intensity weight of the one or more transmission intensity weights is based on a distance of a corresponding ray of the plurality of rays from the center of the at least one virtual beam transmission, and wherein the one or more transmission intensity weights simulate a decrease in energy due to beam divergence
However, P discloses a beam divergence model
P [0010] “A LIDAR beam is characterized by several parameters: e.g. the beam cross-section, the beam divergence, the beam emission pattern and the beam pulse pattern. A LIDAR beam commonly used in automotive has a rectangular or square cross-section within which a laser or multiple lasers have a emission pattern comprising horizontal or vertical rows/stripes or any other suitable pattern including meander shapes, circles, ellipses, spirals, a Gaussian configuration and the like. The beam may originate from a point source or several point sources defining a source area. From the point source or the source area the beam diverges with a certain divergence angle or divergence angles, which may be measured relative to the emission direction of the point source or source area, respectively. If the LIDAR beam is emitted by a point source, the beam has a certain initial source area some distance away from the actual point source. In order to be able to use the method for single laser and multi laser LIDAR beam simulation the beam is always simulated starting from a source area.”
wherein the at least one virtual beam transmission includes a plurality of rays distributed around a center of the at least one virtual beam transmission
P [0011] “In the first step a) the source area is discretized into separate origins. The source area may be discretized into 100,000 or less, preferably 10,000 or less and most preferably 4000 or less origins. Each origin can be the origin of one single ray of the LIDAR beam. Each ray has a certain emitting direction such that the LIDAR beam is simulated as a truncated pyramid of arbitrary cross-sectional shape like a frustum originating at the source area and diverging with a certain angle of divergence from the emission direction of the source area. The direction of each ray may be characterized by two divergence angles, θ and φ. The two divergence angles may each be within one of two perpendicular planes spanned by the emission direction and a respective radial direction perpendicular to the emission direction.” Examiner notes that the divergence angles represent the lateral expansion of the rays from the central axis as they travel forward, where θ is the angle the ray travels in the x-direction, whereas φ is the angle the ray travels in the y-direction.
M and P are analogous to the claimed invention because they both pertain to simulation of a LiDAR sensor.
It would have been obvious to one with ordinary skill in the art to combine the teachings of M and P because P runs high-quality, physics based simulations in accurately simulate a LiDAR beam, to produce data which is as close as possible to real data and simulates a LiDAR beam with a 2D tree canopy in a 2D grid using a probability
map to determine how the light propagates, whereby Lambertian or mirror reflection can be considered. (See P [0004-0005])
M and P do not explicitly disclose wherein one or more transmission intensity weights corresponding to the one or more rays, wherein each transmission intensity weight of the one or more transmission intensity weights is based on a distance of a corresponding ray of the plurality of rays from the center of the at least one virtual beam transmission
However, F discloses wherein one or more transmission intensity weights corresponding to the one or more rays, wherein each transmission intensity weight of the one or more transmission intensity weights is based on a distance of a corresponding ray of the plurality of rays from the center of the at least one virtual beam transmission.
F [Page 93: Section 3.2] “Assuming the incident laser is Gaussian, the normalized intensity distribution is as follows:
PNG
media_image1.png
26
97
media_image1.png
Greyscale
where I(r) is the intensity at a radial distance r from central axis. The shaped annular laser has a characteristic that the central energy of the incident Gaussian is transferred to the external edge of the annular beam while the external edge energy of the incident Gaussian is transferred to the center of the annular beam. Therefore, the intensity distribution of the annular beam is:
PNG
media_image2.png
36
262
media_image2.png
Greyscale
Examiner notes that given the equations above, as the radius increases, the intensity of the ray decreases.
M, P and F are all analogous to the claimed invention because they all pertain with analysis of a LiDAR beam.
It would have been obvious to one with ordinary skill in the art before the effective filing date to combine the teachings of F with M and P because F discloses an optimal beam shaping approach that analyzes the intensity distribution of a Gaussian beam and shapes the beam such that it improves the imaging quality of linear-array LiDAR by providing an oblate U-shaped intensity distribution, which yields approximately a 30% gain in detection area. (See F [Abstract])
Regarding Claim 9: M in view of P, further in view of F discloses the method of claim 8, wherein at least one virtual beam transmission further includes one or more additional rays reflected from a first virtual object other than the one or more virtual objects, and wherein the one or more processors are further configured to:
determine an object intensity parameter corresponding to the first virtual object based on one or more additional virtual beam receptions corresponding to the one or more additional rays
M [0086] “The illustrated process focuses on simulating a scanning LiDAR system. One example system that can be simulated is the Velodyne HDL-64E which has 64 emitter-detector pairs vertically arranged, each of which uses light pulses to measure distance. The basic concept is that each emitter emits a light pulse which travels until it hits a target, and a portion of the light energy is reflected back and received by the detector. Distance is measured by calculating the time of travel and material reflectance is measured through the intensity of the returned pulse. The entire optical assembly rotates on a base to provide a 360-degree azimuth field of view at around 10 Hz with each full “sweep” providing approximately 70 k returns.”
and determine that the object intensity parameter corresponding to the first virtual object fails to satisfy
M [0125] “The output of the model is a ray dropout probability that predicts, for each element in the array, if it returns or not (e.g., with some probability). In some implementations, to simulate LiDAR noise, the computing system can sample from the probability mask to generate the output LiDAR point cloud. Sampling of the probability mask instead of doing direct thresholding has the following benefits: (1) Raydrop can be learned with cross-entropy loss, meaning the estimated probabilities may not be well calibrated. Sampling helps mitigate this issue compared to thresholding. (2) Real lidar data is non-deterministic due to additional noises (atmospheric transmittance, sensor bias, etc.) that the proposed approach may not completely model.”
M [0029] “The simulated LiDAR data can be used, for example, as simulated input for testing autonomous vehicle control systems. The systems and methods of the present disclosure improve both quantitatively and qualitatively the synthesized LiDAR data over solely physics-based rendering. The improved quality of the synthesized LiDAR point cloud demonstrates the potential of this LiDAR simulation approach and application to generating realistic sensor data, which will ultimately improve the safety an autonomous vehicle.”
M doesn’t disclose a threshold intensity value.
However, P discloses a threshold intensity value.
P [0016] “If no (more) intersection of a current ray (originally emitted ray or scattered ray, respectively) occurs during ray-tracing in step c1) then according to step c3) for each of the rays, the next ray is ray-traced in step c1 ). Consequently, each of the selected rays is ray-traced and, if an intersection is found, scattered and, in turn, the resulting scattered ray is traced until no more intersection with an object in the scene occurs or optionally until the lower threshold for the energy of the ray or the maximal number of reflections is reached. At each intersection the reflected energy based on the current ray (originally emitted ray or scattered ray, respectively) is added to the signal of the LIDAR sensor.”
M and P are analogous to the claimed invention because they both pertain to simulation of a LiDAR sensor.
It would have been obvious to one with ordinary skill in the art to combine the teachings of M and P because P runs high-quality, physics based simulations in accurately simulate a LiDAR beam, to produce data which is as close as possible to real data and simulates a LiDAR beam with a 2D tree canopy in a 2D grid using a probability
map to determine how the light propagates, whereby Lambertian or mirror reflection can be considered. (See P [0004-0005])
Regarding Claim 10: M in view of P, further in view of F disclose the method of claim 8.
M does not disclose, wherein the beam divergence model corresponds to a gaussian beam divergence model.
However, P discloses wherein the beam divergence model corresponds to a gaussian beam divergence model.
P [0010] “A LIDAR beam is characterized by several parameters: e.g. the beam cross-section, the beam divergence, the beam emission pattern and the beam pulse pattern. A LIDAR beam commonly used in automotive has a rectangular or square cross-section within which a laser or multiple lasers have a emission pattern comprising horizontal or vertical rows/stripes or any other suitable pattern including meander shapes, circles, ellipses, spirals, a Gaussian configuration and the like. The beam may originate from a point source or several point sources defining a source area. From the point source or the source area the beam diverges with a certain divergence angle or divergence angles, which may be measured relative to the emission direction of the point source or source area, respectively. If the LIDAR beam is emitted by a point source, the beam has a certain initial source area some distance away from the actual point source. In order to be able to use the method for single laser and multi laser LIDAR beam simulation the beam is always simulated starting from a source area.”
M and P are analogous to the claimed invention because they both pertain to simulation of a LiDAR sensor.
It would have been obvious to one with ordinary skill in the art to combine the teachings of M and P because P runs high-quality, physics based simulations in accurately simulate a LiDAR beam, to produce data which is as close as possible to real data and simulates a LiDAR beam with a 2D tree canopy in a 2D grid using a probability
map to determine how the light propagates, whereby Lambertian or mirror reflection can be considered. (See P [0004-0005])
Regarding Claim 11: M in view of P, further in view of F discloses the method of claim 8, wherein the one or more intensity parameters are based on an angle of incidence between a corresponding ray from the one or more rays and a corresponding virtual object from the one or more virtual objects.
M [0096] “In addition to geometric information, the computing system can record sensory metadata 210 for each surfel to be used for intensity and ray drop simulation. This can include, among other information, the incidence angle, raw intensity, transmitted power level, range value as well as a unique ID per beam.”
Regarding Claim 12: M in view of P, further in view of F discloses the method of claim 8, wherein the one or more intensity parameters are based on one or more reflectivity parameters corresponding to the one or more virtual objects.
M [0086] “The illustrated process focuses on simulating a scanning LiDAR system. One example system that can be simulated is the Velodyne HDL-64E which has 64 emitter-detector pairs vertically arranged, each of which uses light pulses to measure distance. The basic concept is that each emitter emits a light pulse which travels until it hits a target, and a portion of the light energy is reflected back and received by the detector. Distance is measured by calculating the time of travel and material reflectance is measured through the intensity of the returned pulse.”
Regarding Claim 15: M discloses a non-transitory computer-readable media comprising instructions stored thereon which, when executed are configured to cause a computer or processor to:
generate, within a simulation environment, at least one virtual beam transmission from a Light Detection and Ranging (LiDAR) sensor;
M [0034] “Certain existing approaches to LiDAR simulation for autonomous driving focus on employing handcrafted 3D primitives (such as buildings, cars, trees, roads). Graphics engines have been utilized to ray cast the scene and create virtual LiDAR data.”
M [0049] “In some implementations, the computing system can account for the rotary motion of the virtual LiDAR system (also known as “rolling shutter effects”) by compensating for motion of the virtual system along the trajectory during the simulated LiDAR sweep. In particular, performing the ray casting can include determining, for each of a plurality of rays, a ray casting location and a ray casting direction based at least in part on the trajectory.”
M [0039] “In particular, aspects of the present disclosure are directed to systems and methods that use a machine-learned model to make an initial three-dimensional point cloud generated using a physics-based approach more realistic. In particular, the machine-learned model can learn to modify the geometry of point clouds (e.g., as exhibited by ray dropouts) generated through ray casting and/or other physics-based approaches to better match ground truth counterparts that were physically collected by LiDAR systems in the real world.”
determine one or more intensity parameters associated with one or more virtual beam receptions by the LiDAR sensor, wherein the one or more virtual beam receptions correspond to one or more rays from the plurality of rays that are reflected from one or more virtual objects;
M [0044] “The computing system can convert the aggregate LiDAR point cloud to a surface element-based three-dimensional mesh. For example, the computing system can perform voxel-based downsampling and normal estimation to perform the conversion. In addition to the geometric information, sensory metadata (e.g., incidence angle, raw intensity, transmitted power level, range value, unique ID per beam, etc.) can be recorded for each surface element (e.g., to be used for intensity simulation).”
M [0086] “The illustrated process focuses on simulating a scanning LiDAR system. One example system that can be simulated is the Velodyne HDL-64E which has 64 emitter-detector pairs vertically arranged, each of which uses light pulses to measure distance. The basic concept is that each emitter emits a light pulse which travels until it hits a target, and a portion of the light energy is reflected back and received by the detector. Distance is measured by calculating the time of travel and material reflectance is measured through the intensity of the returned pulse. The entire optical assembly rotates on a base to provide a 360-degree azimuth field of view at around 10 Hz with each full “sweep” providing approximately 70 k returns.”
adjust the one or more intensity parameters … to yield one or more modified intensity parameters;
M [0155] “At 614, the computing system can modify one or more values of one or more parameters of the machine-learned model based at least in part on the objective function. For example, the objective function can be backpropagated through the model and the values of the parameters can be updated based on a gradient of the objective function.”
and determine at least one object intensity parameter for each of the one or more virtual objects based on the one or more modified intensity parameters.
M [0044] “The computing system can convert the aggregate LiDAR point cloud to a surface element-based three-dimensional mesh. For example, the computing system can perform voxel-based downsampling and normal estimation to perform the conversion. In addition to the geometric information, sensory metadata (e.g., incidence angle, raw intensity, transmitted power level, range value, unique ID per beam, etc.) can be recorded for each surface element (e.g., to be used for intensity simulation).”
send the one or more modified intensity parameters to a perception stack of an autonomous vehicle operating in the simulation environment, wherein the one or more modified intensity parameters, when received by the perception stack, are configured to cause the perception stack to detect and classify the one or more virtual objects
M [0029] “The simulated LiDAR data can be used, for example, as simulated input for testing autonomous vehicle control systems. The systems and methods of the present disclosure improve both quantitatively and qualitatively the synthesized LiDAR data over solely physics-based rendering. The improved quality of the synthesized LiDAR point cloud demonstrates the potential of this LiDAR simulation approach and application to generating realistic sensor data, which will ultimately improve the safety an autonomous vehicle.”
and optimize a performance of a machine learning model that utilizes LiDAR data to detect and identify objects, the machine learning model being optimized based on the one or more modified intensity parameters sent to the perception stack, wherein the machine learning model, after being optimized, is configured to detect and identify objects from LiDAR data captured by a physical LiDAR sensor of a physical autonomous vehicle during operation of the physical autonomous vehicle, the physical LiDAR sensor being simulated by the LiDAR sensor.
M [0029] “The simulated LiDAR data can be used, for example, as simulated input for testing autonomous vehicle control systems. The systems and methods of the present disclosure improve both quantitatively and qualitatively the synthesized LiDAR data over solely physics-based rendering. The improved quality of the synthesized LiDAR point cloud demonstrates the potential of this LiDAR simulation approach and application to generating realistic sensor data, which will ultimately improve the safety an autonomous vehicle.”
M [0041] “Thus, in some implementations, to generate a three-dimensional map of an environment, the computing system can first obtain a plurality of sets of real-world LiDAR data physically collected by one or more LiDAR systems in the environment. For example, these sets of real-world LiDAR data can have been collected by autonomous vehicles and/or non-autonomous vehicles as they traveled through the environment.”
M [0054] “In such fashion, the systems and methods enable the efficient and accurate generation of synthetic—yet realistic—LiDAR data. The ability to generate such synthetic LiDAR data has a number of benefits, including, for example, the ability to test/train autonomous vehicle systems on the synthetic LiDAR data. In particular, LiDAR data can be synthesized for challenging edge cases which can enable more robust testing/training of autonomous vehicle systems, thereby leading to autonomous vehicles which demonstrate improved safety, efficiency, and/or other performance measures.”
M does not disclose using a beam divergence model, wherein the at least one virtual beam transmission includes a plurality of rays distributed around a center of the at least one virtual beam transmission, one or more transmission intensity weights corresponding to the one or more rays, wherein each transmission intensity weight of the one or more transmission intensity weights is based on a distance of a corresponding ray of the plurality of rays from the center of the at least one virtual beam transmission, and wherein the one or more transmission intensity weights simulate a decrease in energy due to beam divergence
However, P discloses a beam divergence model
P [0010] “A LIDAR beam is characterized by several parameters: e.g. the beam cross-section, the beam divergence, the beam emission pattern and the beam pulse pattern. A LIDAR beam commonly used in automotive has a rectangular or square cross-section within which a laser or multiple lasers have a emission pattern comprising horizontal or vertical rows/stripes or any other suitable pattern including meander shapes, circles, ellipses, spirals, a Gaussian configuration and the like. The beam may originate from a point source or several point sources defining a source area. From the point source or the source area the beam diverges with a certain divergence angle or divergence angles, which may be measured relative to the emission direction of the point source or source area, respectively. If the LIDAR beam is emitted by a point source, the beam has a certain initial source area some distance away from the actual point source. In order to be able to use the method for single laser and multi laser LIDAR beam simulation the beam is always simulated starting from a source area.”
wherein the at least one virtual beam transmission includes a plurality of rays distributed around a center of the at least one virtual beam transmission
P [0011] “In the first step a) the source area is discretized into separate origins. The source area may be discretized into 100,000 or less, preferably 10,000 or less and most preferably 4000 or less origins. Each origin can be the origin of one single ray of the LIDAR beam. Each ray has a certain emitting direction such that the LIDAR beam is simulated as a truncated pyramid of arbitrary cross-sectional shape like a frustum originating at the source area and diverging with a certain angle of divergence from the emission direction of the source area. The direction of each ray may be characterized by two divergence angles, θ and φ. The two divergence angles may each be within one of two perpendicular planes spanned by the emission direction and a respective radial direction perpendicular to the emission direction.” Examiner notes that the divergence angles represent the lateral expansion of the rays from the central axis as they travel forward, where θ is the angle the ray travels in the x-direction, whereas φ is the angle the ray travels in the y-direction.
M and P are analogous to the claimed invention because they both pertain to simulation of a LiDAR sensor.
It would have been obvious to one with ordinary skill in the art to combine the teachings of M and P because P runs high-quality, physics based simulations in accurately simulate a LiDAR beam, to produce data which is as close as possible to real data and simulates a LiDAR beam with a 2D tree canopy in a 2D grid using a probability
map to determine how the light propagates, whereby Lambertian or mirror reflection can be considered. (See P [0004-0005])
M and P do not explicitly disclose wherein one or more transmission intensity weights corresponding to the one or more rays, wherein each transmission intensity weight of the one or more transmission intensity weights is based on a distance of a corresponding ray of the plurality of rays from the center of the at least one virtual beam transmission
However, F discloses wherein one or more transmission intensity weights corresponding to the one or more rays, wherein each transmission intensity weight of the one or more transmission intensity weights is based on a distance of a corresponding ray of the plurality of rays from the center of the at least one virtual beam transmission.
F [Page 93: Section 3.2] “Assuming the incident laser is Gaussian, the normalized intensity distribution is as follows:
PNG
media_image1.png
26
97
media_image1.png
Greyscale
where I(r) is the intensity at a radial distance r from central axis. The shaped annular laser has a characteristic that the central energy of the incident Gaussian is transferred to the external edge of the annular beam while the external edge energy of the incident Gaussian is transferred to the center of the annular beam. Therefore, the intensity distribution of the annular beam is:
PNG
media_image2.png
36
262
media_image2.png
Greyscale
Examiner notes that given the equations above, as the radius increases, the intensity of the ray decreases.
M, P and F are all analogous to the claimed invention because they all pertain with analysis of a LiDAR beam.
It would have been obvious to one with ordinary skill in the art before the effective filing date to combine the teachings of F with M and P because F discloses an optimal beam shaping approach that analyzes the intensity distribution of a Gaussian beam and shapes the beam such that it improves the imaging quality of linear-array LiDAR by providing an oblate U-shaped intensity distribution, which yields approximately a 30% gain in detection area. (See F [Abstract])
Regarding Claim 16: M in view of P, further in view of F discloses the non-transitory computer-readable media of claim 15, wherein at least one virtual beam transmission further includes one or more additional rays reflected from a first virtual object other than the one or more virtual objects, and wherein the one or more processors are further configured to:
determine an object intensity parameter corresponding to the first virtual object based on one or more additional virtual beam receptions corresponding to the one or more additional rays
M [0086] “The illustrated process focuses on simulating a scanning LiDAR system. One example system that can be simulated is the Velodyne HDL-64E which has 64 emitter-detector pairs vertically arranged, each of which uses light pulses to measure distance. The basic concept is that each emitter emits a light pulse which travels until it hits a target, and a portion of the light energy is reflected back and received by the detector. Distance is measured by calculating the time of travel and material reflectance is measured through the intensity of the returned pulse. The entire optical assembly rotates on a base to provide a 360-degree azimuth field of view at around 10 Hz with each full “sweep” providing approximately 70 k returns.”
and determine that the object intensity parameter corresponding to the first virtual object fails to satisfy
M [0125] “The output of the model is a ray dropout probability that predicts, for each element in the array, if it returns or not (e.g., with some probability). In some implementations, to simulate LiDAR noise, the computing system can sample from the probability mask to generate the output LiDAR point cloud. Sampling of the probability mask instead of doing direct thresholding has the following benefits: (1) Raydrop can be learned with cross-entropy loss, meaning the estimated probabilities may not be well calibrated. Sampling helps mitigate this issue compared to thresholding. (2) Real lidar data is non-deterministic due to additional noises (atmospheric transmittance, sensor bias, etc.) that the proposed approach may not completely model.”
M [0029] “The simulated LiDAR data can be used, for example, as simulated input for testing autonomous vehicle control systems. The systems and methods of the present disclosure improve both quantitatively and qualitatively the synthesized LiDAR data over solely physics-based rendering. The improved quality of the synthesized LiDAR point cloud demonstrates the potential of this LiDAR simulation approach and application to generating realistic sensor data, which will ultimately improve the safety an autonomous vehicle.”
M doesn’t disclose a threshold intensity value.
However, P discloses a threshold intensity value.
P [0016] “If no (more) intersection of a current ray (originally emitted ray or scattered ray, respectively) occurs during ray-tracing in step c1) then according to step c3) for each of the rays, the next ray is ray-traced in step c1 ). Consequently, each of the selected rays is ray-traced and, if an intersection is found, scattered and, in turn, the resulting scattered ray is traced until no more intersection with an object in the scene occurs or optionally until the lower threshold for the energy of the ray or the maximal number of reflections is reached. At each intersection the reflected energy based on the current ray (originally emitted ray or scattered ray, respectively) is added to the signal of the LIDAR sensor.”
M and P are analogous to the claimed invention because they both pertain to simulation of a LiDAR sensor.
It would have been obvious to one with ordinary skill in the art to combine the teachings of M and P because P runs high-quality, physics based simulations in accurately simulate a LiDAR beam, to produce data which is as close as possible to real data and simulates a LiDAR beam with a 2D tree canopy in a 2D grid using a probability
map to determine how the light propagates, whereby Lambertian or mirror reflection can be considered. (See P [0004-0005])
Regarding Claim 17: M in view of P, further in view of F discloses the non-transitory computer-readable media of claim 15, wherein the one or more intensity parameters are based on an angle of incidence between a corresponding ray from the one or more rays and a corresponding virtual object from the one or more virtual objects.
M [0096] “In addition to geometric information, the computing system can record sensory metadata 210 for each surfel to be used for intensity and ray drop simulation. This can include, among other information, the incidence angle, raw intensity, transmitted power level, range value as well as a unique ID per beam.”
Regarding Claim 18: M in view of P, further in view of F discloses the non-transitory computer-readable media of claim 15, wherein the one or more intensity parameters are based on one or more reflectivity parameters corresponding to the one or more virtual objects.
M [0086] “The illustrated process focuses on simulating a scanning LiDAR system. One example system that can be simulated is the Velodyne HDL-64E which has 64 emitter-detector pairs vertically arranged, each of which uses light pulses to measure distance. The basic concept is that each emitter emits a light pulse which travels until it hits a target, and a portion of the light energy is reflected back and received by the detector. Distance is measured by calculating the time of travel and material reflectance is measured through the intensity of the returned pulse.”
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Claims 1-5, 8-12, and 15-18 are rejected.
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure:
U.S. Patent Application No. 2023/0114731 A1
U.S. Patent Application No. 2021/0208263 A1
Aniceto Belmonte, "Feasibility study for the simulation of beam propagation: consideration of coherent lidar performance," Appl. Opt. 39, 5426-5445 (2000)
Lidar Essential Beam Model for Accurate Width Estimation of Thin Poles (Long, 2020)
U.S. Patent Application 2002/0075763 A1
CN 110175344 A
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Scott T. Tran whose telephone number is (571) 272-8533. The examiner can normally be reached on M-Thurs, 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Renee Chavez, can be reached at (571) 270-1104. The fax phone number for the organization where this application or proceeding is assigned is (571) 273-8300. Informal or draft communication, please label PROPOSED or DRAFT, can be additionally sent to the Examiner’s fax phone number (571) 272-8533.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published a applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
STT
/SCOTT THANH BINH TRAN/Examiner, Art Unit 2186 /RENEE D CHAVEZ/Supervisory Patent Examiner, Art Unit 2186
