Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA
Specification
Applicant is reminded of the proper language and format for an abstract of the disclosure.
The abstract should be in narrative form and generally limited to a single paragraph on a separate sheet within the range of 50 to 150 words in length. The abstract should describe the disclosure sufficiently to assist readers in deciding whether there is a need for consulting the full patent text for details.
The language should be clear and concise and should not repeat information given in the title. It should avoid using phrases which can be implied, such as, “The disclosure concerns,” “The disclosure defined by this invention,” “The disclosure describes,” etc. In addition, the form and legal phraseology often used in patent claims, such as “means” and “said,” should be avoided.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-6, 8, 10-13, 19, 22, 24-26, 32, 33, 37, 38, 40-42, 44-51 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hennecke (Patent No. US 20210109199 A1) in view of Saranin (Patent No. WO 2022086739 A2).
Regarding claim 1, Hennecke teaches A LIDAR system, comprising: at least one laser light source; at least one LIDAR sensor; and at least one processor configured to: (Hennecke, “[0027] Thus, the system controller 23 includes at least one processor and/or processor circuitry (e.g., comparators and digital signal processors (DSPs)) of a signal processing chain for processing data, as well as control circuitry, such as a microcontroller, that is configured to generate control signals. / [0028] The LIDAR scanning system 100 may also include a sensor 26, such as a temperature sensor, that provides temperature sensor information to the system controller 23. For example, sensor 26 may measure a laser temperature of the illumination unit 10, and the system controller 23 may use the measured laser temperature to perform a calibration, determine which SPAD pixels to activate/deactivate, or determine which SPAD pixels correspond to received laser light and which SPAD pixels correspond to received ambient light. / [0030] The illumination unit 10 is a laser array that includes one or more light sources (e.g., laser diodes, light emitting diodes, or laser channels) that are configured to transmit light used for scanning a field of view for objects. The light emitted by the light sources is typically infrared light although light with another wavelength might also be used.”)
cause the at least one laser light source to project laser light toward a field of view of the LIDAR system; (Hennecke, “[0030] The illumination unit 10 is a laser array that includes one or more light sources (e.g., laser diodes, light emitting diodes, or laser channels) that are configured to transmit light used for scanning a field of view for objects. The light emitted by the light sources is typically infrared light although light with another wavelength might also be used. The shape of the light emitted by the light sources may be spread in a direction perpendicular to a scanning direction to form a light beam with an oblong shape extending, lengthwise, perpendicular to the scanning direction. The illumination light transmitted from the light sources may be directed towards a transmitter optics (not illustrated) that is configured to focus each laser onto a MEMS mirror 12, which in turn directs the laser beams into the field of view. The transmitter optics may be, for example, a lens or a prism.”)
receive first signals from the at least one LIDAR sensor indicative of times of flight of the laser light that is reflected from objects in the field of view and is incident on the at least one LIDAR sensor; (Hennecke, “[0034] Upon impinging one or more objects, the transmitted laser light is reflected by backscattering back towards the LIDAR scanning system 100 as a reflected light where the receiver 22 receives the reflected light. The receiver incudes a receiver mirror that receives the reflected light and directs the light along the receiver path. The receiver mirror may be the MEMS mirror 12 or a second MEMS mirror separate from MEMS mirror 12.”)
receive second signals from the at least one LIDAR sensor in response to non-laser light that is reflected from objects in the field of view and is incident on the at least one LIDAR sensor; (Hennecke, “[0058] The light includes both laser light (i.e., signal photons) and ambient light (i.e., ambient photons). By spreading the light across the SPAD array, the number of SPAD pixels that receive the light is increased. Furthermore, ambient photons and signal photons are spread apart from each other, thus reducing the ambient light flux per SPAD pixel. As a result, the probability of detecting signal photons by the SiPM pixel 34 is increased.”)
generate a point cloud including distance information relative to the objects in the field of view of the LIDAR system based on the first signals response (Hennecke, “0048] As noted above, the receiver circuit 24 includes a readout circuit that further includes one or more readout channels coupled to the photodetector array 15. The receiver circuit 24 may receive the electrical signals from the one or more SiPM pixels of the photodetector array 15 and transmit the electrical signals as raw sensor data to the system controller 23 for ToF measurement and generation of object data (e.g., 3D point cloud data).”)
generate an image representative of at least a portion of the field of view of the LIDAR system based on the second signals (Hennecke, “[0086] Alternatively, all SPAD pixels may remain activated, read out, and the sensor information thereof stored in memory. Here, sensor information generated by those SPAD pixels not in the determined location may be separated from sensor information generated by “signal” SPADs and used for other purposes. For example, the spectrum of ambient light can be recorded and used for generating a color image of environment, which may help with sensor fusion and object classification.”)
and reconstruct a three-dimensional representation of the object using the adjusted point cloud. (Hennecke, “[0036] The receiver optical component directs the reflected light onto a further receiver component, such as a spatial filter, to be described in more detail below. Ultimately, the received light is projected onto a photodetector array 15 that is configured to generate electrical measurement signals based on the received light incident thereon. The electrical measurement signals may be used by the system controller 23, received as raw sensor data, for generating a 3D map of the environment and/or other object data based on the reflected light (e.g., via TOF calculations and processing)”)
However, Hennecke is silent about detect an object in the image; identify an area of the point cloud containing the detected object; compare the identified area of the point cloud to the image to select points in the point cloud corresponding to the detected object while adjusting the point cloud to correct inconsistencies with the image in the identified area;
Saranin teaches detect an object in the image; (Saranin, “[0009] The present disclosure also concerns implementing systems and methods for CLF object detection with point pruning. The present solution can be used to operate an autonomous vehicle. In this scenario, the methods comprise: obtaining, by a computing device, a LiDAR dataset generated by a LiDAR system of the autonomous vehicle; and using, by a computing device, the LiDAR dataset and at least one image to detect an object that is in proximity to the autonomous vehicle.”)
identify an area of the point cloud containing the detected object; (Saranin, “[0009] The present disclosure also concerns implementing systems and methods for CLF object detection with point pruning. The present solution can be used to operate an autonomous vehicle. In this scenario, the methods comprise: obtaining, by a computing device, a LiDAR dataset generated by a LiDAR system of the autonomous vehicle; and using, by a computing device, the LiDAR dataset and at least one image to detect an object that is in proximity to the autonomous vehicle.”)
compare the identified area of the point cloud to the image to select points in the point cloud corresponding to the detected object (Saranin, “[00114] The CLF object detection takes full advantage of monocular camera image detections where detections are fused with the LiDAR point cloud. LiDAR data points are projected into the monocular camera frame in order to transfer pixel information to the LiDAR data points, as described above. The transferred information can include, but is not limited to, color, object type and object instance.”)
while adjusting the point cloud to correct inconsistencies with the image in the identified area; (Saranin, “[0009] The present disclosure also concerns implementing systems and methods for CLF object detection with point pruning.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including detect an object in the image; identify an area of the point cloud containing the detected object; compare the identified area of the point cloud to the image to select points in the point cloud corresponding to the detected object while adjusting the point cloud to correct inconsistencies with the image in the identified area; as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
The motivation for the combination is to improve the object detection in the image of scanned image.
Regarding claim 2, Hennecke is silent about The LIDAR system of The LIDAR system of wherein adjusting the point cloud includes removal or addition of points from the generated point cloud.
Saranin teaches The LIDAR system of The LIDAR system of wherein adjusting the point cloud includes removal or addition of points from the generated point cloud. (Saranin, “[0009] The object is detected by generating a pruned LiDAR dataset by reducing a total number of points contained in the LiDAR dataset, and detecting the object in a point cloud defined by the pruned LiDAR dataset.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The LIDAR system of The LIDAR system of wherein adjusting the point cloud includes removal or addition of points from the generated point cloud as taught by Saranin and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 3, Hennecke is silent about The LIDAR system of claim 1, wherein adjusting the point cloud includes updating a confidence level of points from the generated point cloud.
Saranin teaches The LIDAR system of claim 1, wherein adjusting the point cloud includes updating a confidence level of points from the generated point cloud. (Saranin, “[0007] Alternatively or additionally, the matching comprises determining a probability distribution over a set of object detections in which a point of the LiDAR dataset is likely to be, based on at least one confidence value indicating a level of confidence that at least one respective pixel of the at least one image belongs to a given detected object.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The LIDAR system of claim 1, wherein adjusting the point cloud includes updating a confidence level of points from the generated point cloud as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 4, Hennecke is silent about The LIDAR system of claim 1, wherein adjusting the point cloud includes changing a classification of at least one object representation in the generated point cloud.
Saranin teaches The LIDAR system of claim 1, wherein adjusting the point cloud includes changing a classification of at least one object representation in the generated point cloud. (Saranin, “[0090] As shown in FIG. 5, an object classification is performed in block 504 to classify the detected object into one of a plurality of classes and/or sub-classes. The classes can include, but are not limited to, a vehicle class and a pedestrian class. The vehicle class can have a plurality of vehicle sub-classes. The vehicle sub-classes can include, but are not limited to, a bicycle subclass, a motorcycle sub-class, a skateboard sub-class, a roller blade sub-class, a scooter sub-class, a sedan sub-class, an SUV sub-class, and/or a truck sub-class. The object classification is made based on sensor data generated by a LiDAR system (e.g., LiDAR system 264 of FIG. 2) and/or a camera (e.g., camera 262 of FIG. 2) of the vehicle. Techniques for classifying objects based on LiDAR data and/or imagery data are well known in the art. Any known or to be known object classification technique can be used herein without limitation. Information 530 specifying the object’s classification is provided to block 512, in addition to the information 532 indicating the object’s actual speed and direction of travel.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The LIDAR system of claim 1, wherein adjusting the point cloud includes changing a classification of at least one object representation in the generated point cloud as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 5, Hennecke is silent about The LIDAR system of claim 1, wherein adjusting the point cloud includes a reduction in size of at least one object representation in the generated point cloud.
Saranin teaches The LIDAR system of claim 1, wherein adjusting the point cloud includes a reduction in size of at least one object representation in the generated point cloud. (Saranin, “[0012] In those or other scenarios, the pruned LiDAR dataset is generated by downsampling the LiDAR dataset based on point labels assigned to the points. Each of the point labels may comprise at least one of an object class identifier, a color, and/or a unique identifier.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The LIDAR system of claim 1, wherein adjusting the point cloud includes a reduction in size of at least one object representation in the generated point cloud as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 6, Hennecke is silent about The LIDAR system of claim 1, wherein adjusting the point cloud includes a change in shape of at least one object representation in the generated point cloud.
Saranin teaches The LIDAR system of claim 1, wherein adjusting the point cloud includes a change in shape of at least one object representation in the generated point cloud. (Saranin, “[00176] It should be noted that one issue that is difficult to deal with from a pairwise segment merging perspective is fragments of larger vehicles. For example, a large box truck may be observed as multiple fragments that are far apart Due to projection uncertainty, these fragments often do not project into the same image detection mask as the back of the truck, and there is not enough context in order for the merger to combine these fragments based on the merge probabilities. Therefore, the merger 1716 performs additional operations to fit each large detected segment detected as vehicle (e.g., a back of the truck) to a shape model (e.g., a cuboid) in order to estimate the true extent of the detected object. Bounding box estimator may use ground height and lane information from onboard HD map and visual heading from image detection. The estimated cuboid now has enough information to merge fragments based on their overlap area with the estimated cuboid. Another example where cuboids help is segmentation of the buses. A large window area allows laser light to pass through and scan the interior portions of the bus resulting in multiple fragments that are far away from the L-shape of the bus exterior. Upon completing the merge operations, the merger 1716 outputs a plurality of merged segments 1714”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The LIDAR system of claim 1, wherein adjusting the point cloud includes a change in shape of at least one object representation in the generated point cloud as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 8, Hennecke is silent about The LIDAR system of claim 1, wherein identifying the area of the point cloud containing the detected object includes a comparison of a size of an object representation in the point cloud with a size of a object in the image.
Saranin teaches The LIDAR system of claim 1, wherein identifying the area of the point cloud containing the detected object includes a comparison of a size of an object representation in the point cloud with a size of a object in the image. (Saranin, “[00180] An example is a vehicle detection with a pole in front and also a pedestrian behind. LiDAR data points that belong to the true pedestrian object and the pole object will have points labeled as vehicles due to projection errors that occur during the sensor fusion stage. In order to resolve the ambiguity of image detection mask to segment association, projection characteristics are computed for all segments containing LiDAR data points that project into a particular image detection mask. One or more best matches are reported that are likely to correspond to the object detected on the image. This helps eliminate clutter from the set of tracked objects, and reduces tracking pipeline latency and computational requirements. / [00182] There are other cluster features that can be used to identify segments of LiDAR data points that are associated with a pedestrian, a vehicle, a bicyclist, and/or any other moving object. These additional cluster features include a cluster feature H representing a cluster height, a cluster feature £ representing a cluster length, and a cluster feature LTW representing a length- to-width ratio for a cluster. / [00183] Clusters with a height above 2.0 - 2.5 meters are unlikely to be associated with pedestrians. Clusters over 1 meter in length are unlikely to be associated with pedestrians. Clusters with a length-to-width ratio above 4.0 often tend to be associated with buildings and are unlikely associated with pedestrians. Clusters with high cylinder convolution score are likely to be associated with pedestrians.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The LIDAR system of claim 1, wherein identifying the area of the point cloud containing the detected object includes a comparison of a size of an object representation in the point cloud with a size of a object in the image as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 10, Hennecke is silent about The LIDAR system of claim 1, wherein identifying the area of the point cloud containing the detected object includes a comparison of a shape of an object representation in the point cloud with a shape of object in the image.
Saranin teaches The LIDAR system of claim 1, wherein identifying the area of the point cloud containing the detected object includes a comparison of a shape of an object representation in the point cloud with a shape of object in the image. (Saranin, “[00180] An example is a vehicle detection with a pole in front and also a pedestrian behind. LiDAR data points that belong to the true pedestrian object and the pole object will have points labeled as vehicles due to projection errors that occur during the sensor fusion stage. In order to resolve the ambiguity of image detection mask to segment association, projection characteristics are computed for all segments containing LiDAR data points that project into a particular image detection mask. One or more best matches are reported that are likely to correspond to the object detected on the image. This helps eliminate clutter from the set of tracked objects, and reduces tracking pipeline latency and computational requirements. / [00182] There are other cluster features that can be used to identify segments of LiDAR data points that are associated with a pedestrian, a vehicle, a bicyclist, and/or any other moving object. These additional cluster features include a cluster feature H representing a cluster height, a cluster feature £ representing a cluster length, and a cluster feature LTW representing a length- to-width ratio for a cluster. / [00183] Clusters with a height above 2.0 - 2.5 meters are unlikely to be associated with pedestrians. Clusters over 1 meter in length are unlikely to be associated with pedestrians. Clusters with a length-to-width ratio above 4.0 often tend to be associated with buildings and are unlikely associated with pedestrians. Clusters with high cylinder convolution score are likely to be associated with pedestrians.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The LIDAR system of claim 1, wherein identifying the area of the point cloud containing the detected object includes a comparison of a shape of an object representation in the point cloud with a shape of object in the image as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 11, Hennecke is silent about The LIDAR system of claim 1, wherein comparing the identified area of the point cloud and the image identifies an object in the image that is not recognized in the point cloud.
Saranin teaches The LIDAR system of claim 1, wherein comparing the identified area of the point cloud and the image identifies an object in the image that is not recognized in the point cloud. (Saranin, “[00122] The present CLF based solution has many advantages. For example, the present CLF based solution takes full advantage of image detections but does not only rely on image detections or machine learning. This means both separating objects in close proximity and detecting objects that have not been recognized before. This approach combines ML image detections with classical methods for point cloud segmentation”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The LIDAR system of claim 1, wherein comparing the identified area of the point cloud and the image identifies an object in the image that is not recognized in the point cloud as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 12, Hennecke is silent about The LIDAR system of claim 1, wherein the at least one processor comprises a trained neural network for performing a comparison of the point cloud and the image.
Saranin teaches The LIDAR system of claim 1, wherein the at least one processor comprises a trained neural network for performing a comparison of the point cloud and the image. (Saranin, “[00120] Segment Merger: Any machine-learned classification technique can be employed by the present solution to learn which segments should be merged. The machine-learned classification technique includes, but are not limited to, an artificial neural network, a random forest, a decision tree, and/or a support vector machine. The machine-learned classification technique is trained to determine which segments should be merged with each other. The same image detection information that was used in segmentation is now aggregated over the constituent points of the segment in order to compute segment-level features. In addition to that the ground height and lane information features from HD map are also used to aid segment merging.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The LIDAR system of claim 1, wherein the at least one processor comprises a trained neural network for performing a comparison of the point cloud and the image as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 13, Hennecke is silent about The LIDAR system of claim 12, wherein the neural network is trained based on a dataset of captured images and associated point clouds.
Saranin teaches The LIDAR system of claim 12, wherein the neural network is trained based on a dataset of captured images and associated point clouds. (Saranin, “[00173] The machine learned classifier 1714 is trained using a machine learning algorithm that learns when two segments should be merged together in view of one or more features. Any machine learning algorithm can be used herein without limitation. For example, one or more of the following machine learning algorithms is employed here: supervised learning; unsupervised learning; semi-supervised learning; and reinforcement learning. The learned information by the machine learning algorithm can be used to generate rules for determining a probability that two segments should be merged. These rules are then implemented by the machine learned classifier 1714.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The LIDAR system of claim 1, wherein the at least one processor comprises a trained neural network for performing a comparison of the point cloud and the image as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 19, Hennecke teaches The LIDAR system of claim 1, wherein the at least one laser light source includes an array of laser sources. (Saranin, “[0030] The illumination unit 10 is a laser array that includes one or more light sources (e.g., laser diodes, light emitting diodes, or laser channels) that are configured to transmit light used for scanning a field of view for objects. The light emitted by the light sources is typically infrared light although light with another wavelength might also be used. The shape of the light emitted by the light sources may be spread in a direction perpendicular to a scanning direction to form a light beam with an oblong shape extending, lengthwise, perpendicular to the scanning direction. The illumination light transmitted from the light sources may be directed towards a transmitter optics (not illustrated) that is configured to focus each laser onto a MEMS mirror 12, which in turn directs the laser beams into the field of view. The transmitter optics may be, for example, a lens or a prism.”)
Regarding claim 22, Hennecke teaches The LIDAR system of claim 1, wherein the at least one processor is configured to generate the image using an exposure time that is determined per pixel of the field of view. (Hennecke, “[0085] As noted above, when an entire DMD pixel column is activated, each row of the SPAD array receives light. As a result, one or a select few SPAD pixel columns may correspond to the determined location of the laser light spectrum. Thus, this corresponding SPAD column or columns may be activated and read out, while the remaining SPAD columns may be deactivated or ignored. / [0086] Alternatively, all SPAD pixels may remain activated, read out, and the sensor information thereof stored in memory. Here, sensor information generated by those SPAD pixels not in the determined location may be separated from sensor information generated by “signal” SPADs and used for other purposes. For example, the spectrum of ambient light can be recorded and used for generating a color image of environment, which may help with sensor fusion and object classification.”)
Regarding claim 24, Hennecke teaches The LIDAR system of claim 1, wherein the at least one processor is further configured to selectively generate the image representative of the at least a portion of the field of view of the LIDAR system based on one or more sensed environmental conditions. (Hennecke, “[0086] Alternatively, all SPAD pixels may remain activated, read out, and the sensor information thereof stored in memory. Here, sensor information generated by those SPAD pixels not in the determined location may be separated from sensor information generated by “signal” SPADs and used for other purposes. For example, the spectrum of ambient light can be recorded and used for generating a color image of environment, which may help with sensor fusion and object classification. / [0087] Additionally or alternatively, the spectrum of ambient light can be recorded and used for generating a hyperspectral image of environment, which may also help with object classification. Since different spectral components are received in different SPAD columns, these detected spectral components from ambient light (i.e., non-laser light spectral components) may indicate the color of an object that reflects the ambient light. A signal processor at the system controller 23 may receive this color information from the remaining SPAD pixels that do not correspond to the determined location of the laser light spectrum and use the color information to classify a type of object that reflected the ambient light.)
Regarding claim 25, Hennecke teaches The LIDAR system of claim 1, wherein the at least one processor is further configured to forgo generation of the image when a sensed ambient light level is less than a predetermined threshold. (Hennecke, “[0086] Alternatively, all SPAD pixels may remain activated, read out, and the sensor information thereof stored in memory. Here, sensor information generated by those SPAD pixels not in the determined location may be separated from sensor information generated by “signal” SPADs and used for other purposes. For example, the spectrum of ambient light can be recorded and used for generating a color image of environment, which may help with sensor fusion and object classification. / [0087] Additionally or alternatively, the spectrum of ambient light can be recorded and used for generating a hyperspectral image of environment, which may also help with object classification. Since different spectral components are received in different SPAD columns, these detected spectral components from ambient light (i.e., non-laser light spectral components) may indicate the color of an object that reflects the ambient light. A signal processor at the system controller 23 may receive this color information from the remaining SPAD pixels that do not correspond to the determined location of the laser light spectrum and use the color information to classify a type of object that reflected the ambient light.”)
Regarding claim 26, Hennecke teaches The LIDAR system of claim 1, wherein the at least one processor is further configured to forgo the adjustment of one or more aspects of the generated point cloud comparison based on a sensed quality of the image. (Hennecke, “[0086] Alternatively, all SPAD pixels may remain activated, read out, and the sensor information thereof stored in memory. Here, sensor information generated by those SPAD pixels not in the determined location may be separated from sensor information generated by “signal” SPADs and used for other purposes. For example, the spectrum of ambient light can be recorded and used for generating a color image of environment, which may help with sensor fusion and object classification. / [0087] Additionally or alternatively, the spectrum of ambient light can be recorded and used for generating a hyperspectral image of environment, which may also help with object classification. Since different spectral components are received in different SPAD columns, these detected spectral components from ambient light (i.e., non-laser light spectral components) may indicate the color of an object that reflects the ambient light. A signal processor at the system controller 23 may receive this color information from the remaining SPAD pixels that do not correspond to the determined location of the laser light spectrum and use the color information to classify a type of object that reflected the ambient light.”)
Regarding claim 32, Hennecke teaches A method for detecting objects using a LIDAR system, the method comprising: (Hennecke, “[0108] In addition, some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, some one or more of the method steps may be executed by such an apparatus.”)
receiving a point cloud including distance information relative to objects in a field of view of the LIDAR system based on first signals generated by at least one LIDAR sensor in the LIDAR system in response to received laser light return signals reflected from the objects in the field of view; (Hennecke, “[0034] Upon impinging one or more objects, the transmitted laser light is reflected by backscattering back towards the LIDAR scanning system 100 as a reflected light where the receiver 22 receives the reflected light. The receiver incudes a receiver mirror that receives the reflected light and directs the light along the receiver path. The receiver mirror may be the MEMS mirror 12 or a second MEMS mirror separate from MEMS mirror 12. / [0048] As noted above, the receiver circuit 24 includes a readout circuit that further includes one or more readout channels coupled to the photodetector array 15. The receiver circuit 24 may receive the electrical signals from the one or more SiPM pixels of the photodetector array 15 and transmit the electrical signals as raw sensor data to the system controller 23 for ToF measurement and generation of object data (e.g., 3D point cloud data).”)
receiving an image representative of at least a portion of the field of view of the LIDAR system based on second signals generated by the at least one LIDAR sensor in the LIDAR system in response to non-laser light incident upon the at least one LIDAR sensor; (Hennecke, “[0058] The light includes both laser light (i.e., signal photons) and ambient light (i.e., ambient photons). By spreading the light across the SPAD array, the number of SPAD pixels that receive the light is increased. Furthermore, ambient photons and signal photons are spread apart from each other, thus reducing the ambient light flux per SPAD pixel. As a result, the probability of detecting signal photons by the SiPM pixel 34 is increased. / [0086] Alternatively, all SPAD pixels may remain activated, read out, and the sensor information thereof stored in memory. Here, sensor information generated by those SPAD pixels not in the determined location may be separated from sensor information generated by “signal” SPADs and used for other purposes. For example, the spectrum of ambient light can be recorded and used for generating a color image of environment, which may help with sensor fusion and object classification.”)
and reconstruct a three-dimensional representation of the object using the adjusted point cloud. (Hennecke, “[0036] The receiver optical component directs the reflected light onto a further receiver component, such as a spatial filter, to be described in more detail below. Ultimately, the received light is projected onto a photodetector array 15 that is configured to generate electrical measurement signals based on the received light incident thereon. The electrical measurement signals may be used by the system controller 23, received as raw sensor data, for generating a 3D map of the environment and/or other object data based on the reflected light (e.g., via TOF calculations and processing)”)
However, Hennecke is silent about detect an object in the image; identify an area of the point cloud containing the detected object; compare the identified area of the point cloud to the image to select points in the point cloud corresponding to the detected object while adjusting the point cloud to correct inconsistencies with the image in the identified area;
Saranin teaches detect an object in the image; (Saranin, “[0009] The present disclosure also concerns implementing systems and methods for CLF object detection with point pruning. The present solution can be used to operate an autonomous vehicle. In this scenario, the methods comprise: obtaining, by a computing device, a LiDAR dataset generated by a LiDAR system of the autonomous vehicle; and using, by a computing device, the LiDAR dataset and at least one image to detect an object that is in proximity to the autonomous vehicle.”)
identify an area of the point cloud containing the detected object; (Saranin, “[0009] The present disclosure also concerns implementing systems and methods for CLF object detection with point pruning. The present solution can be used to operate an autonomous vehicle. In this scenario, the methods comprise: obtaining, by a computing device, a LiDAR dataset generated by a LiDAR system of the autonomous vehicle; and using, by a computing device, the LiDAR dataset and at least one image to detect an object that is in proximity to the autonomous vehicle.”)
compare the identified area of the point cloud to the image to select points in the point cloud corresponding to the detected object (Saranin, “[00114] The CLF object detection takes full advantage of monocular camera image detections where detections are fused with the LiDAR point cloud. LiDAR data points are projected into the monocular camera frame in order to transfer pixel information to the LiDAR data points, as described above. The transferred information can include, but is not limited to, color, object type and object instance.”)
while adjusting the point cloud to correct inconsistencies with the image in the identified area; (Saranin, “[0009] The present disclosure also concerns implementing systems and methods for CLF object detection with point pruning.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including detect an object in the image; identify an area of the point cloud containing the detected object; compare the identified area of the point cloud to the image to select points in the point cloud corresponding to the detected object while adjusting the point cloud to correct inconsistencies with the image in the identified area; as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 33, Hennecke teaches A non-transitory computer-readable storage medium including stored instructions that, when executed by at least one processor, cause the at least one processor to perform a method for detecting objects using a LIDAR system, the method comprising: (Hennecke, “[0027] The LIDAR scanning system 100 includes a transmitter unit 21 that is responsible for an emitter path of the system 100, and a receiver unit 22 that is responsible for a receiver path of the system 100. The system also includes a system controller 23 that is configured to control components of the transmitter unit 21 and the receiver unit 22, and to receive raw digital data from the receiver unit 22 and perform processing thereon (e.g., via digital signal processing) for generating object data (e.g., point cloud data). Thus, the system controller 23 includes at least one processor and/or processor circuitry (e.g., comparators and digital signal processors (DSPs)) of a signal processing chain for processing data, as well as control circuitry, such as a microcontroller, that is configured to generate control signals / [0086] Alternatively, all SPAD pixels may remain activated, read out, and the sensor information thereof stored in memory. Here, sensor information generated by those SPAD pixels not in the determined location may be separated from sensor information generated by “signal” SPADs and used for other purposes. For example, the spectrum of ambient light can be recorded and used for generating a color image of environment, which may help with sensor fusion and object classification.”)
receiving a point cloud including distance information relative to objects in a field of view of the LIDAR system based on first signals generated by at least one LIDAR sensor in the LIDAR system in response to received laser light return signals reflected from the objects in the field of view; (Hennecke, “[0034] Upon impinging one or more objects, the transmitted laser light is reflected by backscattering back towards the LIDAR scanning system 100 as a reflected light where the receiver 22 receives the reflected light. The receiver incudes a receiver mirror that receives the reflected light and directs the light along the receiver path. The receiver mirror may be the MEMS mirror 12 or a second MEMS mirror separate from MEMS mirror 12. / [0048] As noted above, the receiver circuit 24 includes a readout circuit that further includes one or more readout channels coupled to the photodetector array 15. The receiver circuit 24 may receive the electrical signals from the one or more SiPM pixels of the photodetector array 15 and transmit the electrical signals as raw sensor data to the system controller 23 for ToF measurement and generation of object data (e.g., 3D point cloud data).”)
receiving an image representative of at least a portion of the field of view of the LIDAR system based on second signals generated by the at least one LIDAR sensor in the LIDAR system in response to non-laser light incident upon the at least one LIDAR sensor; (Hennecke, “[0058] The light includes both laser light (i.e., signal photons) and ambient light (i.e., ambient photons). By spreading the light across the SPAD array, the number of SPAD pixels that receive the light is increased. Furthermore, ambient photons and signal photons are spread apart from each other, thus reducing the ambient light flux per SPAD pixel. As a result, the probability of detecting signal photons by the SiPM pixel 34 is increased. / [0086] Alternatively, all SPAD pixels may remain activated, read out, and the sensor information thereof stored in memory. Here, sensor information generated by those SPAD pixels not in the determined location may be separated from sensor information generated by “signal” SPADs and used for other purposes. For example, the spectrum of ambient light can be recorded and used for generating a color image of environment, which may help with sensor fusion and object classification.”)
and reconstruct a three-dimensional representation of the object using the adjusted point cloud. (Hennecke, “[0036] The receiver optical component directs the reflected light onto a further receiver component, such as a spatial filter, to be described in more detail below. Ultimately, the received light is projected onto a photodetector array 15 that is configured to generate electrical measurement signals based on the received light incident thereon. The electrical measurement signals may be used by the system controller 23, received as raw sensor data, for generating a 3D map of the environment and/or other object data based on the reflected light (e.g., via TOF calculations and processing)”)
However, Hennecke is silent about detect an object in the image; identify an area of the point cloud containing the detected object; compare the identified area of the point cloud to the image to select points in the point cloud corresponding to the detected object while adjusting the point cloud to correct inconsistencies with the image in the identified area;
Saranin teaches detect an object in the image; (Saranin, “[0009] The present disclosure also concerns implementing systems and methods for CLF object detection with point pruning. The present solution can be used to operate an autonomous vehicle. In this scenario, the methods comprise: obtaining, by a computing device, a LiDAR dataset generated by a LiDAR system of the autonomous vehicle; and using, by a computing device, the LiDAR dataset and at least one image to detect an object that is in proximity to the autonomous vehicle.”)
identify an area of the point cloud containing the detected object; (Saranin, “[0009] The present disclosure also concerns implementing systems and methods for CLF object detection with point pruning. The present solution can be used to operate an autonomous vehicle. In this scenario, the methods comprise: obtaining, by a computing device, a LiDAR dataset generated by a LiDAR system of the autonomous vehicle; and using, by a computing device, the LiDAR dataset and at least one image to detect an object that is in proximity to the autonomous vehicle.”)
compare the identified area of the point cloud to the image to select points in the point cloud corresponding to the detected object (Saranin, “[00114] The CLF object detection takes full advantage of monocular camera image detections where detections are fused with the LiDAR point cloud. LiDAR data points are projected into the monocular camera frame in order to transfer pixel information to the LiDAR data points, as described above. The transferred information can include, but is not limited to, color, object type and object instance.”)
while adjusting the point cloud to correct inconsistencies with the image in the identified area; (Saranin, “[0009] The present disclosure also concerns implementing systems and methods for CLF object detection with point pruning.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including detect an object in the image; identify an area of the point cloud containing the detected object; compare the identified area of the point cloud to the image to select points in the point cloud corresponding to the detected object while adjusting the point cloud to correct inconsistencies with the image in the identified area; as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 37, Hennecke is silent about The method of claim 32, further comprising classifying the object as one of a car, truck, human, or traffic sign.
Saranin teaches The method of claim 32, further comprising classifying the object as one of a car, truck, human, or traffic sign. (Saranin, “[0090] As shown in FIG. 5, an object classification is performed in block 504 to classify the detected object into one of a plurality of classes and/or sub-classes. The classes can include, but are not limited to, a vehicle class and a pedestrian class. The vehicle class can have a plurality of vehicle sub-classes. The vehicle sub-classes can include, but are not limited to, a bicycle subclass, a motorcycle sub-class, a skateboard sub-class, a roller blade sub-class, a scooter sub-class, a sedan sub-class, an SUV sub-class, and/or a truck sub-class. The object classification is made based on sensor data generated by a LiDAR system (e.g., LiDAR system 264 of FIG. 2) and/or a camera (e.g., camera 262 of FIG. 2) of the vehicle. Techniques for classifying objects based on LiDAR data and/or imagery data are well known in the art. Any known or to be known object classification technique can be used herein without limitation. Information 530 specifying the object’s classification is provided to block 512, in addition to the information 532 indicating the object’s actual speed and direction of travel.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The method of claim 32, further comprising classifying the object as one of a car, truck, human, or traffic sign as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 38, Hennecke is silent about The method of claim 32, wherein identifying the area of the point cloud containing the detected object includes a comparison of a size of the object in the point cloud with a size of a corresponding representation of the object in the image.
Saranin teaches The method of claim 32, wherein identifying the area of the point cloud containing the detected object includes a comparison of a size of the object in the point cloud with a size of a corresponding representation of the object in the image. (Saranin, “[00180] An example is a vehicle detection with a pole in front and also a pedestrian behind. LiDAR data points that belong to the true pedestrian object and the pole object will have points labeled as vehicles due to projection errors that occur during the sensor fusion stage. In order to resolve the ambiguity of image detection mask to segment association, projection characteristics are computed for all segments containing LiDAR data points that project into a particular image detection mask. One or more best matches are reported that are likely to correspond to the object detected on the image. This helps eliminate clutter from the set of tracked objects, and reduces tracking pipeline latency and computational requirements. / [00182] There are other cluster features that can be used to identify segments of LiDAR data points that are associated with a pedestrian, a vehicle, a bicyclist, and/or any other moving object. These additional cluster features include a cluster feature H representing a cluster height, a cluster feature £ representing a cluster length, and a cluster feature LTW representing a length- to-width ratio for a cluster. / [00183] Clusters with a height above 2.0 - 2.5 meters are unlikely to be associated with pedestrians. Clusters over 1 meter in length are unlikely to be associated with pedestrians. Clusters with a length-to-width ratio above 4.0 often tend to be associated with buildings and are unlikely associated with pedestrians. Clusters with high cylinder convolution score are likely to be associated with pedestrians.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The method of claim 32, wherein identifying the area of the point cloud containing the detected object includes a comparison of a size of the object in the point cloud with a size of a corresponding representation of the object in the image as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 40, Hennecke is silent about The method of claim 32, wherein identifying the area of the point cloud containing the detected object includes a comparison of a shape of the first object in the point cloud with a shape of a corresponding representation of the object in the image.
Saranin teaches The method of claim 32, wherein identifying the area of the point cloud containing the detected object includes a comparison of a shape of the first object in the point cloud with a shape of a corresponding representation of the object in the image. (Saranin, “[00180] An example is a vehicle detection with a pole in front and also a pedestrian behind. LiDAR data points that belong to the true pedestrian object and the pole object will have points labeled as vehicles due to projection errors that occur during the sensor fusion stage. In order to resolve the ambiguity of image detection mask to segment association, projection characteristics are computed for all segments containing LiDAR data points that project into a particular image detection mask. One or more best matches are reported that are likely to correspond to the object detected on the image. This helps eliminate clutter from the set of tracked objects, and reduces tracking pipeline latency and computational requirements. / [00182] There are other cluster features that can be used to identify segments of LiDAR data points that are associated with a pedestrian, a vehicle, a bicyclist, and/or any other moving object. These additional cluster features include a cluster feature H representing a cluster height, a cluster feature £ representing a cluster length, and a cluster feature LTW representing a length- to-width ratio for a cluster. / [00183] Clusters with a height above 2.0 - 2.5 meters are unlikely to be associated with pedestrians. Clusters over 1 meter in length are unlikely to be associated with pedestrians. Clusters with a length-to-width ratio above 4.0 often tend to be associated with buildings and are unlikely associated with pedestrians. Clusters with high cylinder convolution score are likely to be associated with pedestrians.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The method of claim 32, wherein identifying the area of the point cloud containing the detected object includes a comparison of a shape of the first object in the point cloud with a shape of a corresponding representation of the object in the image as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 41, Hennecke is silent about The method of claim 32, wherein comparing the identified area of the point cloud to the image is performed by a trained neural network.
Saranin teaches The method of claim 32, wherein comparing the identified area of the point cloud to the image is performed by a trained neural network. (Saranin, “[00120] Segment Merger: Any machine-learned classification technique can be employed by the present solution to learn which segments should be merged. The machine-learned classification technique includes, but are not limited to, an artificial neural network, a random forest, a decision tree, and/or a support vector machine. The machine-learned classification technique is trained to determine which segments should be merged with each other. The same image detection information that was used in segmentation is now aggregated over the constituent points of the segment in order to compute segment-level features. In addition to that the ground height and lane information features from HD map are also used to aid segment merging.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The method of claim 32, wherein comparing the identified area of the point cloud to the image is performed by a trained neural network as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 42, Hennecke is silent about The method of claim 41, wherein the neural network is trained based on a dataset of annotated images to identify various categories of objects.
Saranin teaches The method of claim 41, wherein the neural network is trained based on a dataset of annotated images to identify various categories of objects. (Saranin, “[00173] The machine learned classifier 1714 is trained using a machine learning algorithm that learns when two segments should be merged together in view of one or more features. Any machine learning algorithm can be used herein without limitation. For example, one or more of the following machine learning algorithms is employed here: supervised learning; unsupervised learning; semi-supervised learning; and reinforcement learning. The learned information by the machine learning algorithm can be used to generate rules for determining a probability that two segments should be merged. These rules are then implemented by the machine learned classifier 1714.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The method of claim 41, wherein the neural network is trained based on a dataset of annotated images to identify various categories of objects as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 44, Hennecke is silent about The method of claim 32, wherein adjusting the point cloud includes removal or addition of points in the received point cloud.
Saranin teaches The method of claim 32, wherein adjusting the point cloud includes removal or addition of points in the received point cloud. (Saranin, “[0009] The object is detected by generating a pruned LiDAR dataset by reducing a total number of points contained in the LiDAR dataset, and detecting the object in a point cloud defined by the pruned LiDAR dataset.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The method of claim 32, wherein adjusting the point cloud includes removal or addition of points in the received point cloud as taught by Saranin and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 45, Hennecke is silent about The method of claim 32, wherein adjusting the point cloud includes updating a confidence level of points in the generated point cloud.
Saranin teaches The method of claim 32, wherein adjusting the point cloud includes updating a confidence level of points in the generated point cloud. (Saranin, “[0007] Alternatively or additionally, the matching comprises determining a probability distribution over a set of object detections in which a point of the LiDAR dataset is likely to be, based on at least one confidence value indicating a level of confidence that at least one respective pixel of the at least one image belongs to a given detected object.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The method of claim 32, wherein adjusting the point cloud includes updating a confidence level of points in the generated point cloud as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 46, Hennecke is silent about The method of claim 32, wherein adjusting the point cloud includes changing a classification of at least one object representation in the received point cloud.
Saranin teaches The method of claim 32, wherein adjusting the point cloud includes changing a classification of at least one object representation in the received point cloud. (Saranin, “[0090] As shown in FIG. 5, an object classification is performed in block 504 to classify the detected object into one of a plurality of classes and/or sub-classes. The classes can include, but are not limited to, a vehicle class and a pedestrian class. The vehicle class can have a plurality of vehicle sub-classes. The vehicle sub-classes can include, but are not limited to, a bicycle subclass, a motorcycle sub-class, a skateboard sub-class, a roller blade sub-class, a scooter sub-class, a sedan sub-class, an SUV sub-class, and/or a truck sub-class. The object classification is made based on sensor data generated by a LiDAR system (e.g., LiDAR system 264 of FIG. 2) and/or a camera (e.g., camera 262 of FIG. 2) of the vehicle. Techniques for classifying objects based on LiDAR data and/or imagery data are well known in the art. Any known or to be known object classification technique can be used herein without limitation. Information 530 specifying the object’s classification is provided to block 512, in addition to the information 532 indicating the object’s actual speed and direction of travel.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The method of claim 32, wherein adjusting the point cloud includes changing a classification of at least one object representation in the received point cloud as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 47, Hennecke is silent about The method of claim 32, wherein adjusting the point cloud includes a reduction in size of at least one object representation in the received point cloud.
Saranin teaches The method of claim 32, wherein adjusting the point cloud includes a reduction in size of at least one object representation in the received point cloud. (Saranin, “[0012] In those or other scenarios, the pruned LiDAR dataset is generated by downsampling the LiDAR dataset based on point labels assigned to the points. Each of the point labels may comprise at least one of an object class identifier, a color, and/or a unique identifier.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The method of claim 32, wherein adjusting the point cloud includes a reduction in size of at least one object representation in the received point cloud as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 48, Hennecke is silent about The method of claim 32, wherein adjusting the point cloud includes a change in shape of at least one object representation in the received point cloud.
Saranin teaches The method of claim 32, wherein adjusting the point cloud includes a change in shape of at least one object representation in the received point cloud. (Saranin, “[00176] It should be noted that one issue that is difficult to deal with from a pairwise segment merging perspective is fragments of larger vehicles. For example, a large box truck may be observed as multiple fragments that are far apart Due to projection uncertainty, these fragments often do not project into the same image detection mask as the back of the truck, and there is not enough context in order for the merger to combine these fragments based on the merge probabilities. Therefore, the merger 1716 performs additional operations to fit each large detected segment detected as vehicle (e.g., a back of the truck) to a shape model (e.g., a cuboid) in order to estimate the true extent of the detected object. Bounding box estimator may use ground height and lane information from onboard HD map and visual heading from image detection. The estimated cuboid now has enough information to merge fragments based on their overlap area with the estimated cuboid. Another example where cuboids help is segmentation of the buses. A large window area allows laser light to pass through and scan the interior portions of the bus resulting in multiple fragments that are far away from the L-shape of the bus exterior. Upon completing the merge operations, the merger 1716 outputs a plurality of merged segments 1714”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The method of claim 32, wherein adjusting the point cloud includes a change in shape of at least one object representation in the received point cloud as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 49, Hennecke is silent about The method of claim 32, wherein adjusting the point cloud includes a change in a position of at least one object representation in the received point cloud.
Saranin teaches The method of claim 32, wherein adjusting the point cloud includes a change in a position of at least one object representation in the received point cloud. (Saranin, “[0003] These distance measurements can be used for tracking movements of the object, making predictions as to the objects trajectory, and planning paths of travel for the vehicle based on the predicted objects trajectory.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The method of claim 32, wherein adjusting the point cloud includes a change in a position of at least one object representation in the received point cloud as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 50, Hennecke is silent about The method of claim 32, wherein comparing the identified area identifies an object in the image that was not recognized in the point cloud.
Saranin teaches The method of claim 32, wherein comparing the identified area identifies an object in the image that was not recognized in the point cloud. (Saranin, “[00122] The present CLF based solution has many advantages. For example, the present CLF based solution takes full advantage of image detections but does not only rely on image detections or machine learning. This means both separating objects in close proximity and detecting objects that have not been recognized before. This approach combines ML image detections with classical methods for point cloud segmentation”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The method of claim 32, wherein comparing the identified area identifies an object in the image that was not recognized in the point cloud as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Regarding claim 51, Hennecke is silent about The method of claim 50, wherein adjusting the point cloud includes adjusting the received point cloud to include a representation of the identified object.
Saranin teaches The method of claim 50, wherein adjusting the point cloud includes adjusting the received point cloud to include a representation of the identified object. (Saranin, “[0009] The present disclosure also concerns implementing systems and methods for CLF object detection with point pruning.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The method of claim 50, wherein adjusting the point cloud includes adjusting the received point cloud to include a representation of the identified object as taught by Saranin, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Claim(s) 9, 39 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hennecke (Patent No. US 20210109199 A1) in view of Saranin (Patent No. WO 2022086739 A2), in further view of Gibson (Patent No. US 20190250251 A1).
Regarding claim 9, Hennecke is silent about The LIDAR system of claim 1, wherein identifying the area of the point cloud containing the detected object includes a comparison of a reflectivity of object in the image.
Gibson teaches The LIDAR system of claim 1, wherein identifying the area of the point cloud (Gibson, “[0026] Direct correlation between the 3D point cloud generated by the LiDAR and the color images captured by an RGB (Red, Green, Blue) video camera can be achieved by using an optical beam splitter that sends optical signals simultaneously to both sensors, simplifying the sensor fusion that generates a color point cloud or RGBD data (Red, Green, Blue and Depth).”)
containing the detected object includes a comparison of a reflectivity of object in the image. (Gibson, “[0028] The exemplary LiDAR device includes for example Quanergy™ sensors using Time-of-Flight (TOF) capability to measure the distance and reflectivity of objects and record the data as a reproducible three-dimensional point cloud with intensity information. Operating at the 905 nm wavelength, sensitive detectors calculate the light's bounceback Time-of-Flight (TOF) to measure the object's distance and record the collected data as a reproducible three-dimensional point cloud. The sensor's ability to detect objects that vary in size, shape, and reflectivity is largely unaffected by ambient light/dark, infrared signature, and atmospheric conditions.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The LIDAR system of claim 1, wherein identifying the area of the point cloud containing the detected object includes a comparison of a reflectivity of object in the image as taught by Gibson, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
The motivation for the combination is to improve area identification with the introduction of reflectivity through the image.
Regarding claim 39, Hennecke is silent about The method of claim 32, wherein identifying the area of the point cloud containing the detected object includes a comparison of a reflectivity of the object in the point cloud with a reflectivity of a corresponding representation of the object in the image.
Gibson teaches The LIDAR system of claim 1, wherein identifying the area of the point cloud (Gibson, “[0026] Direct correlation between the 3D point cloud generated by the LiDAR and the color images captured by an RGB (Red, Green, Blue) video camera can be achieved by using an optical beam splitter that sends optical signals simultaneously to both sensors, simplifying the sensor fusion that generates a color point cloud or RGBD data (Red, Green, Blue and Depth).”)
containing the detected object includes a comparison of a reflectivity of the object in the point cloud with a reflectivity of a corresponding representation of the object in the image. (Gibson, “[0028] The exemplary LiDAR device includes for example Quanergy™ sensors using Time-of-Flight (TOF) capability to measure the distance and reflectivity of objects and record the data as a reproducible three-dimensional point cloud with intensity information. Operating at the 905 nm wavelength, sensitive detectors calculate the light's bounceback Time-of-Flight (TOF) to measure the object's distance and record the collected data as a reproducible three-dimensional point cloud. The sensor's ability to detect objects that vary in size, shape, and reflectivity is largely unaffected by ambient light/dark, infrared signature, and atmospheric conditions.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The method of claim 32, wherein identifying the area of the point cloud containing the detected object includes a comparison of a reflectivity of the object in the point cloud with a reflectivity of a corresponding representation of the object in the image as taught by Gibson, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Claim(s) 20, 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hennecke (Patent No. US 20210109199 A1) in view of Saranin (Patent No. WO 2022086739 A2), in further view of Ohzu (Patent No. US 20020118352 A1).
Regarding claim 20, Hennecke is silent about The LIDAR system of claim 1, wherein the at least one processor is configured to cause generation of the image between laser light frame captures of the field of view.
Ohzu teaches The LIDAR system of claim 1, wherein the at least one processor is configured to cause generation of the image between laser light frame captures of the field of view. (Ohzu, “[0019] Using the fast gating feature, it is also possible to capture instantaneously the information on a spatial cross-sectional distribution of substances in the atmosphere at any distance away from the laser radar apparatus (i.e., information on a two-dimensional spatial distribution at a specified distance). If the delay time of the gate from the point in time when the laser was emitted is shifted continuously for successive shots of laser pulsed light using the gating feature, one can also obtain information on continuous spatial cross-sectional distributions; by connecting these spatial distributions with the fast frame feature (high image capture and processing frequencies), a three-dimensional spatial distribution can be obtained instantaneously.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including The LIDAR system of claim 1, wherein the at least one processor is configured to cause generation of the image between laser light frame captures of the field of view as taught by Ohzu, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
The motivation for the combination is to improve image capture and the basic of image process of the invention.
Regarding claim 21, Hennecke teaches The LIDAR system of claim 1, further including at least one deflector configured to scan the field of view with the projected laser light (Hennecke, “[0024] In Light Detection and Ranging (LIDAR) systems, a light source transmits light pulses into a field of view and the light reflects from one or more objects by backscattering. In particular, LIDAR is a direct Time-of-Flight (TOF) system in which the light pulses (e.g., laser beams of infrared light) are emitted into the field of view, and a photodetector array that includes multiple photodetector pixels detects and measures the reflected beams. For example, an array of photodetectors receives reflections from objects illuminated by the light.”)
However, Hennecke is silent about and wherein the at least one processor is configured to generate the image after a predetermined number of scans of the field of view.
Ohzu teaches and wherein the at least one processor is configured to generate the image after a predetermined number of scans of the field of view. (Ohzu, “[0019] Using the fast gating feature, it is also possible to capture instantaneously the information on a spatial cross-sectional distribution of substances in the atmosphere at any distance away from the laser radar apparatus (i.e., information on a two-dimensional spatial distribution at a specified distance). If the delay time of the gate from the point in time when the laser was emitted is shifted continuously for successive shots of laser pulsed light using the gating feature, one can also obtain information on continuous spatial cross-sectional distributions; by connecting these spatial distributions with the fast frame feature (high image capture and processing frequencies), a three-dimensional spatial distribution can be obtained instantaneously.”)
Therefore, it would have been obvious for an ordinary skilled person in the art before the
effective filing date of claimed invention to have modified Hennecke’s art by including and wherein the at least one processor is configured to generate the image after a predetermined number of scans of the field of view as taught by Ohzu, and use that with Hennecke’s Scanning Lidar Receiver with a Silicon Photomultiplier Detector.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to CHAK FUNG A LAM whose telephone number is (571)272-9823. The examiner can
normally be reached Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a
USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use
the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor,
Said Broome can be reached at 5712722931. The fax phone number for the organization where this
application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from
Patent Center. Unpublished application information in Patent Center is available to registered users. To
file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit
https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center
andhttps://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional
questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or
571-272-1000.
/C.A.L./Examiner, Art Unit 2612
/Said Broome/Supervisory Patent Examiner, Art Unit 2612
Prosecution Insights