DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendments
2. The Amendment filled 03/06/2026 in response to Non-Final Office Action mailed 12/08/2025 has been entered.
3. Claims 1-20 are currently pending.
Response to Arguments
4. Applicants’ arguments, see pg. 1-6, with regard to the 102 and 103 rejections of claims 1-20 have been fully considered and are partially persuasive. Specifically, the applicant presents two arguments with regard to the 102 rejections of claims 1, 4-7, 10-11, 16, and 19 with regard to Lautala. The first argument contends that Lautala fails to disclose decimating the LIDAR samples within a predetermined area around the utility object, and the second argument contends that Lautala fails to disclose generating depth-encoded multi-perspective images. With regard to the first argument, see pg. 1-2, the examiner disagrees. Specifically, the applicant highlights that Lautala divides an image into quadrants, and samples high resolution in the area of interest (e.g., quadrans B1 and B2), and decimates and/or down samples in non-interest/background regions (e.g., quadrants B3 and B4). The applicant argues this is the opposite of the claimed invention. However, this is not the case, given the BRI of the claimed language. Specifically, the examiner highlights that the claim language does not preclude that the area around the object of interest cannot be imaged at a high resolution, only that samples within a predetermined area around the utility object are decimated. In this case, quadrants B1 and B2 correspond to interest areas, which include a utility object, and B3 and B4 correspond to a predetermined area around the utility object that do not include the utility object, and thus decimating B3 and B4 teaches decimating the LIDAR samples within a predetermined area around the utility object. Specifically, the examiner highlights that as currently claimed, the claim language fails to accurately define the scope of the predetermined area or what “around” constitutes (e.g., how far from an object, whether that area is everything but the object, etc.), and therefore, the broadest reasonable interpretation of “predetermined area around the utility object” would be directly analogous to B3 and B4. For another example, presented in Lautala, consider Fig. 4. Specifically, the areas 404, 406, and 408 are decimated, and are clearly around utility object 402, and the area 202 is not decimated, which includes the utility object ([par. 0106, ln. 1-30] “Referring to FIG. 4, illustrated is a diagrammatic illustration 400 of an exemplary environment 204 including a power-distribution infrastructure, in accordance with an embodiment of the present disclosure. As shown, the environment 204 comprises the area of interest 202 including a power-line arrangement 402 (including both the powerline and the pole). Typically, the method 100 and/or the system 200 is employed to determine at least the non-interesting areas in the environment 204, wherein the aerial vehicle 302 (shown in FIG. 3) may capture a lower resolution image to reduce the overall size of the image data captured. Further shown, the environment 204 comprises at least three distinct type of areas to be captured apart from the area of interest 202 covering the power-line arrangement 402. Notably, the at least three distinct areas are a ground surface 404, a water body 406 and a forest area 408. Typically, pursuant to the embodiment of the present disclosure, the aerial vehicle 302 is configured to capture lower resolution or quality image data of the ground surface 404, the water body 406 whereas a relatively higher pixel resolution or quality is used for the forest area 408 (for example, to identify the tree type and structure). Hence, the method 100 and/or the system 200 optimizes the size of the dataset by down-sampling or discarding the area 404, 406 and 408 based on the requirement. For example, the forest area 408 requires a full or at least higher resolution (for example, 512×512 pixels) whereas near the areas 404, 406, wherein only a flat ground and water body is present, the image data size may be optimized for e.g., by using low-resolution imaging such as of the order of 4×4, 8×8, 16×16, 32×32 pixels and so forth.”). Therefore, the applicants first argument is not convincing. With regard to applicants second argument, the applicant argues that the orthorectification as disclosed in Lautala is different from depth-encoded multi-perspective images of the claimed invention. The examiner agrees. Specifically, while Lautala may disclose a planar projection in a single direction, Lautala fails to disclose multiple such projections from multiple direction, and likewise, the references of record fail to specifically disclose wherein multi-view projections and depth-encoded images are created. Therefore, Lautala fails to specifically disclose wherein the generated depth-encoded image is multi-perspective, and the 102 rejections of claims 1, 4-7, 10-11, 16, and 19 are withdrawn. However, a new 103 rejection of claims 1, 4-7, 10-11, 16, and 19 are made in view of U.S. Publication No. 2022/0414362 to Lautala et al., and further in view “View-Dependent Dynamic Point Cloud Compression” to Zhu et al.
Claim Rejections - 35 USC § 103
5. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
6. Claims 1, 4-7, 10-11, 16, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Publication No. 2022/0414362 to Lautala et al. (hereinafter Lautala) and further in view of “View-Dependent Dynamic Point Cloud Compression” to Zhu et al. (hereinafter Zhu).
7. Regarding Claim 1, Lautala discloses a method for dynamically capturing and processing utility infrastructure LIDAR data for improved assessment accuracy and processing efficiency, the method comprising ([par. 0049, ln. 1-18] “The present disclosure provides a method for optimizing image data for generating orthorectified image(s) related to an area of interest in an environment… generally, existing raw aerial imagery captured by aerial vehicles (such as satellites, aeroplanes, helicopters, drones, etc.) contains distortions (or image distortions) induced due to several reasons including, but not limited to, sensor orientation, topographical variation, curvature of the earth and so forth… Generally, the orthorectified image is a correctly identified and/or located image, wherein the pixels of the image to be orthorectified using the method are rearranged (or relocated) to accurately place the displaced pixels to their correct position.”):
determining an orientation of a mobile LIDAR system ([par. 0051, ln. 1-8] “…the image data comprises at least one of the list of timestamps, locations, and orientations for each image taken by the aerial vehicle during a given time range for capturing the image data.”);
dynamically capturing, with the mobile LIDAR system, LIDAR data corresponding to a region of interest by collecting LIDAR samples based on the orientation and the region of interest ([par. 0053, ln. 1-33] “…the aerial vehicle is configured to perform three-dimensional movements for capturing the image data from different angles and orientations with respect to the area of interest to enable the system to track or monitor the environment... Typically, the control signal comprises data related to the trajectory or path to be followed by the aerial vehicle to capture the first image dataset related to the area of interest in the environment based on the method.”, [par. 0054, ln. 1-9] “…the image data comprises at least one of selected from: Light Detection and Ranging (LiDAR) data, hyperspectral imaging (HSI) data, Global Navigation System Satellite (GNSS) data, Inertial Measurement Unit (IMU) data… Optionally, the LiDAR unit… is mounted on an aerial vehicle that is employed for capturing a given LiDAR dataset of the environment.”);
identifying a utility object based at least in part on the LIDAR samples ([Fig. 4 and 5], [par 0067, ln. 12-25] “…Herein, the method comprises pre-identifying the non-interesting areas i.e., the locations and/or objects that do not match the need and/or out of the scope of implementation… in a powerline maintenance setup, only the area (for example, including a distance of up to 100 m from the powerlines) surrounding the powerlines comprise the area of interest and all the other objects and areas in the environment are considered to be non-interesting areas. This is done so as to segregate the area of interest from the non-interesting areas in the environment, and selectively process data pertaining to only the area of interest at a high resolution. This reduces the data storage space required, data storage costs incurred, data transfer time and processing time.”, [par. 0069, ln. 1-12] “The method comprises receiving attribute information related to each of the multiple identified objects. Upon identifying each of the multiple objects in the area of interest, the method comprises receiving attribute information related to each of the multiple identified objects. The term “attribute information” refers to the qualitative and/or quantitative characteristic information that may be recorded and/or analysed associated with each of the multiple identified objects. Examples of attribute information include, but is not limited to, location of object, dimensions of object, colour of object, texture of object, type of object, growth rate of object, and so forth.”);
decimating the LIDAR samples within a predetermined area around the utility object ([par 0067, ln. 12-25], [par. 0068, ln. 1-20] “Further… upon pre-identifying the one or more non-interesting areas, the aerial vehicle is configured to capture a relatively lower resolution image for the non-interesting areas while capturing the plurality of images of the area of interest, to reduce the size of the first image dataset... the capturing of low-resolution images for non-interesting areas is applied to a digital elevation model (DEM) of the environment to reduce the DEM complexity in the non-interesting areas for the given capture and thus to reduce the overall size of the first image dataset. The reduction in overall size of the first image data results in reduced memory storage, memory storage costs, associated computation time and power consumption.”, [par. 0073, ln. 1-34] “The method comprises processing the first image dataset to either discard or down-sample areas other than the maximum relevant second area captured therein, to obtain a second image dataset. In an example, the method employs the GNSS data, IMU data, and LiDAR data to determine areas required to be down-sampled or discarded such as areas wherein the image data is captured while the aerial vehicle flies along a tight curve, having high speeds or lying outside the area of interest. Typically, the image data captured under extreme conditions such as tight curves, high speeds, high vibrations (vibrations typically seems to happen when the aerial vehicle takes a hard curve and/or accelerates), high roll/pitch angle (camera pointing to a different location) are considered to be non-interesting or having a bad quality, and thus may be discarded or down-sampled to reduce the size of the image data. Typically, in an implementation scenario, the non-interesting areas i.e., the areas in the environment other than the area encompassed by the maximum relevant second area are either discarded from the first dataset or down-sampled to obtain the second image dataset. For example, the areas in the environment apart from the maximum relevant second area are captured at a lower resolution relative to the captured images in the second image dataset. Beneficially, by processing the first image dataset to either discard or down-sample the non-interesting areas other than the maximum relevant second area captured therein, the overall size of the image dataset to be processed is reduced drastically. Herein, the reduction in the overall size of the dataset results in reduced memory storage costs, associated computation time and power consumption. As a result of the reduced computation time, the reduced memory and power consumption, the present method provides faster and efficient generation of orthorectified image(s).”);
generating depth-encoded {multi-perspective} images of the utility object using the decimated LIDAR samples ([par. 0054, ln. 14-21] “The plurality of data points represent objects (such as, buildings, vegetation, and the like) on and above a ground surface in a three-dimensional space of the environment. Optionally, location of a given data point is expressed as (x, y, z) coordinates along X, Y, and Z axes, respectively of a given coordinate system employed for the environment. Optionally, the plurality of data points is collectively referred to as point clouds.”, [par. 0053, ln. 1-33] see “…capturing the image data from different angles and orientations with respect to the area of interest …”); and
determining at least one condition of the utility object ([par. 0071, ln. 4-21] “In an exemplary scenario, an object such as a tree may be likely to fall onto one or more components of a power distribution infrastructure (such as, the poles and/or the hanging powerlines in the power distribution infrastructure) in proximity of said tree. If any tree falls onto the power distribution infrastructure, it could lead to disruption in delivering electric power and/or could cause fire due to a circuit break. In this regard, it is of critical importance that such risky trees (or objects) are identified and timely removed or trimmed in order to prevent damage and failure of the power distribution infrastructure, for operation of the power distribution infrastructure to be maintained reliably… digitally identifying the risky objects enables efficient management whilst ensuring that operation of the power distribution infrastructure is maintained. This facilitates reduction in cost of vegetation management, better vegetation management planning, and the like.”).
However, Zhu teaches wherein dynamically captured point cloud can encoded by generating depth-encoded multi-perspective images of an object ([pg. 766, Fig. 1], [pg. 766, col. 1, B. Our Approach, par. 1, ln. 1 to col. 2, par. 1, ln. 19] “Therefore, in this paper, we propose the view-dependent DPC compression for networked applications, as sketched in Fig. 1, a.k.a, View-PCC. In general, our work belongs to the category that applies the 3D-to-2D dimensional reduction and HEVC-based 2D video coding. Our focus is to devise high efficiency 3D-to-2D dimensional reduction via hybrid global and local projections to support view-dependent extraction (via partial stream decoding) and adaptation-based streaming. More specifically, we first perform the global orthographic projection to map the original 3D object onto four faces (e.g., front, back, left, right) of a cube, which is referred to as the multi-view projection for mimicking natural observation viewpoints from four different perpendicular orientations (see Fig. 2a). Different from the normal clustering based patch projection used in MPEG video-based point cloud compression (V-PCC) [9] where we could not directly derive individual view by decoding partial substreams because points or voxels belong to the same view may be dispersed among different projection planes due to their diverse normal attributes, our global projection method enables network friendly view-dependent streaming, by directly applying the perspective projection of 3D voxels to a specific image plane. To further minimize the projection loss, i.e., enforcing points captured by the image planes as many as possible, we have applied the multi-layer projection, where an extra image layer (e.g., geometry, and texture plane) of each view is generated by replacing the corresponding pixels at existing layer with their nearest and available neighbors along with the projection direction. Similar nearest neighbor replacement (NNR) strategy is also utilized across views. These would generally guarantee the completeness of each individual view when partially decoding associated streams.”, [pg. 769, Fig. 2, see depth encoded images in (b), mutli-view (a)], [pg. 769, col. 1, III. Multi-View Mutli-Layer Global Projection, par. 1, ln. 1 to col. 2, par. 2, ln. 11] “We have applied the parallel orthographic projection, where the projection lines are perpendicular to the image planes of a cube [45]. Here, we only use the front, back, right and left faces of a cube, to mimic the natural viewing behavior in reality, as shown in Fig. 2a. These cube faces correspond to multiple views or multi-view in practice that can be encoded independently and in parallel. Each view or viewpoint of a 3D point cloud can be well reconstructed by decoding associated partial streams of corresponding projected image plane. Dynamic navigation in 3D space, such as view switch, can be also easily supported by adapting the streams of corresponding image planes. Such adaptation can be implemented using chunk or segment-based transport protocols, such as the Dynamic Adaptive Streaming over HTTP (DASH) [46], MPEG Media Transport (MMT) [47], etc, where image plane sequences are compressed and encapsulated into time-aligned chunks or segments at a variety of quality (or bitrate) scales [48], [49] for efficient network streaming. Each view-dependent image plane captures the corresponding geometry and texture information projected from a 3D point cloud in Fig. 2a. Note that geometry and texture images are compressed separately. Plane oriented projection is fulfilled in a consecutive order of front, back, left and right. For a specific projection orientation, points in 3D point cloud that are the closest to plane, are captured; In practice, points may be already projected to previous planes. In order to maintain the complete and natural surface appearance, nearest neighbors will be chosen as the replacement, if they are available; otherwise, same points are retained for projection…”, [pg. 770, col. 2, A. Patch Re-Processing, par. 2, ln. 1 to par. 5, ln. 5] “2) Projection Plane Prioritization: In addition to the same four planes used for global projection, we introduce the bottom plane as the fifth one to host the local patches only.2 Total five projection planes Pk are applied with k from 0 to 4. Given the fact that total number of local points is far less than those global-projected ones, we have enforced the local projection to a single plane out of those five options. Since the symmetric characteristics of paired left-right and front back planes where left (or front) plane would have the same number of projected points as the right (or back) one, five planes are categorized into three classes, i.e., bottom, left-right and front-back. For any
V
j
, it leads to the prioritized plane list
P
→
j
, of which
P
→
j
(
0
)
is the optimal plane having the least loss, i.e., with the most projected points, or the least averaged Euclidean distance when first two planes having the same and the largest number of projected points. In practice, the optimal plane
P
→
j
(
0
)
may not be the one used for the final compression when taking the projection occlusion, temporal coherency into consideration. Thus,
P
→
j
will be used when patch re-arrangement is required. Note that the same procedure is iteratively processed for all ordered patches. After being projected to a certain plane
P
→
j
(
k
)
, a 3D patch
V
j
with voxel locations (
x
→
j
,
y
→
j
,
z
→
j
) having positive attributes, can be represented by
C
j
,
P
→
j
(
k
)
having 2D pixels at effective locations (
u
→
j
,
v
→
j
,
d
→
j
(
u
→
j
,
v
→
j
)
) with valid geometry and texture components.”, [pg. 774, Fig. 6, see depth-encoded images top]). One of ordinary skill in the art, before the effective filling date of the claimed invention, would specifically recognize Lautala and Zhu as within the same field of image processing for dynamically captured 3D point clouds, and as analogous to the claimed invention. The motivation to combine is disclosed in Zhu, wherein it allows for efficient compression and streaming without loss of quality ([pg. 766, col. 1, par. 2, ln. 3-22] “we attack the DPC compression from an alternative angle. First, we assume the networked point cloud application perspective, where not only the signal redundancy in point cloud data, but also the network transmission priority, can be used for optimizing the rate distortion performance [22]. Second, such network transmission priority is motivated by the biological characteristics of human visual system (HVS) that our eyes can only perceive the media content within current viewport (or field of view- FoV) in the front, which is similar to view-dependent saliency especially for the visible part of the object [23]. Thus, in principle, we only need to stream the desirable view of the DPC per user’s request, by which the point cloud content consumption can be fulfilled by streaming view-dependent content appropriately. Thus, UnEqual Quality (UEQ)-based compression can be performed across views. For example, we could put high quality (e.g., high-bitrate) view within current viewport or FoV, but reduced or low-quality (e.g., low-bitrate) views elsewhere, for bandwidth efficient network streaming without quality of experience (QoE) loss…”, [pg. 767, col. 1, C. Contributions, par. 1, ln. 1-23] “In a short summary, the novelty parts of this work can be highlighted as follows. • View-dependent streaming methodology well fits the viewing characteristics of the HVS, and can be leveraged for significant network resource reduction but without QoE loss (e.g., applying UEQ scales across views). While this is often overlooked in typical DPC compression methods, such as the V-PCC [9]. • Our hybrid global and local projection mechanism applied in this work not only offers the high-efficiency DPC compression, but also enables view-dependent streaming and adaptation over the bandwidth-constrained network. • We offer the NNR based multi-view, multi-layer strategy for global projection, and patch-based local projection (e.g., intra displaced arrangement/inter temporal alignment), to maximize the number of points captured by the image planes for total reconstruction loss reduction, and improve the spatial and temporal coherency for better compression efficiency. • Though the HEVC is used to compress projected images of our View-PCC and V-PCC, our projection approach is fundamentally distant from the V-PCC, leading to very different processing schemes as detailed in Table I.”). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would find this particularly relevant to Lautala, wherein the dataset is transmitted in flight ([par. 0058, ln. 9-19] “Typically, the first image dataset is obtained by imaging via at least one imaging device of the aerial vehicle (such as, airborne or satellite sensors and/or drones comprising the at least one imaging device) on a target area i.e., the area of interest. Additionally, the first image dataset may be received directly from a local or remote system and/or database comprising the first image dataset or the image data. In an example, the data source is a drone or unmanned aerial vehicle (UAV) configured with a LiDAR and/or HSI device to capture and transmit the first image dataset.”), and thus would benefit from efficient compression, reduced loss of quality, and reduced network usage of Zhu by transmitting the LIDAR point cloud in an analogous fashion. One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu through known means, with no change to their respective function, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu to obtain the invention as specified in claim 1.
8. Regarding Claim 4, a combination of Lautala and Zhu teaches the method of claim 1. Lautala further discloses determining an initial position of the utility object based at least in part on one or more of the LIDAR and GIS data ([par. 0051, ln. 1-8] “…the image data comprises at least one of the list of timestamps, locations, and orientations for each image taken by the aerial vehicle during a given time range for capturing the image data. Notably, the image data is orthorectified using the method and mapped to actual mapped locations in the form of a data representation, such as raster file that comprises of each hyperspectral band orthorectified to specified coordinate system.”, [par. 0054, ln. 14-21], [par. 0081, ln. 1-7] “…the aerial vehicle transmits the image data to the data processing arrangement, wherein the image data comprises at least of: the timestamps with location and/or position for each image, sensor arrangement (or model) data, locations of the nearby multiple objects from a digital twin modeling, Digital Elevation Model (DEM) data.”). Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu to obtain the invention as specified in claim 4.
9. Regarding Claim 5, a combination of Lautala and Zhu teaches the method of claim 1. Lautala further discloses compensating the initial position based on one or more of a speed and an orientation of the LIDAR system to correct position errors of the utility object ([par. 0089, ln. 1-36] “… data processing arrangement is further configured to generate the first image dataset… inedited at least two subsets… of the image dataset having overlapped area of the area of interest captured in each of the at least two subsets… is further configured for identifying the at least two subsets of the image dataset having the overlapped area of interest and selecting one of the at least two subsets for capturing the said overlapped area of the area of interest based on a selection criteria, wherein the selection criteria comprises at least one of selected from: speeds of the aerial vehicle while capturing the at least two subsets, orientations of the aerial vehicle while capturing the at least two subsets, lighting conditions while capturing the at least two subsets. In some cases, such as while capturing image data along the area of interest with different orientations and angles causing overlapping of the captured images in the image dataset, the aerial vehicle tends to capture at least two image datasets of the same object, location and/or overlapped area. And, to eliminate the similar images data from the image dataset, the data processing arrangement is further configured for selecting one of the at least two subsets of the capture image dataset based on the selection criteria. Typically, the at least two subsets are compared to select one of the at least two subsets having better attributes and/or conditions while capturing the image dataset, wherein the comparison is based on at least one of the speed of the aerial vehicle, orientations of the aerial vehicle, lighting conditions, weather conditions and so forth. Thus, beneficially, the data processing arrangement processes only one of the at least two subsets i.e., only the areas that have the best capture conditions, to reduce the associated data storage costs and computation time and make the system faster and efficient”). Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu to obtain the invention as specified in claim 5.
10. Regarding Claim 6, a combination of Lautala and Zhu teaches the method of claim 1. Lautala discloses the method further comprises capturing, with a camera ([par. 0055, ln. 1-26] “Herein, the term “hyperspectral data” refers to a spatially sampled dataset comprising a plurality of pixels related to hyperspectral images captured or collected by hyperspectral imaging devices across the electromagnetic spectrum… The term “hyperspectral imaging or HSI” refers to a type of spectral imaging for inferring spectral characteristics of an image, wherein the spectra is divided to N different wavelengths. Notably, the sampling of the hyperspectral data may or may not be spatially regular based on the implementation. However, the irregular spatial sampling of the HSI data used herein may be orthorectified, i.e., normalized in a given plane, such as the x-y plane.”, [par. 0076, ln. 25-30] “…each slice of the second image data comprises at least the HSI camera model parameters, HSI camera calibration parameters, HSI capture timing data, HSI pixels, GPS location and alignment data, and the DEM that covers the area of interest (or the maximum relevant second area).”), one or more images of the region of interest ([par. 0054, ln. 1-9] see “…image data comprises… hyperspectral imaging (HSI) data”); determining one or more positions and directions of the camera corresponding to the capturing of the one or more images ([par. 0076, ln. 25-30], [par. 0079, ln. 1-9] “For example, in cases of diagonal flights, the diagonal flight covers the maximum area and hence it makes the DEM covering it to grow rapidly. Notably, the two or more second image dataset slices are stored as plans in memory. For example, each dataset slice may comprise X lines of Camera trajectory, (x0, y0, x1, y1) bounding box of the DEM, and camera model as simple text data.”, [par. 0081, ln. 1-7]); and determining, based on the one or more positions and directions of the camera and the one or more captured images, a location of the utility object ([par 0067, ln. 12-25], [par. 0069, ln. 1-12], [par. 0076, 25-30], [par. 0077, ln. 1-8] “Notably, the exact data format depends on the orthorectification algorithm to be applied, wherein the data-splitting or division algorithm takes into account at least the trajectory overlapping and object location(s). Typically, the overlapping data allows the method to select the highest quality of available pixels, and the asset locations allow the method to discard the uninteresting parts during capturing of the image data.”, [par. 0081, ln. 1-7]). Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu to obtain the invention as specified in claim 6.
11. Regarding Claim 7, a combination of Lautala and Zhu teaches the method of claim 6. Lautalla further discloses wherein the one or more positions and directions of the camera is determined by using a second GPS receiver and a second accelerometer ([par. 0057, ln. 1-16] “…“GNSS data” refers to Global Navigation Satellite System (GNSS) i.e., a constellation of satellites providing positioning and timing data to GNSS receivers. The term “IMU data” refers to Inertial Measurement Unit (IMU) data i.e., a collection of measurement tools including a plurality of parameters for tracking or locating an object (stationary or moving) in the environment. The IMU unit comprises a plurality of sensors for tracking objects, including accelerometers, gyroscopes, magnetometers and the like to provide the IMU data. For example, the parameters include, but is not limited to, location, height, roll, pitch, yaw, and timing. Optionally, other type of sensors, for measuring/monitoring other parameters may be used for e.g., sensors to monitor exposure levels, sun brightness, weather data, humidity, wind speed, etc.”, [par. 0073, ln. 1-34] see “…the method employs the GNSS data, IMU data, and LiDAR data to determine areas required to be down-sampled…”, [par. 0076, 25-30]). Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu to obtain the invention as specified in claim 7.
12. Regarding Claim 10, a combination of Lautala and Zhu teaches the method of claim 1. Lautala further discloses wherein the mobile LIDAR system comprises one or more of a backpack, a drone, an aircraft, and a terrestrial based LIDAR-mounted vehicle ([par. 0052, ln. 2-11] “… an unmanned or manned aerial vehicle operatively coupled to the data processing arrangement for capturing image data for generating orthorectified image(s). The aerial vehicle may be a drone (such as small drone, rotor drone, fixed wing drone, a quadcopter, a reconnaissance drone and the like). Optionally, the aerial vehicle may be an unmanned aerial vehicle (UAV) such as a drone fixed wing aircraft, a rotary wing aircraft and the like or a manned aerial vehicle implemented as a helicopter, a quatrocopter, an octocopter, and the like.”). Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu to obtain the invention as specified in claim 10.
13. Regarding Claim 11, a combination of Lautala and Zhu teaches the method of claim 1. Lautala further discloses wherein the region of interest is a selectable or predetermined region ([par. 0062, ln. 13-18] “…while using hyperspectral image (HSI) data, the spectrum for each pixel in the hyperspectral image data is identified and compared with existing data or with the help of a human expert to accurately identify find, distinguish, and identify objects, materials and/or detecting processes in the area of interest.”, [par 0067, ln. 12-25]). Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu to obtain the invention as specified in claim 11.
14. Regarding Claim 16, a combination of Lautala and Zhu teaches the method of claim 1. Lautala further discloses applying pre-decimation to areas surrounding the region of interest to facilitate equipment identification ([par. 0073, ln. 1-34]). Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu to obtain the invention as specified in claim 16.
15. Regarding Claim 19, the claim language is analogous to claim 1 with the exception of “A non-transitory computer-readable storage medium storing instructions that are configured to cause one or more processors to perform a method…”, Lautala discloses a non-transitory computer-readable storage medium storing instructions configured to cause one or more processors to perform the method ([par. 0053, ln. 21-23] “The controller includes a combination of hardware and software components... the controller may include a processor, memory and input/output peripherals and software to be executable by the processor and stored in the memory.”, [par. 0079, ln. 7-16] “…the dataset slice may be stored or written as file data on the memory (or hard drive)… The method requires only pointers of data in the memory, whereas any external party may require the data to be stored in the memory (or RAM)...”, [par. 0068, ln. 1-20] “…in some cases, the ground may be considered as a non-interesting area, or may be objects shorter than a predefined height (for example, 20 m) are considered non-interesting, or may be only a 100 m buffer around the objects is considered interesting, etc. depending on the implementation.”). Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the non-transitory computer-readable storage medium of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu to obtain the invention as specified in claim 19.
16. Claims 2-3, 12, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Publication No. 2022/0414362 to Lautala in view of “View-Dependent Dynamic Point Cloud Compression” to Zhu, and further in view of “Accuracy–Power Controllable LiDAR Sensor System with 3D Object Recognition for Autonomous Vehicle” to Lee et al. (hereinafter Lee).
17. Regarding Claim 2, Lautala discloses further determine a speed of the mobile LIDAR system and collecting the LIDAR samples at a {rate} based on the speed ([par. 0065, ln. 9-17] “…the generation of the first image dataset comprises selecting one of the at least two subsets for capturing the said overlapped area of the area of interest based on a selection criteria, wherein the selection criteria comprises at least one of selected from: speeds of the aerial vehicle while capturing the at least two subsets…”, [par. 0073, ln. 1-34] see “…image data captured under extreme conditions such as… high speeds… may be discarded or down-samples…”). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize Lautala discloses selecting and/or collecting LIDAR samples based on the speed of the vehicle, since “collecting” given its broadest reasonable interpretation would encompass the collection of only the LIDAR samples for which speed is not high as disclosed in Lautala. However, Lautala does not specifically disclose wherein the “rate” of the LIDAR is based on the speed, specifically, the collecting of Lautala fails to disclose changing LIDAR acquisition parameters based on the speed, and therefore a “rate” of the LIDAR is not disclosed in Lutalta. Likewise, Zhu does not specifically disclose determining a speed of the mobile LIDAR system and collecting the LIDAR samples at a rate based on the speed.
However, Lee teaches collecting the LIDAR samples at a rate based on the speed of the mobile LIDAR system ([pg. 1, Abstract, par. 1, ln. 7-10] “we propose algorithms to improve the inefficient power consumption of conventional LiDAR sensors, and efficiently reduce power consumption in two ways: (a) controlling the HAR to vary the laser transmission period (TP) of a laser diode (LD) depending on the vehicle’s speed and (b) reducing the static power consumption using a sleep mode, depending on the surrounding environment.”, [pg. 9, Table 3], [pg. 9, 3.2.1 Speed Detection-Based LiDAR Sensor Control, par. 1, ln. 1-16] “LiDAR sensor with the same HAR,
T
p
, detects the object more accurately at a short distance than at a long distance because of the angle at which the laser is transmitted. Therefore, when considering the distance to the object, if the laser’s HAR is low, the LiDAR sensor can be used for short-range object detection, and the higher the laser’s HAR, the more suitable it will be for detecting long-range objects. We designed the LiDAR sensor so that the LD’s
T
p
depends on the vehicle’s speed, as depicted in Algorithm 1. When the vehicle’s speed is faster than 100 km/h, the LiDAR sensor uses the maximum accuracy to detect distant objects at high speed. However, when the vehicle’s speed is slower than 100 km/h,
N
x
is reduced for low-power operation. As shown in Table 3, when the vehicle’s speed is faster than 100 km/h, the LiDAR sensor transmits the laser 580 times to the maximum number of laser transmissions (
N
x
.
m
a
x
) with 0.25° HAR. When the vehicle is driving at medium-high speed (80–100 km/h), the LiDAR sensor transmits the laser 483 times at 0.3° HAR, so our method reduces
N
x
by 16.73%.
N
x
is decreased by 28.62% with 414 laser transmissions and 0.35° HAR, when the vehicle’s speed is medium (60–80 km/h). For medium-low speeds (40–60 km/h), the LD transmits the laser 362 times, which reduce
N
x
by 37.56%, and the HAR is 0.4°. If the vehicle’s speed is slower than 40 km/h, then the minimum number of laser transmissions (
N
x
.
m
i
n
), 322, can decrease
N
x
by 44.48% with a 0.45° HAR.”). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize Lautala, Lee and Zhu as within the same field of image processing for dynamically captured 3D point clouds, and Lautala and Lee as further within the same field of mobile LiDAR imaging methods, and as analogous to the claimed invention. The motivation to combine is disclosed in Lee, wherein it reduces power consumption efficiency ([pg. 1, Abstract, par. 1, ln. 7-10]). One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu, and further combined the method of the combination of Lautala and Zhu with the collecting of LiDAR samples at a rate based on speed of Lee through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu and the collecting of LiDAR samples at a rate based on speed of Lee to obtain the invention as specified in claim 2.
18. Regarding Claim 3, a combination of Lautala, Zhu, and Lee teaches the method of claim 2. Lautala teaches wherein the speed of the mobile LIDAR is determined by using a first GPS receiver, and wherein the orientation of the mobile LIDAR system is determined by using a first accelerometer ([par. 0057, ln. 1-16], [par. 0073, ln. 1-34]). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize that the GNSS and/or IMU system is analogous to a first GPS received that can determine a speed of the vehicle and an accelerometer for determining the orientation of the vehicle, and that said data is specifically used for at least the acquisition of the LiDAR data as disclosed in Lautala. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu and the collecting of LiDAR samples at a rate based on speed of Lee to obtain the invention as specified in claim 2.
19. Regarding Claim 12, Lautala discloses the method of claim 1. Lautala further discloses wherein dynamically capturing the LIDAR data comprises {adjusting a data collection rate to control a point cloud density} and dynamically decimating the LIDAR data based on the region of interest ([par 0067, ln. 12-25], [par. 0068, ln. 1-20], [par. 0073, ln. 1-34]). Analogous to claim 2, Lautala does not specifically disclose adjusting a data collection rate to control a point cloud density. Likewise, while Zhu does not specifically teach adjusting a data collection rate to control a point cloud density.
However, Lee teaches adjusting a data collection rate to control a point cloud density ([pg. 1, Abstract, par. 1, ln. 7-10], [pg. 9, Table 3], [pg. 9, 3.2.1 Speed Detection-Based LiDAR Sensor Control, par. 1, ln. 1-16]). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize that controlling the transmission period of the LiDAR system is directly analogous to controlling the horizontal angular resolution ([pg. 3, 2. Related Works, par. 4, ln. 1-2] “The LiDAR sensor controls and changes the laser transmission period (𝑇𝑃), which is the HAR, according to the vehicle’s speed and surrounding environment.”), and because the resolution of a LiDAR image is equivalent to the number of points acquired by the LiDAR system, you are effectively controlling the horizonal point cloud density ([pg. 13, Fig. 15], see (a) 580 point-(f) 290 point (sleep) which equates the number of points directly to the number of laser transmissions). The motivation to combine would have been obvious to one of ordinary skill in the art, and is discloses in Lee, wherein the accuracy (i.e., density of the point cloud) is directly proportional to the collection rate and power consumption rate ([pg. 7, Fig. 7], [pg. 6, par. 7, ln. 1 to pg. 7, par. 1, ln. 4] “
T
p
refers to the time interval for transmitting the laser at the HAR intervals. As shown in Figure 7, if the HAR is increased by narrowing
T
p
, the LiDAR sensor’s power consumption increases as the accuracy of the object detection increases. In addition, if the HAR is decreased by widening
T
p
, the power consumption decreases as the accuracy decreases. Therefore, because
T
p
of the LD, which determines the HAR, and the accuracy of the object detection are proportional, power consumption increases as
T
p
narrows for the same period of time.”), and therefore, arguments analogous to claim 2 are further applicable to claim 12. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu and the collecting of LiDAR samples at a rate based on speed of Lee to obtain the invention as specified in claim 12.
20. Regarding Claim 20, a combination of Lautala and Zhu teaches the non-transitory computer readable storage medium of claim 19. Lautala further teaches determine a speed of the mobile LIDAR system ([par. 0065, ln. 9-17] [par. 0073, ln. 1-34]); collect the LIDAR samples at a {rate} based on the speed ([par. 0065, ln. 9-17] [par. 0073, ln. 1-34]); {adjust a data collection rate to control a point cloud density}; and dynamically decimate the LIDAR data based on the region of interest ([par 0067, ln. 12-25], [par. 0068, ln. 1-20], [par. 0073, ln. 1-34]). Analogous to claims 2 and 12, Lautala and Zhu does not specifically disclose adjusting a LiDAR collection rate or point cloud density.
However, Lee teaches adjusting a data collection rate to control a point cloud density, and wherein the rate is adjusted based on the speed ([pg. 1, Abstract, par. 1, ln. 7-10], [pg. 9, Table 3], [pg. 9, 3.2.1 Speed Detection-Based LiDAR Sensor Control, par. 1, ln. 1-16]). Rejections analogous to a combination of 2 and 12 are further applicable to claim 20 in view of the non-transitory computer readable storage medium of the combination of Lautala and Zhu. The motivation to combine remains analogous to claims 2 and 12. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the non-transitory computer readable storage medium of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu and the collecting of LiDAR samples at a rate based on speed of Lee to obtain the invention as specified in claim 20.
21. Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over U.S. Publication No. 2022/0414362 to Lautala in view of “View-Dependent Dynamic Point Cloud Compression” to Zhu, and further in view of U.S. Publication No. 2023/0127756 to Desai et al. (hereinafter Desai).
22. Regarding Claim 8, a combination of Lautala and Zhu teaches the method of claim 6. Lautala does not specifically disclose determining the location of the utility object is further based on triangulation using a determined position of the camera. Likewise, Zhu does not specifically disclose determining the location of the utility object is further based on triangulation using a determined position of the camera
However, Desai teaches determining the location of an object is further based on triangulation using a determined position of the camera ([Fig. 9A-C] see electric utility pole, [par. 0044, ln. 1-17] “FIG. 4A illustrates an example SfM map with a correctly mapped key point, according to some embodiments. In trigonometry and geometry, triangulation is the process of determining the location of a point by forming triangles to the point from known points. For purposes of identifying various objects of interest in a visualization mapping (SfM or lidar), many points located within the map may correspond to key points. Example key points of an object located in a viewing space (e.g., of a vehicle) may include, but not be limited to, object corners, edges, surfaces, etc. For illustration purposes only, FIGS. 4A and 4B will be reduced to triangulating a location of a single point. However, the process may be repeated for any number of points in the SfM map. Corresponding depth mapping of these points may assist… an autonomous driving system 100 to understand the environment (e.g., proximate objects) in its immediate path and therefore avoid collisions.”, [par. 0046, ln. 1-13] “A spatial triangle 405 may be formed by two cameras 406 and 408 and a point located within their field of view. Within this spatial triangle 405, the distance between the cameras is the base b1 of the triangle and may be known… cameras 406 and 408 may each retain a geospatial location (e.g., using global positioning satellite (GPS)). By determining a distance between center points on their respective image planes (image plane 1 and image plane 2), the base (b1) of the triangle may be determined. The image plane in a camera is a surface the light is focused onto after passing through a photographic lens… in digital cameras the image plane is the surface of the digital image sensor.”). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize Lautala, Zhu, and Desai as within the same field of image processing for dynamically captured 3D point clouds, and Lautala and Desai as further within the same field of joint LiDAR-camera systems for mobile imaging systems, and as analogous to the claimed invention. Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize triangulation as a standard and widely used technique for determining positions of objects and/or cameras using images. The motivation to combine the method of the combination of Lautala and Zhu with Desai would have been obvious to one of ordinary skill in the art, and is disclosed in Desai, in that the triangulated positions could be used to further improve accuracy by comparison and/or combination with depth/LiDAR data, and wherein the triangulated positions offer an alternative to LiDAR when LiDAR may not be applicable and/or accurate ([Fig. 6] see 608-612, [par. 0069, ln. 1-15] “At step 612, the image processing system compares a first depth of a key point (triangulated point) within the SfM depth map with a common key point in the splatted lidar map to validate these key points. A depth comparison of a depth of a key point in the SfM map and the same key point within the localization prior determines differences in depths between the two modalities for creating a point could map. Incorrectly triangulated points (not the same depth) get marked as outliers if their depths do not lie within a set threshold (e.g., X cm) of the depth obtained from the depth map image (e.g., some minor errors may be allowed for points on a common surface). In a non-limiting example, X may be 5 (cm)…”, [par. 0074-0077, ln. 1-15] “The technology described herein has many benefits… provides a computer solution to a problem (inaccurate depth calculations)… Another benefit is removal of outlier key points within an SfM map and establishing a degree of confidence for points not recognized as outliers... Another benefit is elimination of false detections based on incorrect triangulations or improperly calculated depth maps... Another benefit is reduction of collisions of autonomous vehicles with objects based on a more accurate representation of the object's key points and associated depths.”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu, and further combined the method of the combination of Lautala and Zhu with the triangulation of Desai through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu and the triangulation of Desai to obtain the invention as specified in claim 8.
23. Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over U.S. Publication No. 2022/0414362 to Lautala in view of “View-Dependent Dynamic Point Cloud Compression” to Zhu, and further in view of U.S. Publication No. 2021/0073692 to Saha et al. (hereinafter Saha).
24. Regarding Claim 9, Lautala disclosed the method of claim 1. Lautala does not specifically disclose outputting a work order based on the determining of the at least one condition of the utility object. Likewise, Zhu does not specifically disclose outputting a work order based on the determining of the at least one condition of the utility object.
However, Saha teaches outputting a work order based on the determining of the at least one condition of the utility object ([par. 0187, ln. 1-18] “The example alert module 1504 is configured to receive an indication of the detected condition from each of the detection modules described in reference to the detection module 1502 and, based on the detected condition, transmit an alert for the detected condition to one or more personnel associated with the utility infrastructure, emergency responders, or other individuals and/or computing devices associated with the same… the transmitted alert includes an indication that the detected condition is (or poses) a hazard (or potential hazard) to the utility infrastructure. The alert module 1504 may also be configured to store the detected conditions in a database, from which the condition information may be accessed for a variety of uses including… preemptive de-energization of power line in extreme weather… to avoid ignitions from downed power line or structure or equipment”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize Lautala, Zhu, and Saha as within the same field of image processing for dynamically captured 3D point clouds, and Lautala and Saha as further within the same field of LiDAR imaging for utility infrastructure inspection, and as analogous to the claimed invention. Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize that the notification as disclosed in Saha to perform an action (e.g., de-energization of power line) constitutes a “work order” given the broadest reasonable interpretation of “work order”. Furthermore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to include a “work order” based on the condition of the utility object, since Lautala and Saha are specifically described in the context of regular maintenance of utility infrastructure, and thus in the case where said infrastructure is in need of maintenance (e.g., trees growing too close to power lines, power line down, etc.) it would have been obvious to send a notification “work order” to perform maintenance on the infrastructure. With regards to motivation to combine, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize outputting a work order based on the determining the condition of a utility object would allow for preemptive maintenance and thus reduction of failures within the infrastructure object. One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu, and further combined the method of the combination of Lautala and Zhu with the work orders of Saha, through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu and the work orders of Saha to obtain the invention as specified in claim 9.
25. Claims 13 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Publication No. 2022/0414362 to Lautala in view of “View-Dependent Dynamic Point Cloud Compression” to Zhu, in view of “Accuracy–Power Controllable LiDAR Sensor System with 3D Object Recognition for Autonomous Vehicle” to Lee, and further in view of U.S. Publication No. 2024/0144694 to Alismail et al. (hereinafter Alismail).
26. Regarding Claim 13, a combination of Lautala and Lee teaches the method of claim 12. Lautala, Zhu, and Lee do not specifically disclose wherein dynamically decimating the LIDAR data further utilizes voxel decimation to limit a number of points per grid square.
However, Alismail teaches dynamically decimating the LIDAR data further utilizes voxel decimation to limit a number of points per grid square ([par. 0060, ln. 1-23] “An ICP technique may be generally employed to minimize the difference between two or more point clouds… the methods may also minimize of range-based uncertainty… one point cloud (e.g., point cloud of the reference sweep) may be kept fixed, while the point cloud corresponding to another sweep from the window (e.g., source point clouds), may be transformed to best match the reference point cloud. The ICP technique may iteratively revise the transformation (e.g., combination of translation and rotation) in order to minimize an error metric… the error metric may be… a distance from the source point cloud to the reference point cloud (e.g., the sum of squared differences between the coordinates of the matched pairs)… ICP may align scan data (e.g., point clouds) given an initial guess of the transformation which is subsequently iteratively refined. Specifically, given an initial estimate, ICP minimizes the Euclidean distance between pairs of matching points from both point clouds (present given the overlap between sweeps) in an iterative manner. For aligning more than two point clouds, the ICP may be performed in a pairwise manner with the same reference point cloud.”, [par. 0064, ln. 1-30] “…with a voxelization filter, instead of matching individual points, the system may implement ICP over a pyramid of point-clouds voxelized using multiple voxel sizes ranging from coarse to fine. Voxelization refers to the process of downsampling the input point cloud such that a single point is selected in a voxel (cube) of a given dimension. The point cloud maybe dynamically divided into multiple voxels of different sizes… each voxel may be associated with statistical data representing multiple data points, such as, but not limited to, a number of data points, an average position of the data points, a covariance of the data points, and the like. As such, data received from a sensor (i.e., a point cloud) may be used to populate one or more voxels. For example, the maximum and minimum values of the x, y and z axis of an input point cloud may be calculated, and a three-dimensional bounding box according to these values may be established. The bounding box may be divided into small cubes with the assigned voxel size, such that all points in the small cube are represented as a single point such as the center of gravity of the small cube and/or an average of all points. In this way, multiple points inside the voxel are represented by one point, and the point cloud is reduced. Additionally, in the first step of ICP, voxel (rather than point) correspondences are made. Optionally, voxel shape parameters or features (e.g., surface normal, density, and curvature) are also incorporated to ensure that local 3D structure around the points are considered for the determination of the data association between two point clouds. Surface normal estimation may be performed per voxel-level independently.”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize Lautala, Zhu, Lee, and Alismail as within the same field of image processing for dynamically captured 3D point clouds, and Lautala, Lee, and Alismail as within the same field of mobile point cloud imaging systems, and as analogous to the claimed invention. Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize the voxel down sampling of Alismail to an assigned voxel size is directly analogous to a voxel decimation operation to limit a number of points per gird square. The motivation to combine would have been obvious to one of ordinary skill in the art, and is disclosed in Alismail, wherein voxel decimation reduces processing complexity ([par. 0062, ln. 1-9] “Optionally, the system may reduce processing complexity of the ICP by including data filters such as, without limitation, bounding box filter, voxelization filter, octree grid filter, observation direction filter, surface normal filter, orient normal filter saliency filter, maximum density filter, random sampling filter, distance based filter, normal space sampling filter, shadow point filter, and/or the like. Such filters process an input point cloud into an intermediate point cloud used in the alignment procedure.”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu and the adjusting of the data collection rate to control a point cloud density of Lee, and further combined the method of the combination of Lautala, Zhu, and Lee with the voxel decimation of Alismail, through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Lautala the depth-encoded multi-perspective images from point cloud data of Zhu, the adjusting of the data collection rate to control a point cloud density of Lee, and the voxel decimation of Alismail to obtain the invention as specified in claim 13.
27. Regarding Claim 14, a combination of Lautala, Lee, and Alismail teaches the method of claim 13. Lautala, Zhu, and Lee do not specifically disclose utilizing dynamic octree spatial index in conjunction with a-priori knowledge of a geometry of structures to optimize point selection.
However, Alismail teaches utilizing dynamic octree spatial index in conjunction with a-priori knowledge of a geometry of structures to optimize point selection ([par. 0059, ln. 1-26] “For LiDAR sensors, the scan data resembles point clouds, and the alignment determines the relative transform between the scan data of the reference sweep and each of the other sweeps using, for example, an iterative closest point technique such as, without limitation, point-to-point ICP, point-to-plane ICP, or an alternate ICP technique. Because the reference sweep and each of the other sweeps in the window have at least a partial overlap (based on the selection of the window size as discussed above), the corresponding scans are associated with at least a section of the calibration environment that is common between the sweeps being aligned. As such, the alignment may detect common points and/or features (e.g., edges, planes such as surface normals, voxels, curvature, density, or the like) between the reference sweep point cloud and each of the other point clouds. Specifically, common points and/or features may be detected and aligned such that common points and/or features from the reference sweep and another sweep can be brought into coincidence by one rigid transformation… using the relative transform and the identified common points and/or features in each set of scan data, the reference sweep scan data may be aligned with scan data of each of the other sweeps in the window to aggregate all of the sets of scan data into an aggregate representation of the surrounding environment, for example, a point cloud representation.”, [par. 0060, ln. 1-23], [par. 0061, ln. 1-31] “…the system receives as input a reference point cloud and a source point cloud, an initial estimation of the transformation to align the source point cloud to the reference point cloud, and some criteria for stopping the iterations. The system may perform the ICP technique to generate a refined transformation, for example, the transformation to determine the pose of the vehicle (or the LiDAR) given the calibration environment and the known rotational position of the turntable. For example, for each point in the source point cloud, the system may identify a match point in the reference point cloud (or a selected set). The system may then estimate the combination of rotation and translation (e.g., a transformation function) which will best align each source point to its match found in the previous step. In some embodiments, the system may use a root mean square point to point distance metric minimization technique for estimating the combination of rotation and translation. The system may, optionally, weigh points (e.g., using a cost function) and reject outliers prior to alignment. The system may then transform the source points using the obtained transformation. The system may next repeat these actions (e.g., by re-associating the points, and so on) until a predetermined stopping criteria is met such as, without limitation, convergence (i.e., a transformation between the sweeps is found such that no improvement in the closest neighbors is possible), a maximum number of iterations is reached (e.g., 100, 150, 200, or the like), relative reduction in the estimated parameters within the inner iterations of ICP is under a threshold (e.g., about 1×10.sup.−6), relative reduction is the nearest neighbor distance drops below a threshold (e.g., about 1×10.sup.−3), or the like.”, [par. 0062, ln. 1-9] see “…octree grid filter…”). Specifically, the examiner notes that the octree grid filter of Alismail can likewise be understood to be a “octree spatial index” since octrees recursively decompose a 3D space (e.g. voxel) into a tree structure comprising eight regions, and are effectively indexed as tree nodes. With regard to “…in conjunction with a-priori knowledge of a geometry of structures…”, the examiner notes that ICP of Alismail performs comparison of a reference point clouds to different point clouds, and therefore the octree spatial index of Alismail as part of the ICP is performed in conjunction with a-priori knowledge to optimize point selection ([par. 0059, ln. 1-26], [par. 0060, ln. 1-23], [par. 0061, ln. 1-31]). The motivation to combine remains analogous to claim 13. One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu and the adjusting of the data collection rate to control a point cloud density of Lee, and further combined the method of the combination of Lautala, Zhu, and Lee with the voxel decimation and octree spatial index of Alismail through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu, the adjusting of the data collection rate to control a point cloud density of Lee, and the voxel decimation and octree spatial index of Alismail to obtain the invention as specified in claim 14.
28. Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over U.S. Publication No. 2022/0414362 to Lautala in view of “View-Dependent Dynamic Point Cloud Compression” to Zhu, and further in view of “
P
3
-LOAM: PPP/LiDAR Loosely Coupled SLAM With Accurate Covariance Estimation and Robust RAIM in Urban Canyon Environment” Li et al. (hereinafter Li).
29. Regarding Claim 15, Lautala discloses the method of claim 1. Lautala does not specifically disclose performing precise point positioning (PPP) around identified regions of interest, though one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize Lautala does utilize GNSS data analogous to the data used for PPP to perform decimation and identify object/regions of interest ([par. 0054, ln. 1-9], [par. 0062, ln. 8-13] “…the method comprises using the first image data comprising… the GNSS data and/or the IMU data to identify the multiple objects in the area of interest.”, [par. 0073, ln. 1-34]). Likewise, Zhu does not specifically disclose performing precise point positioning (PPP) around identified regions of interest.
However, Li teaches wherein a coupled LiDAR/PPP system ([pg. 6663, Fig. 1] see GNSS and PPP Algorithm, [pg. 6661, col. 1, par. 3, ln. 20-28] “…PPP technology is more suitable for unmanned systems as PPP is independent of base stations. Single-frequency PPP (SF-PPP) and Dual-frequency PPP (DF-PPP) are two mainstream PPP technologies. Both of them need precise corrections which can be acquired from International GNSS Service Real-Time Service (IGS-RTS) [25] for the satellite orbits and satellite clocks. In autonomous driving, SF-PPP receives considerable attention because of its lower cost.”, [pg. 6663, Fig. 2], [pg. 6662, col. 2, par. 3, ln. 1-10] “The coordinate frames and notations involved in this article will be clarified below. The Earth-centered Earth-fixed coordinate system (ECEF, Frame E , see Fig. 2) rotates with the Earth, taking the Earth’s centroid as the origin. The X-axis of Frame E points to the intersection of the equator and prime meridian. The Earth’s rotation axis is taken as Z-axis, and the North Pole is the positive direction. Then, the Y-axis is perpendicular to the X-Z plane, forming a right-handed coordinate system. The GNSS receiver generally outputs positioning results in Frame E.”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize Lautala, Zhu, and Li as within the same field of image processing for dynamically captured 3D point clouds, and Lautala and Li as further as within the same field of LiDAR GNSS integrated point cloud imaging systems, and as analogous to the claimed invention. Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize the GNSS system of the method of the combination of Lautala and Zhu as being directly analogous to the GNSS system of Li. The motivation to combine the PPP of Li with the GNSS system of the method of the combination of Lautala and Zhu is disclosed in Li, wherein it allows for a reliable conversion of LiDAR local reference positions to a global reference position ([pg. 6661, col. 2, par. 2, ln. 10-28] “For these reasons, the coupling of GNSS will enable the LiDAR-SLAM system to provide more robust localization and mapping results with global earth coordinates in large-scale scenarios. In turn, GNSS is heavily influenced by multipath and NLOS in urban canyon environment, resulting in poor performance. LiDAR-SLAM always performs well in urban canyon environment and gets a relative positioning result. So it is meaningful to fuse GNSS and LiDAR-SLAM: LiDAR-SLAM can improve the reliability and availability of GNSS, on the other hand, GNSS can turn the positioning result of LiDAR-SLAM in a local coordinate system to the global such as World Geodetic System 1984 (WGS-84). But in conventional GNSS and LiDAR-SLAM fusion systems, few works have paid attention to either the calculation of LiDAR-SLAM positioning covariance, or the use of PPP in autonomous driving. In conclusion, our motivation for this paper is complementing PPP and LiDAR-SLAM to obtain more reliable and available positioning results in global coordinate system.”). In combining the PPP of Li, it would have been obvious to one of ordinary skill in the art to perform PPP around identified regions of interest using the GNSS of the method of the combination of Lautala and Zhu, analogous to the region of interest determination already performed in the method of the combination of Lautala and Zhu using the GNSS system. One of ordinary skill in the art, before the effective filing date of the claimed invention, would have combined the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu, and further combined the method of the combination of Lautala and Zhu with the PPP of Li through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu and the PPP of Li to obtain the invention as specified in claim 15.
30. Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over U.S. Publication No. 2022/0414362 to Lautala in view of “View-Dependent Dynamic Point Cloud Compression” to Zhu, and further in view of “An Easy-to-Use Airborne LiDAR Data Filtering Method Based on Cloth Simulation” to Zhang et al. (hereinafter Zhang).
31. Regarding Claim 17, Lautala does not specifically disclose wherein the method comprises one or more of storing the processed LIDAR data in a .las format; converting 64-bit LIDAR data points near a grid center to 32-bit data; storing the processed LIDAR data as a binary large object (BLOB); removing ground from images using a cloth simulation filter (CSF) algorithm; scaling the images to a predefined resolution; utilizing grayscale or HSV color space with color mapping to represent a z-position in the images; and employing one or more Yolo models for classification and object detection. Specifically, Lautala only discloses removing the ground from images, and does not specifically disclose this is done using a CSF ([par. 0068, ln. 1-20]). Likewise, Zhu does not specifically disclose the limitations of claim 17.
However, Zhang discloses wherein the ground can be removed using a CSF ([pg. 3, par. 2, ln. 7-14] “To cope with these problems, this paper proposes a novel filtering algorithm which is capable of approximating the ground surface with a few parameters. Different from other algorithms, the proposed method filters the ground points by simulating a physical process that an virtual cloth drops down to an inverted (upside-down) point cloud. Compared to existing filtering algorithms, the proposed filtering method has some advantages: (1) few parameters are used in the proposed algorithm, and these parameters are easy to understand and set; (2) the proposed algorithm can be applied to various landscapes without determining elaborate filtering parameters; and (3) this method works on raw LiDAR data.”, [pg. 3, 2. Method, par. 1, ln. 1-11] “Our method is based on the simulation of a simple physical process. Imagine a piece of cloth is placed above a terrain, and then this cloth drops because of gravity. Assuming that the cloth is soft enough to stick to the surface, the final shape of the cloth is the DSM (digital surface model). However, if the terrain is firstly turned upside down and the cloth is defined with rigidness, then the final shape of the cloth is the DTM. To simulate this physical process, we employ a technique that is called cloth simulation [26]. Based on this technique, we developed our cloth simulation filtering (CSF) algorithm to extract ground points from LiDAR points. The overview of the proposed algorithm is illustrated in Figure 1. First, the original point cloud is turned upside down, and then a cloth drops to the inverted surface from above. By analyzing the interactions between the nodes of the cloth and the corresponding LiDAR points, the final shape of the cloth can be determined and used as a base to classify the original points into ground and non-ground parts.”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize Lautala, Zhu, and Zhang as within the same field of rendering models using LiDAR data, and as analogous to the claimed invention. The motivation to combine the CSF filter of Zhang with the method of the combination of Lautala and Zhu is disclosed in Zhang, wherein it offers a low complexity determination of the ground points ([pg. 20, 5. Conclusions, par. 1, ln. 3-11] “Compared to conventional filtering algorithms, the parameters are less numerous and are easy to set. Regardless of the complexity of ground objects, the samples were divided into three categories according to the shape of the terrain. Few parameters are needed, and these parameters hardly changed among the three sample categories; only an integer parameter rigidness and a Boolean parameter ST are required to be set by the user. These three groups of parameters exhibit relatively high accuracies for all fifteen samples of the ISPRS benchmark datasets. Another benefit of the CSF algorithm is that the simulated cloth can be directly treated as the final generated DTM for some circumstances, which avoids the interpolation of ground points, and can also recover areas of missing data”). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize that you could identify and decimate the ground, analogous to the process already disclosed in Lautala ([par. 0068, ln. 1-20]), using the ground points as identified by the CSF of Zhang. One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the method and ground removal of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu, and further combined the method of the combination of Lautala and Zhu with the CSF filter for ground point identification of Zhang through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results. The examiner specifically notes that the claim states “…one or more of…”, and therefore, only one of the select groups of limitations is required given the BRI of the claim, and as such, a combination of Lautala, Zhu, and Zhang would teach claim 17.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combined the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu and the CSF filter of Zhang to obtain the invention as specified in claim 17.
32. Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over U.S. Publication No. 2022/0414362 to Lautala “View-Dependent Dynamic Point Cloud Compression” to Zhu, and further in view of “3D YOLO: End-to-End 3D Object Detection Using Point Clouds” to Al Hakim (hereinafter Al Hakim).
33. Regarding Claim 18, Lautala disclose the method of claim 1. Lautala further discloses processing data from regions {in a 3-section grid using a Yolo} model to identify a presence of the utility object ([par. 0062, ln. 1-13] “…for effective monitoring of the powerline, the method comprises identifying the multiple objects in the area of interest from the first image dataset. Typically, the multiple objects are identified using a detection and/or identification algorithm that may or may not be assisted by a human expert or user. Herein, the detection and/or identification algorithm is a machine learning algorithm configured to identify the multiple objects based on the information provided from the capture image data or the first image dataset. Typically, the method comprises using the first image data comprising the LiDAR data, the HSI data, the GNSS data and/or the IMU data to identify the multiple objects in the area of interest”); applying object detection to determine a precise location of the utility object ([par. 0062, ln. 1-13]); and creating a full 3D model of each utility object for assessment ([par. 0071, ln. 4-21], [par. 0074, ln. 1-10] “In an example, such as in cases of a flat terrain or a known flat forest and/or trees, instead of processing each of the pixels in the captured image data (location data for that group of pixels is coming from a digital twin model), a group of pixels based on the attribute information are processed to enable the method to process the image data much faster and efficiently. For instance, consider a whole Digital elevation model (DEM) for an area of 100 m×100 m that is represented by 1000×1000 pixels in the first image dataset.”, [par. 0079, ln. 19-25] “Thus, by joining or combining the second image dataset slice, the method or system is enabled to manage the overlapping data on the fly. Optionally, the DEM based on the second image dataset is turned into a 3D model via at least one tinning algorithm.”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize the DEM of Lautala to comprise the utility objects within an area of interest, and therefore represent a full 3D model of each utility object. Lautala does not specifically disclose wherein the machine learning model for detecting the utility object is a YOLO based model or wherein the regions are in a 3d-section grid. Likewise, Zhu does not specifically disclose processing data from regions in a 3-section grid using a Yolo model.
However, Al Hakim teaches processing data from regions in a 3-section grid using a YOLO model to identify a presence of an object ([pg. 34, Fig. 4.1], [pg. 34, Chapter 4, 3D YOLO, par. 1, ln. 1-5] “This chapter presents extension of YOLO for end-to-end trainable 3D object detection network that takes point cloud as input and yields 3D bounding boxes with class scores without using any hand-crafted features. The proposed model will be denoted 3D YOLO.”, [pg. 35, 4.1 Architecture, par. 1, ln. 1-6] “Briefly, 3D YOLO consists of two networks, the Feature Learning Network (FLN) and a CNN based on You Only Look Once v2 (YOLOv2), as shown in Figure 4.1. The FLN is the same network as used in VoxelNet, which transforms the input point cloud to a new feature space. YOLO takes this new representation of the point cloud as input and outputs class scores and bounding box coordinates.”, [pg. 35, 4.1.1 Feature Learning Network, Network, par. 1, ln. 1-5] “The FLN takes a non-empty voxel
V
i
n
(
i
)
=
{
p
^
1
i
,
…
,
p
^
T
i
}
∈
R
T
×
7
as input and outputs a feature vector with fixed dimension C, denoted as
V
o
u
t
(
i
)
∈
R
C
. We also convert an empty voxel (containing no points) to the zero feature vector. The architecture of FLN is a chain of connected Voxel Feature Encoding (VFE) layers, as shown in Figure 3.15.”, [pg. 36, par. 5, ln. 1-4] “By feeding all non-empty voxels
V
i
n
(
1
)
,
…
,
V
i
n
(
R
)
to the FLN, a list of feature vectors will be obtained
V
o
u
t
(
1
)
,
…
,
V
o
u
t
(
R
)
. This list can be represented as a sparse 4D tensor of size (C × D’ × H’ × W’). After reshaping it to a 3D tensor of size (H’ × W’ × C · D’), it passes through the YOLO network.”, [pg. 37, Fig. 4.3], [pg. 37, 4.1.2 YOLO Network, par. 1, ln. 1-2 and par. 1-9] “We design a new CNN architecture base on YOLOv2 [39] to detect 3D objects in real-time, called 3DNet…The output of 3DNet is a tensor of size (H’/8 × W’/8 × B · (8 + K)), where B is the number of the anchors and K is the number of classes. Each cell in the feature map grid (H’/8 × W’/8) predicts B bounding boxes, confidence scores for each of them and K class scores p1, ..., pK (see Figure 4.3). Predicted bounding boxes are parameterized as refined anchors.”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would specifically recognize Lautala, Zhu and Al Hakim as within the same field of image processing for 3D point clouds, and Lautala and Al Hakim as further within the same field of object detecting using machine learning on 3D point cloud data, and as analogous to the claimed invention. The motivation to substituted the machine learning based utility object detection of the method of the combination of Lautala and Zhu with the 3-section grid-based YOLO object detection of Al Hakim is disclosed in Al Hakim, wherein 3D YOLO achieves significant accuracy while being significantly faster than comparably accurate models ([pg. 45, 5.2 Runtime, par. 1, ln. 1-5] “We compare runtime of 3D YOLO with MV3D [15] and VoxelNet [20], presented in Table 5.1. Since VoxelNet has been tested on a Nvidia Titan X GPU and source code is unavailable, we compared our runtime with an unofficial Tensorflow implementation [43] of VoxelNet on a Nvidia Tesla P100 GPU. The table shows our model is 1.64× faster than VoxelNet.”, [pg. 46, Table 5.1], [pg. 46, 5.3 KITTI Evaluation on Validation Set, par. 1, ln. 1-5] “We evaluate our model only on the Car class across all three difficulty levels (easy, moderate and hard) and compare it with several state-of-the-art 3D object detectors, including Mono3D [18] and 3DOP [19] which use image data only, VeloFCN [17] and VoxelNet [20] which use LiDAR data only and MV3D [15] which was both images and LiDAR data.”, [pg. 46, Table 5.2], [pg. 47, Table 5.3], [pg. 47, par. 1, ln. 1-4] “On the validation set (see Table 5.3), our model outperformed all LiDARbased methods except VoxelNet on both easy and moderate difficulty levels. We see in the tables and Figure 5.1 what is expected, the average position for bird-eye-view has higher value than the 3D world.”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the method and ground removal of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu, and further would have substituted the machine learning model for utility object location detection of the method of the combination Lautala and Zhu with the 3-section grid based YOLO object detection of Al Hakim, through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Lautala with the depth-encoded multi-perspective images from point cloud data of Zhu and the 3-section grid-based YOLO object detection of Al Hakim to obtain the invention as specified in claim 18.
Conclusion
34. The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure. See PTO-892.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAULO ANDRES GARCIA whose telephone number is (703)756-5493. The examiner can normally be reached Mon-Fri, 8-4:30PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached on (571)272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PAULO ANDRES GARCIA/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
Prosecution Insights