DETAILED ACTION
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on April 20th, 2026 has been entered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to independent claims 1 and 12 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4, 10, 12-14, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Obropta (US20180047177) in view of Pelissaro de Lima (“On the Stockpiles Volume Measurement Using a 2D Scanner”) and Robert (US20230007962).
In regards to claim 1, Obropta teaches a system for determining the volume of a Obropta Paragraph [0034] “In some embodiments, the device may include: (1) an imaging sensor (e.g., a color camera, a monochrome camera, a multi-spectral camera, etc.) configured to capture one or more image(s) of a set of harvested specialty crops; (2) a depth sensor (e.g., an ultrasonic sensor, a LIDAR (Light Detection and Ranging) sensor, another imaging sensor, etc.); (3) processing circuitry configured to generate depth information at least in part by using data obtained by the depth sensor (and, in some embodiments, by also using the image(s) obtained by the imaging sensor);”) configured to: receive the image data (Obropta Paragraph [0152] “As shown in FIG. 8, process 800 includes: (1) obtaining an image of a set harvested specialty crops and associated depth information at act 802”), receive the additional information from the light detection and ranging sensor (Obropta Paragraph [0148] “Next, process 700 proceeds to act 704, where the sensor package obtains depth data associated with the image using a depth sensor. Examples of depth sensors are provided herein. The depth data may include ultrasound data, LIDAR data, and/or imaging data.”), generate an estimate of the stockpile volume based on the second estimate of the location and rotational orientation of the image sensor (Obropta Paragraph [0157] “Next, at act 806, the volume of the set of harvested specialty crops may be estimated using the 3D surface model generated at act 804. In some embodiments, the volume may be determined by computing a volume integral of the area underneath the 3D surface model. In some embodiments, e.g., where the surface model includes a collection of cuboids, the volume may be calculated by calculating the volume of portions of the surface model, e.g., one cuboid or section of the mesh at a time, and sum the volumes of the portions.” Examiner note: Given that the stockpile volume is based on a 3D model which was computed based on an image captured from the imaging sensor, the volume must be at least partially based on where the imaging sensor is located and how it is rotated), and provide the estimate of the stockpile volume to a user interface (Obropta Paragraph [0166] “Next, at step 816, any of the information derived during process 800 from the image of the set of harvested specialty crops and the associated depth information is output. In some embodiments, the information may be stored for subsequent use. In some embodiments, the information may be provided to an operator of harvesting equipment (e.g., a harvester or bin piler) and the operator may alter the operation of the harvesting equipment based on the received information.”).
Obropta does not teach a system for determining the volume of a stationary stockpile; generating a first estimate of a location and rotational orientation of the image sensor in relation to the stockpile based on the image data from the image sensor, and generating a second estimate of the location and rotational orientation of the image sensor in relation to the stockpile based on the image data.
However, Pelissaro de Lima teaches a system for determining the volume of a stationary stockpile (Pelissaro de Lima Figures 1 & 2; Abstract “Grain storage in piles is a commonly adopted alternative during super harvests. In these cases, it is important to be able to accurately and quickly estimate the volume of the stockpiles in order to relocate the grains. The most common solutions to this problem use 3D or 2D scanners attached to a displacement system. In this work, a system based on a static 2D scanner, which can be easily transported and operated, is proposed.” Examiner note: Obropta teaches performing volume estimation on moving stockpiles. However, the disclosure of Obropta does not use the movement of the stockpiles to determine their volume. Obropta merely teaches that the stockpiles of their disclosure are moving due to the field of use of harvesting crops with a tractor and truck. Therefore, the method of Obropta does not teach that stockpiles must be moving for their volume estimation to work. Then, when considering Obropta alongside Pelissaro de Lima, it can be seen that the method of Obropta could be applied to the stationary stockpiles of Pelissaro de Lima).
Pelissaro de Lima is considered to be analogous to the claimed invention because they are both in the same field of stockpile volume estimation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Obropta to include the teachings of Pelissaro de Lima, to provide the advantage of lower expected error for larger stockpiles (Pelissaro de Lima Figure 9; Section III “The obtained results, simulating the stockpiles at a reduced scale, leads to an average relative error of 19.6%. Despite being a substantially high error, this result illustrates a good performance of the method if evaluated considering the absolute error of the scanner (see Section II-D). According to the relationships between height and diameter of a pile of grains modeled in (1), in the Figure 9 is illustrated the behavior of the mean relative error as the dimensions of the pile increase, reaching errors about 1.5%.”)
Furthermore, Robert teaches generating a first estimate of a location and rotational orientation of the image sensor in relation to the stockpile based on the image data from the image sensor, (Robert Figure 2 Step 210-240; Paragraph [0023] “At step 210, the photogrammetry application receives a set of images 150 of a scene captured by one or more physical cameras.”; Paragraph [0027] “In a first phase of this step, pairwise camera geometry is estimated. In this phase relative camera pose for each image pair is derived. Then, in a second phase, global camera geometry is estimated. Camera pose is estimated by first estimating rotations for camera(s) (e.g., using global averaging), and then deriving translations for camera(s) using the estimated rotations.”) and generating a second estimate of the location and rotational orientation of the image sensor in relation to the stockpile based on the image data (Robert Figure 2 Step 250; Paragraph [0015] “As used herein, the term “camera”, when used without the modifier “physical”, refers to a mathematical object defined according to a pinhole camera model that describes how an image was captured. There is a one-to-one correspondence between cameras and images captured at different positions and orientations. This is independent of the particular physical camera that may have captured the images.”; Paragraph [0030] “As part of step 250, camera pose and position of keypoints in 3D space may be scaled and georeferenced. Typically, a global transform is generated for all cameras under consideration (i.e. all cameras for direct methods, or a growing subset of cameras for incremental methods) that best maps camera positions in 3D space to actual camera positions.” Examiner note: While paragraph [0030] describes multiple “cameras”, it can be seen from paragraph [0015] that these cameras are not necessarily different imaging sensors, but are instead different locations and orientations of images captured by one or more physical cameras).
Robert is considered to be analogous to the claimed invention because they are in the same field of determining the orientation and location of a plurality of cameras. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of claimed invention to have modified the system of Obropta in view of Pelissaro de Lima to include the teachings of Robert, to provide the advantage of reducing calculation time and therefore saving processing power (Robert Paragraph [0050] “Integrating sensor-derived camera-pose at an early stage of the process may have a number of advantages, as mentioned above. Calculation efficiency may be improved, thereby reducing processor and memory resource consumption. Using the present techniques, it is possible to compute camera rotation in the 3D space of the scene from sensor-derived camera position using only pairwise relations between the given camera and other cameras that observe a common portion of the scene.”).
In regards to claim 2, Obropta in view of Pelissaro de Lima and Robert teaches the system of claim 1, wherein the one or more processors are configured to generate a first estimate of the location and rotational orientation of the image sensor utilizing quaternions (Robert Paragraph [0048] “At step 440, the camera rotation calculation subprocess 134 determines a camera rotation in the 3D space of the scene that best maps unit vectors defined based on differences in the optical centers determined from sensor-derived camera positions (from step 420) to the unit vectors along the epipoles (from step 430). As discussed above, provided there are multiple camera pairs available, an approximate solution may be determined using a singular value decomposition and cancelling the inside diagonal matrix or using unit quaternions”).
In regards to claim 3, Obropta in view of Pelissaro de Lima and Robert teaches the system of claim 2, wherein the one or more processors are configured to generate a first estimate of the location and rotational orientation of the image sensor utilizing image comparison (Robert Paragraph Figure 2 Step 220-230; Paragraph [0024] “At step 220, a feature detection process 131 of the photogrammetry application identifies common image features in individual images across the set of images, and computes information describing these image features.” Paragraph [0025] “At step 230, a feature correspondence and filtering process 132 of the photogrammetry application matches image features across image pairs.”).
In regards to claim 4, Obropta in view of Pelissaro de Lima and Robert teaches the system of claim 1, wherein the one or more processors are configured to generate a first estimate of the location and rotational orientation of the image sensor by comparison of images at different rotational orientations and different locations in relation to the stationary stockpile. (Robert Paragraph Figure 2 Step 220-230; Paragraph [0024] “At step 220, a feature detection process 131 of the photogrammetry application identifies common image features in individual images across the set of images, and computes information describing these image features.” Paragraph [0025] “At step 230, a feature correspondence and filtering process 132 of the photogrammetry application matches image features across image pairs.”).
In regards to claim 10, Obropta in view of Pelissaro de Lima and Robert teaches the system of claim 1, wherein the one or more processors are configured to perform digital surface model generation for volume estimation (Obropta Paragraph [0155] “Next, at act 804, a 3D surface model of the set of harvested specialty crops is generated using the image and depth information received at act 802.”; Paragraph [0157] “Next, at act 806, the volume of the set of harvested specialty crops may be estimated using the 3D surface model generated at act 804.”).
In regards to claim 12, Obropta teaches a method for determining the volume of a Obropta Paragraph [0152] “As shown in FIG. 8, process 800 includes: (1) obtaining an image of a set harvested specialty crops and associated depth information at act 802”); receiving range information data from a range sensor to multiple portions of the surface of the stockpile (Obropta Paragraph [0148] “Next, process 700 proceeds to act 704, where the sensor package obtains depth data associated with the image using a depth sensor. Examples of depth sensors are provided herein. The depth data may include ultrasound data, LIDAR data, and/or imaging data.”); generating with a processor a first estimate of the location of the image sensor in relation to the stockpile based onObropta Paragraph [0067] “In some embodiments, the sensor package 211a may contain any of the numerous types of sensors described herein including, by way of example and not limitation, one or more imaging sensors (e.g., a color camera, a monochrome camera, or a multi-spectral camera, etc.) and one or more depth sensors (e.g., one or more ultrasound sensors, one or more LIDAR sensors, or one or more additional imaging sensors, etc.). In some embodiments, sensor package 211a may be configured to obtain information used for determining the distance between the harvested specialty crops on the conveyor 209 and the sensor package 211a.”); generating with a processor an estimate of the stockpile volume based on the second estimate of the location and rotational orientation of the image sensor (Obropta Paragraph [0157] “Next, at act 806, the volume of the set of harvested specialty crops may be estimated using the 3D surface model generated at act 804. In some embodiments, the volume may be determined by computing a volume integral of the area underneath the 3D surface model. In some embodiments, e.g., where the surface model includes a collection of cuboids, the volume may be calculated by calculating the volume of portions of the surface model, e.g., one cuboid or section of the mesh at a time, and sum the volumes of the portions. “); and providing via a user interface information concerning the volume of the stockpile (Obropta Paragraph [0166] “Next, at step 816, any of the information derived during process 800 from the image of the set of harvested specialty crops and the associated depth information is output. In some embodiments, the information may be stored for subsequent use. In some embodiments, the information may be provided to an operator of harvesting equipment (e.g., a harvester or bin piler) and the operator may alter the operation of the harvesting equipment based on the received information.”).
Obropta does not teach a method for determining the volume of a stationary stockpile; generating with a processor a first estimate of the location of the image sensor in relation to the stockpile based on the image data
However, Pelissaro de Lima teaches a method for determining the volume of a stationary stockpile (Pelissaro de Lima Figures 1 & 2; Abstract “Grain storage in piles is a commonly adopted alternative during super harvests. In these cases, it is important to be able to accurately and quickly estimate the volume of the stockpiles in order to relocate the grains. The most common solutions to this problem use 3D or 2D scanners attached to a displacement system. In this work, a system based on a static 2D scanner, which can be easily transported and operated, is proposed.”).
Pelissaro de Lima is considered to be analogous to the claimed invention because they are both in the same field of stockpile volume estimation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Obropta to include the teachings of Pelissaro de Lima, to provide the advantage of lower expected error for larger stockpiles (Pelissaro de Lima Figure 9; Section III “The obtained results, simulating the stockpiles at a reduced scale, leads to an average relative error of 19.6%. Despite being a substantially high error, this result illustrates a good performance of the method if evaluated considering the absolute error of the scanner (see Section II-D). According to the relationships between height and diameter of a pile of grains modeled in (1), in the Figure 9 is illustrated the behavior of the mean relative error as the dimensions of the pile increase, reaching errors about 1.5%.”)
However, Robert teaches generating with a processor a first estimate of the location of the image sensor in relation to the stockpile based on the image dataRobert Figure 2 Step 210-240; Paragraph [0023] “At step 210, the photogrammetry application receives a set of images 150 of a scene captured by one or more physical cameras.”; Paragraph [0027] “In a first phase of this step, pairwise camera geometry is estimated. In this phase relative camera pose for each image pair is derived. Then, in a second phase, global camera geometry is estimated. Camera pose is estimated by first estimating rotations for camera(s) (e.g., using global averaging), and then deriving translations for camera(s) using the estimated rotations.”); and generating with a processor a second estimate of the locations and rotational orientations of the image sensor in relation to the stockpile based on the image data (Robert Figure 2 Step 250; Paragraph [0015] “As used herein, the term “camera”, when used without the modifier “physical”, refers to a mathematical object defined according to a pinhole camera model that describes how an image was captured. There is a one-to-one correspondence between cameras and images captured at different positions and orientations. This is independent of the particular physical camera that may have captured the images.”; Paragraph [0030] “As part of step 250, camera pose and position of keypoints in 3D space may be scaled and georeferenced. Typically, a global transform is generated for all cameras under consideration (i.e. all cameras for direct methods, or a growing subset of cameras for incremental methods) that best maps camera positions in 3D space to actual camera positions.”).
Robert is considered to be analogous to the claimed invention because they are in the same field of determining the orientation and location of a plurality of cameras. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of claimed invention to have modified the system of Obropta in view of Pelissaro de Lima to include the teachings of Robert, to provide the advantage of reducing calculation time and therefore saving processing power (Robert Paragraph [0050] “Integrating sensor-derived camera-pose at an early stage of the process may have a number of advantages, as mentioned above. Calculation efficiency may be improved, thereby reducing processor and memory resource consumption. Using the present techniques, it is possible to compute camera rotation in the 3D space of the scene from sensor-derived camera position using only pairwise relations between the given camera and other cameras that observe a common portion of the scene.”).
In regards to claim 13, Obropta in view of Pelissaro de Lima and Robert renders obvious the claim limitations as in the consideration of claims 2, 3, and 12 above.
In regards to claim 14, Obropta in view of Pelissaro de Lima and Robert renders obvious the claim limitations as in the consideration of claims 4 and 12 above.
In regards to claim 18, Obropta in view of Pelissaro de Lima and Robert renders obvious the claim limitations as in the consideration of claims 10 and 12 above.
Claims 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Obropta in view of Pelissaro de Lima and Robert as applied to the claims above, and further in view of Lukas (US20100284572).
In regards to claim 5, Obropta in view of Pelissaro de Lima and Robert teaches the system of claim 1, but fails to teach wherein the one or more processors are configured to perform segmentation of planar features from individual scans.
However, Lukas teaches wherein the one or more processors are configured to perform segmentation of planar features from individual scans (Lukas Figure 1 Part 102; Paragraph [0041] “Next, processor 104 determines which cells qualify as planar feature candidates (314). A cell qualifies as a planar feature candidate when a number of data points associated with the cell is equal to or above a threshold. Similar to block 310, cells having a number of data points below a certain threshold are disregarded.” Examiner note: Part 102 is the scanner which produces an individual scan, the second excerpt shows segmenting planar features.).
Lukas is considered to be analogous to the claimed invention because they are in the same field of identifying features having a flat surface. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of claimed invention to have modified the system of Obropta in view of Pelissaro de Lima and Robert to include the teachings of Lukas, to provide the advantage of a system that works well 3D data (such as LIDAR) (Lukas Paragraph [0002] “Point feature based approaches work well for some images. For example, point feature based approaches are a commonly used when dealing with standard intensity images and infrared images. Point feature based approaches, however, do not perform as well with images containing 3D data.”)
In regards to claim 15, Obropta in view of Pelissaro de Lima, Robert, and Lukas renders obvious the claim limitations as in the consideration of claims 5 and 12 above.
Claims 6-7 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Obropta in view of Pelissaro de Lima and Robert as applied to the claims above, and further in view of Tan (US20230115887).
In regards to claim 6, Obropta in view of Pelissaro de Lima and Robert teaches the system of claim 1, but fails to teach wherein the one or more processors are configured to perform image-based coarse registration of sensor scans at a single data collection location.
However, Tan teaches wherein the one or more processors are configured to perform image-based coarse registration of sensor scans at a single data collection location (Tan Figure 5 Step 568 Paragraph [0051] “The method 500 may perform an initial iterative closest point (ICP) process 568 based on the pose adjusted digital twin and the preprocessed union data to perform the coarse alignment between the real world object and the digital twin. FIG. 8A shows the alignment of the object and digital twin after coarse alignment (with the real object shown as an image and the points of the digital twin with coarse alignment shown as squares with each square representing real world scanned 3D points by a scanner (for example, HoloLens) related to the digital twin CAD model. As shown in FIG. 8A, the alignment is still very rough and does not meet the criteria for a good alignment as discussed above. FIG. 8B shows the union of the point clouds for the coarse alignment based on the high confidence points from the pre-trained models shown in FIGS. 7A and 7B.” Examiner note: This excerpt teaches coarse alignment of sensor scans between real world scanned points and a computer model, where the real world scanned points are taken from a single location.).
Tan is considered to be analogous to the claimed invention because they are in the same field of taking real world data and converting it into accurate 3D models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of claimed invention to have modified the system of Obropta in view of Pelissaro de Lima and Robert to include the teachings of Tan, to provide the advantage of a system that can handle precise alignment of real world objects into virtual data (Tan Paragraph [0024] “The system 100 must have one or more computing devices 102 that each may store and execute a client application to communicate over a communications path 104 to (and exchange data with) a backend system 106 that together provide an augmented reality/a mixed reality experience that benefits from the precise/submillimeter alignment of the real-world object and its digital twin.”)
In regards to claim 7, Obropta in view of Pelissaro de Lima and Robert teaches the system of claim 1, but fails to teach wherein the one or more processors are configured to perform feature matching and fine registration of sensor point clouds from a single data collection location.
However, Tan teaches wherein the one or more processors are configured to perform feature matching and fine registration of sensor point clouds from a single data collection location (Tan Figure 5 Step 544; Paragraph [0052] “The method may also use the union of points to refine the submillimeter alignment (544) in which the accumulated high confidence point pairs are obtained from an intersection 800 (also referenced as 700X or 702X in FIGS. 7A and 7B) of all the points from each of the models as shown in FIG. 8B. The intersection is a portion of the point cloud at which each of the models had identified high confidence point pairs. Thus, in the second stage of alignment, the align refinement uses commonly detected regions, the center which has points identified by each of the models that overlap/adjacent to each other as shown in FIG. 8B.” Examiner note: This excerpt teaches that after the initial ICP of the previous step (568), the matching can be further refined and aligned based on matching the detected regions (or features)).
In regards to claim 16, Obropta in view of Pelissaro de Lima, Robert, and Tan renders obvious the claim limitations as in the consideration of claims 6, 7, and 12 above.
Claims 8-9 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Obropta in view of Pelissaro de Lima and Robert as applied to the claims above, and further in view of Su (CN116385507A).
In regards to claim 8, Obropta in view of Pelissaro de Lima and Robert teaches the system of claim 1, but fails to teach wherein the one or more processors are configured to perform coarse registration of point clouds from different data collection locations
However, Su teaches wherein the one or more processors are configured to perform coarse registration of point clouds from different data collection locations (Su Page 2 Paragraph 4 “As a first aspect of the present invention, providing a multi-source point cloud data registration method”; Su Page 2 Step S3 “step S3: by adding scale factor in the 4 PCS algorithm, selecting the same name line segment with different proportions in the laser point cloud and dense image point cloud the target object as the base, so as to finish coarse registration of the dense image point cloud and the laser point cloud, and obtaining coarse registration parameters of the laser point cloud and the dense image point cloud” Examiner note: The first excerpt shows that this system takes multi-source point cloud data, the second excerpt shows coarse registration.)
Su is considered to be analogous to the claimed invention because they are in the same field of taking real world data and converting it into accurate 3D models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of claimed invention to have modified the system of Obropta in view of Pelissaro de Lima and Robert to include the teachings of Su, to provide the advantage of a system that can precisely convert 3D point cloud data into virtual 3D models (Su Abstract “The invention can be applied to the laser point cloud and image data registration, it solves the problem that the different scale point cloud registration precision is low, at the same time, it improves the registration efficiency.”)
In regards to claim 9, Obropta in view of Pelissaro de Lima and Robert teaches the system of claim 1, and wherein the one or more processors are configured to perform feature matching (Obropta Paragraph [0084] “For example, in some embodiments, a correspondence between features in pairs of images may be computed to generate a disparity map, which together with information about the relative position of the two cameras to each other may be used to determine depths of one or more points in the image.”) but fails to teach fine registration of sensor point clouds from different data locations.
However, Su teaches fine registration of sensor point clouds from different data collection locations. (Su Page 2 Step S4 “step S4: constructing a fine registration model based on dual quaternion, so as to finish fine registration of dense image point cloud and the laser point cloud and obtaining fine registration parameters of the laser point cloud and the dense image point cloud coarse registration.”)
Su is considered to be analogous to the claimed invention because they are in the same field of taking real world data and converting it into accurate 3D models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of claimed invention to have modified the system of Obropta in view of Pelissaro de Lima and Robert to include the teachings of Su, to provide the advantage of a system that can precisely convert 3D point cloud data into virtual 3D models (Su Abstract “The invention can be applied to the laser point cloud and image data registration, it solves the problem that the different scale point cloud registration precision is low, at the same time, it improves the registration efficiency.”)
In regards to claim 17, Obropta in view of Pelissaro de Lima, Robert, and Su renders obvious the claim limitations as in the consideration of claims 8, 9, and 12.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Obropta in view of Pelissaro de Lima and Robert as applied to the claims above, and further in view of Weng (US20060147188).
In regards to claim 11, Obropta in view of Pelissaro de Lima and Robert teaches the system of claim 1, but fails to teach an extension pole connected to the sensor package, wherein the extension pole is hand extendable and hand rotatable to raise and rotate the sensor package above the stationary stockpile.
However, Weng teaches an extension pole connected to the sensor package, wherein the extension pole is hand extendable and hand rotatable to raise and rotate the sensor package above the stationary stockpile (Weng Paragraph [0037] “An elongating tube 341 is in between the thimble 34 and the hand rod 35 to extend positions of the digital camera C”; Paragraph [0039] “While in operation, the loading turntable 2 and the subject M can be rotated based on demanded angles per time by way of rotating the turn handle 273 by a single hand, and the digital camera C can be rotated around the subject M based on demanded angles per time by way of twisting the hand rod 35”).
Weng is considered to be analogous to the claimed invention because they are in the same field of a camera system that captures all of a desired target. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of claimed invention to have modified the system of Obropta in view of Pelissaro de Lima and Robert to include the teachings of Weng, to provide the advantage of a camera system that has a wide range of view (Weng Paragraph [0039] “therefore the digital camera C is capable of seriously taking surrounding horizontal and vertical pictures.”)
Claim 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Obropta in view of Pelissaro de Lima and Robert as applied to the claims above, and further in view of Lukas, Tan, and Su.
In regards to claim 19, Obropta in view of Pelissaro de Lima and Robert teaches the method of claim 12, renders obvious the claim limitations “said generating with a processor the first estimate of the location of the image sensor in relation to the stationary stockpile includes utilizing quaternions and image comparison; and said generating with a processor a second estimate of the locations and rotational orientations of the image sensor in relation to the stationary stockpile includes comparison of images at different rotational orientations and different locations in relation to the stationary stockpile” as in the consideration of claims 12, 13, and 14, and teaches performing feature matching (Obropta Paragraph [0084] “For example, in some embodiments, a correspondence between features in pairs of images may be computed to generate a disparity map, which together with information about the relative position of the two cameras to each other may be used to determine depths of one or more points in the image.”).
Obropta in view of Pelissaro de Lima and Robert does not teach wherein said generating with a processor an estimate of the stationary stockpile volume includes performing segmentation of planar features from individual scans, performing image-based coarse registration of sensor scans at a data collection location, and performing fine registration of sensor point clouds from a data collection location, and performing coarse registration of point clouds from different data collection locations, and performing feature matching and fine registration of sensor point clouds from different data collection locations.
However, Lukas teaches performing segmentation of planar features from individual scans (Lukas Figure 1 Part 102; Paragraph [0041] “Next, processor 104 determines which cells qualify as planar feature candidates (314). A cell qualifies as a planar feature candidate when a number of data points associated with the cell is equal to or above a threshold. Similar to block 310, cells having a number of data points below a certain threshold are disregarded.”).
Lukas is considered to be analogous to the claimed invention because they are in the same field of identifying features having a flat surface. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of claimed invention to have modified the system of Obropta in view of Pelissaro de Lima and Robert to include the teachings of Lukas, to provide the advantage of a system that works well 3D data (such as LIDAR) (Lukas Paragraph [0002] “Point feature based approaches work well for some images. For example, point feature based approaches are a commonly used when dealing with standard intensity images and infrared images. Point feature based approaches, however, do not perform as well with images containing 3D data.”)
Furthermore, Tan teaches performing image-based coarse registration of sensor scans at a data collection location, and performing feature matching and fine registration of sensor point clouds from a data collection location (Tan Figure 5 Step 568 Paragraph [0051] “The method 500 may perform an initial iterative closest point (ICP) process 568 based on the pose adjusted digital twin and the preprocessed union data to perform the coarse alignment between the real world object and the digital twin. FIG. 8A shows the alignment of the object and digital twin after coarse alignment (with the real object shown as an image and the points of the digital twin with coarse alignment shown as squares with each square representing real world scanned 3D points by a scanner (for example, HoloLens) related to the digital twin CAD model. As shown in FIG. 8A, the alignment is still very rough and does not meet the criteria for a good alignment as discussed above. FIG. 8B shows the union of the point clouds for the coarse alignment based on the high confidence points from the pre-trained models shown in FIGS. 7A and 7B.”; Paragraph [0052] “The method may also use the union of points to refine the submillimeter alignment (544) in which the accumulated high confidence point pairs are obtained from an intersection 800 (also referenced as 700X or 702X in FIGS. 7A and 7B) of all the points from each of the models as shown in FIG. 8B. The intersection is a portion of the point cloud at which each of the models had identified high confidence point pairs. Thus, in the second stage of alignment, the align refinement uses commonly detected regions, the center which has points identified by each of the models that overlap/adjacent to each other as shown in FIG. 8B.”).
Tan is considered to be analogous to the claimed invention because they are in the same field of taking real world data and converting it into accurate 3D models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of claimed invention to have modified the system of Obropta in view of Pelissaro de Lima, Robert, and Lukas to include the teachings of Tan, to provide the advantage of a system that can handle precise alignment of real world objects into virtual data (Tan Paragraph [0024] “The system 100 must have one or more computing devices 102 that each may store and execute a client application to communicate over a communications path 104 to (and exchange data with) a backend system 106 that together provide an augmented reality/a mixed reality experience that benefits from the precise/submillimeter alignment of the real-world object and its digital twin.”)
Lastly, Su teaches performing coarse registration of point clouds from different data collection locations, and performing fine registration of sensor point clouds from different data collection locations (Su Page 2 Paragraph 4 “As a first aspect of the present invention, providing a multi-source point cloud data registration method”; Su Page 2 Step S3 “step S3: by adding scale factor in the 4 PCS algorithm, selecting the same name line segment with different proportions in the laser point cloud and dense image point cloud the target object as the base, so as to finish coarse registration of the dense image point cloud and the laser point cloud, and obtaining coarse registration parameters of the laser point cloud and the dense image point cloud”; Page 2 Step S4 “step S4: constructing a fine registration model based on dual quaternion, so as to finish fine registration of dense image point cloud and the laser point cloud and obtaining fine registration parameters of the laser point cloud and the dense image point cloud coarse registration.”)
Su is considered to be analogous to the claimed invention because they are in the same field of taking real world data and converting it into accurate 3D models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of claimed invention to have modified the system of Obropta in view of Pelissaro de Lima, Robert, Lukas, and Tan to include the teachings of Su, to provide the advantage of a system that can precisely convert 3D point cloud data into virtual 3D models (Su Abstract “The invention can be applied to the laser point cloud and image data registration, it solves the problem that the different scale point cloud registration precision is low, at the same time, it improves the registration efficiency.”).
In regards to claim 20, Obropta in view of Pelissaro de Lima and Robert teaches the system of claim 1, wherein the one or more processor are configured to: generate a first estimate of the location and rotational orientation of the image sensor utilizing quaternions (Robert Paragraph [0048] “At step 440, the camera rotation calculation subprocess 134 determines a camera rotation in the 3D space of the scene that best maps unit vectors defined based on differences in the optical centers determined from sensor-derived camera positions (from step 420) to the unit vectors along the epipoles (from step 430). As discussed above, provided there are multiple camera pairs available, an approximate solution may be determined using a singular value decomposition and cancelling the inside diagonal matrix or using unit quaternions”); generate a first estimate of the location and rotational orientation of the image sensor utilizing image comparison (Robert Paragraph Figure 2 Step 220-230; Paragraph [0024] “At step 220, a feature detection process 131 of the photogrammetry application identifies common image features in individual images across the set of images, and computes information describing these image features.” Paragraph [0025] “At step 230, a feature correspondence and filtering process 132 of the photogrammetry application matches image features across image pairs.”); generate a first estimate of the location and rotational orientation of the image sensor by comparison of images at different rotational orientations and different locations in relation to the stationary stockpile (Robert Paragraph Figure 2 Step 220-230; Paragraph [0024] “At step 220, a feature detection process 131 of the photogrammetry application identifies common image features in individual images across the set of images, and computes information describing these image features.” Paragraph [0025] “At step 230, a feature correspondence and filtering process 132 of the photogrammetry application matches image features across image pairs.”); performing feature matching (Obropta Paragraph [0084] “For example, in some embodiments, a correspondence between features in pairs of images may be computed to generate a disparity map, which together with information about the relative position of the two cameras to each other may be used to determine depths of one or more points in the image.”); and performing digital surface model generation for volume estimation (Obropta Paragraph [0155] “Next, at act 804, a 3D surface model of the set of harvested specialty crops is generated using the image and depth information received at act 802.”; Paragraph [0157] “Next, at act 806, the volume of the set of harvested specialty crops may be estimated using the 3D surface model generated at act 804.”).
Obropta in view of Pelissaro de Lima and Robert does not teach performing segmentation of planar features from individual scans; performing image-based coarse registration of sensor scans at a single data collection location; performing feature matching and fine registration of sensor point clouds from a first data collection location; performing coarse registration of point clouds from different data collection locations; and performing feature matching and fine registration of sensor point clouds from a second data collection location.
However, Lukas teaches performing segmentation of planar features from individual scans (Lukas Figure 1 Part 102; Paragraph [0041] “Next, processor 104 determines which cells qualify as planar feature candidates (314). A cell qualifies as a planar feature candidate when a number of data points associated with the cell is equal to or above a threshold. Similar to block 310, cells having a number of data points below a certain threshold are disregarded.”).
Lukas is considered to be analogous to the claimed invention because they are in the same field of identifying features having a flat surface. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of claimed invention to have modified the system of Obropta in view of Pelissaro de Lima and Robert to include the teachings of Lukas, to provide the advantage of a system that works well 3D data (such as LIDAR) (Lukas Paragraph [0002] “Point feature based approaches work well for some images. For example, point feature based approaches are a commonly used when dealing with standard intensity images and infrared images. Point feature based approaches, however, do not perform as well with images containing 3D data.”)
Furthermore, Tan teaches performing image-based coarse registration of sensor scans at a single data collection location; performing feature matching and fine registration of sensor point clouds from a first data collection location (Tan Figure 5 Step 568 Paragraph [0051] “The method 500 may perform an initial iterative closest point (ICP) process 568 based on the pose adjusted digital twin and the preprocessed union data to perform the coarse alignment between the real world object and the digital twin. FIG. 8A shows the alignment of the object and digital twin after coarse alignment (with the real object shown as an image and the points of the digital twin with coarse alignment shown as squares with each square representing real world scanned 3D points by a scanner (for example, HoloLens) related to the digital twin CAD model. As shown in FIG. 8A, the alignment is still very rough and does not meet the criteria for a good alignment as discussed above. FIG. 8B shows the union of the point clouds for the coarse alignment based on the high confidence points from the pre-trained models shown in FIGS. 7A and 7B.”; Paragraph [0052] “The method may also use the union of points to refine the submillimeter alignment (544) in which the accumulated high confidence point pairs are obtained from an intersection 800 (also referenced as 700X or 702X in FIGS. 7A and 7B) of all the points from each of the models as shown in FIG. 8B. The intersection is a portion of the point cloud at which each of the models had identified high confidence point pairs. Thus, in the second stage of alignment, the align refinement uses commonly detected regions, the center which has points identified by each of the models that overlap/adjacent to each other as shown in FIG. 8B.”).
Tan is considered to be analogous to the claimed invention because they are in the same field of taking real world data and converting it into accurate 3D models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of claimed invention to have modified the system of Obropta in view of Pelissaro de Lima, Robert, and Lukas to include the teachings of Tan, to provide the advantage of a system that can handle precise alignment of real world objects into virtual data (Tan Paragraph [0024] “The system 100 must have one or more computing devices 102 that each may store and execute a client application to communicate over a communications path 104 to (and exchange data with) a backend system 106 that together provide an augmented reality/a mixed reality experience that benefits from the precise/submillimeter alignment of the real-world object and its digital twin.”)
Lastly, Su teaches performing coarse registration of point clouds from different data collection locations, and performing fine registration of sensor point clouds from different data collection locations (Su Page 2 Paragraph 4 “As a first aspect of the present invention, providing a multi-source point cloud data registration method”; Su Page 2 Step S3 “step S3: by adding scale factor in the 4 PCS algorithm, selecting the same name line segment with different proportions in the laser point cloud and dense image point cloud the target object as the base, so as to finish coarse registration of the dense image point cloud and the laser point cloud, and obtaining coarse registration parameters of the laser point cloud and the dense image point cloud”; Page 2 Step S4 “step S4: constructing a fine registration model based on dual quaternion, so as to finish fine registration of dense image point cloud and the laser point cloud and obtaining fine registration parameters of the laser point cloud and the dense image point cloud coarse registration.”)
Su is considered to be analogous to the claimed invention because they are in the same field of taking real world data and converting it into accurate 3D models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of claimed invention to have modified the system of Obropta in view of Pelissaro de Lima, Robert, Lukas, and Tan to include the teachings of Su, to provide the advantage of a system that can precisely convert 3D point cloud data into virtual 3D models (Su Abstract “The invention can be applied to the laser point cloud and image data registration, it solves the problem that the different scale point cloud registration precision is low, at the same time, it improves the registration efficiency.”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CALEB LOGAN ESQUINO whose telephone number is (703)756-1462. The examiner can normally be reached M-Fr 8:00AM-4:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CALEB L ESQUINO/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights