Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejection – 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 6, 9-12, and 14, is/are rejected under 35 U.S.C. 103 as being unpatentable over Hartung (US 11532070 B2) hereinafter referenced as Hartung, in view of Johnson (US 9251419 B2), hereinafter referenced as Johnson.
Regarding claim 1, Hartung teaches
A method for providing an assessment media for use in assessing accuracy of a geospatial dataset, said method comprising:
“One embodiment includes a method comprising accessing a set of points in images of a plurality of input images; projecting the points to a model; determining residuals for the projected points; and generating an indication of accuracy of an orthorectified mosaic for the plurality of input images based on the determined residuals, the generating the indication of accuracy of the orthorectified mosaic is performed without generating the orthorectified mosaic from the plurality of input images.” (Hartung, col 14 line 11-19);
Hartung discloses a method for determining the accuracy of the orthorectified mosaic of a set of points (reads on a method for providing an assessment media for use in assessing accuracy of a geospatial dataset)
obtaining an ordered set of reference media, the ordered set comprising a first and a last reference media,
“In step 202 of FIG. 3, the system obtains input images. For example, satellites 102, 104, and 106 send to ground computing system 100 various images of portions of the earth's surface or elsewhere. The input images obtained in step 202 include overlapping input images that are from a plurality of sensors at different perspectives.” (Hartung, col 14 line 9-19); “Satellite 102 has captured image 304, satellite 104 has captured image 302, and satellite 106 has captured image 306. Images 302, 304, and 306 are overlapping images in that the content depicted in three images overlap. For example, overlap area 304a depicts the overlap of image 302 and 304, and overlap area 304b depicts the region of overlap between images 304 and 306.” (Hartung, col 4, line 9-15);
PNG
media_image1.png
450
374
media_image1.png
Greyscale
(Hartung, Figure 4)
Hartung obtains a series of input images of portions of the earth's surface (reads on obtaining an ordered set of reference media) wherein the input images also include overlapping images that are from a plurality of sensors at different perspectives such as images 302 the red outline, 304 the green outline, and 306 the blue outline (reads on the ordered set comprising a first and a last reference media)
the first reference media comprising one or more identifiable Ground Control Point, GCP, each GCP being associated to a respective 2D or 3D coordinate, each reference media following the first reference media overlapping or encompassing the preceding reference media; and
“Ground control points (GCPs) are places on the earth's surface that have been precisely surveyed in order to determine the exact longitude and latitude of those locations. Those locations are easy to spot in images.” (Hartung, col 3 line 44-50); “Image overlap area 304a includes a series of tie points 310 and a ground control point 320. Thus, tie points 310 and ground control point 320 are in both images 302 and 304.” (Hartung, col 4 line 17-20);
PNG
media_image2.png
450
374
media_image2.png
Greyscale
(Hartung, Figure 4)
Hartung teach ground control points 320, indicated by the purple outline, in image 302 (reads on the first reference media comprising one or more identifiable Ground Control Point, GCP). Each GCP determines the exact longitude and latitude of those locations (reads on each GCP being associated to a respective 2D or 3D coordinate). Each image following 302 comprises of an overlap area, refer to the red, green, and blue boxes (reads on each reference media following the first reference media overlapping or encompassing the preceding reference media).
generating the assessment media based on establishing respective links from each GCP of the first reference media to a corresponding point in the last reference media,
“generate an indication of accuracy of the resulting mosaic prior to orthorectifying and creating the mosaic by accessing a set of points in the plurality of input images, projecting the points to a model, determining residuals for the projected points, and generating the indication of accuracy of the orthorectified mosaic based on the determined residuals.”(Hartung, Abstract); “Image overlap area 304a includes a series of tie points 310 and a ground control point 320. Thus, tie points 310 and ground control point 320 are in both images 302 and 304. Image overlap area 304b includes ground control point 322 and tie points 314. Thus, tie points 314 and ground control point 322 are in both images 304 and 306. Ground control points 320 and 322 were identified in step 204 of FIG. 3. Tie points 310 and 314 were identified in step 206 of FIG. 3.” (Hartung, col 4 line 17-25);
PNG
media_image3.png
450
356
media_image3.png
Greyscale
(Hartung, Figure 4)
Hartung generates an indication of accuracy which involves establishing respective links from each GCP of the first reference media to a corresponding point in the last reference media through the tie points and ground points present, refer to purple and yellow outlines in Figure 4.
said establishing of respective links comprising correlating, from the first reference media to the last reference media, sequentially pairwise reference media, so that each sequentially preceding reference media of a reference media pair has a known position relative to the sequentially following reference media of the reference media pair and
PNG
media_image4.png
450
356
media_image4.png
Greyscale
(Hartung, Figure 4)
“the system performs automated ground control point matching in the input images… Those points that overlap in the imagery can then be matched into the imagery using any number of correlation mechanisms including (but not limited to) normalized cross correlation, feature matching, mutual information, and/or least-square matcher. The resulting matching point coordinates in image space will be projected to ortho-projection space via a camera model, as discussed below.” (Hartung, col 3 line 45-59); “Images 302, 304, and 306 are overlapping images in that the content depicted in three images overlap. For example, overlap area 304a depicts the overlap of image 302 and 304, and overlap area 304b depicts the region of overlap between images 304 and 306. Images 302, 304, and 306 were captured by satellites 102, 104, and 106 and sent to ground computing system 100 as part of step 202. Image overlap area 304a includes a series of tie points 310 and a ground control point 320. Thus, tie points 310 and ground control point 320 are in both images 302 and 304. Image overlap area 304b includes ground control point 322 and tie points 314. Thus, tie points 314 and ground control point 322 are in both images 304 and 306. Ground control points 320 and 322 were identified in step 204 of FIG. 3. Tie points 310 and 314 were identified in step 206 of FIG. 3. Although FIG. 4 shows seven tie points and two ground control points, it is contemplated that in other embodiments, more or less that number of ground control points and tie points will be included. In many embodiments, overlap regions and images will include many tie points and many ground control points.” (Hartung, col 4 line 11-30);
Hartung’s performs automated ground control point matching for input images (reads on establishing of respective links). Input images are matched into the imagery using any number of correlation mechanisms including normalized cross correlation, feature matching, mutual information, and/or least-square matcher. In the example provided, image 302 and image 304 are matched using tie points and many ground control points refer to purple and yellow outlines in Figure 4. Image 304 and image 306 are matched further as well. This links image 302 to image 306 (reads on from correlating the first reference media to the last reference media, sequentially pairwise reference media, so that each sequentially preceding reference media of a reference media pair. Image 302 is the first reference media and image 306 is the last reference media. Image 302 proceeds image 306 and they are correlated together via the ground point matching). Each image has GCPs representing exact longitude and latitude of places on the earth's surface. By linking the images via tie points and ground points, the system also associates longitude and latitude of each image to the other (reads on so that the sequentially following reference media carries information relating to the 2D or 3D coordinates associated with each GCP, thereby linking the 2D or 3D coordinates associated with each GCP to corresponding 2D or 3D coordinates of the last reference media)
so that the sequentially following reference media carries information relating to the 2D or 3D coordinates associated with each GCP, thereby linking the 2D or 3D coordinates associated with each GCP to corresponding 2D or 3D coordinates of the last reference media via a correlation chain.
“In step 204 of FIG. 3, the system performs automated ground control point matching in the input images… Step 204 includes finding those locations (the ground control points) in various images. In one embodiment, the ground control points can be stored in a library. Those points that overlap in the imagery can then be matched into the imagery using any number of correlation mechanisms including (but not limited to) normalized cross correlation, feature matching, mutual information, and/or least-square matcher. The resulting matching point coordinates in image space will be projected to ortho-projection space via a camera model, as discussed below.” (Hartung, col 3 line 45 – 59);
PNG
media_image2.png
450
374
media_image2.png
Greyscale
(Hartung, Figure 4)
Hartung teaches automated ground control point matching (reads on linking the 2D or 3D coordinates associated with each GCP) in the input images wherein the locations of the points are matched into the imagery in the other input images (reads on corresponding 2D or 3D coordinates of the last reference media). The end product is a chained series of overlapping images see Figure 4 (reads on via a correlation chain).
Hartung fails to explicitly teach iteratively correlating tie points and GCPs.
But Johnson does. Johnson teaches:
iteratively correlating tie points and GCPs
“As will be understood, these major steps are iteratively performed. In other words, for a given area of interest, image data can all be collected before anything further occurs (or archived image data can be used) or a subset of the image data can be obtained. The tie points are identified and extracted in an automated process that at the same time generates geo-coordinate information about the tie points (which now become GCPs). The initial MIN is then created. Subsequently, additional image data is used to (1) determine additional geo-coordinate information about certain ones of the GCPs; and (2) determine geo-coordinate information about new GCPs to be added to the network.” (Johnson, col 4 line 29-40);
Johnston teaches iterative processes for adding and correlating tie points and GCPs to a Metric Information Network.
Johnston BASE is analogous art with respect to Hartung because they are from the same field of endeavor, namely geographic information systems. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Hartung with the feature of Johnston to incorporate iterative processes for adding and correlating tie points and GCPs to a Metric Information Network. A person of ordinary skill in the art would do such in order to improve accuracy of the ground control network.
Regarding claim 2, Hartung in view of Johnston teaches the method of claim 1 and additionally teaches the following. Hartung teaches:
wherein the assessment media is configured to enable determination of 2D and/or 3D coordinates of any subset of the assessment media to a level of accuracy.
“In order to provide for the generation of an orthorectified mosaic of suitable quality…this indication of accuracy of the resulting mosaic is generated by accessing a set of points in the plurality of input images, projecting the points to a model, determining residuals for the projected points, and generating the indication of accuracy of the orthorectified mosaic for the plurality of input images based on the determined residuals.” (Hartung, col 2, line 12-26); “the system performs automated ground control point matching in the input images. Ground control points (GCPs) are places on the earth's surface that have been precisely surveyed in order to determine the exact longitude and latitude of those locations. Those locations are easy to spot in images … Those points that overlap in the imagery can then be matched into the imagery using any number of correlation mechanisms including (but not limited to) normalized cross correlation, feature matching, mutual information, and/or least-square matcher. The resulting matching point coordinates in image space will be projected to ortho-projection space via a camera model, as discussed below.” (Hartung, col 3 line 45 – 59);
Hartung generates an orthorectified mosaic of suitable quality (reads on assessment media) which involves chaining every image with the GCP, representing longitude and latitude places on the earth, through residual propagation (reads on enable determination of 2D and/or 3D coordinates of any subset of the assessment media). This allows the accuracy of any image in the set can be determined to the standard as the GCPs (reads on to a level of accuracy).
Johnson continues and teaches the same level of accuracy as the accuracy of the 2D or 3D coordinates of the GCPs based on the correlation chain.
“Two residual threshold values are utilized: a user defined value and a value calculated using the median value of residual distribution in a given iteration. The use of distribution-based threshold is intended to dampen excessive blunder removal that could happen when the presence of some serious blunder points pulls the tie point ground location further away from the location of good interest points, causing the ground residuals of the good points to increase… For the following iteration, the tie point ground location is recalculated after removing all blunder points detected in the previous iteration. The iterative process is terminated when no more blunder point is found, or when only two interest points remaining in the tie point set.” (Johnson, col 8 line 16-31);
Johnson uses the median value of residual distribution in every given iteration to ensure no more blunder points are found (reads on the same level of accuracy as the accuracy. For this allows every matching GCP and tie points iteration to dampen excessive blunder removal and prevent errors in the entire process (reads on accuracy of the 2D or 3D coordinates of the GCPs based on the correlation chain)
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Hartung with the feature of Johnston to incorporate median value of residual distribution in every given iteration to ensure no more blunder points are found removal and prevent errors. A person of ordinary skill in the art would do such in order to improve accuracy of the ground control network.
Regarding claim 5, Hartung in view of Johnson teaches the method of claim 1 and additionally teaches the following. Hartung teaches:
wherein the first reference media comprises a 2D image containing at least one identifiable GCP.
PNG
media_image5.png
580
613
media_image5.png
Greyscale
PNG
media_image6.png
450
353
media_image6.png
Greyscale
(Hartung, Figure 4)
“Ground control points (GCPs) are places on the earth's surface that have been precisely surveyed in order to determine the exact longitude and latitude of those locations. Those locations are easy to spot in images.” (Hartung, col 3 line 46 - 50); “Image overlap area 304a includes a series of tie points 310 and a ground control point 320. Thus, tie points 310 and ground control point 320 are in both images 302 and 304.” (Hartung, col 4 line 17-20);
Hartung teaches wherein the first reference media comprises a 2D image containing at least one GCP, see Figure 1 and Figure 4. Figure 1 is an example of 2D reference medias. In figure 4, the red outline is the first reference media and the purple outline is the ground control points.
Regarding claim 6, Hartung in view of Johnson teaches the method of claim 5 and additionally teaches the following. Hartung teaches:
wherein the 2D image comprises an image taken by a camera, the image including: an aerial photography; or a drone-photo; or a photography from a hand-held camera.
PNG
media_image5.png
580
613
media_image5.png
Greyscale
PNG
media_image7.png
607
788
media_image7.png
Greyscale
“Satellite 102 has captured image 304, satellite 104 has captured image 302, and satellite 106 has captured image 306.” (Hartung, col 4 line 11-12);
Hartung teaches wherein the 2D image comprises an image taken by a camera, the image including: an aerial photography. Figure 1 is an example of 2D reference medias captured.
Regarding claim 9, Hartung in view of Johnson teaches the method of claim 1 and additionally teaches the following. Hartung teaches:
wherein any of the reference media following the first reference media comprises: an aerial photography; or a drone-photo 2D or stereo; or a photography from a hand-held camera; or a textured 3D model; or a point cloud with or without RGB colors; or a data set formed by LIDAR + photogrammetry reconstruction; or satellite imagery; or synthetic-aperture radar, SAR, imagery.
PNG
media_image7.png
607
788
media_image7.png
Greyscale
“Satellite 102 has captured image 304, satellite 104 has captured image 302, and satellite 106 has captured image 306.” (Hartung, col 4 line 11-12);
Hartung teaches reference media following the first reference media, images 304 and 306, comprise of an aerial photography captured by satellites.
Regarding claim 10, Hartung in view of Johnson teaches the method of claim 1 and additionally teaches the following. Hartung teaches:
wherein correlating sequentially pairwise reference media comprises manually comparing a preceding reference media with the following reference media of a reference media pair.
“The image positions of tie points in the overlap areas can be identified and measured manually or automatically. Automatic tie point generation can be performed using normalized cross correlation, feature matching, mutual information or least squares matching” (Hartung, col 3 line 63-67); “FIG. 4 provides an example of steps 202-206… Images 302, 304, and 306 are overlapping images in that the content depicted in three images overlap. For example, overlap area 304a depicts the overlap of image 302 and 304, and overlap area 304b depicts the region of overlap between images 304 and 306. Images 302, 304, and 306 were captured by satellites 102, 104, and 106 and sent to ground computing system 100 as part of step 202. Image overlap area 304a includes a series of tie points 310 and a ground control point 320. Thus, tie points 310 and ground control point 320 are in both images 302 and 304. Image overlap area 304b includes ground control point 322 and tie points 314. Thus, tie points 314 and ground control point 322 are in both images 304 and 306.” (Hartung, col 4 line 4-23);
Hartung teaches the image positions of tie points in the overlap areas can be identified and measured manually (reads on manually comparing a preceding reference media with the following reference media of a reference media pair. Image 302 is the reference media, and 304 and 306 are following reference media of a reference media pair being compared manually). This is done for images 302, 304, and 306 by preforming steps 202-206 (reads on correlating sequentially pairwise reference media).
Johnson continues to teach:
iteratively correlating tie points and GCPs
“As will be understood, these major steps are iteratively performed. In other words, for a given area of interest, image data can all be collected before anything further occurs (or archived image data can be used) or a subset of the image data can be obtained. The tie points are identified and extracted in an automated process that at the same time generates geo-coordinate information about the tie points (which now become GCPs). The initial MIN is then created. Subsequently, additional image data is used to (1) determine additional geo-coordinate information about certain ones of the GCPs; and (2) determine geo-coordinate information about new GCPs to be added to the network.” (Johnson, col 4 line 29-40);
Johnston teaches iterative processes for adding and correlating tie points and GCPs to a Metric Information Network.
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Hartung with the feature of Johnston to incorporate iterative processes for adding and correlating tie points and GCPs to a Metric Information Network. A person of ordinary skill in the art would do such in order to improve accuracy of the ground control network.
Regarding claim 11, Hartung in view of Johnson teaches the method of claim 1 and additionally teaches the following. Hartung teaches:
wherein iteratively correlating sequentially pairwise reference media comprises registering a preceding reference media with the following reference media of a reference media pair.
PNG
media_image8.png
670
1201
media_image8.png
Greyscale
“each of the images captured by the satellites will include metadata. One example of the metadata is attitude of the satellite (e.g., where the satellite is pointing) and ephemeris data. In one example embodiment, the automatic bundle adjustment includes adjusting the attitude data and the ephemeris data to reduce errors.” (Hartung, col 5 line 37-47); “The effect of the bundle adjustment could also be described with respect to FIG. 5C, which shows two photographs 390 and 392. Photograph 390 is a portion of a mosaic where two images are stitched together, line 394 represents the seam between the two images. As can be seen, there is error in that images 390a and 390b do not line up perfectly along seam 394. For example, road 396 is depicted as misaligned at seam 394. The process of automatic bundle adjustment in step 208 of FIG. 3 seeks to adjust the metadata (e.g., attitude data or ephemeris data) in order to arrive at image 392 where the road 396 is no longer is depicted as misaligned at the seam.” (Hartung, col 6 line 28-39)
Hartung performs automatic bundle adjustment with respect to the satellite images (reads on iteratively correlating). For example, given two misaligned photographs 390a, refer to the red outline, and 390b, refer to the green outline, (reads on sequentially pairwise reference media), the system adjusts the metadata in order to align road 396 to seam 394, refer to the pink arrow and purple box. The final is an aligned seamless image pair 932, refer to the blue outline. (reads on registering a preceding reference media with the following reference media of a reference media pair. 390a is the preceding reference media which was registered with 390b, the following reference media. A reference media pair refers to 390).
Regarding claim 12, Hartung in view of Johnson teaches the method of claim 1 and additionally teaches the following. Hartung teaches:
wherein iteratively correlating sequentially pairwise reference media comprises comparing an ortho-projected rendering of registering a preceding reference media with the following reference media of a reference media pair, wherein optionally also an ortho-projected rendering of the following reference media of a reference media pair is provided before the comparison.
“Thus, FIG. 8 shows that for accessed overlapping images that include a respective tie point, the system will project rays from the source of the input image to the DEM in ortho-projection space, determine latitude and longitude of the intersections points in ortho-projection space where the rays intersect the model and determine residuals for those intersection points (and for the rays) by determining relative distances along plane 632 (which is a plane in ortho-projection space).” (Hartung, col 9 line 7-16);
Hartung processes all the overlapping input images (reads on iteratively correlating sequentially pairwise reference media. The overlapping input images are the sequentially pairwise reference media) by projecting rays from the source of the input image from the to the digital elevation model, DEM, in ortho-projection space. This compares the projected rays of the source image is DEM in ortho-projection space (reads on comparing an ortho-projected rendering of registering a preceding reference media with the following reference media of a reference media pair. The DEM in ortho-projection is the ortho-projected rendering of registering a preceding reference media. The input images from the overlapping images are the following reference media of a reference media pair). This is done before their tie points locations are compared for errors (reads on ortho-projected rendering of the following reference media of a reference media pair is provided before the comparison).
Claim 14 is rejected using the same rationale or bases as applied to claim 1 and the mentioned structure.
Additionally, claim 14 recites the following structure:
A computer program product comprising a non-transitory computer-readable storage medium having thereon a computer program comprising program instructions, the computer program being loadable into a processor and configured to cause the processor to perform.
“In one embodiment, ground computing system 100 includes one or more processors for performing the methods described below and one or more non-transitory processor readable storage mediums that collectively store processor readable code that, when executed by the one or more processors, causes the one or more processors to perform the methods described below.” (Hartung, col 2 line 54-60);
Claim(s) 4, and 16-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hartung in view of Johnson and Janky (US 8942483 B2), hereinafter referenced as Janky.
Regarding claim 4, Hartung in view of Johnston teaches the method of claim 1 and but fail to teach the following: wherein each reference media following the first reference media has a larger geographic extent than the preceding reference media.
But Janky does. Janky teaches:
wherein each reference media following the first reference media has a larger geographic extent than the preceding reference media
PNG
media_image9.png
616
1025
media_image9.png
Greyscale
PNG
media_image10.png
655
734
media_image10.png
Greyscale
“FIG. 5 is an aerial image 505 of the terrestrial scene of FIG. 4A, according to an embodiment. In one embodiment, image 505 comprises an image that has been identified by image identification processor 210 from an image database 110 by searching on the location information tagged onto the image file of image 405.” (Janky, col 14 line 32-37); “in one embodiment, a first image is captured with a camera. This captured image comprises a field of view that includes an object with an identifiable feature.” (Janky, col 22 line 61-64); “FIG. 5, in one embodiment image identification processor 210 of system 100 identifies a second image (e.g. image 505) that corresponds to the first image (e.g., image 405)” (Janky, col 32 line 2-5);
Janky teaches image 405 as the first image and image 505 as the second image. Image 405 is the first reference media. Image 505 is an example of each reference media following image 405. Image 505 is an aerial image which provides a larger geographic extent and context than the preceding reference media, image 405.
Janky is analogous art with respect to Hartung in view of Johnson because they are from the same field of endeavor, namely geographic information systems. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Janky with the feature of Hartung in view of Johnson to incorporate a first and second image, wherein the second image is an aerial image which provides a larger geographic extent and context than the preceding first image. A person of ordinary skill in the art would do such in order to improve position determination.
Regarding claim 16, Hartung in view of Johnson teaches using the method according to claim 1 and additionally teaches the following. Hartung teaches:
A method for assessing accuracy of a geospatial dataset, the method comprising: providing assessment media for use in assessing the accuracy of the geospatial dataset, said assessment media being obtained using the method according to claim 1;
“One embodiment includes a method comprising accessing a set of points in images of a plurality of input images; projecting the points to a model; determining residuals for the projected points; and generating an indication of accuracy of an orthorectified mosaic for the plurality of input images based on the determined residuals, the generating the indication of accuracy of the orthorectified mosaic is performed without generating the orthorectified mosaic from the plurality of input images.” (Hartung, col 14 line 11-19);
Hartung discloses a method for determining the accuracy of the orthorectified mosaic of a set of points (reads on a method for providing an assessment media for use in assessing accuracy of a geospatial dataset)
respective 2D or 3D coordinate of the one or more GCP
“Ground control points (GCPs) are places on the earth's surface that have been precisely surveyed in order to determine the exact longitude and latitude of those locations. Those locations are easy to spot in images.” (Hartung, col 3 line 46 - 50); “Image overlap area 304a includes a series of tie points 310 and a ground control point 320. Thus, tie points 310 and ground control point 320 are in both images 302 and 304.” (Hartung, col 4 line 17-20);
Hartung teaches each GCP determines the exact longitude and latitude of those locations (reads on each GCP being associated to a respective 2D or 3D coordinate.
Janky continues and teaches:
correlating the geospatial dataset to an assessment media subset, the assessment media subset being selected from the provided assessment media, the assessment media subset overlapping or encompassing the geospatial dataset; and
“Image identification processor 210, in one embodiment, operates to identify an image from image database 110 that correlates to the received/selected image being processed for georeferencing. Image identification processor 210 can identify the image from image database 110 based on location information, feature matching/image recognition, or other search criteria. For example, the identified image from image database 110 is, in one embodiment, an aerial image that encompasses or corresponds to a geolocation tagged in an image file of the received/selected image.” (Janky, col 7 line 4-13);
Janky identifies an image from image database 110 that correlates to the received/selected image being processed for georeferencing (reads correlating the geospatial dataset to an assessment media subset, the assessment media subset being selected from the provided assessment media. The image database is the geospatial dataset which is correlated to the received/selected image being processed for georeferencing). The image identification identifies the image based on location feature matching/image recognition, geolocation tag, or other search criteria (reads on the assessment media subset overlapping or encompassing the geospatial dataset. The identified image is the assessment media subset which is selected based on its geolocation tag)
assessing the accuracy of the geospatial dataset based on the correlation between the assessment media subset and the geospatial dataset and the correlation between the assessment media subset and the respective 2D or 3D coordinate of the one or more GCP.
“Registration comparer 235, in one embodiment, compares georeferenced registration information that is received in conjunction with a received/selected image to existing georeferenced registration information that is associated with an identified image to determine which is more accurate.” (Janky, col 9 line 44-47); “a candidate image is selected by or received by image selector/receiver 205. After image identification processor 210 identifies a second image that matches or correlates to the candidate image, and after reference feature determiner 215 determines objects/buildings and/or other reference feature(s) common to the candidate and identified image, feature locator 220 locates and retrieves geographic location information related to the common reference feature (e.g., a building corner). Registration comparer 235 looks for a position fix, like a latitude and longitude (and elevation), either in the aerial view, or elsewhere. In many cases, such a position fix has already been located by feature locator 220 and the accuracy of this position fix has already been vetted by accuracy validator 221.” (Janky, col 10 line 46-59);
Janky teaches an accuracy assessment of the geospatial dataset (reads on assessing the accuracy of the geospatial dataset) based on two correlations: comparing the dataset’s registration information against the assessment media subset’s registration information (reads on correlation between the assessment media subset and the geospatial dataset), and comparing position fix, like a latitude and longitude, of the selected images from the geospatial dataset (reads on correlation between the assessment media subset and the respective 2D or 3D coordinate)
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Janky with the feature of Hartung in view of Johnson to incorporate an accuracy assessment of the geospatial dataset (reads on assessing the accuracy based on two correlations: comparing the dataset’s registration information against the assessment media subset’s registration information, and comparing position fix, like a latitude and longitude, of the selected images from the geospatial dataset. A person of ordinary skill in the art would do such in order to improve position determination.
Regarding claim 17, Hartung in view of Johnson teaches using the method according to claim 16, and additionally teaches of a geo-reference satellite image Hartung teaches:
geo-referenced satellite image.
“these satellites are in different orbits, the perspective of the satellites will be different. In one embodiment, each of the images include metadata. Examples of metadata include attitude of the satellite (or attitude of the sensor on the satellite) and/or ephemeris data for the satellite (and/or ephemeris data for the sensor). In one example embodiment, ephemeris data includes location of the satellite and time of taking the data for the location.” (Hartung, col 3 line 34-43); “Satellite 102 has captured image 304, satellite 104 has captured image 302, and satellite 106 has captured image 306.” (Hartung, col 4 line 11-12);
Hartung teaches a set of images from satellites which includes the location of the satellite, and time of taking the data for the location (comprises a geo-referenced satellite image).
Janky continues and teaches:
wherein the geospatial dataset comprises a georeferenced image
“The image comprises digital image information. The system includes a communicative coupling to a georeferenced images database of images. The image identification processor is configured for identifying a second image from the georeferenced images database that correlates to the first image.” (Janky, Abstract);
Janky teaches a georeferenced images database.
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Janky with the feature of Hartung in view of Johnson to incorporate a georeferenced images database. A person of ordinary skill in the art would do such in order to improve position determination.
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable Hartung in view of Johnson and Scherzinger (US 7541974 B2), hereinafter referenced as Scherzinger.
Regarding claim 3, Hartung in view of Johnson teaches the method of claim 1 and Johnson continues and teaches:
determining, for each traversed step of the correlation chain, a residual error
“a value calculated using the median value of residual distribution in a given iteration…For the following iteration, the tie point ground location is recalculated after removing all blunder points detected in the previous iteration. The iterative process is terminated when no more blunder point is found, or when only two interest points remaining in the tie point set.” (Johson, col 8 line17-31).
Johson calculates a median value of residual distribution (reads on determining a residual error) in a given iteration (reads on for each traversed step of the correlation chain).
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Hartung with the feature of Johnston to incorporate calculates a median value of residual distribution in a given iteration. A person of ordinary skill in the art would do such in order to improve accuracy of the ground control network.
But Hartung in view of Johnson fail to teach the following: wherein generating the assessment media further comprises: traversing the correlation chain; and determining a residual error based on the relative known positions of each reference media pair associated with a traversed step, the residual errors relating to a geospatial error in each reference media with respect to the first reference media.
But Scherzinger does. Scherzinger teaches:
wherein generating the assessment media further comprises: traversing the correlation chain; and
“a. defining a first reference point near the perimeter of the zone; b. locating a GPS receiver at a first GPS receiver location outside the zone near the first reference point; c. determining the position of the GPS receiver at the first GPS receiver location using GPS signals; d. determining the position of the first reference point relative to the GPS receiver at the first GPS receiver location; traversing, from the first GPS receiver position into the zone, to the location of the object using a tracking method that is subject to the accumulation of errors;” (Scherzinger, col 2 line 64 – col 3 line 8);
Scherzinger determines the first reference point relative to the GPS receiver at the first GPS receiver location (reads on correlation chain) and then traverses from the first GPS receiver position into the zone (reads on traversing the correlation chain).
determining a residual error based on the relative known positions of each reference media pair associated with a traversed step, the residual errors relating to a geospatial error in each reference media with respect to the first reference media.
“traversing from the starting point to the target while keeping track of positions using a tracking method that is subject to the accumulation of tracking errors, and en route to the target, reducing the accumulated errors by determining position relative to at least one reference location.” (Scherzinger, col 2 line 57-61); “obtaining relative position information of the object location relative to the first reference point during a traversal inside the zone from the first reference point toward the object, accumulating tracking errors during said traversal; an aiding device for reducing the accumulated tracking errors during traversal to the target inside the zone, by determining position relative to at least one reference location whose position is known.” (Scherzinger, col 4 line 7-14);
Scherzinger determines, for each step traversing from the starting point to the target, a tracking error based on the relative position information of the object location relative to the first reference point during a traversal (reads on a residual error based on the relatively known positions of each reference media pair associated with a traversed step). The tracking errors relating to error accumulate in each reference media with respect to the first reference point.
Scherzinger BASE is analogous art with respect to Hartung in view of Johnson because they are from the same field of endeavor, namely geomatics image processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Hartung in view of Johnson with the feature Scherzinger to incorporate traversing for each step from the starting point to the target, a tracking error based on the relative position information of the object location relative to the first reference point during a traversal, wherein the tracking errors relating to error accumulate in each reference media with respect to the first reference point. A person of ordinary skill in the art would do such in order to improve survey-grade position data.
Claim(s) 7-8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hartung in view of Johnson, Janky, and Ely (US 11538135 B2), hereinafter referenced as Ely.
Regarding claim 7, Hartung in view of Johnson teaches the method of claim 1 but fails to teach wherein the first reference media comprises a 3D model comprising the at least one identifiable GCP.
But Janky does. Janky teaches:
wherein the first reference media comprises at least one identifiable feature
“a first image is captured with a camera. This captured image comprises a field of view that includes an object with an identifiable feature.”’ (Janky, col 22 line 61-64);
Janky teaches wherein the first image comprises at least an object with an identifiable feature.
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Janky with the feature of Hartung in view of Johnson to incorporate a first image comprises an image containing at least an object with an identifiable feature. A person of ordinary skill in the art would do such in order to improve position determination.
But Hartung in view of Johnson and Janky fail to teach wherein the first reference media comprises a 3D model comprising at least one GCP.
However Ely does. Ely teaches the following:
wherein the first reference media comprises a 3D model comprising the at least one GCP.
“The first image 102 has been registered to the 3D point cloud 106. The GCPs 552 are provided as part of registering the first image to the 3D point cloud 106.” (Ely, col 11 line 44-46);
Ely teaches the first image comprises of a 3D point cloud which includes GCPs.
Ely BASE is analogous art with respect to Hartung in view of Johnson and Janky because they are from the same field of endeavor, namely geomatics image processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Hartung in view of Johnson and Janky with the feature of Ely to incorporate a first image comprising of a 3D point cloud which includes GCPs. A person of ordinary skill in the art would do such in order to improve capability to accurately tie multiple 2D images together at a precise 3D location.
Regarding claim 8, Hartung in view of Johnson Janky and Ely and teaches the method of claim 7 and additionally teaches the following. Ely teaches:
wherein the 3D model comprises: a textured 3D model; or a point cloud with or without RGB colors; or a data set formed by LID AR+ photogrammetry reconstruction.
“The first image 102 has been registered to the 3D point cloud 106. The GCPs 552 are provided as part of registering the first image to the 3D point cloud 106.” (Ely, col 11 line 44-46);
Ely teaches the first image comprises of a 3D point cloud.
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Hartung in view of Johnson with the feature of Ely to incorporate a first image comprising of a 3D point cloud which includes GCPs. A person of ordinary skill in the art would do such in order to improve capability to accurately tie multiple 2D images together at a precise 3D location.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hartung in view of Johnson, Ely, and Norahim (Reconstructing 3D model of accident scene using drone image), hereinafter referenced as Norahim.
Regarding claim 13, Hartung in view of Johnson teaches the method of claim 1 but fail to teach the following: further comprising obtaining a 3D model of equipment used for obtaining the one or more GCP, wherein the 3D model is used by the first reference media.
But Ely does. Ely teaches the following:
obtaining a 3D model from a point cloud database used for obtaining the one or more GCP, wherein the 3D model is used by the first reference media.
“Embodiments regard determining image coordinates (of two or more two-dimensional (2D) images) associated with a ground control point (GCP) from a three-dimensional (3D) point set.” (Ely, col line 48-51); “FIG. 1 illustrates, by way of example, a flow diagram of an embodiment of a method 100 for 2D image registration to a 3D point set…The 3D point set 104 can be from a point cloud database (DB) 106. The 3D point set 104 can be of a geographical region that overlaps with a geographical region depicted in the image 102” (Ely, col 33-45); “The first image 102 has been registered to the 3D point cloud 106. The GCPs 552 are provided as part of registering the first image to the 3D point cloud 106.” (Ely, col 11 line 44-46);
Ely obtains a 3D model from a point cloud database associated with a ground control point, wherein the 3D model is used by the first image (reads on obtaining a 3D model from a point cloud database used for obtaining the one or more GCP, wherein the 3D model is used by the first reference media)
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Hartung in view of Johnson with the feature of Ely to obtains a 3D model from a point cloud database associated with a ground control point, wherein the 3D model is used by the first image. A person of ordinary skill in the art would do such in order to improve capability to accurately tie multiple 2D images together at a precise 3D location.
But Hartung in view of Johnson and Ely fail to teach obtaining a 3D model of equipment used for obtaining the one or more GCP.
But Norahim does. Norahim teaches:
obtaining a 3D model of equipment used for obtaining the one or more GCP
“reconstruct a 3D model of an accident scene using an unmanned aerial vehicle (UAV).” (Norahim, Abstract); The GCP is produced to give georeferenced to each image that is collected by the UAV when processing, which will give the exact coordinate to each of the images that is the same as the ground coordinate. (Norahim, Page 6);
Norahim teaches of construct a 3D model using an unmanned aerial vehicle (reads on obtaining a 3D model of equipment). This involves obtaining GCP to georeference each image that is collected by the UAV when processing (reads on obtaining the one or more GCP).
Norahim BASE is analogous art with respect to Hartung in view of Johnson and Ely because they are from the same field of endeavor, namely 3D environment imaging with GCP. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Hartung in view of Johnson and Ely with the feature of Norahim to incorporate construct a 3D model using an unmanned aerial vehicle. A person of ordinary skill in the art would do such in order to improve 3D environments.
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hartung in view of Johnson and U.S. Geological Survey (3D Elevation Program), hereinafter referenced as USGS.
Regarding claim 18, Hartung in view of Johnson teaches using the method according to claim 16 but fail to teach wherein the geospatial dataset comprises a geo-referenced 3D model.
But USGS does. USGS teaches:
wherein in the geospatial dataset comprises a 3D model
“The mission of 3DEP is to respond to growing needs for current high-quality topographic data and three-dimensional (3D) representations of the Nation's natural and constructed features.” (USGS, 3D elevation Program Section);
The USGS provides high-quality topographic data and three-dimensional (3D) representations of the Nation's natural and constructed features (reads on geospatial dataset comprises a 3D model).
USGS BASE is analogous art with respect to Hartung in view of Johnson because they are from the same field of endeavor, namely geographic information. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Hartung in view of Johnson with the feature of USGS to incorporate topographic data and three-dimensional (3D) representations of the Nation's natural and constructed features. A person of ordinary skill in the art would do such in order to provide high-quality topographic data.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUNE NGUYEN whose telephone number is (571)272-8919. The examiner can normally be reached M-TH 7:00AM - 5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at (571) 272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUNE NGOC NGUYEN/Examiner, Art Unit 2618
/DEVONA E FAULK/Supervisory Patent Examiner, Art Unit 2618
Prosecution Insights