DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendments
The amendments to claims 1, 11, and 20 are accepted and entered.
Claims 1-20 are pending regarding this application.
Response to Arguments
Applicant’s amendments, see Claims, filed 09/01/2026, with respect to the Claim Objections applied to claims 1, 11, and 20, have been fully considered and are persuasive. Applicant has sufficiently amended claims 1, 11, and 20 in accordance with the suggestion(s) as outlined in the Non-Final Rejection mailed on 06/01/2026. The Claim Objections of claims 1, 11, and 20 have been withdrawn.
Applicant’s arguments with respect to the amended subject matter in claim(s) 1, 11, and 20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. As a result, please see the inclusion of Jin et al. (U.S. Patent No. 9245201 B1) which has been included in the 103 rejection of claims 1, 11, and 20 below in order to teach the newly added limitation(s) as recited in amended claims 1, 11, and 20. Note that the above reference was included in the Notice of References Cited (PTO-892) mailed on 09/16/2025.
Applicant additionally argues that Rasco fails to teach “generating a template queue based at least in part on the plurality of template scores, the template queue including a set of selected templates” as recited in independent claims 1, 11, and 20. Applicant recites “The claimed template queue includes a set of selected templates which cannot be interpreted as equivalent to the "hierarchical data structure" as taught by Rasco, as it relies on components and weights, not the templates or template queue as recited in claim 1. Instead, Rasco discloses, at paragraph [0113], "break the input images 304 down into a plurality of components 308 ... organizes or arranges the various components 308 of the input images 304 into a first hierarchical data structure 248 such as a Max-Tree 300." That is, Rasco's workflow is directed to classifying image components as built-up/non-built-up structures using Max-Trees and KD-Trees, which is entirely different from the template queue including a set of selected templates recited in claim 1. The hierarchical organization of image components in Rasco is not a queue of selected templates and does not perform any template-selection functionality”. The examiner disagrees with this assertion. Rasco teaches that “components 308 of the first hierarchical data structure 248/300 may be appropriately weighted to indicate the relative degree to which corresponding portions of the one or more reference satellite image data sets do or do not indicate urban or built-up areas” in para. [0121]. Rasco additionally states that “at least some components of the first hierarchical data structure” are defined as structures of interest or non-structures of interest (emphasis added) in para. [0120], which directly implies some of the components are not defined as either a structure of interest or a non-structure of interest. These structures/components (templates) “may be appropriately weighted to indicate the relative degree to which corresponding portions of the one or more reference satellite image data sets do or do not indicate urban or built-up areas” as shown in para. [0121]. As such, this weighting aids in the determination of components which do or do not represent built-up areas, and is used in order to extract components (see the automated extraction system 200 in para. [0120]) wherein the extracted components are components that are classified as identifying structures of interest as shown in para. [0130]. Here, these extracted components act as the selected templates. Therefore, it is clear that the components are arranged in a queue-like fashion such that only extracted components are selected to be considered for use in the resultant image, and furthermore, only a subset of the extracted components are actually used in the resultant image as shown in para. [0120] and [0130]-[0131]. Therefore, the first hierarchical data structure acts as a priority queue for selecting a prioritizing which components get used in the resultant image. While examiner believes the original citation of Rasco teaches the above queue, examiner has further cited para. [0130]-[0131] in the rejection of the above limitation below to add additional context to the teachings of Rasco in regard to the claimed limitation.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7, 10, 11-14, 17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Rasco et. al. (U.S. Publication No. 20160379388 A1), hereinafter Rasco in view of Mayle et al. (U.S. Publication No. 2016/0012317 A1), hereinafter Mayle, Li et al. (CN 109919984 A, see attached English translation), hereinafter Li, and Jin et al. (U.S. Patent No. 9245201 B1), hereinafter Jin.
Regarding claim 1, Rasco teaches a method for image georegistration (Rasco teaches a “method” for extracting structures from satellite imagery data in para. [0061]), the method comprising:
receiving an input image (Rasco teaches receiving input satellite data in para. [0061]);
generating a plurality of templates from the input image, each template of the plurality of templates comprising a portion of the input image, the plurality of templates being associated with a plurality of template scores (Rasco teaches “decomposing, using a processor, at least one input satellite image into a plurality of components (e.g., pixels or groups of pixels) of a first hierarchical data structure (e.g., a space-partitioning data structure for organizing data points, such as a MaxTree)” in para. [0061]; the components here are interpreted as the plurality of templates in the claim language; ***see also applicant’s description of templates as portions of the input image in para. [0039]; Rasco also teaches that the first hierarchical data structure can be weighted (positive or negative) in para. [0121], these weights are interpreted as equivalent to the scores in the claim language. See also para. [0138] which lists different ways the hierarchical data structure can be determined);
generating a template queue based at least in part on the plurality of template scores, the template queue including a set of selected templates (Rasco teaches a first hierarchical data structure in which the components are prioritized based on certain physical aspects, along with crowdsourced data, in para. [0138] and [0061]; the data structure is interpreted as the template queue, and the data structure is based on positive/negative weights as shown in para. [0120]-[0122]. These weights are interpreted as the template scores. This weighting aids in the determination of components which do or do not represent built-up areas, and is used in order to extract components (see the automated extraction system 200 in para. [0120]) wherein the extracted components are components that are classified as identifying structures of interest as shown in para. [0130]. Here, it is clear that the components are arranged in a queue-like fashion such that only extracted components are selected to be considered for use in the resultant image as shown in para. [0120] and [0130]-[0131]; ***see applicant’s specification in which they describe the queue as being a data structure in para. [0042]. See also below citation), the generating a template queue including at least:
generating the template queue by selecting the set of selected templates from the plurality of templates based at least in part on the plurality of template scores (Rasco teaches a first hierarchical data structure (template queue) based on categorizing components (templates) of structures of interest and non structures of interest as shown in para. [0061] and [0120]. Here, Rasco states that “at least some components of the first hierarchical data structure” are defined as structures of interest or non-structures of interest (emphasis added) in para. [0120], which directly implies some of the components are not defined as either a structure of interest or a non-structure of interest. these structures/components (templates) “may be appropriately weighted to indicate the relative degree to which corresponding portions of the one or more reference satellite image data sets do or do not indicate urban or built-up areas” as shown in para. [0121]. Furthermore, only extracted components (structures representing built-up structures) are considered for use in the resultant image, generated by the automated extraction as taught by para. [0120] and [0130]-[0131]. Here, the extracted components exist within the first hierarchical data structure (template queue) and act as the selected templates);
receiving one or more reference images (Rasco teaches a reference data set in para. [0068]. See also the feature elements of the second hierarchical data set as taught in para. [0064] which can be interpreted as reference data);
applying a matching algorithm to the set of selected templates and at least one reference image of the one or more reference images (Rasco teaches a process of comparing the reference and the input image components in para. [0067-0068] and [0130] this process is interpreted as the matching algorithm in the claim language; the weights described in para. [0068] are interpreted as the match scores. Additionally, Rasco teaches matching the components (templates) of the first hierarchical data structure in para. [0120] and [0064] with the feature elements (reference data) of the second hierarchical data structure in para. [0064] and [0126]);
select a collection of templates, each template of the collection of templates meeting one or more selection criteria (Rasco teaches that, “only those pixels of each of the successive images that are determined to be the “best” (e.g., as determined by the ascertained error rates) may be included in the resultant image” in para. [0069] and [0130]; here, the selected pixels are interpreted as the templates in the collection of templates. Rasco additionally teaches only assigning a component as a “built-up” structure when it “ha[s] a value within some percentage of the value of a feature element from the trained second hierarchical data structure” as shown in para. [0064] and para. [0126]. Here, the end result is a collection of extracted components (templates) in the first hierarchy data structure (queue) which fall into the category of being a built up structure and are below the threshold for error rates for the extracted components as further shown in para. [0131]); and
generating an image transform based at least in part on the collection of templates (Rasco teaches that “built-up structure components may be extracted from successive input images of the geographic area… and mapped into the resultant image to allow for an iterative refinement of the resultant image over a plurality of cycles” in para. [0067] and [0131]; this refinement process using the “built-up structures that are associated with error rates below a threshold error rate into a resultant image” is interpreted as equivalent to the image transform based on the collection of templates as claimed in the claim language. See also Applicant’s description of image transform as a reduction of error in para. [0030] of Applicant’s specification);
wherein the method is performed using one or more processors (Rasco teaches the use of processors to carry out the above mapped information in para. [0070]).
Rasco fails to specifically teach, beyond what is shown in the mapping above, wherein each template of the plurality of templates comprising a portion of the input image and being associated with a candidate point having a pixel location in the input image corresponding to a respective template, determining a set of match scores for the set of selected templates by applying a matching algorithm to the set of selected templates and at least one reference image of the one or more reference images; evaluating the set of match scores to select a collection of templates, each template of the collection of templates meeting one or more selection criteria; and generating an image transform based at least in part on the collection of templates; wherein the generating the image transform comprises: selecting a first set of points from a first template in the collection of templates; computing a first transform based at least in part on the first set of selected points and the at least one reference image; applying a nonlinear algorithm to determine a first error associated with the first transform; and setting the image transform as the first transform.
However, Mayle teaches determining a set of match scores for the set of selected templates by applying a matching algorithm to the set of selected templates and at least one reference image of the one or more reference images (Mayle teaches that “feature point matching may also be used to see if a template is found in an arbitrary image. A template may be a real or an artificial image that expresses a pattern to be found in the image” in para. [0056]; Mayle further teaches that a template image may be treated as multiple templates in para. [0121]; Mayle teaches “third and fourth steps [which] match (or attempt to match) feature points from one image to feature points from another image” in para. [0071]; this process is carried out by “computing the nearest neighbors of each descriptor from a first image to descriptors in a second image” wherein “Lowe describes a ratio test that computes the ratio of the smallest distance from a keypoint in a first image to a keypoint in a second image” in para. [0071-0072]; here, the keypoints are interpreted as the multitude of templates as the keypoints represent portions of the template/input image; additionally, this ratio is interpreted as the match score as shown in the claim language);
evaluating the set of match scores to select a collection of templates, each template of the collection of templates meeting one or more selection criteria (Mayle teaches that “a large ratio (Lowe used a threshold of 0.8) may be used to indicate that two keypoints in the second image are similar to the keypoint in the first image. When this condition arises, there is no matching keypoint in the second image to the keypoint in the first image. This process is carried out by comparing every keypoint in one image to the keypoints in the second image” in para. [0072]; here, the threshold is interpreted as the selection criteria in the claim language); and
generating an image transform based at least in part on the collection of templates (Mayle teaches a step which “takes as input the keypoints from the template image [collection of templates] that match the keypoints in the test image, and computes a geometric relationship between the points in each image” in para. [0077]; here, the geometric relationship can be determined through multiple different types of transformation as further shown in para. [0077]);
wherein the generating the image transform comprises:
selecting a first set of points from a first template in the collection of templates (Mayle teaches a step which “takes as input the keypoints from the template image [collection of templates]” in para. [0077]);
computing a first transform based at least in part on the first set of selected points and the at least one reference image (Mayle teaches a step which “takes as input the keypoints from the template image [collection of templates] that match the keypoints in the test image, and computes a geometric relationship between the points in each image” in para. [0077]; here, the geometric relationship can be determined through multiple different types of transformation as further shown in para. [0077]. See also para. [0081] which states, for example, “a valid homography is used to establish the coordinate system relationship between points in the template image with the points in the test image”).
Mayle and Rasco are both considered to be analogous to the claimed invention because they are in the same field of performing image transformations through matching. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Rasco to incorporate the teachings of Mayle and “determining a set of match scores for the set of selected templates by applying a matching algorithm to the set of selected templates and at least one reference image of the one or more reference images; evaluating the set of match scores to select a collection of templates, each template of the collection of templates meeting one or more selection criteria; and generating an image transform based at least in part on the collection of templates; wherein the generating the image transform comprises: selecting a first set of points from a first template in the collection of templates; computing a first transform based at least in part on the first set of selected points and the at least one reference image”. The motivation for doing so would have been to “eliminate candidate matches via checks of coverage, homography, reprojection, number of match points, determinant, SVD ratio”, as suggested by Mayle in para. [0087]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Rasco with Mayle to obtain the invention specified in the above limitations.
Rasco and Mayle fail to teach wherein each template of the plurality of templates comprising a portion of the input image and being associated with a candidate point having a pixel location in the input image corresponding to a respective template and applying a nonlinear algorithm to determine a first error associated with the first transform; and setting the image transform as the first transform.
However, Li teaches applying a nonlinear algorithm to determine a first error associated with the first transform; and setting the image transform as the first transform (Li teaches “obtaining a rigid body transformation matrix of point cloud rough matching by using a sampling consistency algorithm” in para. [0072] and “evaluating the quality of the rough registration rigid body transformation matrix by using an error measurement loss function for the preliminarily estimated corresponding point pairs” as shown in para. [0077]-[0078]. Here, the rough registration rigid body transformation matrix is interpreted as equivalent to the first transform, and the quality of the transform is evaluated by using an error metric loss function and a Levenberg-Marquardt algorithm. This function error metric loss function is interpreted as equivalent to the non-linear algorithm).
Mayle, Rasco, and Li are all considered to be analogous to the claimed invention because they are in the same field of performing image transformations through matching. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Rasco (as modified by Mayle) to incorporate the teachings of Li and include “applying a nonlinear algorithm to determine a first error associated with the first transform; and setting the image transform as the first transform”. The motivation for doing so would have been that “the method can obtain initial matching point pairs based on a local fast point feature histogram descriptor and a sampling consistency algorithm, obtain a coarse registration matrix through an error measurement loss function, and then obtain a fine registration rigid body transformation matrix by combining an iterative closest point algorithm, so that the problem that three-dimensional point cloud cannot be automatically registered under different viewing angles can be effectively solved”, as suggested by Li in para. [0045]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Rasco and Mayle with Li to obtain the invention specified in the above claim limitations.
While Rasco teaches generating a plurality of templates from the input image, Rasco, Mayle, and Li fail to teach wherein each template of the plurality of templates comprising a portion of the input image and being associated with a candidate point having a pixel location in the input image corresponding to a respective template.
However, Jin teaches each template of the plurality of templates comprising a portion of the input image and being associated with a candidate point having a pixel location in the input image corresponding to a respective template (Jin teaches “the algorithm iterates through each candidate match point in the search window and computes the matching score between the template window in the reference image and the template window centered at the candidate match point in the sensed image” in col. 5, lines 14-21. Here, the candidate match point has a pixel location in the sensed image, which is interpreted as equivalent to the claimed input image. See also FIG. 5 and col. 5, line 64 through col. 6 line 19, wherein the above process is repeated until there are no more grid cells left to search. As such, multiple search/template windows are generated, each with a candidate point having a pixel location in the input image corresponding to a respective template. Jin’s teaching of specifically using templates with corresponding match points can be combined with Rasco’s teaching of generating a hierarchical data structure of components, such that the automated extraction process as taught by Rasco could be combined with Jin’s teaching of selecting candidate tie points to aid in the process of filtering out components (templates) which do not have a matching point to the reference).
Mayle, Rasco, Li, and Jin are all considered to be analogous to the claimed invention because they are in the same field of performing image transformations through matching. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Rasco (as modified by Mayle and Li) to incorporate the teachings of Jin and include that “each template of the plurality of templates comprising a portion of the input image and being associated with a candidate point having a pixel location in the input image corresponding to a respective template”. The motivation for doing so would have been that “automatic tie point generation and accurate image-to-image registration improves usability in many applications including georeferencing, change detection and time series analysis, data fusion, the formation of image mosaics, digital elevation model (DEM) extraction, 3D modeling, video compression and motion analysis, etc” wherein “the components are adaptively selected to improve registration using available spatial references, information about the images, and optional user input” and “automatic generated tie point information 110c may be generated based on a small number of manually specified seed tie points that are fed into an automatic tie point generation system to improve the overall accuracy” , as suggested by Jin in col. 1, lines 20-25, col. 2, lines 33-36, and col. 2, lines 56-60, respectively. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Rasco, Mayle, and Li with Jin to obtain the invention specified in claim 1.
Regarding claim 2, Rasco, Mayle, Li, and Jin teach the method of claim 1,
wherein each template of the plurality of templates is associated with a pixel location in the input image (Rasco teaches wherein the components are pixels or groups of pixels in para. [0061], and the components are based on dissimilarity between adjacent nodes in para. [0138]); Rasco further teaches the nodes of the second hierarchical tree being associated with two-dimensional space (location) in para. [0117], which is used to classify the components (templates) in the first hierarchical data structure as shown in para. [0064])),
wherein the generating a template queue based at least in part on the plurality of template scores comprises:
generating the template queue based at least in part on a plurality of pixel locations associated with the plurality of templates (Rasco teaches that the components (templates) are based on groups of pixels, which inherently implies the queues involve pixel locations, since pixel locations must be known in order to determine specific groups of pixels to define as components (see para. [0121]). Since the components involve pixel locations, and the components are analyzed based on pixel similarity (see para. [0126]), it is inherent, that the template queue is also based at least in part on pixel locations associated with the templates).
Regarding claim 3, Rasco, Mayle, Li, and Jin teach the method of claim 2, wherein the generating a template queue based at least in part on the plurality of template scores comprises:
generating a data structure including a plurality of template points corresponding to the plurality of pixel locations of the plurality of templates (Rasco teaches a first hierarchical data structure in para. [0064] which consists of components (pixels), which is based on a trained second hierarchical data structure (para. [0064]), and nodes (pixels) rooted in two-dimensional space in para. [0117]; since the pixels are rooted in two-dimensional space, it is inferred that there is a location associated with them); and
selecting the set of selected templates using the data structure (Rasco teaches that the second hierarchical data structure aids in the analysis of the first hierarchical data structure in which the components (templates) of the first hierarchical data are “selected” using a comparative analysis of both the first and second hierarchical data structure to classify/tag the “built up” structures in para. [0064]; here, the selected components with the “built-up” structure are interpreted as the selected templates. See also para. [0066] and [0067]).
Regarding claim 4, Rasco, Mayle, Li, and Jin teach the method of claim 1, wherein the generating an image transform further comprises:
selecting a second set of points from the first template (Li teaches “repeating the step S303 until an optimal measurement error result is achieved” in para. [0077] wherein step 303 involves selecting a set of points. Here, since this process occurs repeatedly, it is inherent that there exists at least a second set of points chosen from the first template, wherein the first template is defined in para. [0075]) (See also both Rasco and Mayle’s teaching of the template in claim 1);
computing a second transform based at least in part on the second set of selected points and the at least one reference image (Li teaches repeatedly “randomly selecting a corresponding relation representing sampling points from the points, and preliminarily estimating the points to be corresponding points” in para. [0076] (S303). Here, since, as shown in para. [0077], the process outlined in para. [0076] occurs repeatedly until a criteria is met, it is inherent that there exists at least a second transform, wherein the second transform is interpreted as the corresponding relation (“the rough registration rigid body transformation matrix”);
applying the nonlinear algorithm to determine a second error associated with the second transform (Li teaches repeatedly “evaluating the quality of the rough registration rigid body transformation matrix by using an error measurement loss function for the preliminarily estimated corresponding point pairs” in para. [0077] (S304). Here, since, as shown in para. [0077], the error is calculated repeatedly until a criteria is met, it is inherent that there exists at least a second error for the second transform); and
if the second error is lower than the first error, setting the image transform as the second transform (Since Li teaches that the error is calculated until an optimal measurement error result is met, is implied that, if the second error is lower than the first error, and the second error exhibits an optimal measurement error result, the second transform replaces the first transform as the “coarse registration rigid body transformation matrix” as shown in para. [0077]-[0079]).
Regarding claim 7, Rasco, Mayle, Li, and Jin teach the method of claim 1,
wherein the selection criteria include a match score being higher than a predetermined threshold (Mayle teaches “a large ratio (Lowe used a threshold of 0.8) may be used to indicate that two keypoints in the second image are similar to the keypoint in the first image” in para. [0068]; the threshold here is interpreted as equivalent to the selection criteria). Similar motivations as applied to claim 1 can be applied here to claim 7.
Regarding claim 10, Rasco, Mayle, Li, and Jin teach the method of claim 1,
wherein the input image is an image taken from a satellite (Rasco teaches receiving input satellite data in para. [0061]).
Regarding claim 11, Rasco teaches a system for image georegistration (Rasco teaches “a system for combining geographical and economic data extracted from satellite imagery” in para. [0010]), the system comprising:
one or more memories having instructions stored thereon (Rasco teaches memory in para. [0083]); and
one or more processors configured to execute the instructions and perform operations (Rasco teaches processors in para. [0083-0086]) comprise:
receiving an input image (Rasco teaches receiving input satellite data in para. [0061]);
generating a plurality of templates from the input image, the plurality of templates being associated with a plurality of template scores (Rasco teaches “decomposing, using a processor, at least one input satellite image into a plurality of components (e.g., pixels or groups of pixels) of a first hierarchical data structure (e.g., a space-partitioning data structure for organizing data points, such as a MaxTree)” in para. [0061]; the components here are interpreted as the plurality of templates in the claim language; ***see also applicant’s description of templates as portions of the input image in para. [0039]; Rasco also teaches that the first hierarchical data structure can be weighted (positive or negative) in para. [0121], these weights are interpreted as equivalent to the scores in the claim language. See also para. [0138] which lists different ways the hierarchical data structure can be determined);
generating a template queue based at least in part on the plurality of template scores, the template queue including a set of selected templates (Rasco teaches a first hierarchical data structure in which the components are prioritized based on certain physical aspects, along with crowdsourced data, in para. [0138] and [0061]; the data structure is interpreted as the template queue, and the data structure is based on positive/negative weights as shown in para. [0120]-[0122]. These weights are interpreted as the template scores; ***see applicant’s specification in which they describe the queue as being a data structure in para. [0042]. See also below citation), the generating a template queue including at least:
generating the template queue by selecting the set of selected templates from the plurality of templates based at least in part on the plurality of template scores (Rasco teaches a first hierarchical data structure (template queue) based on categorizing structures of interest and non structures of interest as shown in para. [0061] and [0120]. Here, Rasco states that “at least some components of the first hierarchical data structure” are defined as structures of interest or non-structures of interest (emphasis added) in para. [0120], which directly implies some of the components are not defined as either a structure of interest or a non-structure of interest. these structures/components (templates) “may be appropriately weighted to indicate the relative degree to which corresponding portions of the one or more reference satellite image data sets do or do not indicate urban or built-up areas” as shown in para. [0121]. As a result, the first data structure (template queue) used for training is generated by selecting ONLY CERTAIN components (templates) of the data structure based on the template weight (i.e. deciding whether a component is negative or positive). To summarize, the categorizing step 512 involves categorizing certain components as positive or negative based on a comparison between the reference and the input image (template score) and only utilizing the positive/negative components in the training step. See also para. [0122] where a subgroup of the components are used to train the second hierarchical data structure);
receiving one or more reference images (Rasco teaches a reference data set in para. [0068]. See also the feature elements of the second hierarchical data set as taught in para. [0064] which can be interpreted as reference data);
applying a matching algorithm to the set of selected templates and at least one reference image of the one or more reference images (Rasco teaches a process of comparing the reference and the input image components in para. [0067-0068] and [0130] this process is interpreted as the matching algorithm in the claim language; the weights described in para. [0068] are interpreted as the match scores. Additionally, Rasco teaches matching the components (templates) of the first hierarchical data structure in para. [0120] and [0064] with the feature elements (reference data) of the second hierarchical data structure in para. [0064] and [0126]);
select a collection of templates, each template of the collection of templates meeting one or more selection criteria (Rasco teaches that, “only those pixels of each of the successive images that are determined to be the “best” (e.g., as determined by the ascertained error rates) may be included in the resultant image” in para. [0069] and [0130]; here, the selected pixels are interpreted as the templates in the collection of templates. Rasco additionally teaches only assigning a component as a “built-up” structure when it “ha[s] a value within some percentage of the value of a feature element from the trained second hierarchical data structure” as shown in para. [0064] and para. [0126]. Here, the end result is a collection of components (templates) which fall into the category of being a built up structure); and
generating an image transform based at least in part on the collection of templates (Rasco teaches that “built-up structure components may be extracted from successive input images of the geographic area… and mapped into the resultant image to allow for an iterative refinement of the resultant image over a plurality of cycles” in para. [0067] and [0131]; this refinement process using the “built-up structures that are associated with error rates below a threshold error rate into a resultant image” is interpreted as equivalent to the image transform based on the collection of templates as claimed in the claim language. See also Applicant’s description of image transform as a reduction of error in para. [0030] of Applicant’s specification);
wherein the method is performed using one or more processors (Rasco teaches the use of processors to carry out the above mapped information in para. [0070]).
Rasco fails to specifically teach, beyond what is shown in the mapping above, generating a plurality of templates from the input image, each template of the plurality of templates comprising a portion of the input image and being associated with a candidate point having a pixel location in the input image corresponding to a respective template, determining a set of match scores for the set of selected templates by applying a matching algorithm to the set of selected templates and at least one reference image of the one or more reference images; evaluating the set of match scores to select a collection of templates, each template of the collection of templates meeting one or more selection criteria; and generating an image transform based at least in part on the collection of templates; wherein the generating the image transform comprises: selecting a first set of points from a first template in the collection of templates; computing a first transform based at least in part on the first set of selected points and the at least one reference image; applying a nonlinear algorithm to determine a first error associated with the first transform; and setting the image transform as the first transform.
However, Mayle teaches determining a set of match scores for the set of selected templates by applying a matching algorithm to the set of selected templates and at least one reference image of the one or more reference images (Mayle teaches that “feature point matching may also be used to see if a template is found in an arbitrary image. A template may be a real or an artificial image that expresses a pattern to be found in the image” in para. [0056]; Mayle further teaches that a template image may be treated as multiple templates in para. [0121]; Mayle teaches “third and fourth steps [which] match (or attempt to match) feature points from one image to feature points from another image” in para. [0071]; this process is carried out by “computing the nearest neighbors of each descriptor from a first image to descriptors in a second image” wherein “Lowe describes a ratio test that computes the ratio of the smallest distance from a keypoint in a first image to a keypoint in a second image” in para. [0071-0072]; here, the keypoints are interpreted as the multitude of templates as the keypoints represent portions of the template/input image; additionally, this ratio is interpreted as the match score as shown in the claim language);
evaluating the set of match scores to select a collection of templates, each template of the collection of templates meeting one or more selection criteria (Mayle teaches that “a large ratio (Lowe used a threshold of 0.8) may be used to indicate that two keypoints in the second image are similar to the keypoint in the first image. When this condition arises, there is no matching keypoint in the second image to the keypoint in the first image. This process is carried out by comparing every keypoint in one image to the keypoints in the second image” in para. [0072]; here, the threshold is interpreted as the selection criteria in the claim language); and
generating an image transform based at least in part on the collection of templates (Mayle teaches a step which “takes as input the keypoints from the template image [collection of templates] that match the keypoints in the test image, and computes a geometric relationship between the points in each image” in para. [0077]; here, the geometric relationship can be determined through multiple different types of transformation as further shown in para. [0077]);
wherein the generating the image transform comprises:
selecting a first set of points from a first template in the collection of templates (Mayle teaches a step which “takes as input the keypoints from the template image [collection of templates]” in para. [0077]);
computing a first transform based at least in part on the first set of selected points and the at least one reference image (Mayle teaches a step which “takes as input the keypoints from the template image [collection of templates] that match the keypoints in the test image, and computes a geometric relationship between the points in each image” in para. [0077]; here, the geometric relationship can be determined through multiple different types of transformation as further shown in para. [0077]. See also para. [0081] which states, for example, “a valid homography is used to establish the coordinate system relationship between points in the template image with the points in the test image”).
Mayle and Rasco are both considered to be analogous to the claimed invention because they are in the same field of performing image transformations through matching. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Rasco to incorporate the teachings of Mayle and “determining a set of match scores for the set of selected templates by applying a matching algorithm to the set of selected templates and at least one reference image of the one or more reference images; evaluating the set of match scores to select a collection of templates, each template of the collection of templates meeting one or more selection criteria; and generating an image transform based at least in part on the collection of templates; wherein the generating the image transform comprises: selecting a first set of points from a first template in the collection of templates; computing a first transform based at least in part on the first set of selected points and the at least one reference image”. The motivation for doing so would have been to “eliminate candidate matches via checks of coverage, homography, reprojection, number of match points, determinant, SVD ratio”, as suggested by Mayle in para. [0087]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Rasco with Mayle to obtain the invention specified in the above limitations.
Rasco and Mayle fail to teach generating a plurality of templates from the input image, each template of the plurality of templates comprising a portion of the input image and being associated with a candidate point having a pixel location in the input image corresponding to a respective template and applying a nonlinear algorithm to determine a first error associated with the first transform; and setting the image transform as the first transform.
However, Li teaches applying a nonlinear algorithm to determine a first error associated with the first transform; and setting the image transform as the first transform (Li teaches “obtaining a rigid body transformation matrix of point cloud rough matching by using a sampling consistency algorithm” in para. [0072] and “evaluating the quality of the rough registration rigid body transformation matrix by using an error measurement loss function for the preliminarily estimated corresponding point pairs” as shown in para. [0077]-[0078]. Here, the rough registration rigid body transformation matrix is interpreted as equivalent to the first transform, and the quality of the transform is evaluated by using an error metric loss function and a Levenberg-Marquardt algorithm. This function error metric loss function is interpreted as equivalent to the non-linear algorithm).
Mayle, Rasco, and Li are all considered to be analogous to the claimed invention because they are in the same field of performing image transformations through matching. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Rasco (as modified by Mayle) to incorporate the teachings of Li and include “applying a nonlinear algorithm to determine a first error associated with the first transform; and setting the image transform as the first transform”. The motivation for doing so would have been that “the method can obtain initial matching point pairs based on a local fast point feature histogram descriptor and a sampling consistency algorithm, obtain a coarse registration matrix through an error measurement loss function, and then obtain a fine registration rigid body transformation matrix by combining an iterative closest point algorithm, so that the problem that three-dimensional point cloud cannot be automatically registered under different viewing angles can be effectively solved”, as suggested by Li in para. [0045]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Rasco and Mayle with Li to obtain the invention specified in the above claim limitations.
Rasco, Mayle, and Li fail to teach generating a plurality of templates from the input image, each template of the plurality of templates comprising a portion of the input image and being associated with a candidate point having a pixel location in the input image corresponding to a respective template.
However, Jin teaches generating a plurality of templates from the input image, each template of the plurality of templates comprising a portion of the input image and being associated with a candidate point having a pixel location in the input image corresponding to a respective template (Jin teaches “the algorithm iterates through each candidate match point in the search window and computes the matching score between the template window in the reference image and the template window centered at the candidate match point in the sensed image” in col. 5, lines 14-21. Here, the candidate match point has a pixel location in the sensed image, which is interpreted as equivalent to the claimed input image).
Mayle, Rasco, Li, and Jin are all considered to be analogous to the claimed invention because they are in the same field of performing image transformations through matching. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Rasco (as modified by Mayle and Li) to incorporate the teachings of Jin and include “generating a plurality of templates from the input image, each template of the plurality of templates comprising a portion of the input image and being associated with a candidate point having a pixel location in the input image corresponding to a respective template”. The motivation for doing so would have been that “automatic tie point generation and accurate image-to-image registration improves usability in many applications including georeferencing, change detection and time series analysis, data fusion, the formation of image mosaics, digital elevation model (DEM) extraction, 3D modeling, video compression and motion analysis, etc” wherein “the components are adaptively selected to improve registration using available spatial references, information about the images, and optional user input” and “automatic generated tie point information 110c may be generated based on a small number of manually specified seed tie points that are fed into an automatic tie point generation system to improve the overall accuracy” , as suggested by Jin in col. 1, lines 20-25, col. 2, lines 33-36, and col. 2, lines 56-60, respectively. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Rasco, Mayle, and Li with Jin to obtain the invention specified in claim 11.
Regarding claim 12, Rasco, Mayle, Li, and Jin teach the system of claim 11,
wherein each template of the plurality of templates is associated with a pixel location in the input image (Rasco teaches wherein the components are pixels or groups of pixels in para. [0061], and the components are based on dissimilarity between adjacent nodes in para. [0138]); Rasco further teaches the nodes of the second hierarchical tree being associated with two-dimensional space (location) in para. [0117], which is used to classify the components (templates) in the first hierarchical data structure as shown in para. [0064])),
wherein the generating a template queue based at least in part on the plurality of template scores comprises:
generating the template queue based at least in part on a plurality of pixel locations associated with the plurality of templates (Rasco teaches that the components (templates) are based on groups of pixels, which inherently implies the queues involve pixel locations, since pixel locations must be known in order to determine specific groups of pixels to define as components (see para. [0121]). Since the components involve pixel locations, and the components are analyzed based on pixel similarity (see para. [0126]), it is inherent, that the template queue is also based at least in part on pixel locations associated with the templates).
Regarding claim 13, Rasco, Mayle, Li, and Jin teach the system of claim 12, wherein the generating a template queue based at least in part on the plurality of template scores comprises:
generating a data structure including a plurality of template points corresponding to the plurality of pixel locations of the plurality of templates (Rasco teaches a first hierarchical data structure in para. [0064] which consists of components (pixels), which is based on a trained second hierarchical data structure (para. [0064]), and nodes (pixels) rooted in two-dimensional space in para. [0117]); and
selecting the set of selected templates using the data structure (Rasco teaches that the second hierarchical data structure aids in the analysis of the first hierarchical data structure in which the components (templates) of the first hierarchical data are “selected” using a comparative analysis of both the first and second hierarchical data structure to classify/tag the “built up” structures in para. [0064]; here, the selected components with the “built-up” structure are interpreted as the selected templates. See also para. [0066] and [0067]).
Regarding claim 14, Rasco, Mayle, Li, and Jin teach the system of claim 11, wherein the generating an image transform further comprises:
selecting a second set of points from the first template (Li teaches “repeating the step S303 until an optimal measurement error result is achieved” in para. [0077] wherein step 303 involves selecting a set of points. Here, since this process occurs repeatedly, it is inherent that there exists at least a second set of points chosen from the first template, wherein the first template is defined in para. [0075]) (See also both Rasco and Mayle’s teaching of the template in claim 1);
computing a second transform based at least in part on the second set of selected points and the at least one reference image (Li teaches repeatedly “randomly selecting a corresponding relation representing sampling points from the points, and preliminarily estimating the points to be corresponding points” in para. [0076] (S303). Here, since, as shown in para. [0077], the process outlined in para. [0076] occurs repeatedly until a criteria is met, it is inherent that there exists at least a second transform, wherein the second transform is interpreted as the corresponding relation (“the rough registration rigid body transformation matrix”);
applying the nonlinear algorithm to determine a second error associated with the second transform (Li teaches repeatedly “evaluating the quality of the rough registration rigid body transformation matrix by using an error measurement loss function for the preliminarily estimated corresponding point pairs” in para. [0077] (S304). Here, since, as shown in para. [0077], the error is calculated repeatedly until a criteria is met, it is inherent that there exists at least a second error for the second transform); and
if the second error is lower than the first error, setting the image transform as the second transform (Since Li teaches that the error is calculated until an optimal measurement error result is met, is implied that, if the second error is lower than the first error, and the second error exhibits an optimal measurement error result, the second transform replaces the first transform as the “coarse registration rigid body transformation matrix” as shown in para. [0077]-[0079]).
Regarding claim 17, Rasco, Mayle, Li, and Jin teach the system of claim 11,
wherein the selection criteria include a match score being higher than a predetermined threshold (Mayle teaches “a large ratio (Lowe used a threshold of 0.8) may be used to indicate that two keypoints in the second image are similar to the keypoint in the first image” in para. [0068]; the threshold here is interpreted as equivalent to the selection criteria). Similar motivations as applied to claim 11 can be applied here to claim 17.
Regarding claim 20, Rasco teaches a method for image georegistration (Rasco teaches a “method” for extracting structures from satellite imagery data in para. [0061]), the method comprising:
receiving an input image (Rasco teaches receiving input satellite data in para. [0061]);
generating a plurality of templates from the input image, the plurality of templates being associated with a plurality of template scores (Rasco teaches “decomposing, using a processor, at least one input satellite image into a plurality of components (e.g., pixels or groups of pixels) of a first hierarchical data structure (e.g., a space-partitioning data structure for organizing data points, such as a MaxTree)” in para. [0061]; the components here are interpreted as the plurality of templates in the claim language; ***see also applicant’s description of templates as portions of the input image in para. [0039]; Rasco also teaches that the first hierarchical data structure can be weighted (positive or negative) in para. [0121], these weights are interpreted as equivalent to the scores in the claim language. See also para. [0138] which lists different ways the hierarchical data structure can be determined);
generating a template queue based at least in part on the plurality of template scores, the template queue including a set of selected templates (Rasco teaches a first hierarchical data structure in which the components are prioritized based on certain physical aspects, along with crowdsourced data, in para. [0138] and [0061]; the data structure is interpreted as the template queue, and the data structure is based on positive/negative weights as shown in para. [0120]-[0122]. These weights are interpreted as the template scores; ***see applicant’s specification in which they describe the queue as being a data structure in para. [0042]. See also below citation), the generating a template queue including at least:
generating the template queue by selecting the set of selected templates from the plurality of templates based at least in part on the plurality of template scores (Rasco teaches a first hierarchical data structure (template queue) based on categorizing structures of interest and non structures of interest as shown in para. [0061] and [0120]. Here, Rasco states that “at least some components of the first hierarchical data structure” are defined as structures of interest or non-structures of interest (emphasis added) in para. [0120], which directly implies some of the components are not defined as either a structure of interest or a non-structure of interest. these structures/components (templates) “may be appropriately weighted to indicate the relative degree to which corresponding portions of the one or more reference satellite image data sets do or do not indicate urban or built-up areas” as shown in para. [0121]. As a result, the first data structure (template queue) used for training is generated by selecting ONLY CERTAIN components (templates) of the data structure based on the template weight (i.e. deciding whether a component is negative or positive). To summarize, the categorizing step 512 involves categorizing certain components as positive or negative based on a comparison between the reference and the input image (template score) and only utilizing the positive/negative components in the training step. See also para. [0122] where a subgroup of the components are used to train the second hierarchical data structure);
receiving one or more reference images (Rasco teaches a reference data set in para. [0068]. See also the feature elements of the second hierarchical data set as taught in para. [0064] which can be interpreted as reference data);
applying a matching algorithm to the set of selected templates and at least one reference image of the one or more reference images (Rasco teaches a process of comparing the reference and the input image components in para. [0067-0068] and [0130] this process is interpreted as the matching algorithm in the claim language; the weights described in para. [0068] are interpreted as the match scores. Additionally, Rasco teaches matching the components (templates) of the first hierarchical data structure in para. [0120] and [0064] with the feature elements (reference data) of the second hierarchical data structure in para. [0064] and [0126]);
select a collection of templates, each template of the collection of templates meeting one or more selection criteria (Rasco teaches that, “only those pixels of each of the successive images that are determined to be the “best” (e.g., as determined by the ascertained error rates) may be included in the resultant image” in para. [0069] and [0130]; here, the selected pixels are interpreted as the templates in the collection of templates. Rasco additionally teaches only assigning a component as a “built-up” structure when it “ha[s] a value within some percentage of the value of a feature element from the trained second hierarchical data structure” as shown in para. [0064] and para. [0126]. Here, the end result is a collection of components (templates) which fall into the category of being a built up structure); and
generating an image transform based at least in part on the collection of templates (Rasco teaches that “built-up structure components may be extracted from successive input images of the geographic area… and mapped into the resultant image to allow for an iterative refinement of the resultant image over a plurality of cycles” in para. [0067] and [0131]; this refinement process using the “built-up structures that are associated with error rates below a threshold error rate into a resultant image” is interpreted as equivalent to the image transform based on the collection of templates as claimed in the claim language. See also Applicant’s description of image transform as a reduction of error in para. [0030] of Applicant’s specification);
wherein the method is performed using one or more processors (Rasco teaches the use of processors to carry out the above mapped information in para. [0070]).
Rasco fails to specifically teach, beyond what is shown in the mapping above, generating a plurality of templates from the input image, each template of the plurality of templates comprising a portion of the input image and being associated with a candidate point having a pixel location in the input image corresponding to a respective template, determining a set of match scores for the set of selected templates by applying a matching algorithm to the set of selected templates and at least one reference image of the one or more reference images; evaluating the set of match scores to select a collection of templates, each template of the collection of templates meeting one or more selection criteria; and generating an image transform based at least in part on the collection of templates; wherein the generating the image transform comprises: selecting a first set of points from a first template in the collection of templates; computing a first transform based at least in part on the first set of selected points and the at least one reference image; applying a nonlinear algorithm to determine a first error associated with the first transform; and setting the image transform as the first transform.
However, Mayle teaches determining a set of match scores for the set of selected templates by applying a matching algorithm to the set of selected templates and at least one reference image of the one or more reference images (Mayle teaches that “feature point matching may also be used to see if a template is found in an arbitrary image. A template may be a real or an artificial image that expresses a pattern to be found in the image” in para. [0056]; Mayle further teaches that a template image may be treated as multiple templates in para. [0121]; Mayle teaches “third and fourth steps [which] match (or attempt to match) feature points from one image to feature points from another image” in para. [0071]; this process is carried out by “computing the nearest neighbors of each descriptor from a first image to descriptors in a second image” wherein “Lowe describes a ratio test that computes the ratio of the smallest distance from a keypoint in a first image to a keypoint in a second image” in para. [0071-0072]; here, the keypoints are interpreted as the multitude of templates as the keypoints represent portions of the template/input image; additionally, this ratio is interpreted as the match score as shown in the claim language);
evaluating the set of match scores to select a collection of templates, each template of the collection of templates meeting one or more selection criteria (Mayle teaches that “a large ratio (Lowe used a threshold of 0.8) may be used to indicate that two keypoints in the second image are similar to the keypoint in the first image. When this condition arises, there is no matching keypoint in the second image to the keypoint in the first image. This process is carried out by comparing every keypoint in one image to the keypoints in the second image” in para. [0072]; here, the threshold is interpreted as the selection criteria in the claim language); and
generating an image transform based at least in part on the collection of templates (Mayle teaches a step which “takes as input the keypoints from the template image [collection of templates] that match the keypoints in the test image, and computes a geometric relationship between the points in each image” in para. [0077]; here, the geometric relationship can be determined through multiple different types of transformation as further shown in para. [0077]);
wherein the generating the image transform comprises:
selecting a first set of points from a first template in the collection of templates (Mayle teaches a step which “takes as input the keypoints from the template image [collection of templates]” in para. [0077]);
computing a first transform based at least in part on the first set of selected points and the at least one reference image (Mayle teaches a step which “takes as input the keypoints from the template image [collection of templates] that match the keypoints in the test image, and computes a geometric relationship between the points in each image” in para. [0077]; here, the geometric relationship can be determined through multiple different types of transformation as further shown in para. [0077]. See also para. [0081] which states, for example, “a valid homography is used to establish the coordinate system relationship between points in the template image with the points in the test image”).
Mayle and Rasco are both considered to be analogous to the claimed invention because they are in the same field of performing image transformations through matching. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Rasco to incorporate the teachings of Mayle and “determining a set of match scores for the set of selected templates by applying a matching algorithm to the set of selected templates and at least one reference image of the one or more reference images; evaluating the set of match scores to select a collection of templates, each template of the collection of templates meeting one or more selection criteria; and generating an image transform based at least in part on the collection of templates; wherein the generating the image transform comprises: selecting a first set of points from a first template in the collection of templates; computing a first transform based at least in part on the first set of selected points and the at least one reference image”. The motivation for doing so would have been to “eliminate candidate matches via checks of coverage, homography, reprojection, number of match points, determinant, SVD ratio”, as suggested by Mayle in para. [0087]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Rasco with Mayle to obtain the invention specified in the above limitations.
Rasco and Mayle fail to teach generating a plurality of templates from the input image, each template of the plurality of templates comprising a portion of the input image and being associated with a candidate point having a pixel location in the input image corresponding to a respective template and applying a nonlinear algorithm to determine a first error associated with the first transform; and setting the image transform as the first transform.
However, Li teaches applying a nonlinear algorithm to determine a first error associated with the first transform; and setting the image transform as the first transform (Li teaches “obtaining a rigid body transformation matrix of point cloud rough matching by using a sampling consistency algorithm” in para. [0072] and “evaluating the quality of the rough registration rigid body transformation matrix by using an error measurement loss function for the preliminarily estimated corresponding point pairs” as shown in para. [0077]-[0078]. Here, the rough registration rigid body transformation matrix is interpreted as equivalent to the first transform, and the quality of the transform is evaluated by using an error metric loss function and a Levenberg-Marquardt algorithm. This function error metric loss function is interpreted as equivalent to the non-linear algorithm).
Mayle, Rasco, and Li are all considered to be analogous to the claimed invention because they are in the same field of performing image transformations through matching. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Rasco (as modified by Mayle) to incorporate the teachings of Li and include “applying a nonlinear algorithm to determine a first error associated with the first transform; and setting the image transform as the first transform”. The motivation for doing so would have been that “the method can obtain initial matching point pairs based on a local fast point feature histogram descriptor and a sampling consistency algorithm, obtain a coarse registration matrix through an error measurement loss function, and then obtain a fine registration rigid body transformation matrix by combining an iterative closest point algorithm, so that the problem that three-dimensional point cloud cannot be automatically registered under different viewing angles can be effectively solved”, as suggested by Li in para. [0045]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Rasco and Mayle with Li to obtain the invention specified in the above claim limitations.
Rasco, Mayle, and Li fail to teach generating a plurality of templates from the input image, each template of the plurality of templates comprising a portion of the input image and being associated with a candidate point having a pixel location in the input image corresponding to a respective template.
However, Jin teaches generating a plurality of templates from the input image, each template of the plurality of templates comprising a portion of the input image and being associated with a candidate point having a pixel location in the input image corresponding to a respective template (Jin teaches “the algorithm iterates through each candidate match point in the search window and computes the matching score between the template window in the reference image and the template window centered at the candidate match point in the sensed image” in col. 5, lines 14-21. Here, the candidate match point has a pixel location in the sensed image, which is interpreted as equivalent to the claimed input image).
Mayle, Rasco, Li, and Jin are all considered to be analogous to the claimed invention because they are in the same field of performing image transformations through matching. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Rasco (as modified by Mayle and Li) to incorporate the teachings of Jin and include “generating a plurality of templates from the input image, each template of the plurality of templates comprising a portion of the input image and being associated with a candidate point having a pixel location in the input image corresponding to a respective template”. The motivation for doing so would have been that “automatic tie point generation and accurate image-to-image registration improves usability in many applications including georeferencing, change detection and time series analysis, data fusion, the formation of image mosaics, digital elevation model (DEM) extraction, 3D modeling, video compression and motion analysis, etc” wherein “the components are adaptively selected to improve registration using available spatial references, information about the images, and optional user input” and “automatic generated tie point information 110c may be generated based on a small number of manually specified seed tie points that are fed into an automatic tie point generation system to improve the overall accuracy” , as suggested by Jin in col. 1, lines 20-25, col. 2, lines 33-36, and col. 2, lines 56-60, respectively. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Rasco, Mayle, and Li with Jin to obtain the invention specified in claim 20.
Claims 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Rasco et. al. (U.S. Publication No. 2016/0379388 A1), hereinafter Rasco in view of Mayle et al. (U.S. Publication No. 2016/0012317 A1), hereinafter Mayle, Li et al. (CN 109919984 A, see attached English translation), hereinafter Li, Jin et al. (U.S. Patent No. 9245201 B1), hereinafter Jin, and further in view of Holz (U.S. Publication No. 2018/0306587 A1).
Regarding claim 5, Rasco, Mayle, and Li teach the method of claim 1.
Rasco, Mayle, and Li fail to teach determining a subset of inlier templates in the collection of templates; and determining a confidence value of the image transformation based at least in part on the subset of inlier templates and the collection of templates.
However, Holz teaches determining a subset of inlier templates in the collection of templates (Holz teaches “measur[ing] confidence comprising a percentage of occupied cells in the occupancy grid map that are within a threshold distance of lines from the design model that contain the sampled points” in para. [0142]; here, the occupied cells in the map within a threshold distance are interpreted as the subset of inlier templates in the collection of templates); and
determining a confidence value of the image transformation based at least in part on the subset of inlier templates and the collection of templates (Holz teaches that “the computing system may then determine the transformation based on the measured confidence determined for each candidate transformation” in para. [0142]).
Rasco, Mayle, Li, and Holz are both considered to be analogous to the claimed invention because they are in the same field of transforming geographical environments. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Rasco (as modified by Mayle and Li) to incorporate the teachings of Holz and include “determining a subset of inlier templates in the collection of templates; and determining a confidence value of the image transformation based at least in part on the subset of inlier templates and the collection of templates”. The motivation for doing so would have been to “reduce[] errors between cells or points of the occupancy grid map and sampled points from the design model”, as suggested by Holz in para. [0032]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Rasco, Mayle, and Li with Holz to obtain the invention specified in claim 5.
Regarding claim 15, Rasco, Mayle, and Li teach the system of claim 11.
Rasco, Mayle, and Li fail to teach determining a subset of inlier templates in the collection of templates; and determining a confidence value of the image transformation based at least in part on the subset of inlier templates and the collection of templates.
However, Holz teaches determining a subset of inlier templates in the collection of templates (Holz teaches “measur[ing] confidence comprising a percentage of occupied cells in the occupancy grid map that are within a threshold distance of lines from the design model that contain the sampled points” in para. [0142]; here, the occupied cells in the map within a threshold distance are interpreted as the subset of inlier templates in the collection of templates); and
determining a confidence value of the image transformation based at least in part on the subset of inlier templates and the collection of templates (Holz teaches that “the computing system may then determine the transformation based on the measured confidence determined for each candidate transformation” in para. [0142]).
Rasco, Mayle, Li, and Holz are both considered to be analogous to the claimed invention because they are in the same field of transforming geographical environments. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Rasco (as modified by Mayle and Li) to incorporate the teachings of Holz and include “determining a subset of inlier templates in the collection of templates; and determining a confidence value of the image transformation based at least in part on the subset of inlier templates and the collection of templates”. The motivation for doing so would have been to “reduce[] errors between cells or points of the occupancy grid map and sampled points from the design model”, as suggested by Holz in para. [0032]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Rasco, Mayle, and Li with Holz to obtain the invention specified in claim 15.
Claims 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Rasco et. al. (U.S. Publication No. 20160379388 A1), hereinafter Rasco in view of Mayle et al. (U.S. Publication No. 2016/0012317 A1), hereinafter Mayle, Li et al. (CN 109919984 A, see attached English translation), hereinafter Li, Jin et al. (U.S. Patent No. 9245201 B1), hereinafter Jin, and Fan et al. (“Exploiting High Geopositioning Accuracy of SAR Data to Obtain Accurate Geometric Orientation of Optical Satellite Images”), hereinafter Fan.
Regarding claim 6, Rasco, Mayle, Li, and Jin teach the method of claim 1.
Rasco, Mayle, Li, and Jin fail to teach wherein the applying a matching algorithm includes: generating an angle weighted oriented gradients (AWOG) representation of a first template of the plurality of templates; and determining one match score of the set of match scores based at least in part on the AWOG representation of the first template and the at least one reference image.
However, Fan teaches wherein the applying a matching algorithm includes:
generating an angle weighted oriented gradients (AWOG) representation of a first template of the plurality of templates (Fan “AWOG constructs dense descriptors for each pixel, increasing the description ability of image details”…“AWOG uses an angle weighting strategy to distribute the gradient value only into the two most related orientations, largely increasing the distinguishability of the feature descriptors” Section 3.1; “based on the analysis of AWOGPC in Section 5.1, we set the values of m, n, and the size of the template window image as 3, 8, and 61 × 61 pixels” Section 5.2.2; here, while Rasco teaches the plurality templates (see claim 1), Fan teaches the AWOG representation of the pixels; see also FIG. 5); and
determining one match score of the set of match scores based at least in part on the AWOG representation of the first template and the at least one reference image (Fan teaches “using AWOG to match the images” “AWOG is applied to match the roughly geo-rectified image with the reference SAR image” Section 4.(3), and a peak value of the correlation function, which is based on the AWOG representation as shown in Section 3.2, equation (8), (10), and (11); Fan also teaches similarity values, which are interpreted as match scores, with AWOG descriptors in FIG. 6).
Rasco, Mayle, Li, Jin, and Fan are both considered to be analogous to the claimed invention because they are in the same field of transforming geographical environments. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Rasco (as modified by Mayle, Li, and Jin) to incorporate the teachings of Fan and include “wherein the applying a matching algorithm includes: generating an angle weighted oriented gradients (AWOG) representation of a first template of the plurality of templates; and determining one match score of the set of match scores based at least in part on the AWOG representation of the first template and the at least one reference image”. The motivation for doing so would have been to improve calculation efficiency, as suggested by Fan in section 3.2, Phase Correlation Matching. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Rasco, Mayle, Li, and Jin with Fan to obtain the invention specified in claim 6.
Regarding claim 16, Rasco, Mayle, Li, and Jin teach the system of claim 11.
Rasco, Mayle, Li, and Jin fail to teach wherein the applying a matching algorithm includes: generating an angle weighted oriented gradients (AWOG) representation of a first template of the plurality of templates; and determining one match score of the set of match scores based at least in part on the AWOG representation of the first template and the at least one reference image.
However, Fan teaches wherein the applying a matching algorithm includes:
generating an angle weighted oriented gradients (AWOG) representation of a first template of the plurality of templates (Fan “AWOG constructs dense descriptors for each pixel, increasing the description ability of image details”…“AWOG uses an angle weighting strategy to distribute the gradient value only into the two most related orientations, largely increasing the distinguishability of the feature descriptors” Section 3.1; “based on the analysis of AWOGPC in Section 5.1, we set the values of m, n, and the size of the template window image as 3, 8, and 61 × 61 pixels” Section 5.2.2; here, while Rasco teaches the plurality templates (see claim 1), Fan teaches the AWOG representation of the pixels; see also FIG. 5); and
determining one match score of the set of match scores based at least in part on the AWOG representation of the first template and the at least one reference image (Fan teaches “using AWOG to match the images” “AWOG is applied to match the roughly geo-rectified image with the reference SAR image” Section 4.(3), and a peak value of the correlation function, which is based on the AWOG representation as shown in Section 3.2, equation (8), (10), and (11); Fan also teaches similarity values, which are interpreted as match scores, with AWOG descriptors in FIG. 6).
Rasco, Mayle, Li, Jin, and Fan are both considered to be analogous to the claimed invention because they are in the same field of transforming geographical environments. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Rasco (as modified by Mayle, Li, and Jin) to incorporate the teachings of Fan and include “wherein the applying a matching algorithm includes: generating an angle weighted oriented gradients (AWOG) representation of a first template of the plurality of templates; and determining one match score of the set of match scores based at least in part on the AWOG representation of the first template and the at least one reference image”. The motivation for doing so would have been to improve calculation efficiency, as suggested by Fan in section 3.2, Phase Correlation Matching. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Rasco, Mayle, Li, and Jin with Fan to obtain the invention specified in claim 16.
Claim 8 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Rasco et. al. (U.S. Publication No. 20160379388 A1), hereinafter Rasco in view of Mayle et al. (U.S. Publication No. 2016/0012317 A1), hereinafter Mayle, Li et al. (CN 109919984 A, see attached English translation), hereinafter Li, Jin et al. (U.S. Patent No. 9245201 B1), hereinafter Jin, and Safra et al. (U.S. Publciation No. 2009/0019081 A1), hereinafter Safra.
Regarding claim 8, Rasco, Mayle, Li, and Jin teach the method of claim 1.
Mayle further teaches selection criterion associated with templates in the input image. Rasco, Mayle, Li, and Jin fail to teach wherein the selection criteria include a criterion associated with a template distribution in the input image.
However, Safra teaches wherein the selection criteria include a criterion associated with a template distribution in the input image (Safra teaches selecting real-world entities, wherein the locations are chosen “according to a uniform distribution” in para. [0076] in order to analyze error; here, the locations of the entities are interpreted as the templates in the claim language).
Rasco, Mayle, Li, Jin, and Safra are both considered to be analogous to the claimed invention because they are in the same field of transforming geographical environments. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Rasco (as modified by Mayle, Li, and Jin) to incorporate the teachings of Safra and include “wherein the selection criteria include a criterion associated with a template distribution in the input image”. The motivation for doing so would have been to “test [their] methods on data with varying levels of accuracy and incompleteness”, as suggested by Safra in para. [0076]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Rasco, Mayle, Li, and Jin with Safra to obtain the invention specified in claim 8.
Regarding claim 18, Rasco, Mayle, Li, and Jin teach the system of claim 11.
Mayle further teaches selection criterion associated with templates in the input image. Rasco, Mayle, Li, and Jin fail to teach wherein the selection criteria include a criterion associated with a template distribution in the input image.
However, Safra teaches wherein the selection criteria include a criterion associated with a template distribution in the input image (Safra teaches selecting real-world entities, wherein the locations are chosen “according to a uniform distribution” in para. [0076] in order to analyze error; here, the locations of the entities are interpreted as the templates in the claim language).
Rasco, Mayle, Li, Jin, and Safra are both considered to be analogous to the claimed invention because they are in the same field of transforming geographical environments. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Rasco (as modified by Mayle, Li, and Jin) to incorporate the teachings of Safra and include “wherein the selection criteria include a criterion associated with a template distribution in the input image”. The motivation for doing so would have been to “test [their] methods on data with varying levels of accuracy and incompleteness”, as suggested by Safra in para. [0076]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Rasco, Mayle, Li, and Jin with Safra to obtain the invention specified in claim 18.
Claim 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Rasco et. al. (U.S. Publication No. 20160379388 A1), hereinafter Rasco in view of Mayle et al. (U.S. Publication No. 2016/0012317 A1), hereinafter Mayle, Li et al. (CN 109919984 A, see attached English translation), hereinafter Li, Jin et al. (U.S. Patent No. 9245201 B1), hereinafter Jin, and Amann et al. (U.S. Publication No. 2004/0189677 A1), hereinafter Amann.
Regarding claim 9, Rasco, Mayle, Li, and Jin teaches the method of claim 1.
Rasco, Mayle, Li, and Jin fail to teach applying the image transform to the input image to generate a registered image; and exporting the registered image.
However, Amann teaches applying the image transform to the input image to generate a registered image (Amann teaches “transforming, with a graphics processing unit, a rendered image from a first image format to a second image format” in para. [0013]; the rendered image here is interpreted as equivalent to the registered image taught in the claim language); and exporting the registered image (Amann teaches “send[ing] scaled and transformed images to the at least one remote device” in para. [0013]).
Rasco, Mayle, Li, Jin, and Amann are both considered to be analogous to the claimed invention because they are in the same field of transforming images. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Rasco (as modified by Mayle, Li, and Jin) to incorporate the teachings of Amann and include “applying the image transform to the input image to generate a registered image; and exporting the registered image”. The motivation for doing so would have been to “reduce the effect of jagged lines and aliasing” and correcting depth of field effect, as suggested by Amann in para. [0005]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Rasco, Mayle, Li, and Jin with Amann to obtain the invention specified in claim 9.
Regarding claim 19, Rasco, Mayle, Li, and Jin teaches the system of claim 11.
Rasco, Mayle, Li, and Jin fail to teach applying the image transform to the input image to generate a registered image; and exporting the registered image.
However, Amann teaches applying the image transform to the input image to generate a registered image (Amann teaches “transforming, with a graphics processing unit, a rendered image from a first image format to a second image format” in para. [0013]; the rendered image here is interpreted as equivalent to the registered image taught in the claim language); and exporting the registered image (Amann teaches “send[ing] scaled and transformed images to the at least one remote device” in para. [0013]).
Rasco, Mayle, Li, Jin, and Amann are both considered to be analogous to the claimed invention because they are in the same field of transforming images. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Rasco (as modified by Mayle, Li, and Jin) to incorporate the teachings of Amann and include “applying the image transform to the input image to generate a registered image; and exporting the registered image”. The motivation for doing so would have been to “reduce the effect of jagged lines and aliasing” and correcting depth of field effect, as suggested by Amann in para. [0005]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Rasco, Mayle, Li, and Jin with Amann to obtain the invention specified in claim 19.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office
action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the
extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from
the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH
shortened statutory period, then the shortened statutory period will expire on the date the advisory
action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing
date of the advisory action. In no event, however, will the statutory period for reply expire later than
SIX MONTHS from the date of this final action.
Contact
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to KYLA G ALLEN whose telephone number is (703)756-5315. The examiner can
normally be reached M-F 7:30am - 4:30pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a
USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use
the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor,
John Villecco can be reached on (571) 272-7319. The fax phone number for the organization where this
application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from
Patent Center. Unpublished application information in Patent Center is available to registered users. To
file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit
https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and
https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional
questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like
assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or
571-272-1000.
/Kyla Guan-Ping Tiao Allen/
Examiner, Art Unit 2661
/AARON W CARTER/Primary Examiner, Art Unit 2661
