DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This communication is in response to the remarks and amendments filed 02/27/2026. Claims 1-3, 5-15 and 17-20 are pending.
Claim Objections
Claim(s) 1-2, 8 and 11-13 are objected to because of the following informalities:
Claims 1, 2 and 8 as well as 11, 12 and 13 each recite a first, second, third and/or fourth neural networks. Applicant is advised to change numbering order of the neural networks such that it follows a chronological order of a first neural network in the parent claims up to a fourth neural network in the dependent claims to avoid typographical and/or 112b clarity issues.
Appropriate correction is required.
Response to Remarks
Applicant’s arguments with respect to independent claims 1 and 11 have been carefully and respectfully considered in light of the instant amendment, but are not persuasive. Accordingly, this action has been made FINAL.
Claim Rejections - 35 USC § 102 and/or 103
On page 7 of the remarks, Applicant argues “the Office Action (page 4) alleged that Wang disclosed the 3rd and 4th neural networks extracting ground features and aerial features as part of the rotation estimator. However, Wang simply discloses an operation for performing a translation... There is not teaching or suggestion to extract either aerial features or ground features from a neural network, much less Wang teach or suggest third and fourth neural networks as recited in claim 1”.
As described below in the new 103 rejection of Wang and Shi, Wang pg. 2, Col 2 teaches the extraction of ground and aerial features as feature maps (as disclosed in cancelled claim 4) that are now used in combination with Shi to describe a third and fourth neural network to be used for a rotation estimator. In addition, assuming arguendo, the claims do not recite a distinction between each neural network, i.e., do not require each neural network to be operationally different to extract their respective features (ground/aerial). A mere indication of training using ground/aerial features does not constitute a difference in modes of operation to determine a distinction between neural networks, as a CNN (as disclosed by Wang) is capable of extracting ground and aerial features.
Therefore, the argued limitations were written broad such that they read upon the cited references or are shown explicitly by the references. As a result, the claims stand rejected as follows.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 5-8, 11-12, 14-15 and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (“Satellite Image Based Cross-view Localization for Autonomous Vehicle”, 2022) in view of Shi et al. (“Boosting 3-DoF Ground-to-Satellite Camera Localization Accuracy via Geometry-Guided Cross-View Transformer”, 2023).
Regarding claim 1, Wang discloses a system, comprising: a computer that includes a processor and a memory, the memory including instructions executable by the processor to (“Satellite Image Based Cross-view Localization (SIBCL) Deep Neural Network, Wang, pg. 1 Col 2): a ground view image processed by the third neural network, trained with ground features, to extract ground features, and an aerial view image processed by the fourth neural network, trained to extract aerial features, to extract aerial features (“The GaFE employs a Convolutional Neural Network (CNN) to extract feature maps Fs and Fg from the satellite and ground-view images, respectively. We adopt a U-Net structure of CNN that aims to obtain feature maps with original resolution that benefits accurate pose estimation” Wang, pg. 2 Col 2); project a ground feature map and a confidence map corresponding to the ground view image to an aerial feature map (“attention map is used to highlight pixels with potential cross-view correspondences. It assigns low score to temporal-inconsistent objects, like cars, and high scores to building edges and road marks that are identifiable from the satellite image” Wang, pg. 2 Col 2) corresponding to the aerial view image according to the relative rotation to create a projected overhead-view feature map (“sparse ground features and attention by projecting 3D points onto the correspondence views. Fig. 3 depicts the projection of 3D points onto the ground-view and satellite-view images. The projection points on satellite images depends on the pose of the query camera” Wang, pg. 3 Col 2); determine a translation difference between the projected overhead-view feature map and the aerial feature map using spatial correlation (“The alignment between the ground and satellite 3D coordinate systems is calculated using [eq. (4)]” Wang, pg. 3 Col 1; i.e., the alignment between the ground and satellite 3D coordinate systems includes that of a lateral translation, longitudinal translation and a height translation fixed to GPS height and a difference is compared (optimized) from an initial pose to the ground truth pose, see Wang Fig. 4); and determine a high-definition estimated three degree-of-freedom pose of a ground view camera based on the relative rotation and the translation difference (“Our goal is to estimate the 3-DoF pose. we report the median errors in lateral and longitudinal translation (m), yaw rotation” Wang, pg. 4 Col 2; Fig. 1 “We aim to estimate accurate 3-DoF pose of the ground-view camera”).
Wang discloses all of the subject matter as described above except for specifically teaching input, to a rotation estimator that includes a third neural network and a fourth neural network and input the ground features and the aerial features to a pose estimator neural network to estimate a relative rotation between the ground view image and the aerial view image.
However, Shi in the same field of endeavor teaches input, to a rotation estimator that includes a third neural network and a fourth neural network (“The neural optimizer constructed by neural networks can produce accurate and reliable rotation estimations, because a rotation on the input can be magnified on the network output” Shi, pg. 3, Col 2; wherein the neural optimizer is “constructed by two swin transformer layers” which are a form of neural networks, see Shi pg. 5 Col 1) and input the ground features and the aerial features to a pose estimator neural network to estimate a relative rotation between the ground view image and the aerial view image (“Our neural optimizer takes input as the differences between the synthesized and observed feature maps Fg2s−Fs, where Fs denote the observed satellite image features, and outputs a relative pose update based on the current pose estimation” Shi, pg. 5 Col 1; wherein Fg2s contains ground features and F-s is satellite aerial features; see additionally “The neural optimizer constructed by neural networks can produce accurate and reliable rotation estimations, because a rotation on the input can be magnified on the network output” Shi, pg. 3, Col 2; Fig. 2 includes a relative rotation estimation between ground image and satellite aerial image).
Therefore, it would have been obvious to one of ordinary skill in the art to combine Wang and Shi before the effective filing date of the claimed invention. The motivation for this combination of references would have been to estimate the relative rotation between two views in course to fine feature levels repeatedly (Shi, pg. 5 Col 1). This motivation for the combination of Wang and Shi is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim 2, Wang and Shi disclose the system of claim 1, wherein the instructions further comprise instructions to determine the ground feature map and the confidence map from the ground view image with a first neural network and instructions to determine the aerial feature map from the aerial view image with a second neural network (“The GaFE employs a Convolutional Neural Network (CNN) to extract feature maps Fs and Fg from the satellite and ground-view images, respectively.” Wang, pg. 2 Col 2; Fig. 2 GaFE extracts feature and attention maps from ground and satellite views).
Regarding claim 5, Wang and Shi disclose the system of claim 1, wherein the rotation estimator instructions further comprise instructions to project the ground features to the aerial features to create an overhead view projection (“The overhead-view satellite images are approximated as a parallel projection. The projection of 3D real-world objects onto a satellite image is formulated as [Eq. (1)]” Wang, pg. 3 Col 2).
Regarding claim 6, Wang and Shi disclose wherein the rotation estimator instructions further comprise instructions to estimate the relative rotation between the ground features and the overhead view projection with a neural pose optimizer (“a neural pose optimizer to estimate the relative pose between them [ground view and overhead view], especially the relative rotation” Shi, pg. 5 Col 1).
Therefore, combining Wang and Shi would meet the claim limitations for the same reasons as previously discussed in claim 1.
Regarding claim 7, Wang and Shi disclose the system of claim 1, wherein the instructions to estimate the three degree-of-freedom pose of the ground view camera comprise instructions to estimate the three degree-of-freedom pose based on an initial estimate of the three degree-of-freedom pose of the ground view camera (“to optimize pose by successively utilizing features on each level, beginning with the coarsest level and initializing each level with the previous level’s outcome” Wang, pg. 3 Col 2; i.e., the 3-DoF pose is based on an initial estimate to be further optimized).
Regarding claim 8, Wang and Shi disclose the system of claim 2, wherein the first and second neural networks have a U-net architecture (“The GaFE employs a Convolutional Neural Network (CNN) to extract feature maps Fs and Fg from the satellite and ground-view images, respectively. We adopt a U-Net structure of CNN that aims to obtain feature maps with original resolution that benefits accurate pose estimation.” Wang, pg. 2 Col 2).
Regarding claim 11, Wang discloses a method, comprising: a ground view image processed by the third neural network, trained with ground features, to extract ground features, and an aerial view image processed by the fourth neural network, trained to extract aerial features, to extract aerial features (“The GaFE employs a Convolutional Neural Network (CNN) to extract feature maps Fs and Fg from the satellite and ground-view images, respectively. We adopt a U-Net structure of CNN that aims to obtain feature maps with original resolution that benefits accurate pose estimation” Wang, pg. 2 Col 2); projecting a ground feature map and a confidence map corresponding to the ground view image to an aerial feature map (“attention map is used to highlight pixels with potential cross-view correspondences. It assigns low score to temporal-inconsistent objects, like cars, and high scores to building edges and road marks that are identifiable from the satellite image” Wang, pg. 2 Col 2) corresponding to the aerial view image according to the relative rotation to create a projected overhead-view feature map (“sparse ground features and attention by projecting 3D points onto the correspondence views. Fig. 3 depicts the projection of 3D points onto the ground-view and satellite-view images. The projection points on satellite images depends on the pose of the query camera” Wang, pg. 3 Col 2); determining a translation difference between the projected overhead-view feature map and the aerial feature map using spatial correlation (“The alignment between the ground and satellite 3D coordinate systems is calculated using [eq. (4)]” Wang, pg. 3 Col 1; i.e., the alignment between the ground and satellite 3D coordinate systems includes that of a lateral translation, longitudinal translation and a height translation fixed to GPS height and a difference is compared (optimized) from an initial pose to the ground truth pose, see Wang Fig. 4); and determining a high-definition estimated three degree-of-freedom pose of a ground view camera based on the relative rotation and the translation difference (“Our goal is to estimate the 3-DoF pose. we report the median errors in lateral and longitudinal translation (m), yaw rotation” Wang, pg. 4 Col 2; Fig. 1 “We aim to estimate accurate 3-DoF pose of the ground-view camera”).
Wang discloses the subject matter as described above except for specifically teaching inputting, to a rotation estimator that includes a third neural network and a fourth neural network and inputting the ground features and the aerial features to a pose estimator neural network to estimate a relative rotation between the ground view image and the aerial view image.
However, Shi in the same field of endeavor teaches inputting, to a rotation estimator that includes a third neural network and a fourth neural network (“The neural optimizer constructed by neural networks can produce accurate and reliable rotation estimations, because a rotation on the input can be magnified on the network output” Shi, pg. 3, Col 2; wherein the neural optimizer is “constructed by two swin transformer layers” which are a form of neural networks, see Shi pg. 5 Col 1) and inputting the ground features and the aerial features to a pose estimator neural network to estimate a relative rotation between the ground view image and the aerial view image (“Our neural optimizer takes input as the differences between the synthesized and observed feature maps Fg2s−Fs, where Fs denote the observed satellite image features, and outputs a relative pose update based on the current pose estimation” Shi, pg. 5 Col 1; wherein Fg2s contains ground features and F-s is satellite aerial features; see additionally “The neural optimizer constructed by neural networks can produce accurate and reliable rotation estimations, because a rotation on the input can be magnified on the network output” Shi, pg. 3, Col 2; Fig. 2 includes a relative rotation estimation between ground image and satellite aerial image).
Therefore, combining Wang and Shi would meet the claim limitations for the same reasons as previously discussed in claim 1.
Regarding claim 12, Wang and Shi disclose the method of claim 11, further comprising determining the ground feature map and the confidence map from the ground view image with a first neural network and determining the aerial feature map from the aerial view image with a second neural network (“The GaFE employs a Convolutional Neural Network (CNN) to extract feature maps Fs and Fg from the satellite and ground-view images, respectively.” Wang, pg. 2 Col 2; Fig. 2 GaFE extracts feature and attention maps from ground and satellite views).
Regarding claim 14, Wang and Shi disclose the method of claim 11, further comprising randomly rotating and translating aerial training images and training the rotation estimator (“the initial pose is randomly sampled under 30 yaw angle errors and 10m lateral and longitudinal shifts” Wang, pg.5 Col 1) based on the aerial training images and the randomly rotated and translated aerial training images (“The satellite features selected by these masks are matched to the query ground image to determine its pose. Instead of sampling discretized poses, our method produces continuous rotation estimates, and our translation is searched uniformly over the entire search space without being affected by the sample randomness” Shi, pg. 2 Col 2).
Therefore, combining Wang and Shi would meet the claim limitations for the same reasons as previously discussed in claim 1.
Regarding claim 15, Wang and Shi disclose the method of claim 14, further comprising extracting a triangle region of the randomly rotated and translated aerial training images (“The satellite features selected by these masks are matched to the query ground image to determine its pose. Instead of sampling discretized poses, our method produces continuous rotation estimates, and our translation is searched uniformly over the entire search space without being affected by the sample randomness” Shi, pg. 2 Col 2).
Therefore, combining Wang and Shi would meet the claim limitations for the same reasons as previously discussed in claim 1.
Regarding claim 17, Wang and Shi disclose the method of claim 11, wherein estimating the relative rotation includes projecting the ground features to the aerial features to create an overhead view projection (“The overhead-view satellite images are approximated as a parallel projection. The projection of 3D real-world objects onto a satellite image is formulated as [Eq. (1)]” Wang, pg. 3 Col 2).
Regarding claim 18, Wang and Shi disclose the method of claim 17, wherein estimating the relative rotation includes estimating the relative rotation between the ground features and the overhead view projection with a neural pose optimizer (“a neural pose optimizer to estimate the relative pose between them [overhead view and ground view], especially the relative rotation” Shi, pg. 5 Col 1).
Therefore, combining Wang and Shi would meet the claim limitations for the same reasons as previously discussed in claim 1.
Regarding claim 19, Wang and Shi disclose the method of claim 11, wherein the estimated three degree-of-freedom pose of the ground view camera is determined based on an initial estimate of the three degree-of-freedom pose of the ground view camera (“to optimize pose by successively utilizing features on each level, beginning with the coarsest level and initializing each level with the previous level’s outcome” Wang, pg. 3 Col 2; i.e., the 3-DoF pose is based on an initial estimate to be further optimized).
Claim(s) 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. in view of Shi et al. and in further view of Zhang et al. (“Monocular weakly supervised depth and pose estimation method based on multi-information fusion”, 2022).
Regarding claim 3, the combination of Wang and Shi disclose the system of claim 1, except for specifically teaching wherein the instructions further comprise instructions to supervise the system using a combination of self-supervised learning and weak supervision. However, Zhang in the same field of endeavor teaches wherein the instructions further comprise instructions to supervise the system using a combination of self-supervised learning and weak supervision (“combines the cost of weak supervision and self-supervision to construct a new weak supervision joint optimization loss function” Zhang, pg. 3).
Therefore, it would have been obvious to one of ordinary skill in the art to combine Wang, Shi and Zhang before the effective filing date of the claimed invention. The motivation for this combination of references would have been to obtain a more accurate pose estimation by the addition of weak supervision labels (Zhang, pg. 14 and 16). This motivation for the combination of Wang, Shi and Zhang is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim 13, Wang, Shi and Zhang disclose the method of claim 12, further comprising supervising the first and second neural networks using a combination of self-supervised learning and weak supervision (“combines the cost of weak supervision and self-supervision to construct a new weak supervision joint optimization loss function” Zhang, pg. 3).
Therefore, combining Wang, Shi and Zhang would meet the claim limitations for the same reasons as previously discussed in claim 3.
Claim(s) 9-10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. in view of Shi et al. and in further view of Lee et al. (US 20220230353).
Regarding claim 9, Wang and Shi disclose the system of claim 1, wherein the instructions further comprise instructions to output the high-definition estimated three degree-of-freedom pose of the ground view camera (“estimate the 3-DoF pose. we report the median errors in lateral and longitudinal translation (m), yaw rotation” Wang, pg. 4 Col 2; Fig. 1 “We aim to estimate accurate 3-DoF pose of the ground-view camera”)
Wang and Shi fail to specifically teach to operate a vehicle. However, Lee in the same field of endeavor teaches to operate a vehicle (“determining the pose of the vehicle 10 within the environment 50, for automatically controlling the components of the autonomous vehicle 10, and generate control signals to the actuator system 30 to automatically control the components of the autonomous vehicle 10 based on the logic, calculations, methods, and/or algorithms. Although only one controller 34 is shown in FIG. 1, embodiments of the autonomous vehicle 10 can include any number of controllers 34 that communicate over any suitable communication medium or a combination of communication mediums and that cooperate to process the sensor signals, perform logic, calculations, methods, and/or algorithms, and generate control signals to automatically control features of the vehicle 10” Lee, [0044]).
Therefore, it would have been obvious to one of ordinary skill in the art to combine Wang, Shi and Lee before the effective filing date of the claimed invention. The motivation for this combination of references would have been to in order to improve real-time localization of the vehicle navigating through an environment (Lee, [0001]). This motivation for the combination of Wang, Shi and Lee is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim 10, Wang, Shi and Lee disclose the system of claim 9, further comprising a vehicle computer configured to determine a vehicle path upon which to operate the vehicle (“determining the pose of the vehicle 10 within the environment 50, for automatically controlling the components of the autonomous vehicle 10, and generate control signals to the actuator system 30 to automatically control the components of the autonomous vehicle 10 based on the logic, calculations, methods, and/or algorithms. Although only one controller 34 is shown in FIG. 1, embodiments of the autonomous vehicle 10 can include any number of controllers 34 that communicate over any suitable communication medium or a combination of communication mediums and that cooperate to process the sensor signals, perform logic, calculations, methods, and/or algorithms, and generate control signals to automatically control features of the vehicle 10” Lee, [0044]) based on the high-definition estimated three degree-of-freedom pose of the ground view camera and the aerial view image (“estimate the 3-DoF pose. we report the median errors in lateral and longitudinal translation (m), yaw rotation” Wang, pg. 4 Col 2; Fig. 1 “We aim to estimate accurate 3-DoF pose of the ground-view camera”).
Therefore, combining Wang, Shi and Lee would meet the claim limitations for the same reasons as previously discussed in claim 9.
Regarding claim 20, Wang, Shi and Lee disclose the method of claim 11, further comprising outputting the high-definition estimated three degree-of-freedom pose of the ground view camera (“estimate the 3-DoF pose. we report the median errors in lateral and longitudinal translation (m), yaw rotation” Wang, pg. 4 Col 2; Fig. 1 “We aim to estimate accurate 3-DoF pose of the ground-view camera”) to operate a vehicle (“determining the pose of the vehicle 10 within the environment 50, for automatically controlling the components of the autonomous vehicle 10, and generate control signals to the actuator system 30 to automatically control the components of the autonomous vehicle 10 based on the logic, calculations, methods, and/or algorithms. Although only one controller 34 is shown in FIG. 1, embodiments of the autonomous vehicle 10 can include any number of controllers 34 that communicate over any suitable communication medium or a combination of communication mediums and that cooperate to process the sensor signals, perform logic, calculations, methods, and/or algorithms, and generate control signals to automatically control features of the vehicle 10” Lee, [0044]).
Therefore, combining Wang, Shi and Lee would meet the claim limitations for the same reasons as previously discussed in claim 9.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Inquiries
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EMMANUEL SILVA-AVINA whose telephone number is (571)270-0729. The examiner can normally be reached Monday - Friday 11 AM - 8 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EMMANUEL SILVA-AVINA/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights