DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant's submission filed on 16 October 2025 has been entered. Claims 1-3, 8-10, 15 and 16 have been amended. Claims 1-20 are currently pending and have been considered below.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been carefully considered but are moot in view of the new grounds of rejection necessitated by Applicant’s amendments.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee, Andrew, et al. "Long-range pose estimation for aerial refueling approaches using deep neural networks." Journal of aerospace information systems 17.11 (2020): 634-646, hereinafter, “Lee”, and further in view of Hao, Gangtao, et al. "Scale-unambiguous relative pose estimation of space uncooperative targets based on the fusion of three-dimensional time-of-flight camera and monocular camera." Optical Engineering 54.5 (2015): 053112-053112, hereinafter, “Hao”.
As per claim 1, Lee discloses a method comprising:
receiving a two-dimensional (2D) image from an imaging camera of an imaging device of a first device (Lee, Abstract, stereoscopic (stereo) vision systems; Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation; Lee, page 636, III.A.1. Stereo Camera System, Figure 2 shows the stereo camera configuration for the low-resolution EO cameras and the IR cameras. The cameras were configured to trigger on a hardware signal controlled by the collection program ... Images were collected at 10 Hz);
detecting a second device within the 2D image based on 2D image values (Lee, page 634, I. Introduction, allowing the tanker to automatically control the receiver during the refueling process ... the system must have a high-precision relative pose-estimation process that tracks the receiver in real time … we implement a vision system that achieves such requirements ... range from the cameras on a tanker to the refueling contact point. The contact point is approximately 30 m from the stereoscopic (stereo) cameras ... increase the stereo image resolution ... localize the receiver in each image ... image processing steps need only be applied to the region containing the receiver);
receiving a depth image from the imaging device (Lee, Abstract, generate a 3-D point cloud of the target object; Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation, Stereo vision finds features in images and, after a calibration, undistortion, and rectification process, reprojects these features into 3-D space relative to the cameras ... The stereo block matching algorithm locates features in both images and calculates the disparity, or distance in pixel space, between them. Once disparities have been calculated for an image pair, the disparity map can be reprojected into space to create a 3-D point cloud);
determining 2D keypoints of the second device located within the 2D image based on the depth image, the 2D image values, and a previously determined keypoint model (Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation, Stereo vision finds features in images and, after a calibration, undistortion, and rectification process, reprojects these features into 3-D space relative to the cameras ... The stereo block matching algorithm locates features in both images and calculates the disparity, or distance in pixel space, between them. Once disparities have been calculated for an image pair, the disparity map can be reprojected into space to create a 3-D point cloud; Lee, page 637, III.A.4. Pseudotanker and Pseudoreceiver, For pose registration, a reference point cloud (red) is matched onto a sensed point cloud (yellow) using the ICP. Figure 4 shows the reference point cloud for the pseudoreceiver and an example of a sensed point cloud. The reference point cloud is assumed to be a known, uniformly sampled, geometrically accurate model of the approaching receiver);
determining a 6 degree-of-freedom (6DOF) pose using the 2D keypoints and corresponding three-dimensional (3D) keypoints (Lee, Abstract, three-dimensional (3-D) pose-estimation; Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation, Stereo vision finds features in images and, after a calibration, undistortion, and rectification process, reprojects these features into 3-D space relative to the cameras ... The stereo block matching algorithm locates features in both images and calculates the disparity, or distance in pixel space, between them. Once disparities have been calculated for an image pair, the disparity map can be reprojected into space to create a 3-D point cloud ... Once a point cloud has been generated, there are many techniques to perform pose estimation. Since aircraft are rigid bodies, the point-to-point iterative closest point (ICP) [9] was chosen for this work; Lee, page 636, III.A.3. Truth Data, ICP returns a six-degree-of-freedom rigid-body registration); and
outputting a position of at least a component of the second device based on the 6DOF pose (Lee, Abstract, three-dimensional (3-D) pose-estimation; Lee, page 634, I. Introduction, We perform pose estimation with an error of 6 cm at a target 30 m away from the camera system; Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation, Stereo vision finds features in images and, after a calibration, undistortion, and rectification process, reprojects these features into 3-D space relative to the cameras ... The stereo block matching algorithm locates features in both images and calculates the disparity, or distance in pixel space, between them. Once disparities have been calculated for an image pair, the disparity map can be reprojected into space to create a 3-D point cloud ... Once a point cloud has been generated, there are many techniques to perform pose estimation. Since aircraft are rigid bodies, the point-to-point iterative closest point (ICP) [9] was chosen for this work; Lee, page 636, III.A.3. Truth Data, ICP returns a six-degree-of-freedom rigid-body registration; Lee, page 636, III. Ground Experiment Design, Use ICP to register the receiver’s pose; Lee, page 637, III.A.4. Pseudotanker and Pseudoreceiver, For pose registration, a reference point cloud (red) is matched onto a sensed point cloud (yellow) using the ICP; Led, page 637, III.A.5. Running the Experiment, we applied the computer vision pipeline to estimate the pseudoreceiver’s pose. Figure 5 shows an example of registration being visualized in the virtual environment).
Lee further discloses (Lee, page 636, III.A.1. Stereo Camera System, Two separate stereo vision systems comprising two pairs of stereo EO cameras and one pair of IR cameras were employed ... The use of IR cameras provides the opportunity to validate stereo IR cameras as a viable option for stereo vision in the AAR domain) but does not explicitly disclose the following limitation as further recited however Hao discloses
receiving a depth image from a depth camera of the imaging device, wherein the depth camera is a different type of camera than the imaging camera, and wherein the depth image is generated independently of the 2D image (Hao, page 1, 1 Introduction, a 3-D ToF camera … can provide both range and intensity information at video frame rates ... we propose a scale-unambiguous relative pose estimation algorithm for space uncooperative targets based on the fusion of a 3-D ToF camera and a monocular camera. In our work, the monocular camera is used to obtain a high-resolution intensity image to provide frame-to-frame feature points’ tracking. The 3-D ToF camera can provide a range image in real time to reconstruct the depth values of feature points ... in our work, both the observer and target are freely moving);
determining 2D keypoints of the second device located within the 2D image based on the depth image, the 2D image values (Hao, page 2, 2 Feature Points Range Reconstruction, We assume that the monocular camera and 3-D ToF camera are mounted in a rigid binocular configuration. Once the intrinsic and external parameters of the two cameras have been calibrated, the range–intensity image can be registered using ... a geometrical model; Hao, pages 5-6, 4 Overall Scale Factor Estimation Algorithm, the overall scale factor estimation algorithm based on the range-intensity fusion image, which takes the range reconstruction uncertainty of feature points as the measurement noise, is proposed for the final scale-unambiguous pose estimation. As seen in Fig. 3, when the range image of the 3-D ToF camera is mapped onto the 2-D monocular image, each reconstructed feature point will have a range measurement. The 3-D point A on the target body is the one of the feature points and A0 0 is its projection point on the monocular image. The 3-D point O is defined as the original point of the target body frame ... Once the overall scale factor is determined, the scale unambiguous pose can be obtained);
determining a pose using the 2D keypoints and corresponding three-dimensional (3D) keypoints (Hao, page 1, 1 Introduction, Relative pose estimation … is critical for autonomous rendezvous and docking; Hao, pages 5-6, 4 Overall Scale Factor Estimation Algorithm, Once the overall scale factor is determined, the scale unambiguous pose can be obtained; Hao, pages 7-9, 5.2.2 Relative pose estimation, The tracking results are shown in Figs. 9 and 10. It is observed that not only the relative pose estimator proposed tracks orientation and position successfully, but also the 3-D locations of the four feature points are obtained, as shown in Fig. 11. The estimation accuracy of the feature points’ 3-D location is important for the whole pose estimation. Figure 11 compares the estimated 3-D location and the truth 3-D location of each feature point (P1, P2, P3, and P4). Qualitatively it shows good agreement between the truth and the estimation of the feature points 3-D locations at each time step).
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine the teachings of Hao and Lee because they are in the same field of endeavor. One skilled in the art would have been motivated to substitute the monocular camera and 3D time-of-flight camera set as taught by Hao for the electro-optical and infrared camera sets as taught by Lee and as alternate means to determine depth and distance from target airborne craft (Hao, page 1, 1 Introduction).
As per claim 2, Lee and Hao disclose the method of claim 1. Hao discloses wherein the depth camera comprises a range sensing device (Hao, page 1, 1 Introduction, a 3-D ToF camera, which uses an innovative approach to obtain 3-D images, can provide both range and intensity information at video frame rates ... we propose a scale-unambiguous relative pose estimation algorithm for space uncooperative targets based on the fusion of a 3-D ToF camera and a monocular camera … The 3-D ToF camera can provide a range image in real time to reconstruct the depth values of feature points). The motivation would be the same as above in claim 1.
As per claim 3, Lee and Hao disclose the method of claim 1, wherein the depth camera comprises a camera with depth sensing capabilities (Lee, Abstract, generate a 3-D point cloud of the target object using stereo vision; Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation, Stereo vision finds features in images and, after a calibration, undistortion, and rectification process, reprojects these features into 3-D space relative to the cameras ... The stereo block matching algorithm locates features in both images and calculates the disparity, or distance in pixel space, between them. Once disparities have been calculated for an image pair, the disparity map can be reprojected into space to create a 3-D point cloud; Hao, page 1, 1 Introduction, a 3-D ToF camera, which uses an innovative approach to obtain 3-D images, can provide both range and intensity information at video frame rates ... we propose a scale-unambiguous relative pose estimation algorithm for space uncooperative targets based on the fusion of a 3-D ToF camera and a monocular camera … The 3-D ToF camera can provide a range image in real time to reconstruct the depth values of feature points).
As per claim 4, Lee and Hao disclose the method of claim 1, wherein the 2D image values comprise multiple colors or intensities (Lee, Abstract, stereoscopic (stereo) vision systems; Lee, page 634, I. Introduction, The greatest challenge for an AAR computer vision system is the long range from the cameras on a tanker to the refueling contact point. The contact point is approximately 30 m from the stereoscopic (stereo) cameras; Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation, a 3-D model of the environment from2D images ... Stereo vision finds features in images ... The stereo block matching algorithm locates features in both images and calculates the disparity, or distance in pixel space ... Stereo block matching requires a series of pixelwise comparisons).
As per claim 5, Lee and Hao disclose the method of claim 4, wherein the 2D image values comprise red, green, and blue (RGB) values (Lee, page 636, III.A.1. Stereo Camera System, Two separate stereo vision systems comprising two pairs of stereo EO cameras and one pair of IR cameras were employed ... Allied Vision Proscilica GT1290C EO cameras were chosen for the low-resolution EO stereo vision system. The GT1290Cs capture 24 bit red, green, blue (RGB) images at a resolution of 1280 × 960 and have adjustable focal points and apertures).
As per claim 6, Lee and Hao disclose the method of claim 5, further comprising training a neural network using the RGB values and the depth image to produce estimates of 2D keypoints, wherein determining the 2D keypoints uses the neural network (Lee, page 634, I. Introduction, To replicate this nature-based approach, we use a convolutional neural network (CNN) to localize the receiver in each image. Then, the more computationally expensive image processing steps need only be applied to the region containing the receiver. In this paper, we contribute the following: 1) We demonstrate a novel deep learning approach to speed up stereo vision-based point-cloud generation; Lee, page 643, IV. C. CNN Application Procedure, the CNN’s bounding box is used to reduce the computational cost of stereo block matching. Once the stereo images are captured, the left image is downsampled from the original resolution to 512 × 386 and passed as input to the CNN ... The bounding box is then used to mask a precomputed rectification map ... The captured images are remapped using this now-cropped rectification map into a final pair of rectified, undistorted, and cropped images. These images are then passed into OpenCV’s stereo block matcher to generate a disparity map. Finally, the disparity map is reprojected into 3-D space for use as a point cloud for pose registration; Lee, page 644, V. Conclusion, a CNN can identify and label several objects of interest in a stereo image pair and then perform the pose-estimation process quickly on each of them; Lee, page 644, Figure 16).
As per claim 7, Lee and Hao disclose the method of claim 6, wherein training the neural network comprises simulating sensor noise by randomly augmenting the depth image (Lee, page 635, II. C. Camera Resolution and Depth Estimation, In this subsection, we simulate the error in depth reprojection for a single point using extensive open source computer vision library ... With Gaussian noise and a 1 pixel standard deviation in both images, a 1280 × 960 image resulted in a 0.4598 m MAE in distance from the cameras. By using a higher-resolution camera of 4896 × 3264, a 0.38 m MAE is achieved; Lee, page 642, IV. B. 1. Data, The project used 5000 pairs for training and validation and 500 pairs for testing ... Each of the training images was blurred using a 3 × 3 low-pass blurring filter to help prevent the model from overfitting potential sharp edges in the simulated imagery).
As per claim 8, Lee discloses a tanker aircraft (Lee, page 634, I. Introduction, allowing the tanker to automatically control the receiver during the refueling process) comprising:
a camera system comprising an imaging camera configured to generate a two-dimensional (2D) image of a receiving aircraft and a depth image of the receiving aircraft (Lee, Abstract, aerial refueling …generate a 3-D point cloud of the target object using stereo vision; Lee, page 636, III.A.1. Stereo Camera System, Figure 2 shows the stereo camera configuration for the low-resolution EO cameras and the IR cameras. The cameras were configured to trigger on a hardware signal controlled by the collection program ... Images were collected at 10 Hz; Lee, page 634, I. Introduction, allowing the tanker to automatically control the receiver during the refueling process; Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation, Stereo vision finds features in images and, after a calibration, undistortion, and rectification process, reprojects these features into 3-D space relative to the cameras ... The stereo block matching algorithm locates features in both images and calculates the disparity, or distance in pixel space, between them. Once disparities have been calculated for an image pair, the disparity map can be reprojected into space to create a 3-D point cloud);
a processor; and non-transitory computer readable storage media storing code (Lee, Abstract, computer vision pipeline), the code being executable by the processor to perform operations comprising:
detecting the receiving aircraft within the 2D image based on 2D image values (Lee, page 634, I. Introduction, allowing the tanker to automatically control the receiver during the refueling process ... the system must have a high-precision relative pose-estimation process that tracks the receiver in real time … we implement a vision system that achieves such requirements ... range from the cameras on a tanker to the refueling contact point. The contact point is approximately 30 m from the stereoscopic (stereo) cameras ... increase the stereo image resolution ... localize the receiver in each image ... image processing steps need only be applied to the region containing the receiver);
determining 2D keypoints of the receiving aircraft located within the 2D image based on the depth image, the 2D image values, and a previously determined keypoint model (Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation, Stereo vision finds features in images and, after a calibration, undistortion, and rectification process, reprojects these features into 3-D space relative to the cameras ... The stereo block matching algorithm locates features in both images and calculates the disparity, or distance in pixel space, between them. Once disparities have been calculated for an image pair, the disparity map can be reprojected into space to create a 3-D point cloud; Lee, page 637, III.A.4. Pseudotanker and Pseudoreceiver, For pose registration, a reference point cloud (red) is matched onto a sensed point cloud (yellow) using the ICP. Figure 4 shows the reference point cloud for the pseudoreceiver and an example of a sensed point cloud. The reference point cloud is assumed to be a known, uniformly sampled, geometrically accurate model of the approaching receiver);
determining a 6 degree-of-freedom (6DOF) pose using the 2D keypoints and corresponding three-dimensional (3D) keypoints (Lee, Abstract, three-dimensional (3-D) pose-estimation; Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation, Stereo vision finds features in images and, after a calibration, undistortion, and rectification process, reprojects these features into 3-D space relative to the cameras ... The stereo block matching algorithm locates features in both images and calculates the disparity, or distance in pixel space, between them. Once disparities have been calculated for an image pair, the disparity map can be reprojected into space to create a 3-D point cloud ... Once a point cloud has been generated, there are many techniques to perform pose estimation. Since aircraft are rigid bodies, the point-to-point iterative closest point (ICP) [9] was chosen for this work; Lee, page 636, III.A.3. Truth Data, ICP returns a six-degree-of-freedom rigid-body registration); and
outputting a position of at least a component of the receiving aircraft based on the 6DOF pose (Lee, Abstract, three-dimensional (3-D) pose-estimation; Lee, page 634, I. Introduction, We perform pose estimation with an error of 6 cm at a target 30 m away from the camera system; Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation, Stereo vision finds features in images and, after a calibration, undistortion, and rectification process, reprojects these features into 3-D space relative to the cameras ... The stereo block matching algorithm locates features in both images and calculates the disparity, or distance in pixel space, between them. Once disparities have been calculated for an image pair, the disparity map can be reprojected into space to create a 3-D point cloud ... Once a point cloud has been generated, there are many techniques to perform pose estimation. Since aircraft are rigid bodies, the point-to-point iterative closest point (ICP) [9] was chosen for this work; Lee, page 636, III.A.3. Truth Data, ICP returns a six-degree-of-freedom rigid-body registration; Lee, page 636, III. Ground Experiment Design, Use ICP to register the receiver’s pose; Lee, page 637, III.A.4. Pseudotanker and Pseudoreceiver, For pose registration, a reference point cloud (red) is matched onto a sensed point cloud (yellow) using the ICP; Led, page 637, III.A.5. Running the Experiment, we applied the computer vision pipeline to estimate the pseudoreceiver’s pose. Figure 5 shows an example of registration being visualized in the virtual environment).
Lee further discloses (Lee, page 636, III.A.1. Stereo Camera System, Two separate stereo vision systems comprising two pairs of stereo EO cameras and one pair of IR cameras were employed ... The use of IR cameras provides the opportunity to validate stereo IR cameras as a viable option for stereo vision in the AAR domain), but does not explicitly disclose the following limitations as further recited however Hao discloses
a camera system comprising an imaging camera configured to generate a two-dimensional (2D) image of a receiving aircraft and a depth camera configured to generate a depth image of the receiving aircraft, wherein the depth camera is a different type of camera than the imaging camera, and wherein the depth image is generated independently of the 2D image (Hao, page 1, 1 Introduction, Relative pose estimation … is critical for autonomous rendezvous and docking … a 3-D ToF camera … can provide both range and intensity information at video frame rates ... we propose a scale-unambiguous relative pose estimation algorithm for space uncooperative targets based on the fusion of a 3-D ToF camera and a monocular camera. In our work, the monocular camera is used to obtain a high-resolution intensity image to provide frame-to-frame feature points’ tracking. The 3-D ToF camera can provide a range image in real time to reconstruct the depth values of feature points ... in our work, both the observer and target are freely moving);
determining 2D keypoints of the receiving aircraft located within the 2D image based on the depth image, the 2D image values (Hao, page 2, 2 Feature Points Range Reconstruction, We assume that the monocular camera and 3-D ToF camera are mounted in a rigid binocular configuration. Once the intrinsic and external parameters of the two cameras have been calibrated, the range–intensity image can be registered using ... a geometrical model; Hao, pages 5-6, 4 Overall Scale Factor Estimation Algorithm, the overall scale factor estimation algorithm based on the range-intensity fusion image, which takes the range reconstruction uncertainty of feature points as the measurement noise, is proposed for the final scale-unambiguous pose estimation. As seen in Fig. 3, when the range image of the 3-D ToF camera is mapped onto the 2-D monocular image, each reconstructed feature point will have a range measurement. The 3-D point A on the target body is the one of the feature points and A0 0 is its projection point on the monocular image. The 3-D point O is defined as the original point of the target body frame ... Once the overall scale factor is determined, the scale unambiguous pose can be obtained);
determining a pose using the 2D keypoints and corresponding three-dimensional (3D) keypoints (Hao, page 1, 1 Introduction, Relative pose estimation … is critical for autonomous rendezvous and docking; Hao, pages 5-6, 4 Overall Scale Factor Estimation Algorithm, Once the overall scale factor is determined, the scale unambiguous pose can be obtained; Hao, pages 7-9, 5.2.2 Relative pose estimation, The tracking results are shown in Figs. 9 and 10. It is observed that not only the relative pose estimator proposed tracks orientation and position successfully, but also the 3-D locations of the four feature points are obtained, as shown in Fig. 11. The estimation accuracy of the feature points’ 3-D location is important for the whole pose estimation. Figure 11 compares the estimated 3-D location and the truth 3-D location of each feature point (P1, P2, P3, and P4). Qualitatively it shows good agreement between the truth and the estimation of the feature points 3-D locations at each time step).
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine the teachings of Hao and Lee because they are in the same field of endeavor. One skilled in the art would have been motivated to substitute the monocular camera and 3D time-of-flight camera set as taught by Hao for the electro-optical and infrared camera sets as taught by Lee and as alternate means to determine depth and distance from target airborne craft (Hao, page 1, 1 Introduction).
As per claim 9, Lee and Hao disclose the tanker aircraft of claim 8, wherein the depth camera comprises a range sensing device (Lee, page 634, I. Introduction, allowing the tanker to automatically control the receiver during the refueling process; Hao, page 1, 1 Introduction, Relative pose estimation … is critical for autonomous rendezvous and docking … a 3-D ToF camera, which uses an innovative approach to obtain 3-D images, can provide both range and intensity information at video frame rates ... we propose a scale-unambiguous relative pose estimation algorithm for space uncooperative targets based on the fusion of a 3-D ToF camera and a monocular camera … The 3-D ToF camera can provide a range image in real time to reconstruct the depth values of feature points).
As per claim 10, Lee and Hao disclose the tanker aircraft of claim 8, wherein the depth camera comprises a camera with depth sensing capabilities (Lee, Abstract, generate a 3-D point cloud of the target object using stereo vision; Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation, Stereo vision finds features in images and, after a calibration, undistortion, and rectification process, reprojects these features into 3-D space relative to the cameras ... The stereo block matching algorithm locates features in both images and calculates the disparity, or distance in pixel space, between them. Once disparities have been calculated for an image pair, the disparity map can be reprojected into space to create a 3-D point cloud; Hao, page 1, 1 Introduction, a 3-D ToF camera, which uses an innovative approach to obtain 3-D images, can provide both range and intensity information at video frame rates ... we propose a scale-unambiguous relative pose estimation algorithm for space uncooperative targets based on the fusion of a 3-D ToF camera and a monocular camera … The 3-D ToF camera can provide a range image in real time to reconstruct the depth values of feature points).
As per claim 11, Lee and Hao disclose the tanker aircraft of claim 8, wherein the 2D image values comprise multiple colors or intensities (Lee, Abstract, stereoscopic (stereo) vision systems; Lee, page 634, I. Introduction, The greatest challenge for an AAR computer vision system is the long range from the cameras on a tanker to the refueling contact point. The contact point is approximately 30 m from the stereoscopic (stereo) cameras; Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation, a 3-D model of the environment from2D images ... Stereo vision finds features in images ... The stereo block matching algorithm locates features in both images and calculates the disparity, or distance in pixel space ... Stereo block matching requires a series of pixelwise comparisons).
As per claim 12, Lee and Hao disclose the tanker aircraft of claim 11, wherein the 2D image values comprise red, green, and blue (RGB) values (Lee, page 636, III.A.1. Stereo Camera System, Two separate stereo vision systems comprising two pairs of stereo EO cameras and one pair of IR cameras were employed ... Allied Vision Proscilica GT1290C EO cameras were chosen for the low-resolution EO stereo vision system. The GT1290Cs capture 24 bit red, green, blue (RGB) images at a resolution of 1280 × 960 and have adjustable focal points and apertures).
As per claim 13, Lee and Hao disclose the tanker aircraft of claim 12, wherein the processor performs further operations comprising training a neural network using the RGB values and the depth image to produce estimates of 2D keypoints, wherein determining the 2D keypoints uses the neural network (Lee, page 634, I. Introduction, To replicate this nature-based approach, we use a convolutional neural network (CNN) to localize the receiver in each image. Then, the more computationally expensive image processing steps need only be applied to the region containing the receiver. In this paper, we contribute the following: 1) We demonstrate a novel deep learning approach to speed up stereo vision-based point-cloud generation; Lee, page 643, IV. C. CNN Application Procedure, the CNN’s bounding box is used to reduce the computational cost of stereo block matching. Once the stereo images are captured, the left image is downsampled from the original resolution to 512 × 386 and passed as input to the CNN ... The bounding box is then used to mask a precomputed rectification map ... The captured images are remapped using this now-cropped rectification map into a final pair of rectified, undistorted, and cropped images. These images are then passed into OpenCV’s stereo block matcher to generate a disparity map. Finally, the disparity map is reprojected into 3-D space for use as a point cloud for pose registration; Lee, page 644, V. Conclusion, a CNN can identify and label several objects of interest in a stereo image pair and then perform the pose-estimation process quickly on each of them; Lee, page 644, Figure 16).
As per claim 14, Lee and Hao disclose the tanker aircraft of claim 13, wherein training the neural network comprises simulating sensor noise by randomly augmenting the depth image (Lee, page 635, II. C. Camera Resolution and Depth Estimation, In this subsection, we simulate the error in depth reprojection for a single point using extensive open source computer vision library ... With Gaussian noise and a 1 pixel standard deviation in both images, a 1280 × 960 image resulted in a 0.4598 m MAE in distance from the cameras. By using a higher-resolution camera of 4896 × 3264, a 0.38 m MAE is achieved; Lee, page 642, IV. B. 1. Data, The project used 5000 pairs for training and validation and 500 pairs for testing ... Each of the training images was blurred using a 3 × 3 low-pass blurring filter to help prevent the model from overfitting potential sharp edges in the simulated imagery).
As per claim 15, Lee discloses a refueling system (Lee, Abstract, aerial refueling) comprising:
a processor; and non-transitory computer readable storage media storing code (Lee, Abstract, computer vision pipeline), the code being executable by the processor to perform operations comprising:
receiving a two-dimensional (2D) image from an imaging camera of an imaging device of a first device (Lee, Abstract, stereoscopic (stereo) vision systems; Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation, Stereo vision finds features in images; Lee, page 636, III.A.1. Stereo Camera System, Figure 2 shows the stereo camera configuration for the low-resolution EO cameras and the IR cameras. The cameras were configured to trigger on a hardware signal controlled by the collection program ... Images were collected at 10 Hz);
detecting a second device within the 2D image based on 2D image values (Lee, page 634, I. Introduction, allowing the tanker to automatically control the receiver during the refueling process ... the system must have a high-precision relative pose-estimation process that tracks the receiver in real time … we implement a vision system that achieves such requirements ... range from the cameras on a tanker to the refueling contact point. The contact point is approximately 30 m from the stereoscopic (stereo) cameras ... increase the stereo image resolution ... localize the receiver in each image ... image processing steps need only be applied to the region containing the receiver);
receiving a depth image from the imaging device (Lee, Abstract, generate a 3-D point cloud of the target object; Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation, Stereo vision finds features in images and, after a calibration, undistortion, and rectification process, reprojects these features into 3-D space relative to the cameras ... The stereo block matching algorithm locates features in both images and calculates the disparity, or distance in pixel space, between them. Once disparities have been calculated for an image pair, the disparity map can be reprojected into space to create a 3-D point cloud);
determining 2D keypoints of the second device located within the 2D image based on the depth image, the 2D image values, and a previously determined keypoint model (Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation, Stereo vision finds features in images and, after a calibration, undistortion, and rectification process, reprojects these features into 3-D space relative to the cameras ... The stereo block matching algorithm locates features in both images and calculates the disparity, or distance in pixel space, between them. Once disparities have been calculated for an image pair, the disparity map can be reprojected into space to create a 3-D point cloud; Lee, page 637, III.A.4. Pseudotanker and Pseudoreceiver, For pose registration, a reference point cloud (red) is matched onto a sensed point cloud (yellow) using the ICP. Figure 4 shows the reference point cloud for the pseudoreceiver and an example of a sensed point cloud. The reference point cloud is assumed to be a known, uniformly sampled, geometrically accurate model of the approaching receiver);
determining a 6 degree-of-freedom (6DOF) pose using the 2D keypoints and corresponding three-dimensional (3D) keypoints (Lee, Abstract, three-dimensional (3-D) pose-estimation; Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation, Stereo vision finds features in images and, after a calibration, undistortion, and rectification process, reprojects these features into 3-D space relative to the cameras ... The stereo block matching algorithm locates features in both images and calculates the disparity, or distance in pixel space, between them. Once disparities have been calculated for an image pair, the disparity map can be reprojected into space to create a 3-D point cloud ... Once a point cloud has been generated, there are many techniques to perform pose estimation. Since aircraft are rigid bodies, the point-to-point iterative closest point (ICP) [9] was chosen for this work; Lee, page 636, III.A.3. Truth Data, ICP returns a six-degree-of-freedom rigid-body registration); and
outputting a position of at least a component of the second device based on the 6DOF pose (Lee, Abstract, three-dimensional (3-D) pose-estimation; Lee, page 634, I. Introduction, We perform pose estimation with an error of 6 cm at a target 30 m away from the camera system; Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation, Stereo vision finds features in images and, after a calibration, undistortion, and rectification process, reprojects these features into 3-D space relative to the cameras ... The stereo block matching algorithm locates features in both images and calculates the disparity, or distance in pixel space, between them. Once disparities have been calculated for an image pair, the disparity map can be reprojected into space to create a 3-D point cloud ... Once a point cloud has been generated, there are many techniques to perform pose estimation. Since aircraft are rigid bodies, the point-to-point iterative closest point (ICP) [9] was chosen for this work; Lee, page 636, III.A.3. Truth Data, ICP returns a six-degree-of-freedom rigid-body registration; Lee, page 636, III. Ground Experiment Design, Use ICP to register the receiver’s pose; Lee, page 637, III.A.4. Pseudotanker and Pseudoreceiver, For pose registration, a reference point cloud (red) is matched onto a sensed point cloud (yellow) using the ICP; Led, page 637, III.A.5. Running the Experiment, we applied the computer vision pipeline to estimate the pseudoreceiver’s pose. Figure 5 shows an example of registration being visualized in the virtual environment).
Lee further discloses (Lee, page 636, III.A.1. Stereo Camera System, Two separate stereo vision systems comprising two pairs of stereo EO cameras and one pair of IR cameras were employed ... The use of IR cameras provides the opportunity to validate stereo IR cameras as a viable option for stereo vision in the AAR domain) but does not explicitly disclose the following limitations as further recited however Hao discloses
receiving a depth image from a depth camera of the imaging device, wherein the depth camera is a different type of camera than the imaging camera, and wherein the depth image is generated independently of the 2D image (Hao, page 1, 1 Introduction, a 3-D ToF camera … can provide both range and intensity information at video frame rates ... we propose a scale-unambiguous relative pose estimation algorithm for space uncooperative targets based on the fusion of a 3-D ToF camera and a monocular camera. In our work, the monocular camera is used to obtain a high-resolution intensity image to provide frame-to-frame feature points’ tracking. The 3-D ToF camera can provide a range image in real time to reconstruct the depth values of feature points ... in our work, both the observer and target are freely moving);
determining 2D keypoints of the second device located within the 2D image based on the depth image, the 2D image values (Hao, page 2, 2 Feature Points Range Reconstruction, We assume that the monocular camera and 3-D ToF camera are mounted in a rigid binocular configuration. Once the intrinsic and external parameters of the two cameras have been calibrated, the range–intensity image can be registered using ... a geometrical model; Hao, pages 5-6, 4 Overall Scale Factor Estimation Algorithm, the overall scale factor estimation algorithm based on the range-intensity fusion image, which takes the range reconstruction uncertainty of feature points as the measurement noise, is proposed for the final scale-unambiguous pose estimation. As seen in Fig. 3, when the range image of the 3-D ToF camera is mapped onto the 2-D monocular image, each reconstructed feature point will have a range measurement. The 3-D point A on the target body is the one of the feature points and A0 0 is its projection point on the monocular image. The 3-D point O is defined as the original point of the target body frame ... Once the overall scale factor is determined, the scale unambiguous pose can be obtained);
determining a pose using the 2D keypoints and corresponding three-dimensional (3D) keypoints (Hao, page 1, 1 Introduction, Relative pose estimation … is critical for autonomous rendezvous and docking; Hao, pages 5-6, 4 Overall Scale Factor Estimation Algorithm, Once the overall scale factor is determined, the scale unambiguous pose can be obtained; Hao, pages 7-9, 5.2.2 Relative pose estimation, The tracking results are shown in Figs. 9 and 10. It is observed that not only the relative pose estimator proposed tracks orientation and position successfully, but also the 3-D locations of the four feature points are obtained, as shown in Fig. 11. The estimation accuracy of the feature points’ 3-D location is important for the whole pose estimation. Figure 11 compares the estimated 3-D location and the truth 3-D location of each feature point (P1, P2, P3, and P4). Qualitatively it shows good agreement between the truth and the estimation of the feature points 3-D locations at each time step).
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine the teachings of Hao and Lee because they are in the same field of endeavor. One skilled in the art would have been motivated to substitute the monocular camera and 3D time-of-flight camera set as taught by Hao for the electro-optical and infrared camera sets as taught by Lee and as alternate means to determine depth and distance from target airborne craft (Hao, page 1, 1 Introduction).
As per claim 16, Lee and Hao disclose the refueling system of claim 15, wherein the depth camera comprises a range sensing device or a camera with depth sensing capabilities (Hao, page 1, 1 Introduction, a 3-D ToF camera, which uses an innovative approach to obtain 3-D images, can provide both range and intensity information at video frame rates ... we propose a scale-unambiguous relative pose estimation algorithm for space uncooperative targets based on the fusion of a 3-D ToF camera and a monocular camera … The 3-D ToF camera can provide a range image in real time to reconstruct the depth values of feature points; Lee, Abstract, generate a 3-D point cloud of the target object using stereo vision; Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation, Stereo vision finds features in images and, after a calibration, undistortion, and rectification process, reprojects these features into 3-D space relative to the cameras ... The stereo block matching algorithm locates features in both images and calculates the disparity, or distance in pixel space, between them. Once disparities have been calculated for an image pair, the disparity map can be reprojected into space to create a 3-D point cloud).
As per claim 17, Lee and Hao disclose the refueling system of claim 15, wherein the 2D image values comprise multiple colors or intensities (Lee, Abstract, stereoscopic (stereo) vision systems; Lee, page 634, I. Introduction, The greatest challenge for an AAR computer vision system is the long range from the cameras on a tanker to the refueling contact point. The contact point is approximately 30 m from the stereoscopic (stereo) cameras; Lee, page 635, II. B. Six-Degree-of-Freedom Pose Estimation, a 3-D model of the environment from2D images ... Stereo vision finds features in images ... The stereo block matching algorithm locates features in both images and calculates the disparity, or distance in pixel space ... Stereo block matching requires a series of pixelwise comparisons).
As per claim 18, Lee and Hao disclose the refueling system of claim 17, wherein the 2D image values comprise red, green, and blue (RGB) values (Lee, page 636, III.A.1. Stereo Camera System, Two separate stereo vision systems comprising two pairs of stereo EO cameras and one pair of IR cameras were employed ... Allied Vision Proscilica GT1290C EO cameras were chosen for the low-resolution EO stereo vision system. The GT1290Cs capture 24 bit red, green, blue (RGB) images at a resolution of 1280 × 960 and have adjustable focal points and apertures).
As per claim 19, Lee and Hao disclose the refueling system of claim 18, wherein the processor performs further operations comprising training a neural network using the RGB values and the depth image to produce estimates of 2D keypoints, wherein determining the 2D keypoints uses the neural network (Lee, page 634, I. Introduction, To replicate this nature-based approach, we use a convolutional neural network (CNN) to localize the receiver in each image. Then, the more computationally expensive image processing steps need only be applied to the region containing the receiver. In this paper, we contribute the following: 1) We demonstrate a novel deep learning approach to speed up stereo vision-based point-cloud generation; Lee, page 643, IV. C. CNN Application Procedure, the CNN’s bounding box is used to reduce the computational cost of stereo block matching. Once the stereo images are captured, the left image is downsampled from the original resolution to 512 × 386 and passed as input to the CNN ... The bounding box is then used to mask a precomputed rectification map ... The captured images are remapped using this now-cropped rectification map into a final pair of rectified, undistorted, and cropped images. These images are then passed into OpenCV’s stereo block matcher to generate a disparity map. Finally, the disparity map is reprojected into 3-D space for use as a point cloud for pose registration; Lee, page 644, V. Conclusion, a CNN can identify and label several objects of interest in a stereo image pair and then perform the pose-estimation process quickly on each of them; Lee, page 644, Figure 16).
As per claim 20, Lee and Hao disclose the refueling system of claim 19, wherein training the neural network comprises simulating sensor noise by randomly augmenting the depth image (Lee, page 635, II. C. Camera Resolution and Depth Estimation, In this subsection, we simulate the error in depth reprojection for a single point using extensive open source computer vision library ... With Gaussian noise and a 1 pixel standard deviation in both images, a 1280 × 960 image resulted in a 0.4598 m MAE in distance from the cameras. By using a higher-resolution camera of 4896 × 3264, a 0.38 m MAE is achieved; Lee, page 642, IV. B. 1. Data, The project used 5000 pairs for training and validation and 500 pairs for testing ... Each of the training images was blurred using a 3 × 3 low-pass blurring filter to help prevent the model from overfitting potential sharp edges in the simulated imagery).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TRACY MANGIALASCHI whose telephone number is (571)270-5189. The examiner can normally be reached M-F, 9:30AM TO 6:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be reached at (571) 272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TRACY MANGIALASCHI/Examiner, Art Unit 2668
/VU LE/Supervisory Patent Examiner, Art Unit 2668