Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Applicant’s election with traverse of Group II, including claims 13-20, in the reply filed on 11/14/2025 is acknowledged. Upon reconsideration, and after conducting a search with respect to the elected Group II claims, Examiner has determined that no significant additional search would be required to examine the non-elected claims. Accordingly, the restriction requirement is withdrawn, and claims 1-20 will be examined together as set forth in this Office Action.
Specification
The abstract of the disclosure is objected to because of it is not limited to a single paragraph preferably within the range of 50 to 150 words in length. Note the inclusion of the ANNEX A section in the provided abstract necessitates the objection. A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 12 and 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 12 and 20 recite “determining a centroid based on a respective amount of the fiducial marker associated with each visual representation value”, which lacks antecedent basis. For example, the claims do not previously recite any visual representation values, or describe fiducial markers as being associated with such values. Thus, it is unclear what “visual representation value” is meant to refer to or how such values are associated with the fiducial markers for purposes of determining the centroid. For examination purposes, the limitation will be interpreted to as “determining a centroid based on a respective amount of the fiducial marker associated with a visual representation value.”
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 13 is rejected under 35 U.S.C. 102(a)(1) as being anticipated by Mozer et al. (US 20190197908 A1), (hereinafter Mozer).
Regarding claim 13, Mozer teaches an unmanned aerial vehicle comprising:
a processing circuit (Mozer, “The control system can include one or more microcontrollers, microprocessors, digital signal processors, application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or any general-purpose or special-purpose circuitry that can be programmed or configured to perform the various functions described herein.”, pgs. 1 and 2, paragraph 0020) configured to:
descend the unmanned aerial vehicle toward a landing platform, wherein a plurality of fiducial markers are disposed on the landing platform (Mozer, “Computer vision techniques are used in accordance with one or more embodiments to improve the precision of the autonomous drone landing, and thus the reliability of a successful docking event with a docking station. In accordance with one or more embodiments, one or more fiducial markers, such as light-emitting beacons, of known position and arrangement are configured at the landing target.”, pg. 2, paragraph 0021, lines 1-7,);
detect, as the unmanned aerial vehicle descends, at least one fiducial marker among a first subset of the plurality of fiducial markers when the unmanned aerial vehicle is within a first altitude range from the landing platform; and detect, as the unmanned aerial vehicle descends, at least one fiducial marker among a second subset of the plurality of fiducial markers when the unmanned aerial vehicle is within a second altitude range from the landing platform (Mozer, “The landing procedure for an aircraft in this scenario naturally involves starting at farther distances and approaching towards the target until the aircraft has landed... In accordance with one or more embodiments, to overcome this technical hurdle, a set of progressively smaller constellations are used that are appropriate for each stage of the descent, guiding the aircraft into its final, precise location. By way of example, as shown in FIG. 6, such constellations can comprise a series of nested circles 144 (each circle comprising multiple fiducials 140 arranged in a circular pattern) with decreasing diameters. FIG. 5 shows constellations comprising a series of squares 142 (each square comprising multiple fiducials 140 arranged in a square pattern) with decreasing dimensions. FIG. 7 shows a series of lines 146 (each line comprising multiple fiducials 140 arranged in a line). Suitable fiducials systems could include any combination or permutation of fiducial constellations that get progressively smaller (i.e. closer to the center point of the camera FOY) as the aircraft approaches the landing target.”, pg. 2, paragraph 0028-0031, see Fig. 4 and 6, During drone descends, subsets of fiducial markers, such as the nested circles of Fig. 6, are detected within different altitude ranges.).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Mozer et al. (US 20190197908 A1) in view of Jourdan et al. (US 20200301445 A1), (hereinafter Jourdan).
Regarding claim 14, Mozer teaches the unmanned aerial vehicle of claim 13, wherein the first subset comprises a first fiducial marker and a second fiducial maker having the same or similar sizes, and wherein the second subset comprises a third fiducial marker and a fourth fiducial marker having the same or similar sizes and are smaller than the first and the second fiducial markers (Mozer, “By way of example, as shown in FIG. 6, such constellations can comprise a series of nested circles 144 (each circle comprising multiple fiducials 140 arranged in a circular pattern) with decreasing diameters.”, pg. 2, paragraph 0031, lines 5-8, Each subset or nested circle contains fiducial markers of the same size with the inner rings having a smaller constellation diameter than those of the outer rings.), and
wherein the processing circuit is further configured to: land the unmanned aerial vehicle (Mozer, “At step 210, the vision system provides the position/orientation information to the flight controller, which guides the drone to the landing site. These steps are continuously repeated until the drone has successfully landed at the landing site. The camera 114 continuously captures images, e.g., at 50 frames per second. The image analysis described above is repeated for each frame”, pg. 3, paragraphs 0046-0047).
Mozer does not teach detect at least one fiducial marker among a third subset of the plurality of fiducial markers when the unmanned aerial vehicle is on the landing platform.
However, Jourdan teaches detect at least one fiducial marker among a third subset of the plurality of fiducial markers when the unmanned aerial vehicle is on the landing platform (Jourdan, “The fiducial disturbance module 268 ensures that the UAV 100 recalibrates its stored position in the event that it is moved more than a threshold amount while landed. The fiducial disturbance module 268 causes the UAV 100 to determine its navigation solution using the FNS 204 immediately prior to landing, or while landed. Then, the fiducial disturbance module 268 causes the UAV 100 to monitor its position while landed (using any of the available navigation systems, e.g., GPS and/or IMU). If the fiducial disturbance module 268 detects that the position of the UAV 100 changes more than a permissible threshold level (e.g., 0.1 meter, 0.5 meter, 1.0 meter, etc.), then it invalidates the navigation solution determined immediately prior to landing. Subsequently, the fiducial disturbance module 268 causes the UAV 100 to re-acquire its position using one or more available navigation systems. For example, the UAV 100 may re-acquire its position by imaging a fiducial marker with the camera 140.”, pg. 13, paragraph 0090, Fiducial markers are used after landing to verify that the UAV has not moved within a given threshold.).
Mozer teaches detecting fiducial markers to guide a landing operation of a UAV on a landing platform (Mozer, paragraphs 0027-0031, see Figs. 4 and 6). Mozer does not teach detecting additional fiducial markers when the UAV is on the landing platform. Jourdan teaches monitoring a UAV’s position with respect to a landing platform after a landing operation has been performed (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Mozer to include UAV position monitoring post landing as taught by Jourdan (Jourdan, pg. 13, paragraph 0090). The motivation for doing so would have been to maintain correct alignment between the UAV and the landing platform after landing, thereby reducing the risk of misalignment during subsequent take-off operations. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Mozer with Jourdan to obtain the invention as specified in claim 14.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Mozer et al. (US 20190197908 A1) in view of Stabler et al. (US 20160257424 A1), (hereinafter Stabler).
Regarding claim 15, Mozer teaches the unmanned aerial vehicle of claim 13, wherein the processing circuit is further configured to adjust a trajectory of the unmanned aerial vehicle and/or a landing position of the unmanned aerial vehicle based at least on detections of one or more of the plurality of fiducial markers as the unmanned aerial vehicle descends (Mozer, “Computer vision techniques are used in accordance with one or more embodiments to improve the precision of the autonomous drone landing, and thus the reliability of a successful docking event with a docking station. In accordance with one or more embodiments, one or more fiducial markers, such as light-emitting beacons, of known position and arrangement are configured at the landing target. The fiducials along with the camera 114 mounted on the drone aircraft in a known position and orientation, enable highspeed state estimation of the aircraft relative to the landing target. This state estimate, i.e., relative position and/or orientation, is used to control the aircraft precisely during the descent until successful landing has been achieved.”, pg. 2, paragraph 0021).
Mozer does not teach wherein the landing platform comprises a launch and recovery box (LRB), and wherein the processing circuit is further configured to: receive orientation data from the LRB; orient the unmanned aerial vehicle based on the orientation data; and descend the unmanned aerial vehicle toward the landing platform with the unmanned aerial vehicle oriented according to the orientation data.
However, Stabler teaches wherein the landing platform comprises a launch and recovery box (LRB), and wherein the processing circuit is further configured to: receive orientation data from the LRB; orient the unmanned aerial vehicle based on the orientation data; and descend the unmanned aerial vehicle toward the landing platform with the unmanned aerial vehicle oriented according to the orientation data (Stabler, “The present disclosure provides systems and methods that permit an unmanned aerial vehicle (UAV) to land on a landing pad of a landing station and take off from the landing pad.”, pg. 1, paragraph 0007, lines 1-4, “The landing pad can further comprise a plurality of markers in a predetermined geometric configuration on the landing pad. The markers can be detectable by the UAV to identify a location of the landing pad relative to a given location of the UAV. The UAV can detect one or more markers in the plurality of markers with a vision sensor on-board the UAV. The vision sensor can comprise a camera. The camera can be a charge-coupled device (CCD) camera. The camera can be a complementary metal-oxide semiconductor (CMOS) camera. Alternatively, the vision sensor can be provided on the landing pad and the UAV can comprise a plurality of markers in a given or predetermined geometric configuration on the UAV.”, pg. 6, paragraph 068, “The UAV can comprise one or more computer processors configured to calculate the orientation and/or position of the UAV relative to a detected marker. In some cases, the computer processors can be on-board the UAV. Alternatively, the computer processors can be off-board the UAV. The computer processors can be on the landing pad.”, pg. 6, paragraph 0073, lines 1-9, A launch and recovery system includes collecting images through a camera fixed to the landing pad and using a processor fixed to the landing pad to detect fiducial markers on the UAV to determine orientation information for landing. This orientation information includes guidance or control information which is communicated to the UAV to facilitate landing operations.).
Mozer teaches guiding a UAV to a landing target by detecting fiducial markers (Mozer, see paragraphs 0027-0031, see Fig. 9). Stabler teaches a launch and recovery system that determines orientation information for transmittal to a UAV for executing landing operations (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Mozer to include the launch and recovery system as taught by Stabler (Stabler, pg. 6, paragraph 068 and pg. 6, paragraph 0073, lines 1-9). The motivation for doing so would have been to perform fiducial marker detection and orientation calculations at the launch and recovery system rather than onboard the UAV, thereby reducing UAV computational load. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Mozer with Stabler to obtain the invention as specified in claim 15.
Claims 16 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Mozer et al. (US 20190197908 A1) in view of Zhenglong et al. (“Pose estimation for multicopters based on monocular vision and AprilTag”, 2018 37th Chinese Control Conference (CCC). IEEE, 2018), (hereinafter Zhenglong) and further in view of Tang et al. (US 20220084238 A1), (hereinafter Tang) and Hui et al. (“Vision-based autonomous navigation approach for unmanned aerial vehicle transmission-line inspection”, International Journal of Advanced Robotic Systems, 2018), (hereinafter Hui).
Regarding claim 16, Mozer teaches the unmanned aerial vehicle of claim 13, further comprising a camera configured to capture an image (Mozer, “The drone aircraft 100 includes a control system 106 for controlling operation of the aircraft, a battery 108 for powering the aircraft, a set of rotors 110 driven by motors 112, a camera 114, and sensors 116.”, pg. 1, paragraph 0018, lines 3-7).
Mozer does not teach wherein the processing circuit is further configured to: identify one or more image contours in the image; determine a position of a first fiducial marker of the plurality of fiducial markers in the image; and determine a pose associated with the first fiducial marker based at least on the one or more image contours.
However, Zhenglong teaches wherein the processing circuit is further configured to: identify one or more image contours in the image; determine a position of a first fiducial marker of the plurality of fiducial markers in the image; and determine a pose associated with the first fiducial marker based at least on the one or more image contours (Zhenglong, “In computer vision, determining the relative pose between a 2D-tag and a calibrated camera is often known as a Perspective-n-Point (PnP) problem from n 2D-2D point correspondences [8]… we used the map composed of AprilTags and a downward-looking camera to estimate the pose of multicopters with a Kalman filter based on a linear constant-velocity process model, which is a kind of recursive methods.”, pg. 4717, 2nd column, 2nd full paragraph, “The Fig.8(a) is the input image. The first step is to detect line segments in the image. As shown in Fig.8(b) and (c), the approach computes the gradient direction and magnitude at every pixel. Using a graph-based method [20], pixels with similar gradient directions and magnitude are clustered into components shown in in Fig.8(d).”, pg. 4721, 1st column, 2nd full paragraph, lines 1-6, see Figs. 7 and 8, “Taking camera lens distortion into consideration, let (u, v) be the ideal pixel image coordinates, and (
u
~
,
v
~
) the corresponding real observed image coordinates. The ideal points are the projection of the model points according to the pinhole model… Here, the projection of the camera is from the 2D tag to a 2D image. Therefore, we use (13) and (15) to compute the relative pose between the tag plane and the image plane.”, pg. 4719, 1st column 2nd full paragraph, lines 1-5, Fiducial marker detection includes extracting contours corresponding to marker boundaries/line segments from input images and determining pixel positions of the marker. Once detected, planar feature points of the fiducial marker are projected using a pinhole camera model to determine the pose relative to the camera.).
Mozer teaches guiding a UAV to a landing target by detecting fiducial markers, namely light-emitting beacons, by estimating a relative pose between a camera on the UAV and the fiducial marker (Mozer, “In accordance with one or more embodiments, one or more fiducial markers, such as light-emitting beacons, of known position and arrangement are configured at the landing target. “, pg. 2, paragraph 0021, lines 4-7, see paragraphs 0027-0031, see Fig. 9). Mozer does not teach determining the pose of the fiducial markers using contour-based detection. Zhenglong teaches estimating a relative pose between a camera of a UAV and April tags by detecting marker contours and estimating the pose using the detected markers (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the fiducial marker pose estimation of Mozer to include pose estimation based on detected fiducial marker contours as taught by Zhenglong (Zhenglong, pg. 4717, 2nd column, 2nd full paragraph, pg. 4721, 1st column, 2nd full paragraph, lines 1-6, see Figs. 7 and 8). The motivation for doing so would have been to enable pose estimation based on marker geometry in addition to emitted light, thereby improving landing capabilities in environment where active illumination may be undesirable or impractical. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Mozer with Zhenglong to obtain the invention as specified above.
Mozer in view of Zhenglong does not teach project, based at least on the position, models associated with one or more contours of the first fiducial marker into an image plane of the camera to obtain one or more model contours; and determine a pose associated with the first fiducial marker based at least on the one or more image contours and the one or more model contours.
However, Tang teaches project, based at least on the position, models associated with one or more contours of the first fiducial marker into an image plane of the camera to obtain one or more model contours; and determine a pose associated with the first fiducial marker based at least on the one or more image contours and the one or more model contours (Tang, “FIG . 5 is an illustration 100 of how the PnP process may be implemented in this example to obtain the 3D pose of the object 16. The illustration 100 shows a 3D object 106, representing the object 16, at a ground truth or real location. The object 106 is observed by a camera 112, representing the camera 20 , and projected as a 2D object image 108 on a 2D image plane 110 , where the object image 108 represents the image 94 and where points 102 on the image 108 are feature points predicted by the neural network 78, representing the points 96. The illustration 100 also shows a virtual 3D CAD model 114 of the object 16 having feature points 104 at the same location as the feature points 96 that is randomly placed in front of the camera 112 and is projected as a 2D model image 116 on the image plane 110 also including projected feature points 118. The CAD model 114 is rotated and translated in front of the camera 112 , which rotates and translates the model image 116 in an attempt to minimize the distance between each of the feature points 118 on the model image 116 and the corresponding feature points 102 on the object image 108 , i.e. , align the images 116 and 108. Once the model image 116 is aligned with the object image 108 as best as possible, the pose of the CAD model 114 with respect to the camera 112 is the estimated 3D pose of the object 16.”, pg. 3, paragraph 0022, A 3D object model is projected into the camera image plane and pose is estimated by determining the rotation and translation which best aligns model feature points with detected image feature points.).
Mozer in view of Zhenglong teaches detecting contours of fiducial markers and estimating pose by projecting feature points of the fiducial marker into the image plane of a camera (Zhenglong, pg. 4719, 1st column 2nd full paragraph, lines 1-5, see Fig. 7). Mozer in view of Zhenglong does not teach projecting models associated with fiducial marker contours to obtain model contours. Tang teaches estimating object pose by projecting 3D models into the image plane of a camera and aligning projected model features with detected image features (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Mozer in view of Zhenglong to project a 3D fiducial marker model, rather than only feature points, for model-based pose estimation as taught by Tang (Tang, pg. 3, paragraph 0022, see Fig. 5). The motivation for doing so would have been to rely on additional geometric features of the projected model, thereby improving the reliability of pose estimation. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Mozer in view of Zhenglong with Tang to obtain the invention as specified above.
Mozer in view of Zenglong and further in view of Tang does not teach wherein the first fiducial marker is partially occluded in the image by a portion of the unmanned aerial vehicle.
However, Hei teaches wherein the first fiducial marker is partially occluded in the image by a portion of the unmanned aerial vehicle (Hui, “This article presents an autonomous navigation approach based on a transmission tower for unmanned aerial vehicle (UAV) power line inspection. For this complex vision task, a perspective navigation model, which plays an important role in the description and analysis of the flight strategy, is introduced.”, see abstract, lines 1-3, “At initialization stage, by detection, the UAV determines first the position of the next tower. Thereafter, the PTZ adjusts the pitch and yaw angle to make the camera optical axis point to the center of the detected transmission tower. At the stage of UAV heading adjustment,
p
A
is constantly calculated by eqaution (5) until it reaches the same horizontal position as VP. Since PTZ actively tracks tower, VP position has almost no change during the adjustment of flight heading… At the inspection stage, the UAV flies along the expected heading. With the detection-tracking visual strategy enabled, the PTZ constantly adjusts the yaw and pitch to track the next tower until the relative yaw angle
ψ
p
u
between camera optical axis and the expected heading reaches 90◦. During the flight, once the tracking fails, UAV will immediately brake. After that, the tracker will be initialized again using the recent detection result”, pg. 8, 2nd column, 3rd and 4th full paragraphs, “At the position A, PTZ, whose camera optical axis is denoted by the red arrow, adjusts the yaw and pitch angle to situate the detected transmission tower in the image center. Then, UAV rotates to the expected heading defined by VP. In order to avoid the occlusion to the camera caused by landing gear, from waypoint B, UAV continues to fly along the expected direction and simultaneously begins to rotate.”, pg. 12, 1st column, 2nd full paragraph, lines 10-17, see Fig. 12(a), A PTZ camera is mounted on a UAV and used to perform object detection for visual navigation. Active camera control mitigates the impact of images collected under partial occlusions, including self-occlusion caused by the UAV itself, by dynamically adjusting the viewing direction in order to collect sufficient visual information for navigation.).
Mozer in view of Zenglong and further in view of Tang teaches collecting images using a fixed camera on a UAV to guide a landing target by detecting fiducial markers (Mozer, see paragraphs 0027-0031, see Fig. 9). Mozer in view of Zenglong and further in view of Tang does not teach fiducial markers being partially occluded in the images. Hui teaches mounting a PTZ camera on a UAV and performing dynamic camera control to mitigate the impact of partially occluded objects for collected images, including instances of self-occlusion caused by UAV landing gear (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have replaced the fixed camera of Mozer in view of Zenglong and further in view of Tang with the PTZ camera and dynamic camera control as taught by Hui (Hui, pg. 8, 2nd column, 3rd and 4th full paragraphs, pg. 12, 1st column, 2nd full paragraph, lines 10-17, see Fig. 12(a)). The motivation for doing so would have been to improve robustness of fiducial marker detection under partial occlusion, thereby improving the reliability of UAV landing. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Mozer in view of Zhenglong and further in view of Tang with Hui to obtain the invention as specified in claim 16.
Regarding claim 17, Mozer in view of Zhenglong and further in view of Tang and Hei teaches the unmanned aerial vehicle of claim 16, wherein the models comprise three-dimensional (3D) models and the image plane comprises a two-dimensional (2D) image plane of the camera, wherein the processing circuit is further configured to perform contour matching on the one or more image contours and the one or more model contours, and wherein the pose is based on the contour matching (Tang, “The CAD model 114 is rotated and translated in front of the camera 112, which rotates and translates the model image 116 in an attempt to minimize the distance between each of the feature points 118 on the model image 116 and the corresponding feature points 102 on the object image 108 , i.e. , align the images 116 and 108. Once the model image 116 is aligned with the object image 108 as best as possible, the pose of the CAD model 114 with respect to the camera 112 is the estimated 3D pose of the object 16.”, pg. 3, paragraph 0022, The combination of Mozer in view of Zhenglong and further in view of Tang and Hui would perform rotation and translation matching between detected marker contours and stored 3D model marker contours for pose estimation.).
Claims 1 and 4 are rejected under 35 U.S.C. 103 as being unpatentable over Zhenglong et al. (“Pose estimation for multicopters based on monocular vision and AprilTag”, 2018 37th Chinese Control Conference (CCC). IEEE, 2018) in view of Tang et al. (US 20220084238 A1).
Regarding claim 1, Zhenglong teaches a method comprising:
capturing, by a camera of an unmanned aerial vehicle, a first image (Zhenglong, “In computer vision, determining the relative pose between a 2D-tag and a calibrated camera is often known as
a Perspective-n-Point (PnP) problem from n 2D-2D point correspondences [8]… we used the map composed of AprilTags and a downward-looking camera to estimate the pose of multicopters with a Kalman filter based on a linear constant-velocity process model, which is a kind of recursive methods.”, pg. 4717, 2nd column, 2nd full paragraph, see Fig. 2, camera, and Fig. 8(a), input image);
identifying one or more first image contours in the first image; determining a first position of a fiducial marker in the first image (Zhenglong, “The Fig.8(a) is the input image. The first step is to detect line segments in the image. As shown in Fig.8(b) and (c), the approach computes the gradient direction and magnitude at every pixel. Using a graph-based method [20], pixels with similar gradient directions and magnitude are clustered into components shown in in Fig.8(d).”, pg. 4721, 1st column, 2nd full paragraph, lines 1-6, see Figs. 7 and 8, Fiducial marker detection includes extracting contours corresponding to marker boundaries/line segments from input images and determining pixel positions of the marker.);
determining a first pose associated with the fiducial marker based at least on the one or more first image contours (Zhenglong, “Taking camera lens distortion into consideration, let (u, v) be the ideal pixel image coordinates, and (
u
~
,
v
~
) the corresponding real observed image coordinates. The ideal points are the projection of the model points according to the pinhole model… Here, the projection of the camera is from the 2D tag to a 2D image. Therefore, we use (13) and (15) to compute the relative pose between the tag plane and the image plane.”, pg. 4719, 1st column 2nd full paragraph, lines 1-5, Once detected, planar feature points of the fiducial marker are projected using a pinhole camera model to determine the pose relative to the camera.).
Zhenglong does not teach projecting, based at least on the first position, models associated with one or more contours of the fiducial marker into an image plane of the camera to obtain one or more first model contours; and determining a first pose associated with the fiducial marker based at least on the one or more first image contours and the one or more first model contours.
However, Tang teaches projecting, based at least on the first position, models associated with one or more contours of the fiducial marker into an image plane of the camera to obtain one or more first model contours; and determining a first pose associated with the fiducial marker based at least on the one or more first image contours and the one or more first model contours (Tang, “FIG . 5 is an illustration 100 of how the PnP process may be implemented in this example to obtain the 3D pose of the object 16. The illustration 100 shows a 3D object 106, representing the object 16, at a ground truth or real location. The object 106 is observed by a camera 112, representing the camera 20 , and projected as a 2D object image 108 on a 2D image plane 110 , where the object image 108 represents the image 94 and where points 102 on the image 108 are feature points predicted by the neural network 78, representing the points 96. The illustration 100 also shows a virtual 3D CAD model 114 of the object 16 having feature points 104 at the same location as the feature points 96 that is randomly placed in front of the camera 112 and is projected as a 2D model image 116 on the image plane 110 also including projected feature points 118. The CAD model 114 is rotated and translated in front of the camera 112 , which rotates and translates the model image 116 in an attempt to minimize the distance between each of the feature points 118 on the model image 116 and the corresponding feature points 102 on the object image 108 , i.e. , align the images 116 and 108. Once the model image 116 is aligned with the object image 108 as best as possible, the pose of the CAD model 114 with respect to the camera 112 is the estimated 3D pose of the object 16.”, pg. 3, paragraph 0022, A 3D object model is projected into the camera image plane and pose is estimated by determining the rotation and translation which best aligns model feature points with detected image feature points.).
Zhenglong teaches detecting contours of fiducial markers and estimating pose by projecting feature points of the fiducial marker into the image plane of a camera (Zhenglong, pg. 4719, 1st column 2nd full paragraph, lines 1-5, see Fig. 7). Zhenglong does not teach projecting models associated with fiducial marker contours to obtain model contours. Tang teaches estimating object pose by projecting 3D models into the image plane of a camera and aligning projected model features with detected image features (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Zhenglong to project a 3D fiducial marker model, rather than only feature points, for model-based pose estimation as taught by Tang (Tang, pg. 3, paragraph 0022, see Fig. 5). The motivation for doing so would have been to rely on additional geometric features of the projected model, thereby improving the reliability of pose estimation. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Zhenglong with Tang to obtain the invention as specified in claim 1.
Regarding claim 4, Zhenglong in view of Tang teaches the method of claim 1, wherein the models comprise three- dimensional (3D) models and the image plane comprises a two-dimensional (2D) image plane of the camera (Zhenglong, “The object 106 is observed by a camera 112, representing the camera 20 , and projected as a 2D object image 108 on a 2D image plane 110, where the object image 108 represents the image 94 and where points 102 on the image 108 are feature points predicted by the neural network 78, representing the points 96. The illustration 100 also shows a virtual 3D CAD model 114 of the object 16 having feature points 104 at the same location as the feature points 96 that is randomly placed in front of the camera 112 and is projected as a 2D model image 116 on the image plane 110 also including projected feature points 118.”, pg. 3, paragraph 0022).
Claims 2 and 3 are rejected under 35 U.S.C. 103 as being unpatentable over Zhenglong et al. (“Pose estimation for multicopters based on monocular vision and AprilTag”, 2018 37th Chinese Control Conference (CCC). IEEE, 2018) in view of Tang et al. (US 20220084238 A1) and further in view of Hui et al. (“Vision-based autonomous navigation approach for unmanned aerial vehicle transmission-line inspection”, International Journal of Advanced Robotic Systems, 2018).
Regarding claim 2, Zhenglong in view of Tang teaches The method of claim 1, further comprising: capturing, by the camera, a second image to track the fiducial marker; identifying one or more second image contours in the second image; determining a second position of the fiducial marker in the second image; projecting, based at least on the second position, the models into the image plane of the camera to obtain one or more second model contours; and determining a second pose associated with the fiducial marker based at least on the one or more second image contours and the one or more second model contours (Zhenglong, “Here, we used the algorithm described in Section III to process the image and estimate the pose of the multicopter based on the images captured by the downward-looking camera.”, pg. 4721, 2nd column, 1st full paragraph, The UAV collects successive images and performs image processing and pose estimation with respect to fiducial markers for each collected image.).
Zhenglong in view of Tang does not teach wherein the fiducial marker is partially occluded in the first image and/or the second image.
However, Hui teaches wherein the fiducial marker is partially occluded in the first image and/or the second image (Hui, “This article presents an autonomous navigation approach based on a transmission tower for unmanned aerial vehicle (UAV) power line inspection. For this complex vision task, a perspective navigation model, which plays an important role in the description and analysis of the flight strategy, is introduced.”, see abstract, lines 1-3, “At initialization stage, by detection, the UAV determines first the position of the next tower. Thereafter, the PTZ adjusts the pitch and yaw angle to make the camera optical axis point to the center of the detected transmission tower. At the stage of UAV heading adjustment,
p
A
is constantly calculated by eqaution (5) until it reaches the same horizontal position as VP. Since PTZ actively tracks tower, VP position has almost no change during the adjustment of flight heading… At the inspection stage, the UAV flies along the expected heading. With the detection-tracking visual strategy enabled, the PTZ constantly adjusts the yaw and pitch to track the next tower until the relative yaw angle
ψ
p
u
between camera optical axis and the expected heading reaches 90◦. During the flight, once the tracking fails, UAV will immediately brake. After that, the tracker will be initialized again using the recent detection result”, pg. 8, 2nd column, 3rd and 4th full paragraphs, “At the position A, PTZ, whose camera optical axis is denoted by the red arrow, adjusts the yaw and pitch angle to situate the detected transmission tower in the image center. Then, UAV rotates to the expected heading defined by VP. In order to avoid the occlusion to the camera caused by landing gear, from waypoint B, UAV continues to fly along the expected direction and simultaneously begins to rotate.”, pg. 12, 1st column, 2nd full paragraph, lines 10-17, see Fig. 12(a), A PTZ camera is mounted on a UAV and used to perform object detection for visual navigation. Active camera control mitigates the impact of images collected under partial occlusions, including self-occlusion caused by the UAV itself, by dynamically adjusting the viewing direction in order to collect sufficient visual information for navigation.).
Zenglong in view of Tang teaches collecting images using a fixed camera on a UAV to guide a landing target by detecting fiducial markers (Zhenglong, pg. 1, 2nd column, 2nd full paragraph, lines 17-21, see Figs. 2 and 3). Zenglong in view of Tang does not teach fiducial markers being partially occluded in the images. Hui teaches mounting a PTZ camera on a UAV and performing dynamic camera control to mitigate the impact of partially occluded objects for collected images, including instances of self-occlusion caused by UAV landing gear (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have replaced the fixed camera of Zenglong in view of Tang with the PTZ camera and dynamic camera control as taught by Hui (Hui, pg. 8, 2nd column, 3rd and 4th full paragraphs, pg. 12, 1st column, 2nd full paragraph, lines 10-17, see Fig. 12(a)). The motivation for doing so would have been to improve robustness of fiducial marker detection under partial occlusion, thereby improving the reliability of UAV landing. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Zenglong in view of Tang with Hui to obtain the invention as specified in claim 2.
Regarding claim 3, Zhenglong in view of Tang teaches the method of claim 1. Zhenglong in view of Tang does not teach wherein the fiducial marker is partially occluded in the first image, and wherein the fiducial marker is partially occluded by a portion of the unmanned aerial vehicle.
However, similar to the analysis made for claims 2 and 16, Hui teaches wherein the fiducial marker is partially occluded in the first image, and wherein the fiducial marker is partially occluded by a portion of the unmanned aerial vehicle (Hui, pg. 8, 2nd column, 3rd and 4th full paragraphs, pg. 12, 1st column, 2nd full paragraph, lines 10-17, see Fig. 12(a)).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have replaced the fixed camera of Zenglong in view of Tang with the PTZ camera and dynamic camera control as taught by Hui (Hui, pg. 8, 2nd column, 3rd and 4th full paragraphs, pg. 12, 1st column, 2nd full paragraph, lines 10-17, see Fig. 12(a)). The motivation for doing so would have been to improve robustness of fiducial marker detection under partial occlusion, thereby improving the reliability of UAV landing. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Zenglong in view of Tang with Hui to obtain the invention as specified in claim 3.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Zhenglong et al. (“Pose estimation for multicopters based on monocular vision and AprilTag”, 2018 37th Chinese Control Conference (CCC). IEEE, 2018), (hereinafter Zhenglong) in view of Tang et al. (US 20220084238 A1) and further in view of Liu et al. (CN 111775152 A), (hereinafter Liu).
Regarding claim 5, Zhenglong in view of Tang teaches the method of claim 1, further comprising performing contour matching on the one or more first image contours and the one or more first model contours, wherein the first pose is based on the contour matching (Tang, “The CAD model 114 is rotated and translated in front of the camera 112, which rotates and translates the model image 116 in an attempt to minimize the distance between each of the feature points 118 on the model image 116 and the corresponding feature points 102 on the object image 108 , i.e. , align the images 116 and 108. Once the model image 116 is aligned with the object image 108 as best as possible, the pose of the CAD model 114 with respect to the camera 112 is the estimated 3D pose of the object 16.”, pg. 3, paragraph 0022, The combination of Zhenglong in view of Tang would perform rotation and translation matching between detected marker contours and stored 3D model marker contours for pose estimation.).
Zhenglong in view of Tang does not teach wherein the contour matching is based on Hu moments.
However, Liu teaches wherein the contour matching is based on Hu moments (Liu, “S201: Acquire a two-dimensional image and three-dimensional point cloud data of the target workpiece through the three-dimensional measurement system; S202: Perform Gaussian filter processing on the two-dimensional image of the target workpiece and then perform Canny edge detection to obtain edge information of the target workpiece… S204: Divide each edge connected area and determine a number, traverse each edge connected area, and calculate the Hu moment of each edge connected area; S205. Calculate the similarity between the edge connected region and the Hu moment of the preset standard workpiece contour, filter out the edge connected regions whose calculated value is less than the threshold, and determine the grabbing order of the target workpiece according to the calculated value and number it. The grabbing sequence is grabbing first with a smaller calculated value, grabbing later with a larger calculated value”, pg. 8, lines 11-27, Hu moment values are computed for detected object contours. These values are used to identify valid object contours via filtering and determine an order for grasping.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the contour matching of Zhenglong in view of Tang to include Hu moment filtering as taught by Liu (Liu, pg. 8, lines 11-27). The motivation for doing so would have been filter out invalid or unsuccessful contours prior to performing translation and rotation matching, thereby improving the efficiency of contour matching. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Zhenglong in view of Tang with Liu to obtain the invention as specified above.
Zhenglong in view of Tang does not teach wherein the first pose is determined based on iterative closest point processes.
However, Liu further teaches wherein the first pose is determined based on iterative closest point processes (ICP) (Liu, “The fine matching method adopts the ICP algorithm and uses the distance threshold, ie the closest point principle, to find the corresponding points in the two sets of point clouds; estimate the new transformation matrix (rotation and translation) by minimizing the distance error of all corresponding points; use the rough matching result As the initial value, iteratively update the transformation matrix value until it converges to the optimal transformation.”, pg. 11, lines 10-14, Iterative closest point (ICP) alignment and matching is performing between a detected object and a corresponding model of the object for pose estimation.).
Zhenglong in view of Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the contour matching of Zhenglong in view of Tang to determine marker pose using ICP as taught by Liu (Liu, pg. 11, lines 10-14). The motivation for doing so would have been to apply known techniques for precise matching, thereby improving the accuracy of pose estimation (as suggested by Liu, “Based on the initial rotation and translation relationship, the ICP algorithm is used to accurately match the three-dimensional coordinates of the corner points of the target workpiece with the three-dimensional point cloud of the target workpiece layer by layer from low to high to obtain a precise matching result.”, pg. 10, lines 58-60). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Zhenglong in view of Tang in view of Liu with the further teaching of Liu above to obtain the invention as specified in claim 5.
Claims 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Zhenglong et al. (“Pose estimation for multicopters based on monocular vision and AprilTag”, 2018 37th Chinese Control Conference (CCC). IEEE, 2018) in view of Tang et al. (US 20220084238 A1) and further in view of Mozer et al. (US 20190197908 A1).
Regarding claim 6, Zhenglong in view of Tang teaches the method of claim 1. Zhenglong in view of Tang does not teach wherein the fiducial marker is disposed on a landing platform, wherein the fiducial marker is a first fiducial marker of a plurality of fiducial markers on the landing platform, the method further comprising: descending the unmanned aerial vehicle toward the landing platform; and as the unmanned aerial vehicle descends, detecting at least one fiducial marker among a first subset of the plurality of fiducial markers when the unmanned aerial vehicle is within a first altitude range from the landing platform, wherein the first subset comprises the first fiducial marker and a second fiducial marker of the plurality of fiducial markers.
However, Mozer teaches wherein the fiducial marker is disposed on a landing platform, wherein the fiducial marker is a first fiducial marker of a plurality of fiducial markers on the landing platform, the method further comprising: descending the unmanned aerial vehicle toward the landing platform; and as the unmanned aerial vehicle descends, detecting at least one fiducial marker among a first subset of the plurality of fiducial markers when the unmanned aerial vehicle is within a first altitude range from the landing platform, wherein the first subset comprises the first fiducial marker and a second fiducial marker of the plurality of fiducial markers (Mozer, “Computer vision techniques are used in accordance with one or more embodiments to improve the precision of the autonomous drone landing, and thus the reliability of a successful docking event with a docking station. In accordance with one or more embodiments, one or more fiducial markers, such as light-emitting beacons, of known position and arrangement are configured at the landing target.”, pg. 2, paragraph 0021, lines 1-7, “The landing procedure for an aircraft in this scenario naturally involves starting at farther distances and approaching towards the target until the aircraft has landed... In accordance with one or more embodiments, to overcome this technical hurdle, a set of progressively smaller constellations are used that are appropriate for each stage of the descent, guiding the aircraft into its final, precise location. By way of example, as shown in FIG. 6, such constellations can comprise a series of nested circles 144 (each circle comprising multiple fiducials 140 arranged in a circular pattern) with decreasing diameters. FIG. 5 shows constellations comprising a series of squares 142 (each square comprising multiple fiducials 140 arranged in a square pattern) with decreasing dimensions. FIG. 7 shows a series of lines 146 (each line comprising multiple fiducials 140 arranged in a line). Suitable fiducials systems could include any combination or permutation of fiducial constellations that get progressively smaller (i.e. closer to the center point of the camera FOY) as the aircraft approaches the landing target.”, pg. 2, paragraph 0028-0031, see Fig. 4 and 6, During drone descends, subsets of fiducial markers, such as the nested circles constellations of Fig. 6, are detected within different altitude ranges to execute a landing operation.).
Zhenglong in view of Tang teaches using a UAV to detect fiducial markers for pose estimation (Zhenglong, “we used the map composed of AprilTags and a downward looking camera to estimate the pose of multicopters…”, pg. 4717, 2nd column, 2nd full paragraph, lines 19-20, see Figs. 2 and 3), but does not teach detecting fiducial markers to execute a landing operation on a landing platform. Mozer teaches detecting constellations of fiducial markers to guide a landing operation of a UAV on a landing platform (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the fiducial marker detection of Zhenglong in view of Tang to be used for executing a landing operation of a UAV as taught by Mozer (Mozer, pg. 2, paragraph 0028-0031, see Fig. 4 and 6). The motivation for doing so would have been improve the precision of autonomous drone landing (as suggested by Mozer, “Computer vision techniques are used in accordance with one or more embodiments to improve the precision of the autonomous drone landing, and thus the reliability of a successful docking event with a docking station.”, pg. 2, paragraph 0021, lines 1-4). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Zhenglong in view of Tang with Mozer above to obtain the invention as specified in claim 6.
Regarding claim 7, Zhenglong in view of Tang and further in view of Mozer teaches the method of claim 6, further comprising, as the unmanned aerial vehicle descends, detecting at least one fiducial marker among a second subset of the plurality of fiducial markers when the unmanned aerial vehicle is within a second altitude range from the landing platform, wherein the second subset comprises a third fiducial marker of the plurality of fiducial markers and a fourth fiducial marker of the plurality of fiducial markers, wherein the first and the second fiducial markers have the same or similar sizes, and wherein the third and the fourth fiducial markers have the same or similar sizes and are smaller than the first and the second fiducial markers (Mozer, “By way of example, as shown in FIG. 6, such constellations can comprise a series of nested circles 144 (each circle comprising multiple fiducials 140 arranged in a circular pattern) with decreasing diameters.”, pg. 2, paragraph 0031, lines 5-8, Each subset or nested circle contains fiducial markers of the same size with the inner rings having a smaller constellation diameter than those of the outer rings.).
Regarding claim 9, Zhenglong in view of Tang teaches the method of claim 1. Zhenglong in view of Tang does not teach wherein the fiducial marker is disposed on a landing platform, wherein the fiducial marker is a first fiducial marker of a plurality of fiducial markers on the landing platform, the method further comprising adjusting a trajectory of the unmanned aerial vehicle and/or a landing position of the unmanned aerial vehicle based at least on detections of one or more of the plurality of fiducial markers as the unmanned aerial vehicle descends.
However, Mozer teaches wherein the fiducial marker is disposed on a landing platform, wherein the fiducial marker is a first fiducial marker of a plurality of fiducial markers on the landing platform, the method further comprising adjusting a trajectory of the unmanned aerial vehicle and/or a landing position of the unmanned aerial vehicle based at least on detections of one or more of the plurality of fiducial markers as the unmanned aerial vehicle descends (Mozer, “Computer vision techniques are used in accordance with one or more embodiments to improve the precision of the autonomous drone landing, and thus the reliability of a successful docking event with a docking station. In accordance with one or more embodiments, one or more fiducial markers, such as light-emitting beacons, of known position and arrangement are configured at the landing target.”, pg. 2, paragraph 0021, lines 1-7, “The landing procedure for an aircraft in this scenario naturally involves starting at farther distances and approaching towards the target until the aircraft has landed... In accordance with one or more embodiments, to overcome this technical hurdle, a set of progressively smaller constellations are used that are appropriate for each stage of the descent, guiding the aircraft into its final, precise location. By way of example, as shown in FIG. 6, such constellations can comprise a series of nested circles 144 (each circle comprising multiple fiducials 140 arranged in a circular pattern) with decreasing diameters. FIG. 5 shows constellations comprising a series of squares 142 (each square comprising multiple fiducials 140 arranged in a square pattern) with decreasing dimensions. FIG. 7 shows a series of lines 146 (each line comprising multiple fiducials 140 arranged in a line). Suitable fiducials systems could include any combination or permutation of fiducial constellations that get progressively smaller (i.e. closer to the center point of the camera FOY) as the aircraft approaches the landing target.”, pg. 2, paragraph 0028-0031, see Fig. 4 and 6, During drone descends, subsets of fiducial markers, such as the nested circles constellations of Fig. 6, are detected within different altitude ranges to adjust control of a UAV to perform a landing operation.).
Zhenglong in view of Tang teaches using a UAV to detect fiducial markers for pose estimation (Zhenglong, “we used the map composed of AprilTags and a downward looking camera to estimate the pose of multicopters…”, pg. 4717, 2nd column, 2nd full paragraph, lines 19-20, see Figs. 2 and 3), but does not teach detecting fiducial markers to adjust a trajectory or landing position of a UAV during descent. Mozer teaches detecting constellations of fiducial markers to control a UAV during a landing operation to a landing platform (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the fiducial marker detection of Zhenglong in view of Tang to be used for executing a landing operation of a UAV as taught by Mozer (Mozer, pg. 2, paragraph 0028-0031, see Fig. 4 and 6). The motivation for doing so would have been improve the precision of autonomous drone landing (as suggested by Mozer, “Computer vision techniques are used in accordance with one or more embodiments to improve the precision of the autonomous drone landing, and thus the reliability of a successful docking event with a docking station.”, pg. 2, paragraph 0021, lines 1-4). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Zhenglong in view of Tang with Mozer above to obtain the invention as specified in claim 9.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Zhenglong et al. (“Pose estimation for multicopters based on monocular vision and AprilTag”, 2018 37th Chinese Control Conference (CCC). IEEE, 2018) in view of Tang et al. (US 20220084238 A1) and further in view of Mozer et al. (US 20190197908 A1) and Jourdan et al. (US 20200301445 A1).
Regarding claim 8, Zhenglong in view of Tang and further in view of Mozer teaches the method of claim 6, further comprising; landing the unmannded aerial vehicle (Mozer, “Computer vision techniques are used in accordance with one or more embodiments to improve the precision of the autonomous drone landing, and thus the reliability of a successful docking event with a docking station.”, pg. 2, paragraph 0021, lines 1-4).
Zhenglong in view of Tang and further in view of Mozer does not teach detecting at least one fiducial marker among a second subset of the plurality of fiducial markers when the unmanned aerial vehicle is on the landing platform.
However, Jourdan teaches detect at least one fiducial marker among a second subset of the plurality of fiducial markers when the unmanned aerial vehicle is on the landing platform (Jourdan, “The fiducial disturbance module 268 ensures that the UAV 100 recalibrates its stored position in the event that it is moved more than a threshold amount while landed. The fiducial disturbance module 268 causes the UAV 100 to determine its navigation solution using the FNS 204 immediately prior to landing, or while landed. Then, the fiducial disturbance module 268 causes the UAV 100 to monitor its position while landed (using any of the available navigation systems, e.g., GPS and/or IMU). If the fiducial disturbance module 268 detects that the position of the UAV 100 changes more than a permissible threshold level (e.g., 0.1 meter, 0.5 meter, 1.0 meter, etc.), then it invalidates the navigation solution determined immediately prior to landing. Subsequently, the fiducial disturbance module 268 causes the UAV 100 to re-acquire its position using one or more available navigation systems. For example, the UAV 100 may re-acquire its position by imaging a fiducial marker with the camera 140.”, pg. 13, paragraph 0090, Fiducial markers are used after landing to verify that the UAV has not moved within a given threshold.).
Zhenglong in view of Tang and further in view of Mozer teaches detecting fiducial markers to guide a landing operation of a UAV on a landing platform (Mozer, paragraphs 0027-0031, see Figs. 4 and 6), but does not teach detecting additional fiducial markers when the UAV is on the landing platform. Jourdan teaches monitoring a UAV’s position with respect to a landing platform after a landing operation has been performed (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Zhenglong in view of Tang and further in view of Mozer to include UAV position monitoring post landing as taught by Jourdan (Jourdan, pg. 13, paragraph 0090). The motivation for doing so would have been to maintain correct alignment between the UAV and the landing platform after landing, thereby reducing the risk of misalignment during subsequent take-off operations. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Zhenglong in view of Tang and further in view of Mozer with Jourdan to obtain the invention as specified in claim 8.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Zhenglong et al. (“Pose estimation for multicopters based on monocular vision and AprilTag”, 2018 37th Chinese Control Conference (CCC). IEEE, 2018) in view of Tang et al. (US 20220084238 A1) and further in view of Stabler et al. (US 20160257424 A1).
Regarding claim 10, Zhenglong in view of Tang teaches the method of claim 1. Zhenglong in view of Tang does not teach wherein the fiducial marker is disposed on a landing platform, wherein the landing platform comprises a launch and recovery box (LRB), the method further comprising: receiving orientation data from the LRB; orienting the unmanned aerial vehicle based on the orientation data; and descending the unmanned aerial vehicle toward the landing platform with the unmanned aerial vehicle oriented according to the orientation data.
However, Stabler teaches wherein the fiducial marker is disposed on a landing platform, wherein the landing platform comprises a launch and recovery box (LRB), the method further comprising: receiving orientation data from the LRB; orienting the unmanned aerial vehicle based on the orientation data; and descending the unmanned aerial vehicle toward the landing platform with the unmanned aerial vehicle oriented according to the orientation data (Stabler, “The present disclosure provides systems and methods that permit an unmanned aerial vehicle (UAV) to land on a landing pad of a landing station and take off from the landing pad.”, pg. 1, paragraph 0007, lines 1-4, “The landing pad can further comprise a plurality of markers in a predetermined geometric configuration on the landing pad. The markers can be detectable by the UAV to identify a location of the landing pad relative to a given location of the UAV. The UAV can detect one or more markers in the plurality of markers with a vision sensor on-board the UAV. The vision sensor can comprise a camera. The camera can be a charge-coupled device (CCD) camera. The camera can be a complementary metal-oxide semiconductor (CMOS) camera. Alternatively, the vision sensor can be provided on the landing pad and the UAV can comprise a plurality of markers in a given or predetermined geometric configuration on the UAV.”, pg. 6, paragraph 068, “The UAV can comprise one or more computer processors configured to calculate the orientation and/or position of the UAV relative to a detected marker. In some cases, the computer processors can be on-board the UAV. Alternatively, the computer processors can be off-board the UAV. The computer processors can be on the landing pad.”, pg. 6, paragraph 0073, lines 1-9, A launch and recovery system includes collecting images through a camera fixed to the landing pad and using a processor fixed to the landing pad to detect fiducial markers on the UAV to determine orientation information for landing. This orientation information includes guidance or control information which is communicated to the UAV to facilitate landing operations.).
Zhenglong in view of Tang teaches using a UAV to detect fiducial markers for pose estimation (Zhenglong, “we used the map composed of AprilTags and a downward looking camera to estimate the pose of multicopters…”, pg. 4717, 2nd column, 2nd full paragraph, lines 19-20, see Figs. 2 and 3). Stabler teaches a launch and recovery system that includes determining orientation information based on detected markers for transmittal to a UAV for executing landing operations (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Zhenglong in view of Tang to include the launch and recovery system for executing landing as taught by Stabler (Stabler, pg. 6, paragraph 068 and pg. 6, paragraph 0073, lines 1-9). The motivation for doing so would have been to perform fiducial marker detection and orientation calculations at the launch and recovery system rather than onboard the UAV, thereby reducing UAV computational load. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Zhenglong in view of Tang with Stabler to obtain the invention as specified in claim 10.
Allowable Subject Matter
Claims 11, 18 and 19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Examiner notes that no prior art was applied against claims 12 and 20. However, claims 12 and 20 stands rejected under 35 U.S.C. 112(b) due to indefiniteness, and cannot be considered to be allowable subject matter because the scope of the claims is not clearly defined.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CONNOR LEVI HANSEN whose telephone number is (703)756-5533. The examiner can normally be reached Monday-Friday 9:00-5:00 (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CONNOR L HANSEN/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672