DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are pending.
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claim(s) 1-2 and 12-13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al (US20220315242A1) in view of Hu et al (CN115542727A, 2022).
Regarding claims 1 and 12, Liu teaches an apparatus comprising:
a camera mounted on an air mobility and configured to obtain images for an area in front of the air mobility; and
(Liu, " aerial vehicle environment 100 includes onboard visual equipment 106 (e.g., camera, forward-looking infrared (FLIR), millimeter wave radar, etc.)", [0029]; " a plurality of ground images of a ground path ahead of the vehicle", [0007]; an onboard camera configured to obtain images of the path ahead of the air mobility)
one or more processors connected with the camera; and a non-transitory storage medium storing computer-readable instructions that, when executed by the one or more processors, enable the one or more processors to:
determine a flight status of the air mobility, based on flight phase information of the air mobility,
(Liu, "determine when a vehicle is on final approach to a landing runway based on monitored aerial vehicle status information", [0007]; "The determination regarding whether the vehicle is on the final approach segment can be made based on aerial vehicle status information", [0038]; Hu, "course alignment, approach, before drifting, drifting and touchdown.", p5; evaluating and determining the flight status based on flight phase information)
obtain runway position information about a runway position of a runway for the air mobility from a first image obtained by the camera, in response to the flight status being an approach phase, and
(Liu, "process, responsive to determining that the vehicle is on final approach to the landing runway, a plurality of ground images of a ground path ahead of the vehicle retrieved from an image sensor on the vehicle; identify a lane in the processed ground images; determine whether the identified lane from the processed ground images corresponds to an assigned runway for landing", [0007]; processing a first (initial) image of the plurality of images during the approach phase to obtain the runway's position)
obtain centerline position information about a centerline position of a centerline of the runway from a second image obtained by the camera, in response to the flight status being a ground roll phase.
(Liu, "track during landing, based on the processed ground images, a left and a right side edge, a front edge, and a runway center line of the assigned runway", [0007]; Hu, "After the fixed-wing aircraft finishes landing, the conventional controller is activated until the flight ends.", p5; tracking the centerline from subsequent images during landing; incorporating Hu into Liu would lead to switching processing states in response to entering the post-touchdown ground roll phase to extract the centerline position from a second image)
(In summary, Liu teaches a visual landing aid system that processes images during an approach phase to identify the runway and tracks the centerline during landing, but does not expressly have disclosure of dedicating specific processing logic exclusively triggered by the post-touchdown "ground roll phase." Hu teaches a flight control system that dynamically switches controllers based on explicit flight phases, including an approach phase, landing phases, and a post-landing phase (i.e., after the aircraft finishes landing/touchdown until the flight ends))
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the phase-based switching logic of Hu into the visual landing system of Liu in order to optimize image processing and control algorithms for the unique dynamics of each flight phase, thereby using a subsequent (second) image to track the centerline specifically during the ground roll phase for stable post-touchdown rollout. The combination of Liu and Hu also teaches other enhanced capabilities.
Regarding claims 2 and 13, the combination of Liu and Hu teaches its/their respective base claim(s).
The combination further teaches the apparatus of claim 1, wherein the instructions further enable the one or more processors to distinguish between the approach phase and the ground roll phase based on whether the air mobility touches down on the runway.
(Hu, "in the landing flailing stage, whether the fixed-wing aircraft Complete landing monitoring, wherein, when the fixed-wing aircraft finishes landing, turn off the anti-ground disturbance controller and enable the conventional controller, otherwise, when the fixed-wing aircraft fails to complete the landing, keep the enabled anti-ground disturbance controller", p5; distinguishing the flight phase transitioning into the post-landing ground roll phase by explicitly monitoring whether the aircraft finishes landing and touches down to trigger the switch in the control logic)
Claim(s) 3-11 and 14-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al (US20220315242A1) in view of Hu et al (CN115542727A, 2022) and further in view of Cheng et al (CN111145362A, 2020) and Evans et al (US20220198703A1).
Regarding claims 3 and 14, the combination of Liu and Hu teaches its/their respective base claim(s).
The combination does not expressly disclose but the combination of Cheng and Evans teaches the apparatus of claim 1, wherein the instructions further enable the one or more processors to:
transform the first image into a first bird's eye view (BEV) image; and
(Cheng, "perspective projection transformation is carried out on a map picture...by combining flight attitude information and sensor internal parameters (a principal point, a focal length and the like) obtained by calibration of a multi-mode video sensor", p3; Evans, "transforming the camera coordinates for the at least one camera to corresponding runway-framed local coordinates", [0008]; Cheng teaches perspective projection using attitude and intrinsic parameters to produce a registered overhead runway view; Evans teaches transforming camera coordinates to a runway-plane frame, both are the inverse perspective mapping underlying BEV. A POSITA would apply either technique to generate the first BEV image, a predictable result.)
determine a position error between the air mobility and a reference point of the runway for the air mobility, based on a coordinate error between a reference projection point corresponding to coordinates onto which a camera principal point is projected on the first BEV image and the reference point of the runway.
(Cheng, "adopting a Normalized Total Gradient (NTG) as an error energy function of affine transformation between the floating image and the reference image, and optimally solving the error energy function through an iterative method to obtain affine transformation parameters under the condition of minimum error energy function", p4; Evans, [0083]: " match the interest points on the runway or the runway marking in the image, to corresponding points on the runway or the runway marking that have known runway-framed local coordinates; and perform a perspective-n-point (PnP) estimation, using the interest points and the known runway-framed local coordinates, to determine the current pose estimate of the at least one camera and thereby the aircraft relative to the runway or the runway marking", [0170]; Evans, [0090]: “(c, d)=screen center (x, y); o=scaling factor; (p, q, r)=rotation angles around (x, y, z); x=reference point world runway-framed local coordinate x”, [0090]; Cheng teaches coordinate error minimization between camera projection and runway reference; Evans teaches matching image-projected interest points to known runway reference coordinates. Under § 103, combining Cheng's principal-point-based projection with Evans's runway-framed reference framework teaches this limitation.)
(In summary, Liu and Hu teach a vision-based runway approach and landing system but do not expressly disclose a BEV-based lateral position measurement framework or HSV-based centerline extraction. Cheng and Evans are in the same field of airborne visual runway guidance. Cheng teaches perspective projection of runway imagery using aircraft attitude parameters and camera intrinsics to produce a geometrically registered overhead view, and HSV decomposition for runway feature extraction. Evans teaches segmentation-based runway classification, PnP-based pose estimation using known runway-framed coordinates, and centerline-referenced lateral deviation determination)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the BEV image generation and HSV processing of Cheng and the segmentation, PnP coordinate correspondence, and centerline tracking techniques of Evans into the vision-based runway approach and landing system of Liu and Hu in order to improve the runway position accuracy of Liu and Hu during approach and ground roll. The combination of Liu, Hu, Cheng and Evans also teaches other enhanced capabilities.
Regarding claim 4, the combination of Liu, Hu, Cheng and Evans teaches its/their respective base claim(s).
The combination further teaches the apparatus of claim 3, wherein the instructions further enable the one or more processors to transform the first image into the first BEV image based on at least one of or any combination of yaw of the air mobility, pitch of the air mobility, and roll of the air mobility.
(Cheng, "the flight attitude information includes pitch, roll, and yaw data", p2; "pitching, rolling and yawing information are converted into an earth-centered earth-fixed (ECEF) coordinate system, and a synthetic visual map picture under a virtual camera is calculated by combining relative rotation translation parameters between a multi-mode video sensor and a body coordinate system", p6; using pitch, roll, and yaw to perform the perspective projection transformation generating the overhead view.)
Regarding claims 5 and 15, the combination of Liu, Hu, Cheng and Evans teaches its/their respective base claim(s).
The combination further teaches the apparatus of claim 3, wherein the instructions further enable the one or more processors to:
obtain principal point coordinates of the camera on a heading coordinate system; and
(Cheng, "sensor internal parameters (a principal point, a focal length and the like) obtained by calibration of a multi-mode video sensor", p3; "relative rotation translation parameters between a multi-mode video sensor and a body coordinate system", p6; Evans, (c, d)=screen center (x, y); o=scaling factor; (p, q, r)=rotation angles around (x, y, z)", [0090]; Cheng teaches the principal point in the body-coordinate projection; Evans defines the screen center within a projection using body-axis rotation angles. Combining Cheng's principal point calibration with Evans's rotation-based framework would lead to expressing the principal point in the heading (body-frame) coordinate system)
determine the reference projection point on the first BEV image based on a coordinate point corresponding to the principal point coordinates of the camera on the heading coordinate system.
Regarding claims 6 and 16, the combination of Liu, Hu, Cheng and Evans teaches its/their respective base claim(s).
The combination further teaches the apparatus of claim 5, wherein the instructions further enable the one or more processors to:
obtain a projection distance between the camera and the runway along a projection direction of the camera; and
(Evans, "the pose of the aircraft relative to the runway 202 may be expressed in 3DOF as a combination of an x coordinate (distance to the runway), y coordinate (altitude), and lateral angular deviation θLAT", [0060]; "o=scaling factor; (p, q, r)=rotation angles around (x, y, z); x=reference point world runway-framed local coordinate x; and z=runway-framed local coordinate z", [0090]; the PnP estimation determines the camera-to-runway distance component as part of the 3DOF pose, with the scaling factor o and z coordinate encoding the projection distance along the camera optical axis)
obtain the principal point coordinates of the camera on the heading coordinate system, based on the projection distance and the camera principal point on an image coordinate system.
(Cheng, "sensor internal parameters (a principal point, a focal length and the like) obtained by calibration of a multi-mode video sensor", p3; Evans, (a, b)=pixel position (x, y); (c, d)=screen center (x, y); o=scaling factor; (p, q, r)=rotation angles around (x, y, z); x=reference point world runway-framed local coordinate x; and z=runway-framed local coordinate z", [0090]; the projection equation in [0090] shows the screen center (c, d), image-plane principal point, and scaling factor o (projection distance) as co-parameters. Inverting this relation uses the projection distance and image principal point to locate the principal point in the heading coordinate system, which is a direct algebraic consequence of the projection framework of Cheng and Evans)
Regarding claims 7 and 17, the combination of Liu, Hu, Cheng and Evans teaches its/their respective base claim(s).
The combination further teaches the apparatus of claim 3, wherein the instructions further enable the one or more processors to:
obtain a scale factor, based on an actual length of a runway starting line and an image length of the runway starting line on the first BEV image; and
(Evans, "P0x,z=(0,0); P1x,z=([P0x+runway_length], P0z); P2x,z=([P0x+runway_length], [P0z+runway_width]); and P3x,z=([P0x, [P0z+runway_width])", [0089]; "o=scaling factor", [0090]; the PnP estimation uses known actual runway dimensions (runway_length, runway_width) mapped to their image pixel positions, with an explicit scaling factor o. Obviously, deriving the image-to-actual scale factor from this known-actual-length vs. image-length correspondence would be a predictable result)
determine an actual distance error between an extension line in a heading direction of the air mobility and a center point of the runway starting line based on the scale factor and the coordinate error.
(Evans, "the lateral angular deviation θLAT of the aircraft from a centerline 210 of the runway. The vertical angular deviation may be determined by the slope of the edge, a width over distance (number of pixels) relation", [0102]; "o=scaling factor; (p, q, r)=rotation angles around (x, y, z); x=reference point world runway-framed local coordinate x", [0090]; computing the lateral deviation of the heading extension from the runway reference using the projection scaling factor and the image-coordinate offset. Applying the scale factor o ("o=scaling factor", [0090]) to the coordinate error yields the actual distance error, a direct application of Evans's projection equation in [0090])
Regarding claims 8 and 18, the combination of Liu, Hu, Cheng and Evans teaches its/their respective base claim(s).
The combination further teaches the apparatus of claim 7, wherein the instructions further enable the one or more processors to determine the center point of the runway starting line as the reference point.
(Evans, "only two points (e.g. two threshold center points) may be needed for a single solution for the lateral and (if the runway length is known) the vertical angular deviation. The threshold center points here refer to the center point of each threshold marking (on per approach end)", [0090]; using the center point of each runway threshold marking, i.e., the center of the runway starting/threshold line, as the reference point for pose computation)
Regarding claim 9, the combination of Liu, Hu, Cheng and Evans teaches its/their respective base claim(s).
The combination further teaches the apparatus of claim 3, wherein the instructions further enable the one or more processors to classify the runway on the first image, based on segmentation.
(Evans, "performing an object detection and segmentation in which at least one of the runway on the airfield, or a runway marking on the runway, is detected in the image, and in which a mask is produced that includes a segment of pixels of the image assigned to an object class for the runway or the runway marking", [0016]; classifying runway pixels via object detection and segmentation, producing an object-class mask)
Regarding claims 10 and 19, the combination of Liu and Hu teaches its/their respective base claim(s).
The combination of Liu, Hu, Cheng and Evans further teaches apparatus of claim 1, wherein the instructions further enable the one or more processors to:
transform the second image into a second BEV image, based on a second image processing algorithm;
(Cheng, "By applying the perspective transformation model and the accurate registration model, the pixel position of any pixel point of the multi-modal video frame in the map picture can be positioned.", p6; Evans, "the pose-estimation engine is configured to transform the camera coordinates for the at least one camera to corresponding runway-framed local coordinates", [0068]; perspective-to-runway-plane transformation may be applied at any flight phase as, e.g., a second image processing algorithm during ground roll, a predictable extension)
extract the centerline of the runway, the centerline being included in the second BEV image; and
(Evans, "the pose-estimation engine 402 configured to determine the current pose estimate includes the pose-estimation engine configured to determine edges of the mask 410 that correspond to sides of the runway, and angular deviations of the edges from a centerline of the image". [0101]; "a centerline 210 of the runway 202", [0059]; extracting the runway centerline from runway images by detecting edge deviations from it)
represent the centerline of the runway on a coordinate axis of the second BEV image.
(Evans, "the lateral angular deviation θLAT of the aircraft from a centerline 210 of the runway", [0102]; "the pose of the aircraft 200 may be expressed in two degrees of freedom (2DOF) relative to a glideslope 208 and a centerline 210 of the runway 202", [0059]; using the runway centerline as the reference axis (i.e., θLAT = 0) of the runway coordinate system, representing it on a coordinate axis)
Regarding claims 11 and 20, the combination of Liu, Hu, Cheng and Evans teaches its/their respective base claim(s).
The combination further teaches the apparatus of claim 10, wherein the instructions further enable the one or more processors to:
transform the second BEV image into a hue-saturation-value (HSV) image; and
(Cheng, "performing HSV color space decomposition on the corrected map picture to obtain decomposed chrominance, saturation and luminance components, and performing image fusion on the luminance components and the multi-modal video", p2; transforming a registered overhead runway image into the HSV color space; a person having ordinary skill in the art would apply this decomposition to the second BEV image for the same runway marking enhancement purpose)
extract the centerline of the runway from the HSV image.
(Cheng, "HSV chromaticity, saturation and brightness decomposition is applied to the registered synthetic visual virtual map picture, and a single-channel map picture brightness image and a single-channel multi-mode image are extracted and fused", p6; Evans, "angular deviations of the edges from a centerline of the image", [0037]; Cheng teaches extracting the V-channel (brightness image) from HSV decomposition of a runway image; a person having ordinary skill in the art would extract the centerline (as taught by Evans) from this V channel, since runway centerline markings are bright white paint on low-brightness asphalt of the runway, the V channel being the most discriminating for this purpose)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANXUN YANG whose telephone number is (571)272-9874. The examiner can normally be reached on MON-FRI: 8AM-5PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272- 1000.
/JIANXUN YANG/
Primary Examiner, Art Unit 2662 7/11/2026
Prosecution Insights