DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is a Non-Final Action on the Merits. Claims 1-20 are currently pending and are addressed below.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on September 11th, 2025 and April 13th, 2026 has been considered and entered.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. § 101 because the claimed invention is directed to a judicial exception (i.e., an abstract idea) without significantly more.
In sum, claims 1-20 are rejected under 35 U.S.C. §101 because the claimed invention is directed to a judicial exception to patentability (i.e., a law of nature, a natural phenomenon, or an abstract idea) and do not include an inventive concept that is something “significantly more” than the judicial exception under the January 2019 patentable subject matter eligibility guidance (2019 PEG) analysis which follows.
Under the 2019 PEG step 1 analysis, it must first be determined whether the claims are directed to one of the four statutory categories of invention (i.e., process, machine, manufacture, or composition of matter). Applying step 1 of the analysis for patentable subject matter to the claims, it is determined that the claims are directed to the statutory category of a process. Therefore, we proceed to step 2A, Prong 1.
Revised Guidance Step 2A – Prong 1
Under the 2019 PEG step 2A, Prong 1 analysis, it must be determined whether the claims recite an abstract idea that falls within one or more designated categories of patent ineligible subject matter (i.e., organizing human activity, mathematical concepts, and mental processes) that amount to a judicial exception to patentability.
Here, with respect to independent claims 1, 9, and 15, the claims recite the abstract idea of tracking an aircraft, and mentally determine “tracking, by the computing device, a position of the aircraft while the aircraft is approaching the docking area using the LIDAR sensor data and the simulated two-dimensional image”, where these claims fall within one or more of the three enumerated 2019 PEG categories of patent ineligible subject matter, specifically, a mental process, that can be performed in the human mind since each of the above steps could alternatively be performed in the human mind or with the aid of pen and paper. This conclusion follows from CyberSource Corp. v. Retail Decisions, Inc., where our reviewing court held that section 101 did not embrace a process defined simply as using a computer to perform a series of mental steps that people, aware of each step, can and regularly do perform in their heads. 654 F.3d 1366, 1373 (Fed. Cir. 2011); see also In re Grams, 888 F.2d 835, 840–41 (Fed. Cir. 1989); In re Meyer, 688 F.2d 789, 794–95 (CCPA 1982); Elec. Power Group, LLC v. Alstom S.A., 830 F. 3d 1350, 1354–1354 (Fed. Cir. 2016) (“we have treated analyzing information by steps people go through in their minds, or by mathematical algorithms, without more, as essentially mental processes within the abstract-idea category”).
Additionally, mental processes remain unpatentable even when automated to reduce the burden on the user of what once could have been done with pen and paper. See CyberSource, 654 F.3d at 1375 (“That purely mental processes can be unpatentable, even when performed by a computer, was precisely the holding of the Supreme Court in Gottschalk v. Benson.”). These limitations, as drafted, are a simple process that under their broadest reasonable interpretation, covers the performance of the limitations of the mind. For example, the claim limitation encompasses mentally tracking an aircraft based off of the information provided by the car’s sensors while traveling, or alternatively, mentally tracking an aircraft based on observations by a human.
For example, a human could mentally and with the aid of pen and paper track an aircraft.
Revised Guidance Step 2A – Prong 2
Under the 2019 PEG step 2A, Prong 2 analysis, the identified abstract idea to which the claim is directed does not include limitations that integrate the abstract idea into a practical application, since the additional elements of a vehicle sensors and a computing device are merely generic components used as a tool (“apply it”) to implement the abstract idea. (See, e.g., MPEP §2106.05(f)). See Alice, 573 U.S. at 223 (“[T]he mere recitation of a generic computer cannot transform a patent-ineligible abstract idea into a patent-eligible invention.”)
In addition, the limitation “capturing, by a light detection and ranging (LIDAR) sensor, LIDAR sensor data of an aircraft while the aircraft is approaching a docking area of an airport; receiving, from the LIDAR sensor, the sensor data of the aircraft while the aircraft is approaching the docking area; retrieving, by a computing device, a simulated two-dimensional image of the aircraft generated based on simulated data of the aircraft” constitutes insignificant presolution activity that merely gathers data and, therefore, do not integrate the exception into a practical application. See In re Bilski, 545 F.3d 943, 963 (Fed. Cir. 2008) (en banc), aff' d on other grounds, 561 U.S. 593 (2010) (characterizing data gathering steps as insignificant extra-solution activity); see also CyberSource, 654 F.3d at 1371–72 (noting that even if some physical steps are required to obtain information from a database (e.g., entering a query via a keyboard, clicking a mouse), such data-gathering steps cannot alone confer patentability); OIP Techs., Inc. v. Amazon.com, Inc., 788 F.3d 1359, 1363 (Fed. Cir. 2015) (presenting offers and gathering statistics amounted to mere data gathering). Accord Guidance, 84 Fed. Reg. at 55 (citing MPEP § 2106.05(g)).
Furthermore, the limitation “displaying an image including the position of the aircraft while the aircraft is approaching the docking area to provide navigation guidance to a pilot” is insignificant post-solution activity. The Supreme Court guides that the “prohibition against patenting abstract ideas ‘cannot be circumvented by attempting to limit the use of the formula to a particular technological environment' or [by] adding ‘insignificant postsolution activity.' ” Bilski, 561 U.S. at 610–11 (quoting Diehr, 450 U.S. at 191–92).
Displaying an image is mere insignificant extra-solution activity, as supported by the MPEP 2106.05(g), see printing or downloading generated menus, Ameranth, 842 F.3d at 1241-42, 120 USPQ2d at 1854-55. Mere instruction to apply an exception using generic computer components cannot provide an inventive concept.
In addition, merely “[u]sing a computer to accelerate an ineligible mental process does not make that process patent-eligible.” Bancorp Servs., L.L.C. v. Sun Life Assur. Co. of Canada (U.S.), 687 F.3d 1266, 1279 (Fed. Cir. 2012); see also CLS Bank Int’l v. Alice Corp. Pty. Ltd., 717 F.3d 1269, 1286 (Fed. Cir. 2013) (en banc) (“simply appending generic computer functionality to lend speed or efficiency to the performance of an otherwise abstract concept does not meaningfully limit claim scope for purposes of patent eligibility.”), aff’d, 573 U.S. 208 (2014). Accordingly, the additional element of a processor does not transform the abstract idea into a practical application of the abstract idea.
Revised Guidance Step 2B
Under the 2019 PEG step 2B analysis, the additional elements are evaluated to determine whether they amount to something “significantly more” than the recited abstract idea. (i.e., an innovative concept). Here, the additional elements, such as: a computing device and sensors does not amount to an innovative concept since, as stated above in the step 2A, Prong 2 analysis, the claims are simply using the additional elements as a tool to carry out the abstract idea (i.e., “apply it”) on a computer or computing device and/or via software programming. (See, e.g., MPEP §2106.05(f)). The additional elements are specified at a high level of generality to simply implement the abstract idea and are not themselves being technologically improved. (See, e.g., MPEP §2106.05 I.A.). See Alice, 573 U.S. at 223 (“[T]he mere recitation of a generic computer cannot transform a patent-ineligible abstract idea into a patent-eligible invention.”). Thus, these elements, taken individually or together, do not amount to “significantly more” than the abstract ideas themselves.
The additional elements of the dependent claims 2-8, 10-15, and 17-20 merely refine and further limit the abstract idea of the independent claims and do not add any feature that is an “inventive concept” which cures the deficiencies of their respective parent claim under the 2019 PEG analysis. None of the dependent claims considered individually, including their respective limitations, include an “inventive concept” of some additional element or combination of elements sufficient to ensure that the claims in practice amount to something “significantly more” than patent-ineligible subject matter to which the claims are directed.
The elements of the instant claimed invention, when taken in combination do not offer substantially more than the sum of the functions of the elements when each is taken alone. The claims as a whole, do not amount to significantly more than the abstract idea itself because the claims do not effect an improvement to another technology or technical field; the claims do not amount to an improvement to the functioning of an electronic device itself which implements the abstract idea (e.g., the general purpose computer and/or the computer system which implements the process are not made more efficient or technologically improved); the claims do not perform a transformation or reduction of a particular article to a different state or thing (i.e., the claims do not use the abstract idea in the claimed process to bring about a physical change. See, e.g., Diamond v. Diehr, 450 U.S. 175 (1981), where a physical change, and thus patentability, was imparted by the claimed process; contrast, Parker v. Flook, 437 U.S. 584 (1978), where a physical change, and thus patentability, was not imparted by the claimed process); and the claims do not move beyond a general link of the use of the abstract idea to a particular technological environment (e.g., “for tracking an aircraft . . . sensors” claim 16).
Accordingly, claims 1-20 are rejected under 35 USC 101 as being drawn to an abstract idea without significantly more, and thus are ineligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Pal (US 20230267753 A1) (“Pal”) in view of Deng (CN 105335764 A) (“Deng”) (Translation Attached).
With respect to claim 1, Pal teaches a method for tracking an aircraft, comprising:
capturing, by a light detection and ranging (LIDAR) sensor, LIDAR sensor data of an aircraft while the aircraft is approaching a docking area of an airport; receiving, from the LIDAR sensor, the sensor data of the aircraft while the aircraft is approaching the docking area (See at least Pal Paragraphs 34-35 “FIG. 1 illustrates a number of aircraft bridge area arrangements that can be used in accordance with one or more embodiments of the present disclosure. In FIG. 1 , the three arrangements at the top illustrate a number of embodiments using one sensor array (e.g., a combination of at least one camera image sensor and at least one laser/radar sensor, described herein as a LIDAR/Radar sensor, but embodiments are not so limited and the array could be one image sensor, such as a camera) with aircraft 101 located in bridge areas 100. The bottom three arrangements show embodiments having multiple sensor arrays, as will be discussed in more detail below. Each arrangement has several common elements. For example, all of the arrangements include a bridge area wherein at least a portion of the bridge area is within the field of view 104 of at least one sensor of a sensor array 102. Although described herein as a “bridge area,” the term bridge area as used herein merely means a parking space for an aircraft located on the airside apron of a gate at an airport and although described herein using the term “bridge,” the space may sometimes be in a location where, for example, passengers and crew access the aircraft via stairs from the tarmac and a bridge is not used. The embodiments of the present disclosure may be used in such arrangements as well.” | Paragraph 44 “As shown in FIG. 1 , areas of the aircraft such as the fuselage 103, the tail 111, the wings 113, and the engines 115 can be in the field of view of the camera image sensor 104 (e.g., illustrated in the top right arrangement). As discussed below, characteristics of these aircraft features and characteristics of other features can be used to identify the aircraft's type and/or subtype using the processes discussed herein.” | Paragraph 64 “An aircraft detection process is shown at 635, wherein a neural network-based (or any other method) object detection model can be developed to detect an approaching aircraft from the camera captured input video frames. The model is trained with set of aircraft images as a positive class and any other natural images as a negative class. The model learns the overall pattern of an aircraft shape which is agnostic to its class type. In the presence of an approaching aircraft, model starts processing of subsequent steps or else it skips processing of the following steps and loops back again to the aircraft detection logic for the next video frame. A sample detected aircraft is shown in FIG. 7A.” | Paragraph 101 “In some embodiments, the docking system will search the area it can sense for that aircraft type until it comes into the area. Once an aircraft is sensed, the docking system then checks to see if it is the correct type of aircraft. In some embodiments, camera image data and LIDAR/Radar sensor data can be compared to identify whether an aircraft is present in both the camera image data and the LIDAR/Radar sensor data. This can, for example, be done before merging the camera image data and the LIDAR/Radar sensor data. In this way, computing resources will not be wasted on analysis of objects that are not aircraft.”);
retrieving, by a computing device, a simulated two-dimensional image of the aircraft generated based on simulated data of the aircraft (See at least Pal FIGS. 4-5 and Paragraphs 53-56 “Before generating synthetic video of new aircraft, a 3D model 425 of the aircraft is needed. This can be developed using 3D simulation software and will be used for applying geometric parameters. Estimated rotation and scaling parameters with respect to the camera center are applied on the 3D aircraft model of the new aircraft at center of the model to fit the new aircraft correctly into the scale, position, and orientation of the bridge area viewed in the modified images. This produces rotated and scaled views of the new aircraft for different video frames. However, a 2D projection 426 (i.e., display screenshot) of this 3D model does not consider the aircraft backgrounds, nearby other aircraft, ground vehicles, different objects, and the taxi path. Hence, in the summing junction or synthesis parameter accumulator block of FIG. 4 all required parameters are captured, followed by performing image addition to superimpose a 2D projected new aircraft on a docking scenario (where no approaching aircraft is present) at the centroid position (m) calculated by translation parameters 428. In some embodiments a photometric variance 429 can be applied to blend the color dynamics of the images. Further, in various embodiments climate effects can be added to create images that have different climate 430. The generated synthetic frames produce new aircraft model docking scenario 431 as shown in FIG. 5 . This process can be repeated for all frames of images data, such as a stream of frames from a video. FIG. 5 illustrates a method flow for creating aircraft model data in accordance with one or more embodiments of the present disclosure. A new aircraft to be added to the system is shown in the image on the left side of FIG. 5 . Measurements are taken from the image regarding the aircraft to allow the system to create a 3D model of the new aircraft. The 3D model is then translated into a 2d projection that is scaled, positioned, and oriented in the same manner as the old aircraft in the original image (such scaling, position, and orientation data can be taken from the original image and stored in memory). In this manner, the synthetic image can look identical or substantially similar to an original image (except for the different aircraft type) and can, therefore, be used to train the system to identify the new aircraft and identify the aircraft when it actually approaches the bridge area.”); and
displaying an image while the aircraft is approaching the docking area to provide navigation guidance to a pilot (See at least Pal FIG. 3 and Paragraph 46 “FIG. 3 illustrates a display providing aircraft and alignment details to the pilot in accordance with one or more embodiments of the present disclosure. The device 312 is an advanced VDGS, meaning that it contains a image sensory, such as a camera and another sensor type, as described above. These sensors can, for example, be located in area 314 of the device 312. The device also includes a display 316 for displaying various information. For example, the display can present an aircraft type identifier 317 (e.g., A380). This can aid a pilot and/or ground crew in determining if the correct aircraft is approaching the bridge area. In the example of FIG. 3 , the display also includes a depiction of the lead-in line at stop line 316, position with respect thereto 320, and direction to the pilot 318 as to how to adjust the aircraft to align with the lead-in line.”).
Pal, however, fails to explicitly disclose tracking, by the computing device, a position of the aircraft while the aircraft is approaching the docking area using the LIDAR sensor data and the simulated two-dimensional image; and displaying an image including the position of the aircraft while the aircraft is approaching the docking area to provide navigation guidance to a pilot.
Deng teaches tracking, by the computing device, a position of the aircraft while the aircraft is approaching the docking area using the LIDAR sensor data and the simulated two-dimensional image; and displaying an image including the position of the aircraft while the aircraft is approaching the docking area to provide navigation guidance to a pilot (See at least Deng Paragraph 134 “Referring to Figure 5, which is a flowchart of the aircraft parking guidance process of the vision subsystem of the present invention. The scene configuration above the dotted horizontal line in the diagram can be executed after system installation but before berth guidance. The video image preprocessing, aircraft capture, aircraft tracking, aircraft positioning, aircraft identification and authentication, information fusion, and information display below the dotted horizontal line are executed during berth guidance. Among them, aircraft tracking, aircraft positioning, information fusion, and information display within the dashed box are executed and updated in real time during berth guidance. Before the machine vision verification step S2, image preprocessing can be performed, that is, using common image processing methods, including brightness correction, noise reduction, etc., to improve the visual effect of the image, increase the clarity of image components, or make the image more conducive to processing by computing devices.” | Paragraph 177 “In this embodiment, the aircraft tracking and positioning step S6 may also be included. This involves continuously tracking and updating in real-time the video images of the aircraft engine and nose wheel of the aircraft to be parked, captured in step S21, to achieve real-time positioning of the aircraft and accurately determine the degree of deviation of the aircraft from the guide line 42. The aircraft tracking and positioning step S6 may further include:” | Paragraph 182 “Step S32: Aircraft capture. The laser scanning system 1 continuously scans the capture area in the set aircraft parking scene in both horizontal and vertical directions, comparing the current scan line with the previous frame scan parameters until the docked aircraft 5 is captured, and the position of the aircraft's nose 51 is calculated. Horizontal scanning: The horizontal motor 13 rotates, driving the horizontal laser reflector 14 to rotate, causing the target point measured by the laser ranging device 3 to move horizontally. If the vertical motor 11 does not rotate, the horizontal motor 13 rotates, and the laser ranging device 3 continuously performs ranging, obtaining a set of ranging data in the horizontal direction, which constitutes one horizontal scan. Vertical scanning: The vertical motor 11 rotates, driving the vertical laser reflector 12 to rotate, causing the target point measured by the laser ranging device 3 to move vertically. If the horizontal motor 13 does not rotate, the vertical motor 11 rotates, and the laser ranging device 3 continuously performs ranging, obtaining a set of ranging data in the vertical direction, which constitutes one vertical scan.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Pal to include tracking, by the computing device, a position of the aircraft while the aircraft is approaching the docking area using the LIDAR sensor data and the simulated two-dimensional image; and displaying an image including the position of the aircraft while the aircraft is approaching the docking area to provide navigation guidance to a pilot, as taught by Deng as disclosed above, in order to ensure accurate aircraft navigation (Deng Paragraph 8 “The technical problem to be solved by the present invention is to provide a dock-entry aircraft type identification and verification system and method for detecting, identifying and verifying moving objects in aircraft parking guidance, so as to overcome the above-mentioned defects in the prior art.”).
With respect to claim 2, Pal in view of Deng teach that the simulated two-dimensional image is generated by: simulating LIDAR data as a three-dimensional image of the aircraft; and projecting the simulated LIDAR data over a two-dimensional image of the aircraft (See at least Pal FIGS. 4-5 and Paragraphs 53-56 “Before generating synthetic video of new aircraft, a 3D model 425 of the aircraft is needed. This can be developed using 3D simulation software and will be used for applying geometric parameters. Estimated rotation and scaling parameters with respect to the camera center are applied on the 3D aircraft model of the new aircraft at center of the model to fit the new aircraft correctly into the scale, position, and orientation of the bridge area viewed in the modified images. This produces rotated and scaled views of the new aircraft for different video frames. However, a 2D projection 426 (i.e., display screenshot) of this 3D model does not consider the aircraft backgrounds, nearby other aircraft, ground vehicles, different objects, and the taxi path. Hence, in the summing junction or synthesis parameter accumulator block of FIG. 4 all required parameters are captured, followed by performing image addition to superimpose a 2D projected new aircraft on a docking scenario (where no approaching aircraft is present) at the centroid position (m) calculated by translation parameters 428. In some embodiments a photometric variance 429 can be applied to blend the color dynamics of the images. Further, in various embodiments climate effects can be added to create images that have different climate 430. The generated synthetic frames produce new aircraft model docking scenario 431 as shown in FIG. 5 . This process can be repeated for all frames of images data, such as a stream of frames from a video. FIG. 5 illustrates a method flow for creating aircraft model data in accordance with one or more embodiments of the present disclosure. A new aircraft to be added to the system is shown in the image on the left side of FIG. 5 . Measurements are taken from the image regarding the aircraft to allow the system to create a 3D model of the new aircraft. The 3D model is then translated into a 2d projection that is scaled, positioned, and oriented in the same manner as the old aircraft in the original image (such scaling, position, and orientation data can be taken from the original image and stored in memory). In this manner, the synthetic image can look identical or substantially similar to an original image (except for the different aircraft type) and can, therefore, be used to train the system to identify the new aircraft and identify the aircraft when it actually approaches the bridge area.”).
With respect to claim 3, Pal in view of Deng teach that the method further includes: calibrating the LIDAR sensor to capture the LIDAR sensor data of the aircraft while the aircraft is approaching the docking area; receiving information corresponding to a trajectory of the aircraft; and simulating the three-dimensional image of the aircraft using the calibrated sensor and the information corresponding to the trajectory of the aircraft (See at least Pal Paragraph 34 “FIG. 1 illustrates a number of aircraft bridge area arrangements that can be used in accordance with one or more embodiments of the present disclosure. In FIG. 1 , the three arrangements at the top illustrate a number of embodiments using one sensor array (e.g., a combination of at least one camera image sensor and at least one laser/radar sensor, described herein as a LIDAR/Radar sensor, but embodiments are not so limited and the array could be one image sensor, such as a camera) with aircraft 101 located in bridge areas 100. The bottom three arrangements show embodiments having multiple sensor arrays, as will be discussed in more detail below.” | Paragraph 93 “This 6D of pose estimation can be accomplished by, for example, orienting the projection of the 3D model within an image plane by using a calibrated camera (wherein the image plane is defined as part of the calibration process and wherein the image plane is associated with a real world plane of the bridge area) and then comparing the projection with image contours of the aircraft (from the camera image data). From this data, the six dimensions of the aircraft in the image can be determined.”).
With respect to claim 4, Pal in view of Deng teach the three-dimensional image is simulated prior to an initiation of a docking operation of the aircraft (See at least Deng Paragraph 128 “The setup for the vision system 6 includes the following steps: After system installation and before berth guidance, the scene needs to be defined in the actual environment. A black and white striped ruler is laid next to the guide line 42, with the black and white stripes having the same length interval, and a maximum length interval of 1m. Depending on the resolution of the camera device 61, a more precise ruler with length intervals of 0.5m, 0.25m, etc., can be used. The total length of the ruler should not exceed the range for calculating the distance to the aircraft position, typically 50m. Other tasks are performed by pre-written software. The software opens and displays the image captured by the camera device 61, and marks the relevant areas by manually drawing lines, selection boxes, and points, and saves the records.” | Paragraph 134 “The scene configuration above the dotted horizontal line in the diagram can be executed after system installation but before berth guidance. The video image preprocessing, aircraft capture, aircraft tracking, aircraft positioning, aircraft identification and authentication, information fusion, and information display below the dotted horizontal line are executed during berth guidance. Among them, aircraft tracking, aircraft positioning, information fusion, and information display within the dashed box are executed and updated in real time during berth guidance.Before the machine vision verification step S2, image preprocessing can be performed, that is, using common image processing methods, including brightness correction, noise reduction, etc., to improve the visual effect of the image, increase the clarity of image components, or make the image more conducive to processing by computing devices.” | Paragraph 140 “Step S22, Aircraft identification and authentication: Before the aircraft enters the apron, the airport sends the aircraft type information to the berth system. The aircraft identification and authentication process verifies this aircraft type information by analyzing the image. The aircraft to be parked captured in step S21 is subjected to aircraft type and identity verification to further ensure that the aircraft 5 to be parked in the dock is parked safely and accurately. See Figures 6 and 7. Figure 6 is a flowchart of aircraft identification and verification, and Figure 7 is a schematic diagram of layered image structure.” | Paragraph 181 “The aircraft parking scenario configuration above the horizontal line in the diagram can be executed after system installation but before parking guidance. The data preprocessing, aircraft capture, aircraft guidance, aircraft tracking, aircraft identification and authentication, information fusion, and information display below the horizontal line can be executed during parking guidance. Among them, the aircraft guidance, aircraft tracking, information fusion, and information display within the dashed box are executed and updated in real time during parking guidance.”)
With respect to claim 5, Pal in view of Deng teach that the method further includes: feeding the simulated two-dimensional image into a perception model; and predicting positioning information of the aircraft using the perception model (See at least Pal FIGS. 4-5 and Paragraphs 53-56 “Before generating synthetic video of new aircraft, a 3D model 425 of the aircraft is needed. This can be developed using 3D simulation software and will be used for applying geometric parameters. Estimated rotation and scaling parameters with respect to the camera center are applied on the 3D aircraft model of the new aircraft at center of the model to fit the new aircraft correctly into the scale, position, and orientation of the bridge area viewed in the modified images. This produces rotated and scaled views of the new aircraft for different video frames. However, a 2D projection 426 (i.e., display screenshot) of this 3D model does not consider the aircraft backgrounds, nearby other aircraft, ground vehicles, different objects, and the taxi path. Hence, in the summing junction or synthesis parameter accumulator block of FIG. 4 all required parameters are captured, followed by performing image addition to superimpose a 2D projected new aircraft on a docking scenario (where no approaching aircraft is present) at the centroid position (m) calculated by translation parameters 428. In some embodiments a photometric variance 429 can be applied to blend the color dynamics of the images. Further, in various embodiments climate effects can be added to create images that have different climate 430. The generated synthetic frames produce new aircraft model docking scenario 431 as shown in FIG. 5 . This process can be repeated for all frames of images data, such as a stream of frames from a video. FIG. 5 illustrates a method flow for creating aircraft model data in accordance with one or more embodiments of the present disclosure. A new aircraft to be added to the system is shown in the image on the left side of FIG. 5 . Measurements are taken from the image regarding the aircraft to allow the system to create a 3D model of the new aircraft. The 3D model is then translated into a 2d projection that is scaled, positioned, and oriented in the same manner as the old aircraft in the original image (such scaling, position, and orientation data can be taken from the original image and stored in memory). In this manner, the synthetic image can look identical or substantially similar to an original image (except for the different aircraft type) and can, therefore, be used to train the system to identify the new aircraft and identify the aircraft when it actually approaches the bridge area.”).
With respect to claim 6, Pal in view of Deng teach that the positioning information includes information corresponding to a wheel position and bounding box of the aircraft (See at least Pal FIG. 7 and Paragraphs 67-68 “FIG. 7A illustrates a detected aircraft bounding box having an approaching aircraft located therein. FIG. 7B illustrates an instance segmentation predicted mask of an approaching aircraft's body parts, and FIG. 7C illustrates a complete mask of the approaching aircraft to get a total pixel count. A body part segmentation process is illustrated at 637. From the above generated instance body part segmented mask a plurality of components can be counted such as: aircraft engine, wheels, etc. and geometrical features can be measured such as: span of wings, engine position compared to wings, etc. This step helps in deriving intuitive information for docking guidance generation (i.e., if it has detected 4 engines with long wing separation, then it is a large aircraft). Perhaps it is a A320 or other such large aircraft approaching. All this information is passed to subsequent modules.”).
With respect to claim 7, Pal in view of Deng teach that the image is displayed in real-time while the aircraft is approaching the docking area (See at least Pal FIGS. 4-5 and Paragraphs 53-56 “Before generating synthetic video of new aircraft, a 3D model 425 of the aircraft is needed. This can be developed using 3D simulation software and will be used for applying geometric parameters. Estimated rotation and scaling parameters with respect to the camera center are applied on the 3D aircraft model of the new aircraft at center of the model to fit the new aircraft correctly into the scale, position, and orientation of the bridge area viewed in the modified images. This produces rotated and scaled views of the new aircraft for different video frames. However, a 2D projection 426 (i.e., display screenshot) of this 3D model does not consider the aircraft backgrounds, nearby other aircraft, ground vehicles, different objects, and the taxi path. Hence, in the summing junction or synthesis parameter accumulator block of FIG. 4 all required parameters are captured, followed by performing image addition to superimpose a 2D projected new aircraft on a docking scenario (where no approaching aircraft is present) at the centroid position (m) calculated by translation parameters 428. In some embodiments a photometric variance 429 can be applied to blend the color dynamics of the images. Further, in various embodiments climate effects can be added to create images that have different climate 430. The generated synthetic frames produce new aircraft model docking scenario 431 as shown in FIG. 5 . This process can be repeated for all frames of images data, such as a stream of frames from a video. FIG. 5 illustrates a method flow for creating aircraft model data in accordance with one or more embodiments of the present disclosure. A new aircraft to be added to the system is shown in the image on the left side of FIG. 5 . Measurements are taken from the image regarding the aircraft to allow the system to create a 3D model of the new aircraft. The 3D model is then translated into a 2d projection that is scaled, positioned, and oriented in the same manner as the old aircraft in the original image (such scaling, position, and orientation data can be taken from the original image and stored in memory). In this manner, the synthetic image can look identical or substantially similar to an original image (except for the different aircraft type) and can, therefore, be used to train the system to identify the new aircraft and identify the aircraft when it actually approaches the bridge area.”).
With respect to claim 8, Pal in view of Deng teach that the LIDAR sensor data is captured by a moveable LIDAR sensor from various locations (See at least Pal FIGS. 4-5 and Paragraphs 53-56 “Before generating synthetic video of new aircraft, a 3D model 425 of the aircraft is needed. This can be developed using 3D simulation software and will be used for applying geometric parameters. Estimated rotation and scaling parameters with respect to the camera center are applied on the 3D aircraft model of the new aircraft at center of the model to fit the new aircraft correctly into the scale, position, and orientation of the bridge area viewed in the modified images. This produces rotated and scaled views of the new aircraft for different video frames. However, a 2D projection 426 (i.e., display screenshot) of this 3D model does not consider the aircraft backgrounds, nearby other aircraft, ground vehicles, different objects, and the taxi path. Hence, in the summing junction or synthesis parameter accumulator block of FIG. 4 all required parameters are captured, followed by performing image addition to superimpose a 2D projected new aircraft on a docking scenario (where no approaching aircraft is present) at the centroid position (m) calculated by translation parameters 428. In some embodiments a photometric variance 429 can be applied to blend the color dynamics of the images. Further, in various embodiments climate effects can be added to create images that have different climate 430. The generated synthetic frames produce new aircraft model docking scenario 431 as shown in FIG. 5 . This process can be repeated for all frames of images data, such as a stream of frames from a video. FIG. 5 illustrates a method flow for creating aircraft model data in accordance with one or more embodiments of the present disclosure. A new aircraft to be added to the system is shown in the image on the left side of FIG. 5 . Measurements are taken from the image regarding the aircraft to allow the system to create a 3D model of the new aircraft. The 3D model is then translated into a 2d projection that is scaled, positioned, and oriented in the same manner as the old aircraft in the original image (such scaling, position, and orientation data can be taken from the original image and stored in memory). In this manner, the synthetic image can look identical or substantially similar to an original image (except for the different aircraft type) and can, therefore, be used to train the system to identify the new aircraft and identify the aircraft when it actually approaches the bridge area.”).
With respect to claim 9, Pal teaches a method for tracking an aircraft, comprising:
capturing, by a light detection and ranging (LIDAR) sensor, LIDAR sensor data of an aircraft while the aircraft is approaching a docking area of the airport, wherein the LIDAR sensor is positioned within the docking area (See at least Pal Paragraphs 34-35 “FIG. 1 illustrates a number of aircraft bridge area arrangements that can be used in accordance with one or more embodiments of the present disclosure. In FIG. 1 , the three arrangements at the top illustrate a number of embodiments using one sensor array (e.g., a combination of at least one camera image sensor and at least one laser/radar sensor, described herein as a LIDAR/Radar sensor, but embodiments are not so limited and the array could be one image sensor, such as a camera) with aircraft 101 located in bridge areas 100. The bottom three arrangements show embodiments having multiple sensor arrays, as will be discussed in more detail below. Each arrangement has several common elements. For example, all of the arrangements include a bridge area wherein at least a portion of the bridge area is within the field of view 104 of at least one sensor of a sensor array 102. Although described herein as a “bridge area,” the term bridge area as used herein merely means a parking space for an aircraft located on the airside apron of a gate at an airport and although described herein using the term “bridge,” the space may sometimes be in a location where, for example, passengers and crew access the aircraft via stairs from the tarmac and a bridge is not used. The embodiments of the present disclosure may be used in such arrangements as well.” | Paragraph 44 “As shown in FIG. 1 , areas of the aircraft such as the fuselage 103, the tail 111, the wings 113, and the engines 115 can be in the field of view of the camera image sensor 104 (e.g., illustrated in the top right arrangement). As discussed below, characteristics of these aircraft features and characteristics of other features can be used to identify the aircraft's type and/or subtype using the processes discussed herein.” | Paragraph 64 “An aircraft detection process is shown at 635, wherein a neural network-based (or any other method) object detection model can be developed to detect an approaching aircraft from the camera captured input video frames. The model is trained with set of aircraft images as a positive class and any other natural images as a negative class. The model learns the overall pattern of an aircraft shape which is agnostic to its class type. In the presence of an approaching aircraft, model starts processing of subsequent steps or else it skips processing of the following steps and loops back again to the aircraft detection logic for the next video frame. A sample detected aircraft is shown in FIG. 7A.” | Paragraph 101 “In some embodiments, the docking system will search the area it can sense for that aircraft type until it comes into the area. Once an aircraft is sensed, the docking system then checks to see if it is the correct type of aircraft. In some embodiments, camera image data and LIDAR/Radar sensor data can be compared to identify whether an aircraft is present in both the camera image data and the LIDAR/Radar sensor data. This can, for example, be done before merging the camera image data and the LIDAR/Radar sensor data. In this way, computing resources will not be wasted on analysis of objects that are not aircraft.”);
retrieving, by a computing device, a simulated two-dimensional image of the aircraft generated based on simulated data of the aircraft (See at least Pal FIGS. 4-5 and Paragraphs 53-56 “Before generating synthetic video of new aircraft, a 3D model 425 of the aircraft is needed. This can be developed using 3D simulation software and will be used for applying geometric parameters. Estimated rotation and scaling parameters with respect to the camera center are applied on the 3D aircraft model of the new aircraft at center of the model to fit the new aircraft correctly into the scale, position, and orientation of the bridge area viewed in the modified images. This produces rotated and scaled views of the new aircraft for different video frames. However, a 2D projection 426 (i.e., display screenshot) of this 3D model does not consider the aircraft backgrounds, nearby other aircraft, ground vehicles, different objects, and the taxi path. Hence, in the summing junction or synthesis parameter accumulator block of FIG. 4 all required parameters are captured, followed by performing image addition to superimpose a 2D projected new aircraft on a docking scenario (where no approaching aircraft is present) at the centroid position (m) calculated by translation parameters 428. In some embodiments a photometric variance 429 can be applied to blend the color dynamics of the images. Further, in various embodiments climate effects can be added to create images that have different climate 430. The generated synthetic frames produce new aircraft model docking scenario 431 as shown in FIG. 5 . This process can be repeated for all frames of images data, such as a stream of frames from a video. FIG. 5 illustrates a method flow for creating aircraft model data in accordance with one or more embodiments of the present disclosure. A new aircraft to be added to the system is shown in the image on the left side of FIG. 5 . Measurements are taken from the image regarding the aircraft to allow the system to create a 3D model of the new aircraft. The 3D model is then translated into a 2d projection that is scaled, positioned, and oriented in the same manner as the old aircraft in the original image (such scaling, position, and orientation data can be taken from the original image and stored in memory). In this manner, the synthetic image can look identical or substantially similar to an original image (except for the different aircraft type) and can, therefore, be used to train the system to identify the new aircraft and identify the aircraft when it actually approaches the bridge area.”); and
displaying an image while the aircraft is approaching the docking area to provide navigation guidance to a pilot (See at least Pal FIG. 3 and Paragraph 46 “FIG. 3 illustrates a display providing aircraft and alignment details to the pilot in accordance with one or more embodiments of the present disclosure. The device 312 is an advanced VDGS, meaning that it contains a image sensory, such as a camera and another sensor type, as described above. These sensors can, for example, be located in area 314 of the device 312. The device also includes a display 316 for displaying various information. For example, the display can present an aircraft type identifier 317 (e.g., A380). This can aid a pilot and/or ground crew in determining if the correct aircraft is approaching the bridge area. In the example of FIG. 3 , the display also includes a depiction of the lead-in line at stop line 316, position with respect thereto 320, and direction to the pilot 318 as to how to adjust the aircraft to align with the lead-in line.”).
Pal, however, fails to explicitly disclose tracking, by the computing device, a position of the aircraft while the aircraft is approaching the docking area using the LIDAR sensor data and the simulated two-dimensional image; and displaying an image including the position of the aircraft while the aircraft is approaching the docking area to provide navigation guidance to a pilot.
Deng teaches tracking, by the computing device, a position of the aircraft while the aircraft is approaching the docking area using the LIDAR sensor data and the simulated two-dimensional image; and displaying an image including the position of the aircraft while the aircraft is approaching the docking area to provide navigation guidance to a pilot (See at least Deng Paragraph 134 “Referring to Figure 5, which is a flowchart of the aircraft parking guidance process of the vision subsystem of the present invention. The scene configuration above the dotted horizontal line in the diagram can be executed after system installation but before berth guidance. The video image preprocessing, aircraft capture, aircraft tracking, aircraft positioning, aircraft identification and authentication, information fusion, and information display below the dotted horizontal line are executed during berth guidance. Among them, aircraft tracking, aircraft positioning, information fusion, and information display within the dashed box are executed and updated in real time during berth guidance. Before the machine vision verification step S2, image preprocessing can be performed, that is, using common image processing methods, including brightness correction, noise reduction, etc., to improve the visual effect of the image, increase the clarity of image components, or make the image more conducive to processing by computing devices.” | Paragraph 177 “In this embodiment, the aircraft tracking and positioning step S6 may also be included. This involves continuously tracking and updating in real-time the video images of the aircraft engine and nose wheel of the aircraft to be parked, captured in step S21, to achieve real-time positioning of the aircraft and accurately determine the degree of deviation of the aircraft from the guide line 42. The aircraft tracking and positioning step S6 may further include:” | Paragraph 182 “Step S32: Aircraft capture. The laser scanning system 1 continuously scans the capture area in the set aircraft parking scene in both horizontal and vertical directions, comparing the current scan line with the previous frame scan parameters until the docked aircraft 5 is captured, and the position of the aircraft's nose 51 is calculated. Horizontal scanning: The horizontal motor 13 rotates, driving the horizontal laser reflector 14 to rotate, causing the target point measured by the laser ranging device 3 to move horizontally. If the vertical motor 11 does not rotate, the horizontal motor 13 rotates, and the laser ranging device 3 continuously performs ranging, obtaining a set of ranging data in the horizontal direction, which constitutes one horizontal scan. Vertical scanning: The vertical motor 11 rotates, driving the vertical laser reflector 12 to rotate, causing the target point measured by the laser ranging device 3 to move vertically. If the horizontal motor 13 does not rotate, the vertical motor 11 rotates, and the laser ranging device 3 continuously performs ranging, obtaining a set of ranging data in the vertical direction, which constitutes one vertical scan.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Pal to include tracking, by the computing device, a position of the aircraft while the aircraft is approaching the docking area using the LIDAR sensor data and the simulated two-dimensional image; and displaying an image including the position of the aircraft while the aircraft is approaching the docking area to provide navigation guidance to a pilot, as taught by Deng as disclosed above, in order to ensure accurate aircraft navigation (Deng Paragraph 8 “The technical problem to be solved by the present invention is to provide a dock-entry aircraft type identification and verification system and method for detecting, identifying and verifying moving objects in aircraft parking guidance, so as to overcome the above-mentioned defects in the prior art.”).
With respect to claim 10, Pal in view of Deng teach the LIDAR sensor data is three-dimensional data captured by the LIDAR sensor in real-time as the aircraft is approaching the docking area (See at least Pal Paragraphs 34-35 “FIG. 1 illustrates a number of aircraft bridge area arrangements that can be used in accordance with one or more embodiments of the present disclosure. In FIG. 1 , the three arrangements at the top illustrate a number of embodiments using one sensor array (e.g., a combination of at least one camera image sensor and at least one laser/radar sensor, described herein as a LIDAR/Radar sensor, but embodiments are not so limited and the array could be one image sensor, such as a camera) with aircraft 101 located in bridge areas 100. The bottom three arrangements show embodiments having multiple sensor arrays, as will be discussed in more detail below. Each arrangement has several common elements. For example, all of the arrangements include a bridge area wherein at least a portion of the bridge area is within the field of view 104 of at least one sensor of a sensor array 102. Although described herein as a “bridge area,” the term bridge area as used herein merely means a parking space for an aircraft located on the airside apron of a gate at an airport and although described herein using the term “bridge,” the space may sometimes be in a location where, for example, passengers and crew access the aircraft via stairs from the tarmac and a bridge is not used. The embodiments of the present disclosure may be used in such arrangements as well.” | Paragraph 44 “As shown in FIG. 1 , areas of the aircraft such as the fuselage 103, the tail 111, the wings 113, and the engines 115 can be in the field of view of the camera image sensor 104 (e.g., illustrated in the top right arrangement). As discussed below, characteristics of these aircraft features and characteristics of other features can be used to identify the aircraft's type and/or subtype using the processes discussed herein.” | Paragraph 64 “An aircraft detection process is shown at 635, wherein a neural network-based (or any other method) object detection model can be developed to detect an approaching aircraft from the camera captured input video frames. The model is trained with set of aircraft images as a positive class and any other natural images as a negative class. The model learns the overall pattern of an aircraft shape which is agnostic to its class type. In the presence of an approaching aircraft, model starts processing of subsequent steps or else it skips processing of the following steps and loops back again to the aircraft detection logic for the next video frame. A sample detected aircraft is shown in FIG. 7A.” | Paragraph 101 “In some embodiments, the docking system will search the area it can sense for that aircraft type until it comes into the area. Once an aircraft is sensed, the docking system then checks to see if it is the correct type of aircraft. In some embodiments, camera image data and LIDAR/Radar sensor data can be compared to identify whether an aircraft is present in both the camera image data and the LIDAR/Radar sensor data. This can, for example, be done before merging the camera image data and the LIDAR/Radar sensor data. In this way, computing resources will not be wasted on analysis of objects that are not aircraft.”);
With respect to claim 11, Pal in view of Deng teach that the simulated two-dimensional image is generated by: simulating LIDAR data as a three-dimensional point cloud of the aircraft; andprojecting the three-dimensional point cloud onto a two-dimensional image of the aircraft (See at least Pal FIGS. 4-5 and Paragraphs 53-56 “Before generating synthetic video of new aircraft, a 3D model 425 of the aircraft is needed. This can be developed using 3D simulation software and will be used for applying geometric parameters. Estimated rotation and scaling parameters with respect to the camera center are applied on the 3D aircraft model of the new aircraft at center of the model to fit the new aircraft correctly into the scale, position, and orientation of the bridge area viewed in the modified images. This produces rotated and scaled views of the new aircraft for different video frames. However, a 2D projection 426 (i.e., display screenshot) of this 3D model does not consider the aircraft backgrounds, nearby other aircraft, ground vehicles, different objects, and the taxi path. Hence, in the summing junction or synthesis parameter accumulator block of FIG. 4 all required parameters are captured, followed by performing image addition to superimpose a 2D projected new aircraft on a docking scenario (where no approaching aircraft is present) at the centroid position (m) calculated by translation parameters 428. In some embodiments a photometric variance 429 can be applied to blend the color dynamics of the images. Further, in various embodiments climate effects can be added to create images that have different climate 430. The generated synthetic frames produce new aircraft model docking scenario 431 as shown in FIG. 5 . This process can be repeated for all frames of images data, such as a stream of frames from a video. FIG. 5 illustrates a method flow for creating aircraft model data in accordance with one or more embodiments of the present disclosure. A new aircraft to be added to the system is shown in the image on the left side of FIG. 5 . Measurements are taken from the image regarding the aircraft to allow the system to create a 3D model of the new aircraft. The 3D model is then translated into a 2d projection that is scaled, positioned, and oriented in the same manner as the old aircraft in the original image (such scaling, position, and orientation data can be taken from the original image and stored in memory). In this manner, the synthetic image can look identical or substantially similar to an original image (except for the different aircraft type) and can, therefore, be used to train the system to identify the new aircraft and identify the aircraft when it actually approaches the bridge area.”).
With respect to claim 12, Pal in view of Deng teach that the simulated two-dimensional image is simulated to be in the docking area within which the LIDAR sensor is positioned (See at least Pal Paragraphs 80-81 “In embodiments of the present disclosure, an image processing system is used to create an initial aircraft 3D model (these functions are shown at 954 and 956 of FIG. 9 ) and LIDAR/Radar point cloud analytics (these functions are shown at 948 and 950 of FIG. 9 ), utilized by a LIDAR/Radar module, provide the point cloud for the aircraft that is nearing the bridge area. In some embodiments, the embodiment includes machine learning/deep learning capabilities wherein both of the data outputs get fused by the data fusion module to derive the hybrid accurate aircraft model and/or type and/or subtype detection as well as position and orientation of the aircraft. In some embodiments, the LIDAR/Radar point cloud creation, image processing 3D model creation, and data fusion can be real-time processes, depending on the computing and networking speed of the docking system, which derives the speed and position of the aircraft. This will provide the input to the central docking system controller to provide the clear information to use in pilot unit display messages that can be used to provide directional and/or speed guidance to the pilot.”)
With respect to claim 13, Pal in view of Deng teach that the simulated two-dimensional image is fed into a perception model and moved along a known trajectory of the aircraft to train the perception model (See at least Pal FIGS. 4-5 and Paragraphs 53-56 “Before generating synthetic video of new aircraft, a 3D model 425 of the aircraft is needed. This can be developed using 3D simulation software and will be used for applying geometric parameters. Estimated rotation and scaling parameters with respect to the camera center are applied on the 3D aircraft model of the new aircraft at center of the model to fit the new aircraft correctly into the scale, position, and orientation of the bridge area viewed in the modified images. This produces rotated and scaled views of the new aircraft for different video frames. However, a 2D projection 426 (i.e., display screenshot) of this 3D model does not consider the aircraft backgrounds, nearby other aircraft, ground vehicles, different objects, and the taxi path. Hence, in the summing junction or synthesis parameter accumulator block of FIG. 4 all required parameters are captured, followed by performing image addition to superimpose a 2D projected new aircraft on a docking scenario (where no approaching aircraft is present) at the centroid position (m) calculated by translation parameters 428. In some embodiments a photometric variance 429 can be applied to blend the color dynamics of the images. Further, in various embodiments climate effects can be added to create images that have different climate 430. The generated synthetic frames produce new aircraft model docking scenario 431 as shown in FIG. 5 . This process can be repeated for all frames of images data, such as a stream of frames from a video. FIG. 5 illustrates a method flow for creating aircraft model data in accordance with one or more embodiments of the present disclosure. A new aircraft to be added to the system is shown in the image on the left side of FIG. 5 . Measurements are taken from the image regarding the aircraft to allow the system to create a 3D model of the new aircraft. The 3D model is then translated into a 2d projection that is scaled, positioned, and oriented in the same manner as the old aircraft in the original image (such scaling, position, and orientation data can be taken from the original image and stored in memory). In this manner, the synthetic image can look identical or substantially similar to an original image (except for the different aircraft type) and can, therefore, be used to train the system to identify the new aircraft and identify the aircraft when it actually approaches the bridge area.”).
With respect to claim 14, Pal in view of Deng teach that the trained perception model is used to predict a position and height of the aircraft as part of the provided navigation guidance to the pilot (See at least Pal FIGS. 4-5 and Paragraphs 53-56 “Before generating synthetic video of new aircraft, a 3D model 425 of the aircraft is needed. This can be developed using 3D simulation software and will be used for applying geometric parameters. Estimated rotation and scaling parameters with respect to the camera center are applied on the 3D aircraft model of the new aircraft at center of the model to fit the new aircraft correctly into the scale, position, and orientation of the bridge area viewed in the modified images. This produces rotated and scaled views of the new aircraft for different video frames. However, a 2D projection 426 (i.e., display screenshot) of this 3D model does not consider the aircraft backgrounds, nearby other aircraft, ground vehicles, different objects, and the taxi path. Hence, in the summing junction or synthesis parameter accumulator block of FIG. 4 all required parameters are captured, followed by performing image addition to superimpose a 2D projected new aircraft on a docking scenario (where no approaching aircraft is present) at the centroid position (m) calculated by translation parameters 428. In some embodiments a photometric variance 429 can be applied to blend the color dynamics of the images. Further, in various embodiments climate effects can be added to create images that have different climate 430. The generated synthetic frames produce new aircraft model docking scenario 431 as shown in FIG. 5 . This process can be repeated for all frames of images data, such as a stream of frames from a video. FIG. 5 illustrates a method flow for creating aircraft model data in accordance with one or more embodiments of the present disclosure. A new aircraft to be added to the system is shown in the image on the left side of FIG. 5 . Measurements are taken from the image regarding the aircraft to allow the system to create a 3D model of the new aircraft. The 3D model is then translated into a 2d projection that is scaled, positioned, and oriented in the same manner as the old aircraft in the original image (such scaling, position, and orientation data can be taken from the original image and stored in memory). In this manner, the synthetic image can look identical or substantially similar to an original image (except for the different aircraft type) and can, therefore, be used to train the system to identify the new aircraft and identify the aircraft when it actually approaches the bridge area.”).
With respect to claim 15, Pal in view of Deng teach the trained perception model is used to derive a wheel position and bounding box of the aircraft as part of the provided navigation guidance to the pilot (See at least Pal FIGS. 4-5 and Paragraph 46 “FIG. 3 illustrates a display providing aircraft and alignment details to the pilot in accordance with one or more embodiments of the present disclosure. The device 312 is an advanced VDGS, meaning that it contains a image sensory, such as a camera and another sensor type, as described above. These sensors can, for example, be located in area 314 of the device 312. The device also includes a display 316 for displaying various information. For example, the display can present an aircraft type identifier 317 (e.g., A380). This can aid a pilot and/or ground crew in determining if the correct aircraft is approaching the bridge area. In the example of FIG. 3 , the display also includes a depiction of the lead-in line at stop line 316, position with respect thereto 320, and direction to the pilot 318 as to how to adjust the aircraft to align with the lead-in line.” | Paragraphs 53-56 “Before generating synthetic video of new aircraft, a 3D model 425 of the aircraft is needed. This can be developed using 3D simulation software and will be used for applying geometric parameters. Estimated rotation and scaling parameters with respect to the camera center are applied on the 3D aircraft model of the new aircraft at center of the model to fit the new aircraft correctly into the scale, position, and orientation of the bridge area viewed in the modified images. This produces rotated and scaled views of the new aircraft for different video frames. However, a 2D projection 426 (i.e., display screenshot) of this 3D model does not consider the aircraft backgrounds, nearby other aircraft, ground vehicles, different objects, and the taxi path. Hence, in the summing junction or synthesis parameter accumulator block of FIG. 4 all required parameters are captured, followed by performing image addition to superimpose a 2D projected new aircraft on a docking scenario (where no approaching aircraft is present) at the centroid position (m) calculated by translation parameters 428. In some embodiments a photometric variance 429 can be applied to blend the color dynamics of the images. Further, in various embodiments climate effects can be added to create images that have different climate 430. The generated synthetic frames produce new aircraft model docking scenario 431 as shown in FIG. 5 . This process can be repeated for all frames of images data, such as a stream of frames from a video. FIG. 5 illustrates a method flow for creating aircraft model data in accordance with one or more embodiments of the present disclosure. A new aircraft to be added to the system is shown in the image on the left side of FIG. 5 . Measurements are taken from the image regarding the aircraft to allow the system to create a 3D model of the new aircraft. The 3D model is then translated into a 2d projection that is scaled, positioned, and oriented in the same manner as the old aircraft in the original image (such scaling, position, and orientation data can be taken from the original image and stored in memory). In this manner, the synthetic image can look identical or substantially similar to an original image (except for the different aircraft type) and can, therefore, be used to train the system to identify the new aircraft and identify the aircraft when it actually approaches the bridge area.”).
With respect to claim 16, Pal teaches a system for tracking an aircraft, comprising: a light detection and ranging (LIDAR) sensor configured to capture LIDAR sensor data of an aircraft while the aircraft is approaching a docking area of an airport; and a computing device configured to
capturing, by a light detection and ranging (LIDAR) sensor, LIDAR sensor data of an aircraft while the aircraft is approaching a docking area of an airport; receiving, from the LIDAR sensor, the sensor data of the aircraft while the aircraft is approaching the docking area (See at least Pal Paragraphs 34-35 “FIG. 1 illustrates a number of aircraft bridge area arrangements that can be used in accordance with one or more embodiments of the present disclosure. In FIG. 1 , the three arrangements at the top illustrate a number of embodiments using one sensor array (e.g., a combination of at least one camera image sensor and at least one laser/radar sensor, described herein as a LIDAR/Radar sensor, but embodiments are not so limited and the array could be one image sensor, such as a camera) with aircraft 101 located in bridge areas 100. The bottom three arrangements show embodiments having multiple sensor arrays, as will be discussed in more detail below. Each arrangement has several common elements. For example, all of the arrangements include a bridge area wherein at least a portion of the bridge area is within the field of view 104 of at least one sensor of a sensor array 102. Although described herein as a “bridge area,” the term bridge area as used herein merely means a parking space for an aircraft located on the airside apron of a gate at an airport and although described herein using the term “bridge,” the space may sometimes be in a location where, for example, passengers and crew access the aircraft via stairs from the tarmac and a bridge is not used. The embodiments of the present disclosure may be used in such arrangements as well.” | Paragraph 44 “As shown in FIG. 1 , areas of the aircraft such as the fuselage 103, the tail 111, the wings 113, and the engines 115 can be in the field of view of the camera image sensor 104 (e.g., illustrated in the top right arrangement). As discussed below, characteristics of these aircraft features and characteristics of other features can be used to identify the aircraft's type and/or subtype using the processes discussed herein.” | Paragraph 64 “An aircraft detection process is shown at 635, wherein a neural network-based (or any other method) object detection model can be developed to detect an approaching aircraft from the camera captured input video frames. The model is trained with set of aircraft images as a positive class and any other natural images as a negative class. The model learns the overall pattern of an aircraft shape which is agnostic to its class type. In the presence of an approaching aircraft, model starts processing of subsequent steps or else it skips processing of the following steps and loops back again to the aircraft detection logic for the next video frame. A sample detected aircraft is shown in FIG. 7A.” | Paragraph 101 “In some embodiments, the docking system will search the area it can sense for that aircraft type until it comes into the area. Once an aircraft is sensed, the docking system then checks to see if it is the correct type of aircraft. In some embodiments, camera image data and LIDAR/Radar sensor data can be compared to identify whether an aircraft is present in both the camera image data and the LIDAR/Radar sensor data. This can, for example, be done before merging the camera image data and the LIDAR/Radar sensor data. In this way, computing resources will not be wasted on analysis of objects that are not aircraft.”);
retrieving, by a computing device, a simulated two-dimensional image of the aircraft generated based on simulated data of the aircraft (See at least Pal FIGS. 4-5 and Paragraphs 53-56 “Before generating synthetic video of new aircraft, a 3D model 425 of the aircraft is needed. This can be developed using 3D simulation software and will be used for applying geometric parameters. Estimated rotation and scaling parameters with respect to the camera center are applied on the 3D aircraft model of the new aircraft at center of the model to fit the new aircraft correctly into the scale, position, and orientation of the bridge area viewed in the modified images. This produces rotated and scaled views of the new aircraft for different video frames. However, a 2D projection 426 (i.e., display screenshot) of this 3D model does not consider the aircraft backgrounds, nearby other aircraft, ground vehicles, different objects, and the taxi path. Hence, in the summing junction or synthesis parameter accumulator block of FIG. 4 all required parameters are captured, followed by performing image addition to superimpose a 2D projected new aircraft on a docking scenario (where no approaching aircraft is present) at the centroid position (m) calculated by translation parameters 428. In some embodiments a photometric variance 429 can be applied to blend the color dynamics of the images. Further, in various embodiments climate effects can be added to create images that have different climate 430. The generated synthetic frames produce new aircraft model docking scenario 431 as shown in FIG. 5 . This process can be repeated for all frames of images data, such as a stream of frames from a video. FIG. 5 illustrates a method flow for creating aircraft model data in accordance with one or more embodiments of the present disclosure. A new aircraft to be added to the system is shown in the image on the left side of FIG. 5 . Measurements are taken from the image regarding the aircraft to allow the system to create a 3D model of the new aircraft. The 3D model is then translated into a 2d projection that is scaled, positioned, and oriented in the same manner as the old aircraft in the original image (such scaling, position, and orientation data can be taken from the original image and stored in memory). In this manner, the synthetic image can look identical or substantially similar to an original image (except for the different aircraft type) and can, therefore, be used to train the system to identify the new aircraft and identify the aircraft when it actually approaches the bridge area.”); and
displaying an image while the aircraft is approaching the docking area to provide navigation guidance to a pilot (See at least Pal FIG. 3 and Paragraph 46 “FIG. 3 illustrates a display providing aircraft and alignment details to the pilot in accordance with one or more embodiments of the present disclosure. The device 312 is an advanced VDGS, meaning that it contains a image sensory, such as a camera and another sensor type, as described above. These sensors can, for example, be located in area 314 of the device 312. The device also includes a display 316 for displaying various information. For example, the display can present an aircraft type identifier 317 (e.g., A380). This can aid a pilot and/or ground crew in determining if the correct aircraft is approaching the bridge area. In the example of FIG. 3 , the display also includes a depiction of the lead-in line at stop line 316, position with respect thereto 320, and direction to the pilot 318 as to how to adjust the aircraft to align with the lead-in line.”).
Pal, however, fails to explicitly disclose tracking, by the computing device, a position of the aircraft while the aircraft is approaching the docking area using the LIDAR sensor data and the simulated two-dimensional image; and displaying an image including the position of the aircraft while the aircraft is approaching the docking area to provide navigation guidance to a pilot.
Deng teaches tracking, by the computing device, a position of the aircraft while the aircraft is approaching the docking area using the LIDAR sensor data and the simulated two-dimensional image; and displaying an image including the position of the aircraft while the aircraft is approaching the docking area to provide navigation guidance to a pilot (See at least Deng Paragraph 134 “Referring to Figure 5, which is a flowchart of the aircraft parking guidance process of the vision subsystem of the present invention. The scene configuration above the dotted horizontal line in the diagram can be executed after system installation but before berth guidance. The video image preprocessing, aircraft capture, aircraft tracking, aircraft positioning, aircraft identification and authentication, information fusion, and information display below the dotted horizontal line are executed during berth guidance. Among them, aircraft tracking, aircraft positioning, information fusion, and information display within the dashed box are executed and updated in real time during berth guidance. Before the machine vision verification step S2, image preprocessing can be performed, that is, using common image processing methods, including brightness correction, noise reduction, etc., to improve the visual effect of the image, increase the clarity of image components, or make the image more conducive to processing by computing devices.” | Paragraph 177 “In this embodiment, the aircraft tracking and positioning step S6 may also be included. This involves continuously tracking and updating in real-time the video images of the aircraft engine and nose wheel of the aircraft to be parked, captured in step S21, to achieve real-time positioning of the aircraft and accurately determine the degree of deviation of the aircraft from the guide line 42. The aircraft tracking and positioning step S6 may further include:” | Paragraph 182 “Step S32: Aircraft capture. The laser scanning system 1 continuously scans the capture area in the set aircraft parking scene in both horizontal and vertical directions, comparing the current scan line with the previous frame scan parameters until the docked aircraft 5 is captured, and the position of the aircraft's nose 51 is calculated. Horizontal scanning: The horizontal motor 13 rotates, driving the horizontal laser reflector 14 to rotate, causing the target point measured by the laser ranging device 3 to move horizontally. If the vertical motor 11 does not rotate, the horizontal motor 13 rotates, and the laser ranging device 3 continuously performs ranging, obtaining a set of ranging data in the horizontal direction, which constitutes one horizontal scan. Vertical scanning: The vertical motor 11 rotates, driving the vertical laser reflector 12 to rotate, causing the target point measured by the laser ranging device 3 to move vertically. If the horizontal motor 13 does not rotate, the vertical motor 11 rotates, and the laser ranging device 3 continuously performs ranging, obtaining a set of ranging data in the vertical direction, which constitutes one vertical scan.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Pal to include tracking, by the computing device, a position of the aircraft while the aircraft is approaching the docking area using the LIDAR sensor data and the simulated two-dimensional image; and displaying an image including the position of the aircraft while the aircraft is approaching the docking area to provide navigation guidance to a pilot, as taught by Deng as disclosed above, in order to ensure accurate aircraft navigation (Deng Paragraph 8 “The technical problem to be solved by the present invention is to provide a dock-entry aircraft type identification and verification system and method for detecting, identifying and verifying moving objects in aircraft parking guidance, so as to overcome the above-mentioned defects in the prior art.”).
With respect to claim 17, Pal in view of Deng teach that the computing device is configured to generates the simulated two-dimensional image by using a point cloud to correlate simulated LIDAR data points to points in a two-dimensional image of the aircraft (See at least Pal FIGS. 4-5 and Paragraphs 53-56 “Before generating synthetic video of new aircraft, a 3D model 425 of the aircraft is needed. This can be developed using 3D simulation software and will be used for applying geometric parameters. Estimated rotation and scaling parameters with respect to the camera center are applied on the 3D aircraft model of the new aircraft at center of the model to fit the new aircraft correctly into the scale, position, and orientation of the bridge area viewed in the modified images. This produces rotated and scaled views of the new aircraft for different video frames. However, a 2D projection 426 (i.e., display screenshot) of this 3D model does not consider the aircraft backgrounds, nearby other aircraft, ground vehicles, different objects, and the taxi path. Hence, in the summing junction or synthesis parameter accumulator block of FIG. 4 all required parameters are captured, followed by performing image addition to superimpose a 2D projected new aircraft on a docking scenario (where no approaching aircraft is present) at the centroid position (m) calculated by translation parameters 428. In some embodiments a photometric variance 429 can be applied to blend the color dynamics of the images. Further, in various embodiments climate effects can be added to create images that have different climate 430. The generated synthetic frames produce new aircraft model docking scenario 431 as shown in FIG. 5 . This process can be repeated for all frames of images data, such as a stream of frames from a video. FIG. 5 illustrates a method flow for creating aircraft model data in accordance with one or more embodiments of the present disclosure. A new aircraft to be added to the system is shown in the image on the left side of FIG. 5 . Measurements are taken from the image regarding the aircraft to allow the system to create a 3D model of the new aircraft. The 3D model is then translated into a 2d projection that is scaled, positioned, and oriented in the same manner as the old aircraft in the original image (such scaling, position, and orientation data can be taken from the original image and stored in memory). In this manner, the synthetic image can look identical or substantially similar to an original image (except for the different aircraft type) and can, therefore, be used to train the system to identify the new aircraft and identify the aircraft when it actually approaches the bridge area.”).
With respect to claim 18, Pal in view of Deng teach that the system includes a virtual camera configured to collect different viewpoints of the aircraft as it is approaching the docking area (See at least Pal Paragraphs 34-35 “FIG. 1 illustrates a number of aircraft bridge area arrangements that can be used in accordance with one or more embodiments of the present disclosure. In FIG. 1 , the three arrangements at the top illustrate a number of embodiments using one sensor array (e.g., a combination of at least one camera image sensor and at least one laser/radar sensor, described herein as a LIDAR/Radar sensor, but embodiments are not so limited and the array could be one image sensor, such as a camera) with aircraft 101 located in bridge areas 100. The bottom three arrangements show embodiments having multiple sensor arrays, as will be discussed in more detail below. Each arrangement has several common elements. For example, all of the arrangements include a bridge area wherein at least a portion of the bridge area is within the field of view 104 of at least one sensor of a sensor array 102. Although described herein as a “bridge area,” the term bridge area as used herein merely means a parking space for an aircraft located on the airside apron of a gate at an airport and although described herein using the term “bridge,” the space may sometimes be in a location where, for example, passengers and crew access the aircraft via stairs from the tarmac and a bridge is not used. The embodiments of the present disclosure may be used in such arrangements as well.” | Paragraph 44 “As shown in FIG. 1 , areas of the aircraft such as the fuselage 103, the tail 111, the wings 113, and the engines 115 can be in the field of view of the camera image sensor 104 (e.g., illustrated in the top right arrangement). As discussed below, characteristics of these aircraft features and characteristics of other features can be used to identify the aircraft's type and/or subtype using the processes discussed herein.” | Paragraph 64 “An aircraft detection process is shown at 635, wherein a neural network-based (or any other method) object detection model can be developed to detect an approaching aircraft from the camera captured input video frames. The model is trained with set of aircraft images as a positive class and any other natural images as a negative class. The model learns the overall pattern of an aircraft shape which is agnostic to its class type. In the presence of an approaching aircraft, model starts processing of subsequent steps or else it skips processing of the following steps and loops back again to the aircraft detection logic for the next video frame. A sample detected aircraft is shown in FIG. 7A.” | Paragraph 101 “In some embodiments, the docking system will search the area it can sense for that aircraft type until it comes into the area. Once an aircraft is sensed, the docking system then checks to see if it is the correct type of aircraft. In some embodiments, camera image data and LIDAR/Radar sensor data can be compared to identify whether an aircraft is present in both the camera image data and the LIDAR/Radar sensor data. This can, for example, be done before merging the camera image data and the LIDAR/Radar sensor data. In this way, computing resources will not be wasted on analysis of objects that are not aircraft.”).
With respect to claim 19, Pal in view of Deng teach that the image displayed is generated by projecting the captured LIDAR sensor data onto the simulated two-dimensional image (See at least Pal FIGS. 4-5 and Paragraphs 53-56 “Before generating synthetic video of new aircraft, a 3D model 425 of the aircraft is needed. This can be developed using 3D simulation software and will be used for applying geometric parameters. Estimated rotation and scaling parameters with respect to the camera center are applied on the 3D aircraft model of the new aircraft at center of the model to fit the new aircraft correctly into the scale, position, and orientation of the bridge area viewed in the modified images. This produces rotated and scaled views of the new aircraft for different video frames. However, a 2D projection 426 (i.e., display screenshot) of this 3D model does not consider the aircraft backgrounds, nearby other aircraft, ground vehicles, different objects, and the taxi path. Hence, in the summing junction or synthesis parameter accumulator block of FIG. 4 all required parameters are captured, followed by performing image addition to superimpose a 2D projected new aircraft on a docking scenario (where no approaching aircraft is present) at the centroid position (m) calculated by translation parameters 428. In some embodiments a photometric variance 429 can be applied to blend the color dynamics of the images. Further, in various embodiments climate effects can be added to create images that have different climate 430. The generated synthetic frames produce new aircraft model docking scenario 431 as shown in FIG. 5 . This process can be repeated for all frames of images data, such as a stream of frames from a video. FIG. 5 illustrates a method flow for creating aircraft model data in accordance with one or more embodiments of the present disclosure. A new aircraft to be added to the system is shown in the image on the left side of FIG. 5 . Measurements are taken from the image regarding the aircraft to allow the system to create a 3D model of the new aircraft. The 3D model is then translated into a 2d projection that is scaled, positioned, and oriented in the same manner as the old aircraft in the original image (such scaling, position, and orientation data can be taken from the original image and stored in memory). In this manner, the synthetic image can look identical or substantially similar to an original image (except for the different aircraft type) and can, therefore, be used to train the system to identify the new aircraft and identify the aircraft when it actually approaches the bridge area.”).
With respect to claim 20, Pal in view of Deng teach that the computing device is configured to feed the simulated two-dimensional image into a perception model and predict positioning information of the aircraft using the perception model (See at least Pal FIGS. 4-5 and Paragraphs 53-56 “Before generating synthetic video of new aircraft, a 3D model 425 of the aircraft is needed. This can be developed using 3D simulation software and will be used for applying geometric parameters. Estimated rotation and scaling parameters with respect to the camera center are applied on the 3D aircraft model of the new aircraft at center of the model to fit the new aircraft correctly into the scale, position, and orientation of the bridge area viewed in the modified images. This produces rotated and scaled views of the new aircraft for different video frames. However, a 2D projection 426 (i.e., display screenshot) of this 3D model does not consider the aircraft backgrounds, nearby other aircraft, ground vehicles, different objects, and the taxi path. Hence, in the summing junction or synthesis parameter accumulator block of FIG. 4 all required parameters are captured, followed by performing image addition to superimpose a 2D projected new aircraft on a docking scenario (where no approaching aircraft is present) at the centroid position (m) calculated by translation parameters 428. In some embodiments a photometric variance 429 can be applied to blend the color dynamics of the images. Further, in various embodiments climate effects can be added to create images that have different climate 430. The generated synthetic frames produce new aircraft model docking scenario 431 as shown in FIG. 5 . This process can be repeated for all frames of images data, such as a stream of frames from a video. FIG. 5 illustrates a method flow for creating aircraft model data in accordance with one or more embodiments of the present disclosure. A new aircraft to be added to the system is shown in the image on the left side of FIG. 5 . Measurements are taken from the image regarding the aircraft to allow the system to create a 3D model of the new aircraft. The 3D model is then translated into a 2d projection that is scaled, positioned, and oriented in the same manner as the old aircraft in the original image (such scaling, position, and orientation data can be taken from the original image and stored in memory). In this manner, the synthetic image can look identical or substantially similar to an original image (except for the different aircraft type) and can, therefore, be used to train the system to identify the new aircraft and identify the aircraft when it actually approaches the bridge area.”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IBRAHIM ABDOALATIF ALSOMAIRY whose telephone number is (571)272-5653. The examiner can normally be reached M-F 7:30-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Faris Almatrahi can be reached at 313-446-4821. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/IBRAHIM ABDOALATIF ALSOMAIRY/ Examiner, Art Unit 3667 /KENNETH J MALKOWSKI/Primary Examiner, Art Unit 3667
Prosecution Insights