DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1, 7-16, and 19-20 have been amended.
Claims 1-20 are still pending for consideration.
Response to Arguments
Applicant’s arguments have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1, 3-4, 6, 9-11, and 13-20 are rejected under 35 U.S.C. 103 as being unpatentable over Ramakrishnan et al. (US 20210331695 A1) in view of Coppock et al. (US 20180137361 A1), and further in view of Brouns et al. (US 20220092320 A1).
Regarding claim 1, Ramakrishnan et al. teaches a device comprising: one or more processors (see para [0078]; “the object detection and tracking framework 100 may be implemented in conjunction with a special purpose computer …devices include processors (e.g., a single or multiple microprocessors), memory, nonvolatile storage, input devices, and output devices”) configured to: obtain bounding box data indicating a first location history of a first bounding box associated with a first object detected in images (see para [0033]; “The deep learning model 140 performs frame analysis and assignment tracking 152 to detect objects 104 in an output of the camera 112, which is an instantaneous raw RGB image. Object detection on these raw RGB images is achieved by identifying an object's bounding box and 2D locations in each image frame”, see also para [0038]; “These fused detections are used to create and assign tracks 105 for objects 104 to monitor an object's movement across a field of view 103”, and claim 3; “identifying a two-dimensional location and a bounding box for each object in the image frame”); process, using a trained model, at least the first location history (see para [0018]; “analyze input data 110 from multiple types of sensors within a deep learning model 140, to detect, identify, and track objects 104 the presence”, see also para [0030]; “generate output data 170 that represents predictions based on the objects 104 and tracks 105”). However, Ramakrishnan et al. does not teach to detect an aircraft flight preparation event of an aircraft; generate an output indicating the aircraft flight preparation event; generate tagged training data by tagging the first location history as associated with the aircraft flight preparation event; and update the trained model based on the tagged training data.
In the same field of endeavor Coppock et al. teaches to detect an aircraft flight preparation event of an aircraft (see para [0026]; “One example detector/tracker module can be the additional flight turn support objects detector/tracker 116. The additional flight turn support objects detector/tracker 116 can detect and/or track any object that supports determining flight turns. One example detector/tracker module can be the aircraft detector/tracker 118. The aircraft detector/tracker 118 can detect and/or track aircraft. The one or more detector/tracker modules can extract information about the detected and/or tracked object from the video stream”); generate an output indicating the aircraft flight preparation event (see para [0039]; “a signal indicative of an issue with the event can be provided based on the analysis”, see also para [0040]; “when the determination is made that the aircraft is parked, a notification that the aircraft is parked can be created. For example the analytics device 106 can create a notification that the aircraft is parked when the determination that the aircraft is parked is made”). Accordingly, it would have been obvious to a person of ordinary skill in the art, at the time of the invention to modify the use of a method that applies machine learning techniques in a deep learning model to fuse data from multiple sensors to detect and identify objects of Ramakrishnan et al. in view of a method for analyzing at least one phase of an aircraft turn at an airport of Coppock et al. in order to improve object localization and temporal tracking in successive image frame (see para [0039]). However, Ramakrishnan et al. and Coppock et al. as a whole does not teach generate tagged training data by tagging the first location history as associated with the aircraft flight preparation event; and update the trained model based on the tagged training data.
In the same field of endeavor, Brouns et al. teaches generate tagged training data by tagging the first location history as associated with the aircraft flight preparation event (see para [0029]; “The visual object tracker takes as input a cropped region of the image corresponding to the location of an object instance in one frame of the video and then outputs the locations of all other object instances associated with the same object in the other frames of the video”, see also para [0015]; “Trajectories of centroid position and bounding box size of the object instances in the object track are analyzed to determine whether the track is realistic for a roadside object” see para [0019]; “automatically assigned as the ground-truth annotation for the corresponding object track”, and para [0039]; “Get the trajectory of the bounding box centroid of the object instances in the object track” and para [0080]; “the new ground-truth annotations are added to the training set to re-train the classification model” Note: generating ground truth training data by obtaining an object track containing a trajectory of successive bounding box locations, determining classification for the object track, and automatically assigning the determined classification to the corresponding object track as a ground-truth annotation); and update the trained model based on the tagged training data (see para [0080]; “the new ground-truth annotations are added to the training set to re-train the classification model, after which the cycle continues with Step 4 until sufficient ground-truth annotations are generated”). Accordingly, it would have been obvious to a person of ordinary skill in the art, at the time of the invention to modify the use of a method that applies machine learning techniques in a deep learning model to fuse data from multiple sensors to detect and identify objects of Ramakrishnan et al. in view of a method for analyzing at least one phase of an aircraft turn at an airport of Coppock et al. and further in view of a method for generating ground-truth annotations for object detection and classification for roadside objects in video data of Brouns et al. in order to reduce the amount of manual effort involved in creating said ground-truth annotations (see para [0029]).
Regarding claim 3, the rejection of claim 1 is incorporated herein.
Ramakrishnan et al. in the combination further teach wherein the bounding box data includes a first sequence of locations indicating a first path of the first bounding box (see para [0033]; “Object detection on these raw RGB images is achieved by identifying an object's bounding box and 2D locations in each image frame”).
Regarding claim 4, the rejection of claim 1 is incorporated herein.
Coppock et al. in the combination further teaches wherein the one or more processors are configured to receive the images from a camera that captures a scene at an aircraft stand at an airport (see para [0028]; “a camera for detecting and/or tracking aircraft can be placed in the center of an aircraft parking spot”, see also para [0035]; “one or more video streams can be received from one or more cameras. For example, the analytics device 106 can receive one or more video streams from one or more cameras”).
Regarding claim 6, the rejection of claim 1 is incorporated herein.
Ramakrishnan et al. in the combination further teach wherein the one or more processors are configured to perform object detection on the images to: detect the first object (see para [0002]; “to detect and identify objects”); and generate the first bounding box (see para [0050]; “identify a specific bounding box of the detected and tracked object 104 across multiple sensor measurements”).
Regarding claim 9, the rejection of claim 1 is incorporated herein.
Ramakrishnan et al. in the combination further teach wherein the bounding box data includes a second location history of a second bounding box associated with a second object detected in the images (see claim 3; “identifying a two-dimensional location and a bounding box for each object in the image frame”, see also claim 1; “generating a track for each detected object”), and wherein the one or more processors are configured to process, using the trained model, the first location history and the second location history to detect the aircraft flight preparation event (see para [0030]; “The plurality of data processing modules 134 together comprise at least a portion of the deep learning model 140, which represents an application or one or more machine learning and artificial intelligence techniques that are used to detect objects 104, assign, confirm, predict and follow tracks”).
Regarding claim 10, the rejection of claim 1 is incorporated herein.
Ramakrishnan et al. in the combination further teach wherein the bounding box data includes a third location history of a third bounding box associated with a third object detected in the images (see para [0033]; “Object detection on these raw RGB images is achieved by identifying an object's bounding box and 2D locations in each image frame”, see claim 1; “generating a track for each detected object”);
Coppock et al. in the combination further teaches and wherein the one or more processors are configured to: process, using the trained model (see para [0035]; “The one or more feature detection techniques can include …a deep learning algorithm”), the first location history and the third location history to detect a second aircraft flight preparation event of the aircraft (see para [0005]; “The method includes tracking the one or more objects to determine an event associated with at least one video stream of the one or more video streams based on the one or more objects and the data, wherein the event is associated with at least one phase of an aircraft turn at an airport”); and generate a second output indicating the second aircraft flight preparation event (see para [0039]; “At (612), a signal indicative of an issue with the event can be provided based on the analysis. For example, the analytics device 106 can provide a signal indicative of an issue with the event based on the analysis. As another example, the control system 800 can provide a signal indicative of an issue with the event based on the analysis”).
Regarding claim 11, the rejection of claim 1 is incorporated herein. Coppock et al. in the combination further teach wherein the aircraft flight preparation event includes at least one of: start of refueling, end of refueling, start of baggage loading, end of baggage loading, start of de-icing, end of de-icing, opening of a door of the aircraft, closing of the door of the aircraft, start of meal loading, end of meal loading, passenger embarkation, or passenger disembarkation (see para [0027]; “The personnel detector/tracker 110 can detect and/or track ground services crew, such as a crewmember that marshals aircraft, a crewmember that handles baggage, a mechanic, a crewmember that refuels aircraft, etc”, see also para [0032]; “the monitored sensors can happen during an occurrence of an action, such as an aircraft door opening”).
Regarding claim 13, the rejection of claim 1 is incorporated herein.
Coppock in the combination further teaches wherein the aircraft flight preparation event includes at least one of: arrival of the aircraft, departure of the aircraft, arrival of a truck, departure of the truck, a change in a configuration of the truck, start of refueling, end of refueling, start of baggage loading, end of baggage loading, start of de-icing, end of de-icing, opening of a door of the aircraft, closing of the door of the aircraft, start of meal loading, end of meal loading, passenger embarkation, passenger disembarkation, extension of a bridge, or retraction of the bridge (see para [0052]; “the one or more operational constraints can include, for example, unavailable equipment, available equipment, aircraft turn paths foreclosed in light of current weather conditions, possible aircraft turn paths in light of current weather conditions, aircraft turn paths foreclosed in light of other aircraft, possible aircraft turn paths in light of other aircraft, etc”, see also para [0027]; “The personnel detector/tracker 110 can detect and/or track ground services crew, such as a crewmember that marshals aircraft, a crewmember that handles baggage, a mechanic, a crewmember that refuels aircraft, etc. One example detector/tracker module can be the vehicle detector/tracker 112. The vehicle detector/tracker 112 can detect and/or track a service vehicle, such as a baggage cart, a fuel truck, etc”).
Regarding claim 14, the rejection of claim 1 is incorporated herein.
Coppock in the combination further teaches wherein the one or more processors are configured to: obtain additional sensor data; and selectively generate an alert based on determining whether the additional sensor data indicates the aircraft flight preparation event (see para [0032]; “The event management system/database 124 can receive additional events from the other type of aviation event generators.. The other type of aviation event generators 126 can include monitoring sensors and communicating the results of the monitored sensors. Communication of the monitored sensors can happen during an occurrence of an action, such as an aircraft door opening. The event management system/database 124 can correlate the events received from the analytics device 104 and the events received from the other type of aviation event generators 126”, see also para [0029]; “the notification creator can create a notification when a position of the aircraft reaches or exceeds trigger parameters. … a notification can be created and sent (e.g., transmitted, transferred, etc.) to a user”).
Regarding claim 15, the scope of claim 15 is fully encompassed by the scope of claim 1, accordingly, the rejection of claim 1 is fully applicable here.
Regarding claim 16, the rejection of claim 15 is incorporated herein.
Coppock in the combination further teaches wherein the one or more processors are configured to: obtain additional sensor data; and selectively generate an alert based on determining whether the additional sensor data indicates the aircraft flight preparation event (see para [0032]; “The event management system/database 124 can receive additional events from the other type of aviation event generators…. The other type of aviation event generators 126 can include monitoring sensors and communicating the results of the monitored sensors. Communication of the monitored sensors can happen during an occurrence of an action, such as an aircraft door opening. The event management system/database 124 can correlate the events received from the analytics device 104 and the events received from the other type of aviation event generators 126”, see also para [0029]; “the notification creator can create a notification when a position of the aircraft reaches or exceeds trigger parameters. … a notification can be created and sent (e.g., transmitted, transferred, etc.) to a user”).
Regarding claim 17, the rejection of claim 16 is incorporated herein.
Coppock in the combination further teaches further comprising a door, wherein the additional sensor data indicates whether the door is open (see para [0032]; “Communication of the monitored sensors can happen during an occurrence of an action, such as an aircraft door opening”).
Regarding claim 18, the rejection of claim 16 is incorporated herein.
Coppock in the combination further teaches further comprising at least an adaptable wing, wherein the additional sensor data indicates a detected position of the adaptable wing.
Regarding claim 19, the scope of claim 19 is fully encompassed by the scope of claim 1, accordingly, the rejection of claim 1 is fully applicable here.
Regarding claim 20, the rejection of claim 19 is incorporated herein.
Coppock in the combination further teaches wherein the first object includes an aircraft, a fuel truck, a baggage cart, a de-icing truck, a catering truck, a pushback tug, a snow plow, an airstair, an air bridge, an apron bus, a belt loader, a container loader, a water truck, a lavatory service vehicle, a ground power unit, an air start unit, or a fire truck (see para [0027]; “detect and/or track ground services crew, such as a crewmember that marshals aircraft, a crewmember that handles baggage, a mechanic, a crewmember that refuels aircraft, etc. One example detector/tracker module can be the vehicle detector/tracker 112. The vehicle detector/tracker 112 can detect and/or track a service vehicle, such as a baggage cart, a fuel truck, etc… The additional flight turn support objects detector/tracker 116 can detect and/or track any object that supports determining flight turns. One example detector/tracker module can be the aircraft detector/tracker 118”).
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Ramakrishnan et al. and Coppock et al. in view of Brouns et al. as applied in claim 1 above, and further in view of Olson (US 7023469 B1).
Regarding claim 2, the rejection of claim 1 is incorporated herein. The combination of Ramakrishnan et al., Coppock et al. and Brouns et al. as a whole does not teach wherein the bounding box data includes a first image indicating a first path of a first midpoint of the first bounding box.
In the same field of endeavor, Olson teaches wherein the bounding box data includes a first image (see para claim 26; “displaying on the reference image the path of movement of the object”) indicating a first path of a first midpoint of the first bounding box (see col.3, lines 22-25; “the image processing system 27 can use the position in the image of the midpoint of the lower side of the object's bounding box in order to identify how far the object is from the camera”, see also col. 6, lines 38-41; “For each detected object such as the person 86, the image processing section 27 also determines the Cartesian coordinates within each image of the midpoint of the lower side of the bounding box for that detected object”, and col. 10, lines 21-24; “the trace 113 represents the movement of the midpoint of the lower side of the bounding box 87, and thus is an accurate representation of where the person 86 walked”). Accordingly, it would have been obvious to a person of ordinary skill in the art, at the time of the invention to modify the use of a method that applies machine learning techniques in a deep learning model to fuse data from multiple sensors to detect and identify objects of Ramakrishnan et al. in view of a method for analyzing at least one phase of an aircraft turn at an airport of Coppock et al. and automatic video monitoring system which selectively saves information of Olson in order to intelligently save selected information that is meaningful but minimizes storage capacity (see claim 26).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Ramakrishnan et al. in view of Coppock et al. as applied in claim 1 above, and further in view of Sankrithi et al. (US 6405975 B1).
Regarding claim 5, the rejection of claim 1 is incorporated herein. The combination of Ramakrishnan et al. and Coppock et al. as a whole does not teach wherein the one or more processors are configured to receive the images from a camera coupled to the aircraft.
In the same field of endeavor, Sankrithi et al. teaches wherein the one or more processors are configured to receive the images from a camera coupled to the aircraft (Abstract; “The system includes at least one camera mounted on the airplane for generating video images of at last one gear with tires”, see also col.10, lines18-20; “the processor 520 receives digitized or nondigitized images generated by the cameras 524”). Accordingly, it would have been obvious to a person of ordinary skill in the art, at the time of the invention to modify the use of a method that applies machine learning techniques in a deep learning model to fuse data from multiple sensors to detect and identify objects of Ramakrishnan et al. in view of a method for analyzing at least one phase of an aircraft turn at an airport of Coppock et al. and further in view of a method for generating ground-truth annotations for object detection and classification for roadside objects in video data of Brouns et al. and automatic video monitoring system which selectively saves information of Sankrithi et al. in order to intelligently save selected information that is meaningful but minimizes storage capacity (see Abstract).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Ramakrishnan et al. and Coppock et al. in view of Brouns et al. as applied in claim 1 above, and further in view of Gonzalez et al. (US 20200377232 A1).
Regarding claim 7, the rejection of claim 1 is incorporated herein. The combination of Ramakrishnan et al., Coppock et al. and Brouns et al. as a whole does not teach wherein the aircraft flight preparation event corresponds to one of a plurality of events that are tracked by an aircraft turnaround management system, and wherein the aircraft turnaround management system updates a status of the aircraft flight preparation event based on the output.
In the same field of endeavor, Gonzalez et al. teaches wherein the aircraft flight preparation event corresponds to one of a plurality of events that are tracked by an aircraft turnaround management system (see para [0033]; “a turnaround monitoring system that includes a turnaround analysis control unit that receives turnaround data from one or more cameras and one or more sensors and determines a turnaround status of an aircraft based on the turnaround data …The turnaround data includes information regarding various turnaround aspects, such as passenger information (for example, passengers disembarking from a first flight and passengers boarding a subsequent second flight), luggage, catering, fueling, cleaning, crew, and the like”), and wherein the aircraft turnaround management system updates a status of the aircraft flight preparation event based on the output (see para [0045]; “The turnaround analysis control unit 130 analyzes the turnaround data to determine turnaround status of the aircraft 102 between flights”). Accordingly, it would have been obvious to a person of ordinary skill in the art, at the time of the invention to modify the use of a method that applies machine learning techniques in a deep learning model to fuse data from multiple sensors to detect and identify objects of Ramakrishnan et al. in view of a method for analyzing at least one phase of an aircraft turn at an airport of Coppock et al. and further in view of a method for generating ground-truth annotations for object detection and classification for roadside objects in video data of Brouns et al. and a method monitor a turnaround of an aircraft at an airport of Gonzalez et al. in order to determine a status of a particular turnaround services (see para [0033]).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Ramakrishnan et al. and Coppock et al. in view of Brouns et al. Gonzalez et al. as applied in claims 1, and 7 above, and further in view of Agrawal et al. (US 20170011638 A1).
Regarding claim 8, the rejection of claim 7 is incorporated herein. The combination of Ramakrishnan et al., Coppock et al. Brouns et al. and Gonzalez et al. as a whole does not teach wherein the aircraft turnaround management system determines a predicted delay based on the status of the aircraft flight preparation event and planned timing data associated with the plurality of events.
In the same field of endeavor, Gonzalez et al. teaches wherein the aircraft turnaround management system determines a predicted delay based on the status of the event and planned timing data associated with the plurality of events (see para [0028]; “The delay prediction module 216 may determine an aircraft departure delay. The aircraft departure delay may be understood as delay in scheduled departure time of the aircraft. The aircraft departure delay may be caused by one or more scheduled turnaround activities. Further, the delay prediction module 216 may determine the aircraft departure delay by analyzing the time deviation of the scheduled turnaround activities”). Accordingly, it would have been obvious to a person of ordinary skill in the art, at the time of the invention to modify the use of a method that applies machine learning techniques in a deep learning model to fuse data from multiple sensors to detect and identify objects of Ramakrishnan et al. in view of a method for analyzing at least one phase of an aircraft turn at an airport of Coppock et al. and further in view of a method for generating ground-truth annotations for object detection and classification for roadside objects in video data of Brouns et al. and a method monitor a turnaround of an aircraft at an airport of Gonzalez et al. and further in view of system for monitoring scheduled turnaround activities and alerting on time deviation from the scheduled turnaround activities of Agrawal et al. in order to avoid incorrect recording of the time taken to perform turnaround activities (see para [0028]).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Ramakrishnan et al., Coppock et al. in view of Brouns et al. as applied in claim 1 above, and further in view of Cetin et al. (US 8866910 B1).
Regarding claim 12, the rejection of claim 1 is incorporated herein. The combination of Ramakrishnan et al., Coppock et al. and Brouns et al. as a whole does not teach wherein the trained model is trained to detect a particular location pattern corresponding to historical location patterns associated with the aircraft flight preparation event.
In the same field of endeavor, Cetin et al. teaches wherein the trained model is trained to detect a particular location pattern corresponding to historical location patterns associated with the aircraft flight preparation event (see col. 2, lines 52-57; “trajectories consisting of coordinates of blobs are determined. The trajectories are transformed into a corrected image domain, and then each trajectory is fed to a set of Markov Models (MM), which are trained with prior trajectory data corresponding to regular and unusual motion trajectories of moving objects”, see also col. 11, lines 56; “calculating movement trajectories…. training a plurality of state transition probability models with prior trajectory data… and assigning an unusual event classification”). Accordingly, it would have been obvious to a person of ordinary skill in the art, at the time of the invention to modify the use of a method that applies machine learning techniques in a deep learning model to fuse data from multiple sensors to detect and identify objects of Ramakrishnan et al. in view of a method for analyzing at least one phase of an aircraft turn at an airport of Coppock et al. and further in view of a method for generating ground-truth annotations for object detection and classification for roadside objects in video data of Brouns et al. and image processing in a wide-angle video camera, and to tracking moving regions and detecting unusual motion activity within the field of view of the camera of Cetin et al. in order to produce highest probability of unusual events (see col. 2, lines 52-57).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WINTA GEBRESLASSIE whose telephone number is (571)272-3475. The examiner can normally be reached Monday-Friday9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at 571-270-5180. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WINTA GEBRESLASSIE/Examiner, Art Unit 2677
Prosecution Insights