Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 07/27/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Amendment
The amendments and associated applicant arguments/ remarks filed on 7/27/2026 were received and considered.
Claims 1, 2, 4, 11, 12, and 18 have been amended.
Claims 10 and 19 have been cancelled.
Claims 21 and 22 have been added.
Claims 1-9, 11-18, 19-22 are pending.
Response to Arguments
Applicant’s arguments, see Remarks, filed 7/27/2026, with respect to the rejection(s) of claim(s) 1-9, 11-18, and 20 under USC 103 have been fully considered.. However, upon further consideration, a new ground(s) of rejection is made in view of Greene (US 9437112 B1).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-9, 11-18, 20-22 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US 11131987 B2) referred to as Wang hereinafter and further in view of Mead et al. (US 11526390 B2) referred to as Mead hereinafter and further in view of Greene (US 9437112 B1).
Regarding claim 1, Wang teaches A method of monitoring images generated by a second processing system redundant to a first processing system, the method comprising: (“An object-based integrity verification method for detecting hazardously misleading information contained in an output image of a graphics processing unit comprises steps of: identifying a plurality of safety-critical graphic objects in the output image;” Wang, abstract) and (“Independent and redundant sensors are used for pilot and co-pilot” Wang, col. 15, lines 38-39)
identifying a measured value for a parameter based on sensor data obtained via a sensing arrangement; (“Sensor data inputs and ADC function block 505 comprises airspeed, altitude, air temperature, vertical speed, angle of attack and barometric pressure sensor sources, which are processed by the ADC; and turn coordinator, heading indicator and attitude indicator information from Attitude and Heading Reference System (AHRS not shown).” Wang, col. 15, lines 32-37)
verifying the first plurality of characters includes a first set of one or more displayed characters corresponding to the measured value; (“Inside each EDU, the integrity verification procedure comprises three related processes. Graphic commands received from both IMA computers 650 and 660 are evaluated and compared, as shown in step 730. If a selected graphic command source is missing, inconsistent or invalid, as indicated in step 732, the EDU will switch to an alternative command source, and annunciate the fault condition, as indicated in step 734. EDU also verifies the integrity of the local user inputs from keypad and touch screen if applicable, as indicated in step 740. EDU monitors the local user input to detect inconsistent (such as excessive jittering) or invalid (such as prolonged switch bouncing) input as indicated in step 742.” Wang, col. 17, lines 54-66)
and after verifying the first plurality of characters includes the first set of one or more displayed characters corresponding to the measured value: (“provision of an object-based integrity verification system and processes to track and verify the integrity of each safety-critical graphic object depicted on the safety-critical display to detect any hazardously misleading information generated by the graphic processing unit” Wang, col. 5, lines 22-26)
and generating an alert in response to an absence of the second set of one or more displayed characters corresponding to the first set of one or more displayed characters within the second plurality of characters. (“The local processor in function block 1004 select one of the ports as its primary input and output. The video input section of the function block 1002 provides signal conversion from fiber to copper if it is applicable and sends the digital video signal to function block 1004 through data link 1014. The local processor in function block 1004 verifies the signal integrity of the primary video input signal using the embedded sequential frame counter and CRC code. If the primary video input is missing or invalid, the second video input channel will be activated, and the pilot will be alerted.” Wang, col. 20, lines 3-12)
However, Wang does not teach extracting, from a first frame of image data generated by the first processing system based at least in part on the sensor data for presentation on a first display device, a first plurality of characters displayed within the first frame; extracting, from a second frame of second image data generated by the second processing system for presentation on a second display device concurrent to the first frame of image data, a second plurality of characters displayed within the second frame; verifying whether the second plurality of characters extracted from the second frame includes a second set of one or more displayed characters corresponding to the first set of one or more displayed characters extracted from the first frame generated by the first processing system based at least in part on the sensor data;
Mead teaches extracting, from a first frame of image data generated by the first processing system based at least in part on the sensor data (“an imaging sensor (e.g., imaging sensor from imaging device 322 from FIG. 3)” Mead, col.8, lines 37-38) for presentation on a first display device, a first plurality of characters displayed within the first frame; (“a system for verifying the integrity, continuity, and availability (ICA) of information displayed on an uncertified display is provided. The system comprises a server that is certified to generate mission critical data for display; an adapter device configured to house an uncertified display device that provides the uncertified display; and a screen capture system configured to capture an image displayed on the uncertified display; wherein the system is configured to: create, on the server, application data (which may include a user interface) for display on the uncertified display device that includes a subliminal symbol that is periodically embedded in a few out of multiple tens of frames (e.g., 2 out of every 120 frames) and that is camouflaged in the application data using steganography (e.g., two-frame steganography or other multi-frame steganography techniques)” Mead, col. 1, lines 48-60) and (“The example server 504 includes one or more processors configured by programming instructions on computer readable media, one or more application modules 508 configured to generate high integrity data for display, and verification logic 510 configured to validate information displayed on the uncertified display device 502 and cause an annunciation indicating a loss of ICA when a loss of ICA is detected.” Mead, col. 10, lines 37-44)
extracting, from a second frame of second image data generated by the second processing system for presentation on a second display device concurrent to the first frame of image data, a second plurality of characters displayed within the second frame; (“determine whether the subliminal symbol is detected in captured images of the application data displayed on the uncertified display device by extracting the symbol from the captured images and comparing the extracted symbol to an expected symbol, wherein the captured images were captured using a screen capture system that is configured to capture images of the application data displayed on the display screen of the uncertified display device that includes the subliminal symbol that is camouflaged using steganography” Mead, col. 2, lines 50-59) and (“The example application server 102 also includes a data integrity module 112 that is configured to monitor the image displayed on an uncertified cockpit display when critical aeronautical information is transmitted from a high integrity avionics application module 108 to the uncertified cockpit display device (e.g., PED 106) to determine whether a problem exists with the display of the mission critical data on the uncertified display device. The example data integrity module 112 is configured to determine whether a problem exists with the display of the mission critical data on the uncertified display device 106 by verifying the integrity, continuity, and availability (ICA) of the mission critical data displayed on the uncertified display device 106.” Mead, col. 5, lines 48-60)
verifying whether the second plurality of characters extracted from the second frame includes a second set of one or more displayed characters corresponding to the first set of one or more displayed characters extracted from the first frame generated by the first processing system based at least in part on the sensor data; (“The example system 700 includes a server 704 that is certified as a high integrity device and an adapter device 706 that may also be certified as a high integrity device. The example server 704 includes one or more processors configured by programming instructions on computer readable media, one or more application modules 708 configured to generate high integrity data for display, and verification logic 710 configured to validate information displayed on the uncertified display device 702 and cause an annunciation indicating a loss of ICA when a loss of ICA is detected. Instead of using an optical sensing device (e.g., optical sensing device 512) trained on the display screen 714 of the uncertified display device 702 to capture an image displayed on the display screen 714, the example system 700 generates the captured image using operating system support from the low integrity device (for example a screen capture or snapshot function). The example adapter device 706 is further configured with an annunciation screen 716 configured to overlay the display screen 714 of the uncertified display device 702 and annunciate a loss of ICA when a loss of ICA is detected.” Mead, col. 14, lines 40-60)
Wang and Mead are combinable because they are from the same field of endeavor, comparing characters in image analysis.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Wang in light of Mead’s extract, from a first frame of image data generated by the first processing system based at least in part on the sensor data. One would have been motivated to do so because it can improve continuity and/or to inform maintenance systems to improve availability. (Mead, col. 15, lines 46-47)
However, the combination of Wang and Mead does not teach second image data generated by the second processing system based at least in part on second sensor data obtained via a second sensing arrangement; second plurality of characters extracted from the second frame generated by the second processing system based at least in part on the second sensor data
Greene teaches second image data generated by the second processing system based at least in part on second sensor data obtained via a second sensing arrangement; (“The integrated avionics system 100 may employ cross-checking of the primary flight information (e.g., attitude information, altitude information, etc.) to determine if the primary flight information to be furnished to either of the PFDs 102 is incorrect. In implementations, cross-checking may be accomplished through software-based automatic continual comparison of the primary flight information provided by the AHRS 106 and ADC 108. In this manner, a “miss-compare” condition can be explicitly and proactively annunciated to warn the pilot when attitude information displayed by either PFD 102 sufficiently disagrees. The first PFD 102(1), the second PFD 102(2), and/or the MFD 104 may receive additional data aggregated by the avionics unit 110 from a one or more of a plurality of systems communicatively coupled with the avionics unit 110. For example, the avionics unit 110 may be communicatively coupled with and may aggregate data received from one or more of: an Automatic Dependent Surveillance-Broadcast (ADS-B) system 114, Traffic Collision Avoidance System (TCAS) 116, and a Traffic Information Services-Broadcast (TIS-B) system 118.” Col. 6, lines 53-67, col. 7, lines 1-6), PFD 102(2) is the claimed second sensing arrangement
second plurality of characters extracted from the second frame generated by the second processing system based at least in part on the second sensor data (“One or more avionics units 110 (e.g., a single integrated avionics unit (IAU) is illustrated) may aggregate the primary flight information from the AHRSs 106 and ADCs 108 and provide the information to the PFDs 102 via an avionics data bus 112.” Greene, col. 5, lines 24-27) and (“the integrated avionics system 100 illustrated in FIG. 1 employs two PFDs 102 that may receive primary flight information from redundant AHRSs 106 and ADCs 108, via the avionics unit 110. The integrated avionics system 100 is configured so that the first PFD 102(1) receives a first set of primary flight information aggregated by the avionics unit 110 from a first AHRS 106(1) and ADC 108(1). Similarly, the second PFD 102(2) receives a second set of primary flight information aggregated by the avionics unit 110 from a second AHRS 106(2) and ADC 108(2). Additionally, although a single avionics unit 110 and a single avionics data bus 112 are illustrated in FIG. 1, it is contemplated that redundant IAU's and/or redundant data buses may be employed for communication between the various components of the integrated avionics system 100.” Greene, col. 6, 28-42)
Wang, Mead, and Greene are combinable because they are from the same field of endeavor, comparing screens in image analysis.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Wang and Mead in light of Greene’s second processing system based at least in part on second sensor data obtained via a second sensing arrangement. One would have been motivated to do so because it can provide flight crew members with a detailed, accurate and real-time depiction of air traffic in the vicinity of the aircraft. (Greene, col 1, lines 45-47)
Regarding claim 2, Wang in view of Mead and Greene teaches wherein the first processing system is configurable to render the first set of one or more displayed characters corresponding to the measured value at a first location within the first frame of image data and the second processing system is configurable to render the second set of one or more displayed characters corresponding to the parameter at a second location within the second frame of second image data, wherein the first location and the second location are different. (“FIG. 11 shows a second redundant system embodiment of the present invention with the GPU function block integrated in the IMA computer. Cockpit display system 900 comprises four substantially identical Electronics Display Units (EDU) 902, 904, 906 and 908, and two independent Integrated Modular Avionics (IMA) computers 950, and 960. Each of the IMA computers 950 and 960, comprises a plurality of CPUs, GPUs, User Application (UA) software and object-based integrity verification systems of the present invention. Each of the IMA computers 950 and 960 has two independent sensor input sources 992 and 994. Each of input sources 992 and 994 comprises a group of sensor inputs selected from a group comprising airspeed, altitude, air temperature, vertical speed, angle of attack, barometric pressure, turn coordinator, heading indicator, attitude indicator, navigation, engine data and other aircraft information.” Wang, col. 19, lines 4-20)
Greene teaches the parameter based at least in part on the second sensor data from the second sensing arrangement. (“One or more avionics units 110 (e.g., a single integrated avionics unit (IAU) is illustrated) may aggregate the primary flight information from the AHRSs 106 and ADCs 108 and provide the information to the PFDs 102 via an avionics data bus 112.” Greene, col. 5, lines 24-27) and (“the integrated avionics system 100 illustrated in FIG. 1 employs two PFDs 102 that may receive primary flight information from redundant AHRSs 106 and ADCs 108, via the avionics unit 110. The integrated avionics system 100 is configured so that the first PFD 102(1) receives a first set of primary flight information aggregated by the avionics unit 110 from a first AHRS 106(1) and ADC 108(1). Similarly, the second PFD 102(2) receives a second set of primary flight information aggregated by the avionics unit 110 from a second AHRS 106(2) and ADC 108(2).” Greene, col. 6, lines 28-37)
Regarding claim 3, Wang in view of Mead and Greene teaches wherein generating the alert comprises augmenting the image data generated by the first processing system to include a graphical user notification proximate the first set of one or more displayed characters corresponding to the measured value. (“During the run time, EDU processor 806 sends GPU 818 graphic drawing commands to render the display output in the video memory 820 based on the verified command and keyboard input 828. The EDU processor 806 also sends the graphic drawing commands to the object-based integrity verification processor 816 to render the required reference image, such as reference ID code image 425, as shown in FIG. 6.” Wang, col. 18, lines 55-62)
Regarding claim 4, Wang in view of Mead and Greene teaches wherein generating the alert comprises augmenting the second image data generated by the second processing system to include a graphical user notification proximate the second set of one or more displayed characters corresponding to a second measured value for the parameter. (“If the cockpit display system is designed according to the ARINC 661 protocol, the EDU processor 806 configures the display screen based on a Graphic User Interface (GUI) screen Definition File (DF) and a graphic library stored in system memory 808. Based on the DF file, the EDU processor 806 generates a verification database, and sends it to the object-based integrity verification processor 816. The verification database is, then, stored in the verification memory 814.” Wang, col. 18, lines 46-54) and (“Video frame parameters and a plurality of safety-critical graphic objects rendered by the GPU in section 545 are verified to detect any image fault, such as image freezing, incorrect safety-critical graphic object's shape, size, color, orientation and readability problems.” Wang, col. 16, lines 18-23)
Greene teaches parameter based on the second sensor data. (“One or more avionics units 110 (e.g., a single integrated avionics unit (IAU) is illustrated) may aggregate the primary flight information from the AHRSs 106 and ADCs 108 and provide the information to the PFDs 102 via an avionics data bus 112.” Greene, col. 5, lines 24-27) and (“the integrated avionics system 100 illustrated in FIG. 1 employs two PFDs 102 that may receive primary flight information from redundant AHRSs 106 and ADCs 108, via the avionics unit 110. The integrated avionics system 100 is configured so that the first PFD 102(1) receives a first set of primary flight information aggregated by the avionics unit 110 from a first AHRS 106(1) and ADC 108(1). Similarly, the second PFD 102(2) receives a second set of primary flight information aggregated by the avionics unit 110 from a second AHRS 106(2) and ADC 108(2).” Greene, col. 6, lines 28-37)
Regarding claim 5, Wang in view of Mead and Greene teaches wherein generating the alert comprises generating an aural alert. (“The object-based verification process 550 determines the alarm level, such as warning, caution and advisory required.” Wang, col. 16, lines 25-27)
Regarding claim 6, Mead teaches wherein: extracting the first plurality of characters displayed within the first frame comprises performing character recognition on the first frame of image data to obtain the first plurality of characters; (“The comparing the extracted symbol to an expected symbol may comprise one or more of: calculating an image characteristic for both the extracted symbol and the expected symbol; calculating a similarity measure between the extracted symbol and the expected symbol; calculating histograms for each of the extracted symbol and the expected symbol;” Mead, col. 20, lines 38-44)
and extracting the second plurality of characters displayed within the second frame comprises performing character recognition on the second frame of second image data to obtain the second plurality of characters. (“The example process 900 further includes generating an expected image (917) at the server from the application data (operation 916). The example process includes calculating one or more image characteristics 915 from the captured image 913 (operation 914) at the server and calculating one or more image characteristics 919 from the expected image characteristic 917 (operation 918) at the server. The captured image characteristic 915 and the expected image characteristic 919 are compared (operation 920) at the server.” Mead, col. 17, lines 49-57)
Regarding claim 7, Wang in view of Mead and Greene teaches wherein verifying the first plurality of characters includes the first set of one or more displayed characters corresponding to the measured value comprises identifying an extracted character subset within the first plurality of characters that matches the measured value for the parameter. (“If verification sub-step 342S is used, any object can be independently selected or deselected within the verification zone. When verification sub-step 342S is used, a subset of objects can be selected for one video frame, and different subset of objects can be selected for the next video frame… Several widgets may be grouped together as one object, such as the bank scale 292, especially if they remain unchanged in all video frames.” Wang, col. 6, lines 62-66, col. 7, lines 6-9)
Regarding claim 8, Mead teaches wherein verifying whether the second plurality of characters includes the second set of one or more displayed characters comprises validating the second frame of second image data in response to identifying a second extracted character subset within the second plurality of characters that matches the extracted character subset. (“The example system 500 is further configured to capture, e.g., using the optical sensing device 512, an image of the display screen 514 of the uncertified display device 502; transfer, to the validation logic 510 (e.g., from the optical sensing device 512), the captured image of the display as validation data 507; determine, at the high integrity server 504, an expected display image to be displayed on the uncertified display device 502;” Mead, col. 10, lines 60-67)
Regarding claim 9, Wang teaches wherein generating the alert comprises generating the alert in response to an absence of a second extracted character subset within the second plurality of characters that matches the extracted character subset. (“At step 350, the fault condition will be categorized based on the severity and the duration of the fault condition, as well as the flight circumstance. The original display window ID, layer ID, object ID and parameter ID can be retrieved from the verification database according to the ID code of the failed object. Therefore, the impact and severity of this fault related to the roll, pitch and slip/skid parameters can be determined. Proper warning signal, such as warning, alarm or caution, can be invoked in a timely manner to alert the pilot the location and failure condition in details.” Wang, col. 11, lines 36-45)
Regarding claim 11, Wang teaches A method of monitoring a co-pilot forward display generated by a second processing system redundant to a first processing system generating a pilot forward display, the method comprising: (“An object-based integrity verification method for detecting hazardously misleading information contained in an output image of a graphics processing unit comprises steps of: identifying a plurality of safety-critical graphic objects in the output image;” Wang, abstract) and (“Independent and redundant sensors are used for pilot and co-pilot” Wang, col. 15, lines 38-39)
identifying a measured value for a parameter based on sensor data obtained via a sensing arrangement; (“Sensor data inputs and ADC function block 505 comprises airspeed, altitude, air temperature, vertical speed, angle of attack and barometric pressure sensor sources, which are processed by the ADC; and turn coordinator, heading indicator and attitude indicator information from Attitude and Heading Reference System (AHRS not shown).” Wang, col. 15, lines 32-37)
verifying the first plurality of characters includes a character subset comprising one or more displayed characters matching the measured value; (“Inside each EDU, the integrity verification procedure comprises three related processes. Graphic commands received from both IMA computers 650 and 660 are evaluated and compared, as shown in step 730. If a selected graphic command source is missing, inconsistent or invalid, as indicated in step 732, the EDU will switch to an alternative command source, and annunciate the fault condition, as indicated in step 734. EDU also verifies the integrity of the local user inputs from keypad and touch screen if applicable, as indicated in step 740. EDU monitors the local user input to detect inconsistent (such as excessive jittering) or invalid (such as prolonged switch bouncing) input as indicated in step 742.” Wang, col. 17, lines 54-66)
and automatically generating an alert in response to an absence of a second character subset within the second plurality of characters that matches the character subset matching the measured value from the pilot forward display. (“The local processor in function block 1004 select one of the ports as its primary input and output. The video input section of the function block 1002 provides signal conversion from fiber to copper if it is applicable and sends the digital video signal to function block 1004 through data link 1014. The local processor in function block 1004 verifies the signal integrity of the primary video input signal using the embedded sequential frame counter and CRC code. If the primary video input is missing or invalid, the second video input channel will be activated, and the pilot will be alerted.” Wang, col. 20, lines 3-12)
However, Wang does not teach extracting, from a first frame of image data of the pilot forward display generated by the first processing system based at least in part on the sensor data for presentation on a first display device, a first plurality of characters displayed within the first frame; extracting, from a second frame of second image data generated by the second processing system for presentation on a second display device concurrent to the first frame of image data, a second plurality of characters displayed within the second frame; verifying whether the second plurality of characters extracted from the second frame includes a second set of one or more displayed characters corresponding to the first set of one or more displayed characters extracted from the first frame generated by the first processing system based at least in part on the sensor data;
Mead teaches extracting, from a first frame of image data generated by the first processing system based at least in part on the sensor data (“an imaging sensor (e.g., imaging sensor from imaging device 322 from FIG. 3)” Mead, col.8, lines 37-38) for presentation on a first display device, a first plurality of characters displayed within the first frame; (“a system for verifying the integrity, continuity, and availability (ICA) of information displayed on an uncertified display is provided. The system comprises a server that is certified to generate mission critical data for display; an adapter device configured to house an uncertified display device that provides the uncertified display; and a screen capture system configured to capture an image displayed on the uncertified display; wherein the system is configured to: create, on the server, application data (which may include a user interface) for display on the uncertified display device that includes a subliminal symbol that is periodically embedded in a few out of multiple tens of frames (e.g., 2 out of every 120 frames) and that is camouflaged in the application data using steganography (e.g., two-frame steganography or other multi-frame steganography techniques)” Mead, col. 1, lines 48-60) and (“The example server 504 includes one or more processors configured by programming instructions on computer readable media, one or more application modules 508 configured to generate high integrity data for display, and verification logic 510 configured to validate information displayed on the uncertified display device 502 and cause an annunciation indicating a loss of ICA when a loss of ICA is detected.” Mead, col. 10, lines 37-44)
extracting, from a second frame of image data of the co-pilot forward display generated by the second processing system for presentation on a second display device concurrent to the first frame of image data, a second plurality of characters displayed within the second frame; (“determine whether the subliminal symbol is detected in captured images of the application data displayed on the uncertified display device by extracting the symbol from the captured images and comparing the extracted symbol to an expected symbol, wherein the captured images were captured using a screen capture system that is configured to capture images of the application data displayed on the display screen of the uncertified display device that includes the subliminal symbol that is camouflaged using steganography” Mead, col. 2, lines 50-59) and (“The example application server 102 also includes a data integrity module 112 that is configured to monitor the image displayed on an uncertified cockpit display when critical aeronautical information is transmitted from a high integrity avionics application module 108 to the uncertified cockpit display device (e.g., PED 106) to determine whether a problem exists with the display of the mission critical data on the uncertified display device. The example data integrity module 112 is configured to determine whether a problem exists with the display of the mission critical data on the uncertified display device 106 by verifying the integrity, continuity, and availability (ICA) of the mission critical data displayed on the uncertified display device 106.” Mead, col. 5, lines 48-60)
verifying whether the second plurality of characters extracted from the second frame includes a second set of one or more displayed characters corresponding to the first set of one or more displayed characters extracted from the first frame generated by the first processing system based at least in part on the sensor data; (“The example system 700 includes a server 704 that is certified as a high integrity device and an adapter device 706 that may also be certified as a high integrity device. The example server 704 includes one or more processors configured by programming instructions on computer readable media, one or more application modules 708 configured to generate high integrity data for display, and verification logic 710 configured to validate information displayed on the uncertified display device 702 and cause an annunciation indicating a loss of ICA when a loss of ICA is detected. Instead of using an optical sensing device (e.g., optical sensing device 512) trained on the display screen 714 of the uncertified display device 702 to capture an image displayed on the display screen 714, the example system 700 generates the captured image using operating system support from the low integrity device (for example a screen capture or snapshot function). The example adapter device 706 is further configured with an annunciation screen 716 configured to overlay the display screen 714 of the uncertified display device 702 and annunciate a loss of ICA when a loss of ICA is detected.” Mead, col. 14, lines 40-60)
Wang and Mead are combinable because they are from the same field of endeavor, comparing characters in image analysis.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Wang in light of Mead’s extract, from a first frame of image data generated by the first processing system based at least in part on the sensor data. One would have been motivated to do so because it can improve continuity and/or to inform maintenance systems to improve availability. (Mead, col. 15, lines 46-47)
However, the combination of Wang and Mead does not teach second image data generated by the second processing system based at least in part on second sensor data obtained via a second sensing arrangement; second plurality of characters extracted from the second frame generated by the second processing system based at least in part on the second sensor data,
Greene teaches second image data generated by the second processing system based at least in part on second sensor data obtained via a second sensing arrangement; (“The integrated avionics system 100 may employ cross-checking of the primary flight information (e.g., attitude information, altitude information, etc.) to determine if the primary flight information to be furnished to either of the PFDs 102 is incorrect. In implementations, cross-checking may be accomplished through software-based automatic continual comparison of the primary flight information provided by the AHRS 106 and ADC 108. In this manner, a “miss-compare” condition can be explicitly and proactively annunciated to warn the pilot when attitude information displayed by either PFD 102 sufficiently disagrees. The first PFD 102(1), the second PFD 102(2), and/or the MFD 104 may receive additional data aggregated by the avionics unit 110 from a one or more of a plurality of systems communicatively coupled with the avionics unit 110. For example, the avionics unit 110 may be communicatively coupled with and may aggregate data received from one or more of: an Automatic Dependent Surveillance-Broadcast (ADS-B) system 114, Traffic Collision Avoidance System (TCAS) 116, and a Traffic Information Services-Broadcast (TIS-B) system 118.” Col. 6, lines 53-67, col. 7, lines 1-6), PFD 102(2) is the claimed second sensing arrangement
second plurality of characters extracted from the second frame generated by the second processing system based at least in part on the second sensor data (“One or more avionics units 110 (e.g., a single integrated avionics unit (IAU) is illustrated) may aggregate the primary flight information from the AHRSs 106 and ADCs 108 and provide the information to the PFDs 102 via an avionics data bus 112.” Greene, col. 5, lines 24-27) and (“the integrated avionics system 100 illustrated in FIG. 1 employs two PFDs 102 that may receive primary flight information from redundant AHRSs 106 and ADCs 108, via the avionics unit 110. The integrated avionics system 100 is configured so that the first PFD 102(1) receives a first set of primary flight information aggregated by the avionics unit 110 from a first AHRS 106(1) and ADC 108(1). Similarly, the second PFD 102(2) receives a second set of primary flight information aggregated by the avionics unit 110 from a second AHRS 106(2) and ADC 108(2). Additionally, although a single avionics unit 110 and a single avionics data bus 112 are illustrated in FIG. 1, it is contemplated that redundant IAU's and/or redundant data buses may be employed for communication between the various components of the integrated avionics system 100.” Greene, col. 6, 28-42)
Wang, Mead, and Greene are combinable because they are from the same field of endeavor, comparing screens in image analysis.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Wang and Mead in light of Greene’s second processing system based at least in part on second sensor data obtained via a second sensing arrangement. One would have been motivated to do so because it can provide flight crew members with a detailed, accurate and real-time depiction of air traffic in the vicinity of the aircraft. (Greene, col 1, lines 45-47)
Regarding claim 12, Wang in view of Mead and Greene teaches automatically generating the alert comprises automatically generating the alert when the second plurality of characters displayed within the second frame of image data of the co-pilot forward display includes the second character subset for the parameter different from the character subset matching the measured value for the parameter from the pilot forward display. (“If the primary video input is missing or invalid, the second video input channel will be activated, and the pilot will be alerted.” Wang, col. 20, lines 11-12), (“Inside each EDU, the integrity verification procedure comprises three related processes. Graphic commands received from both IMA computers 650 and 660 are evaluated and compared, as shown in step 730.” Wang, col. 17, lines 34-37), and fig. 9
Regarding claim 13, Wang in view of Mead and Greene teaches wherein the first processing system comprises display software configurable to render the character subset for the parameter at a relative position on the pilot forward display different from a second relative position of the second character subset for the parameter on the co-pilot forward display. (“FIG. 11 shows a second redundant system embodiment of the present invention with the GPU function block integrated in the IMA computer. Cockpit display system 900 comprises four substantially identical Electronics Display Units (EDU) 902, 904, 906 and 908, and two independent Integrated Modular Avionics (IMA) computers 950, and 960. Each of the IMA computers 950 and 960, comprises a plurality of CPUs, GPUs, User Application (UA) software and object-based integrity verification systems of the present invention. Each of the IMA computers 950 and 960 has two independent sensor input sources 992 and 994. Each of input sources 992 and 994 comprises a group of sensor inputs selected from a group comprising airspeed, altitude, air temperature, vertical speed, angle of attack, barometric pressure, turn coordinator, heading indicator, attitude indicator, navigation, engine data and other aircraft information.” Wang, col. 19, lines 4-20) and fig. 8
Regarding claim 14, Wang in view of Mead and Greene teaches wherein generating the alert comprises augmenting the pilot forward display to include a graphical user notification proximate the character subset. (“During the run time, EDU processor 806 sends GPU 818 graphic drawing commands to render the display output in the video memory 820 based on the verified command and keyboard input 828. The EDU processor 806 also sends the graphic drawing commands to the object-based integrity verification processor 816 to render the required reference image, such as reference ID code image 425, as shown in FIG. 6.” Wang, col. 18, lines 55-62)
Regarding claim 15, refer to the explanation of claim 5.
Regarding claim 16, Wang in view of Mead and Greene teaches and verifying the first plurality of characters includes the character subset comprising one or more displayed characters matching the measured value comprises identifying an extracted character subset within the first plurality of characters that matches the measured value for the parameter. (“If verification sub-step 342S is used, any object can be independently selected or deselected within the verification zone. When verification sub-step 342S is used, a subset of objects can be selected for one video frame, and different subset of objects can be selected for the next video frame… Several widgets may be grouped together as one object, such as the bank scale 292, especially if they remain unchanged in all video frames.” Wang, col. 6, lines 62-66, col. 7, lines 6-9)
However, Wang does not teach wherein: extracting the first plurality of characters displayed within the first frame comprises performing character recognition on the first frame of image data to obtain the first plurality of characters
Mead teaches wherein: extracting the first plurality of characters displayed within the first frame comprises performing character recognition on the first frame of image data to obtain the first plurality of characters; (“The comparing the extracted symbol to an expected symbol may comprise one or more of: calculating an image characteristic for both the extracted symbol and the expected symbol; calculating a similarity measure between the extracted symbol and the expected symbol; calculating histograms for each of the extracted symbol and the expected symbol;” Mead, col. 20, lines 38-44)
Regarding claim 17, Wang teaches and automatically generating the alert comprises automatically generating the alert in response to a difference between a second extracted character subset within the second plurality of characters corresponding to the parameter and the extracted character subset within the first plurality of characters that matches the measured value for the parameter. (“If the primary video input is missing or invalid, the second video input channel will be activated, and the pilot will be alerted.” Wang, col. 20, lines 11-12), (“Inside each EDU, the integrity verification procedure comprises three related processes. Graphic commands received from both IMA computers 650 and 660 are evaluated and compared, as shown in step 730.” Wang, col. 17, lines 34-37), and fig. 9
However, Wang does not teach wherein: extracting the second plurality of characters displayed within the second frame comprises performing character recognition on the second frame of image data to obtain the second plurality of characters;
Mead teaches wherein: extracting the second plurality of characters displayed within the second frame comprises performing character recognition on the second frame of image data to obtain the second plurality of characters; (“The example process 900 further includes generating an expected image (917) at the server from the application data (operation 916). The example process includes calculating one or more image characteristics 915 from the captured image 913 (operation 914) at the server and calculating one or more image characteristics 919 from the expected image characteristic 917 (operation 918) at the server. The captured image characteristic 915 and the expected image characteristic 919 are compared (operation 920) at the server.” Mead, col. 17, lines 49-57)
Regarding claim 18, Wang in view of Mead and Greene teaches A non-transitory computer-readable medium having computer-executable instructions stored thereon that, when executed by a processing system, cause the processing system to: (“The example application server 102 includes at least one processor and a computer-readable storage device or media encoded with programming instructions for configuring the at least one processor.” Wang, col. 5, lines 34-37)
Regarding rest of claim 18, refer to the explanation of claims 1 and 11.
Regarding claim 20, Wang in view of Mead and Greene teaches wherein the first processing system is configurable to render the first set of one or more displayed characters corresponding to the measured value for the parameter at a first position within the first frame of image data and the second processing system is configurable to render the second set of one or more displayed characters corresponding to the parameter at a second position within the second frame of second image data, wherein the first position and the second position are different. (“FIG. 11 shows a second redundant system embodiment of the present invention with the GPU function block integrated in the IMA computer. Cockpit display system 900 comprises four substantially identical Electronics Display Units (EDU) 902, 904, 906 and 908, and two independent Integrated Modular Avionics (IMA) computers 950, and 960. Each of the IMA computers 950 and 960, comprises a plurality of CPUs, GPUs, User Application (UA) software and object-based integrity verification systems of the present invention. Each of the IMA computers 950 and 960 has two independent sensor input sources 992 and 994. Each of input sources 992 and 994 comprises a group of sensor inputs selected from a group comprising airspeed, altitude, air temperature, vertical speed, angle of attack, barometric pressure, turn coordinator, heading indicator, attitude indicator, navigation, engine data and other aircraft information.” Wang, col. 19, lines 4-20)
Regarding claim 21, Wang in view of Mead and Greene teaches wherein: the first processing system is configured to render the first set of one or more displayed characters on a first side of the first frame; the second processing system is configured to render the second set of one or more displayed characters on a second side of the second frame; and the first side is different from the second side. (“Primary Flight Display (PFD) is a modern aircraft instrument in the cockpit. PFD presents the most critical flight information, such as attitude, airspeed, altitude, vertical speed and heading direction, to the pilot on an integrated display. There are also Multi-Function Displays (MFD) used for navigation, weather, terrain avoidance, engine displays, and crew alerting system. FIG. 1 shows a simplified cockpit crew station 10 in a typical commercial aircraft. Primary flight display (PFD #1) 12 and multi-function display (MFD #1) 14 are installed in front of the pilot on the left side of the cockpit. Multi-function display (MFD #2) 18 and primary flight display (PFD #2) 20 are installed in front of the co-pilot on the right side of the cockpit. Standby flight instruments 16 are installed between MFDs 14 and 18.” Wang, col. 1, lines 27-40)
Regarding claim 22, Greene teaches wherein the second sensing arrangement is redundant to the sensing arrangement. (“the integrated avionics system 100 employs redundant sources of primary flight information to assure the availability of the information to the pilot, and to allow for cross-checking of the sources of the information. For example, the integrated avionics system 100 illustrated in FIG. 1 employs two PFDs 102 that may receive primary flight information from redundant AHRSs 106 and ADCs 108, via the avionics unit 110. The integrated avionics system 100 is configured so that the first PFD 102(1) receives a first set of primary flight information aggregated by the avionics unit 110 from a first AHRS 106(1) and ADC 108(1). Similarly, the second PFD 102(2) receives a second set of primary flight information aggregated by the avionics unit 110 from a second AHRS 106(2) and ADC 108(2). Additionally, although a single avionics unit 110 and a single avionics data bus 112 are illustrated in FIG. 1, it is contemplated that redundant IAU's and/or redundant data buses may be employed for communication between the various components of the integrated avionics system 100.” Greene, col. 6, lines 28-42)
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PARDIS SOHRABY whose telephone number is (571)270-0809. The examiner can normally be reached Monday - Friday 9 am till 6pm.
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PARDIS SOHRABY/ Examiner, Art Unit 2664
/GANDHI THIRUGNANAM/Primary Examiner, Art Unit 2672
Prosecution Insights