DETAILED ACTION
Applicant's has amended claim 4. Claims 1-20 are pending.
Response to Arguments
Applicant’s arguments filed on July 16, 2026 regarding amended claim 4 have been fully considered but are moot in view of the new ground(s) of rejection. The amended claim resulted in changes to the scope and contents; therefore, the grounds of rejection are modified accordingly.
Regarding claim 6, Applicant asserts that “Claim 6 recites the term "a first image capture operation". Withdrawal of the objection is requested.” (Remarks, Pg. 7).
After further review of claim 6, examiner agrees with Applicant regarding statement above. Therefore, prior objection to claim 6 has been withdrawn.
Regarding claim 1, Applicant asserts that “the cited portions of Rix and Lozano, individually or in combination, fail to disclose each and every element of claim 1” (Remarks, Pg. 7-9).
Examiner respectfully disagrees.
As previously set forth in the last Office action, Rix in Par. [0001, 13, 0270-274, 422-426, 512-521, 541-549, 551-559] discloses refueling devices, systems and methods for use in in-flight refueling system that comprises an imaging system for providing (i.e. generating) image data, including acquired 2D images and/or stereoscopic images and/or 3D images (i.e. generate one or more images) of an object, or a part (i.e. portion, section, region, etc.) thereof (i.e. one or more images that each depict at least a portion of an object), for example, including Light Detection And Ranging (LIDAR) units, stereoscopic cameras, conventional cameras, (i.e. an imaging sensor configured to generate one or more images that each depict at least a portion of an object), for example, and the object includes markers (i.e. object that includes multiple reflective markers) that comprise a retro-reflective surface that reflects incident beams and provides a strong (i.e. high) intensity reflection when illuminated (i.e. an imaging sensor configured to generate one or more images that each depict at least a portion of an object that includes multiple reflective markers), for example, including a first set of electromagnetic energy modules (i.e. light sources) configured for illuminating (i.e. projecting light, transmitting light, etc.) a field of regard (i.e. a field of view, scene, environment, etc.) with electromagnetic energy, for example, such as laser energy (i.e. a light source configured to project light), for example, including an imaging (i.e. vision, visual, optical, etc.) system for providing image data of an object, or a part thereof, for example, includes a control computer system (i.e. a processor) programmed to (i.e. a vision processor configured to) process image data acquired, for example, including acquired 2D images and/or stereoscopic images and/or 3D images of an object (i.e. receive a first image of the one or more images), or a part thereof, for example, and the object includes markers that comprise a retro-reflective surface that reflects incident beams and provides a strong (i.e. high) intensity reflection when illuminated (i.e. receive a first image of the one or more images, the first image captured during projection of the light from the light source, the first image captured during projection of the light from the light source), for example, including repeated determination of spatial data (i.e. position, location, etc.) of the object (i.e. an estimate of a location of the object), or a part thereof, for example, based on spatial disposition determination enabling data, such as acquired 2D images and/or stereoscopic images and/or 3D images obtained by data received from LIDAR units, stereoscopic cameras, or conventional cameras (i.e. and determine an estimate of a location of the object based on the first image), for example, including spatial disposition of a receiver aircraft with respect to an engagement area related thereto and/or determination of the spatial dispositions of a refueling device with respect to an engagement enabling position, for example.
Rix discloses the device, as indicated above, but fails to teach “and a filter positioned in a field of view of the imaging sensor and configured to pass at least a portion of the light projected by the light source and reflected by the multiple reflective markers” as further recited in claim 1.
However, as previously set forth in the last Office action, Lozano in Par. [0032-33, 61-62, 76-80] teaches an imaging system that includes a narrow bandpass filter (i.e. a filter) to eliminate light other than that used to light up, and to be reflected in, various objects of a scenario (i.e. field of view, scene, environment, etc.), for example, including a camera (i.e. the imaging sensor) that has, before reaching its sensor (i.e. a filter positioned in a field of view of the imaging sensor), a very narrow band pass filter that only lets wavelengths (λ2) pass that are very close to those generated when reflected by various objects in an environment, including light pluses of filtered wavelengths reflected objects around the aircraft (i.e. multiple reflective markers), for example, including points of light reflected on the environment/scenario generated as a result of structured lighting that is generated, for example, and the result is therefore a point cloud corresponding to points detected in the sensor when reflected from a structured illuminator (i.e. a filter positioned in a field of view of the imaging sensor and configured to pass at least a portion of the light projected by the light source and reflected by the multiple reflective markers).
RIX and Lozano are considered to be analogous art because they pertain to image processing applications. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the variety of refueling devices, systems and methods for use in in-flight refueling that comprises an imaging system for providing image data of an object that includes markers that comprise a retro-reflective surface that reflects incident beams and provides a strong intensity reflection when illuminated and including repeated determination of spatial data of the object (as disclosed by RIX) with and a filter positioned in a field of view of the imaging sensor and configured to pass at least a portion of the light projected by the light source and reflected by the multiple reflective markers (as taught by Lozano, Abstract, Par. [0032-33, 61-62, 76-80]) to eliminate light other than that used to light up, and to be reflected in, various objects of a scenario (Lozano, Abstract, Par. [032-33, 61-62, 76-80]), as previously set forth in the last Office action.
As indicated above, the combined teachings of Rix and Lozano, as a whole, teach a similar inventive concept as the one being recited in the feature limitations of claim 1.
Therefore, based on above rationale, examiner believes rejection of claim 1 previously set forth in the last Office action was proper as nothing prevents Rix and Lozano, as a whole, from being applied to reject claim 1 and all prior rejections of original claims, including claim 1, are hereby maintained as indicated below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 6, 11, 13, and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over RIX et al. (US PG Publication No. 2019/0359345 A1), hereafter referred to as RIX, in view of Lozano et al. (US PG Publication No. 2019/0118963 A1), hereafter referred to as Lozano, for the same reasons as set forth in the last Office action (revised grounds are restated below for convenience).
Regarding claim 1, RIX discloses a device (Par. [0001]: systems and methods for refueling air vehicles, especially aircraft, in particular for refueling aircraft during flight; Par. [0013]: a variety of refueling devices, systems and methods for use in in-flight refueling) comprising:
an imaging system (Par. [0270]: system comprises an imaging system) including:
an imaging sensor configured to generate one or more images that each depict at least a portion of an object that includes multiple reflective markers (Par. [0270-271]: system comprises an imaging system for providing said data including at least image data corresponding to a field of regard with respect to the refueling device. The data acquisition system can be in the form of an imaging system… said imaging system is configured for providing at least one of 2D images, stereoscopic images, and 3D images of a volume defined by said field of regard; Par. [0422-426]: the data acquisition system is in the form of imaging system 150, in particular configured for providing imaging data of any object coming within a field of regard (FOR) aft of the refueling device 100… such an object is the receiver aircraft 20 and more particularly a part AP thereof including the fuel receptacle 22… FLADAR units 151 illuminate the sensing volume 159 and any object therein, in particular part AP of the receiver aircraft 20 and thereafter acquire suitable image data corresponding thereto which is sent to control computer system 155 for processing to provide the aforesaid control signals for controlling the refueling device 100… by means of the FLADAR units 151, a 3D image of the areas AP is reconstructed… the imaging system 150 can comprise any other suitable imaging system (for example, but not limited to, systems providing 2D images and/or stereoscopic images and/or 3D images of (including reconstruction of 3D data corresponding to) the sensing volume 159, in particular but not limited to images that are updated in real time, for example in the form of a video stream) that operate to provide suitable data to the control computer system 155; Par. [0512-521]: the data acquisition system is in the form of imaging system 289B, in particular configured for providing imaging data of any object coming within a field of regard (FOR) aft of the refueling device 200B… imaging system 289B is for example similar to imaging system 350 as described herein… and comprises one or more Light Detection And Ranging (LIDAR) units… the LIDAR unit 351 can be replaced with any other suitable imaging system 350 that provides depth data and electromagnetic intensity data of objects within the sensing volume 359 (including, but not limited to, Flash LADAR, 3D Flash LIDAR Camera, etc.). In still further alternative variations of this example, the LIDAR unit 351 can be replaced with other suitable imaging systems such as stereoscopic cameras, conventional cameras, various radar systems… fuel receptacle marker 342 is at a fixed and known geometrical relationship with respect to the fuel receptacle 22, and is electromagnetically visible to the imaging system 350… the fuel receptacle marker 342 comprises a retro-reflective surface that reflects incident beams along the same path, and thus provides a strong intensity reflection of the respective reflected beam when illuminated by a beam… retro-reflective surface may be provided via a retro-reflective material affixed to a certain known location visible to the imaging system 350 on the receiver aircraft 20; Par. [0543-544]: LIDAR unit 351 is configured for providing depth data and electromagnetic data (for example electromagnetic intensity data) relating to any object within the sensing volume 359, in particular relating to one or more of the nozzle module 317, nozzle 316 and part AP… LIDAR unit 351 comprises a fiber laser unit 352 configured for generating and radiating a laser beam B1… Each time the outgoing beam B1 impinges on an object within the sensing volume 359, a reflected beam B2 returns along a path similar to that of the outgoing beam B1; Par. [0551-559]: the time interval between the outgoing beam B1 and the return beam B2 being detected by the detector 356 provides a measure of the depth of the part of the object which reflected the beam, thereby providing the depth data. In addition, the detector 356 also detects the intensity of the part of the object which reflected the beam to provide the intensity data… the LIDAR unit 351 can be replaced with any other suitable imaging system 350 that provides depth data and electromagnetic intensity data of objects within the sensing volume 359 (including, but not limited to, Flash LADAR, 3D Flash LIDAR Camera, etc.)… the LIDAR unit 351 can be replaced with other suitable imaging systems such as stereoscopic cameras, conventional cameras… boom tip marker 340 comprises a retro-reflective surface that reflects incident beams along the same path, and thus provides a strong intensity reflection of the respective beam B2 when illuminated by beam B1… the fuel receptacle marker 342 comprises a retro-reflective surface that reflects incident beams along the same path, and thus provides a strong intensity reflection of the respective beam B2 when illuminated by beam B1… control computer system 355 is further configured for operating on the depth data and the electromagnetic intensity data to enable identification of a second part of the depth data that corresponds to the high intensity reflection originating from the boom tip marker 340, which in turn enables identification of the part of a third depth data that corresponds to nozzle 316 since the relative spatial relationship between the boom tip marker 340 and nozzle 316 is known; Par. [0554-557]: the data acquisition system 360 further includes boom tip marker 340, comprised in the fuel delivery nozzle module 317 proximate to the nozzle 316. The boom tip marker 340 is at a fixed and known geometrical relationship with respect to the nozzle 316, independently of the relative motion between the boom system 300 and the tanker aircraft 12′, and is electromagnetically visible to the imaging system 350… the boom tip marker 340 comprises a retro-reflective surface that reflects incident beams along the same path, and thus provides a strong intensity reflection of the respective beam B2 when illuminated by beam B1… the fuel receptacle marker 342 comprises a retro-reflective surface that reflects incident beams along the same path, and thus provides a strong intensity reflection of the respective beam B2 when illuminated by beam B1; an imaging sensor configured to generate one or more images that each depict at least a portion of an object that includes multiple reflective markers (e.g. refueling devices, systems and methods for use in in-flight refueling system that comprises an imaging system for providing (i.e. generating) image data, including acquired 2D images and/or stereoscopic images and/or 3D images (i.e. generate one or more images) of an object, or a part (i.e. portion, section, region, etc.) thereof (i.e. one or more images that each depict at least a portion of an object), for example, including Light Detection And Ranging (LIDAR) units, stereoscopic cameras, conventional cameras, (i.e. an imaging sensor configured to generate one or more images that each depict at least a portion of an object), for example, and the object includes markers (i.e. object that includes multiple reflective markers) that comprise a retro-reflective surface that reflects incident beams and provides a strong (i.e. high) intensity reflection when illuminated (i.e. an imaging sensor configured to generate one or more images that each depict at least a portion of an object that includes multiple reflective markers), as indicated above), for example);
a light source configured to project light (Par. [0270-274]: system comprises an imaging system for providing said data including at least image data corresponding to a field of regard with respect to the refueling device. The data acquisition system can be in the form of an imaging system… wherein said imaging system comprises or is operatively connected to a computing system configured for identifying a fuel receptacle of a receiver aircraft within said field of regard from said image data, and for determining a spatial disposition of said nozzle with respect to the fuel receptacle… wherein said imaging system comprises a first set of electromagnetic energy modules configured for illuminating said field of regard with electromagnetic energy (for example laser energy), and a second set of electromagnetic energy modules configured for receiving electromagnetic energy from said illuminated field of regard; Par. [0423-424]: imaging system 150 comprises two pairs of flash ladar units 151, also referred to interchangeably herein as FLADAR units… the FLADAR units 151 illuminate the sensing volume 159 and any object therein, in particular part AP of the receiver aircraft 20; Par. [0562-563]: operation of the imaging system 350 and control computer system 355 will be described… the LIDAR unit 351 illuminates the sensing volume 359 and any object therein, in particular part AP of the receiver aircraft 20; a light source configured to project light (e.g. refueling devices, systems and methods for use in in-flight refueling system comprises an imaging system including a first set of electromagnetic energy modules (i.e. light sources) configured for illuminating (i.e. projecting light, transmitting light, etc.) a field of regard (i.e. a field of view, scene, environment, etc.) with electromagnetic energy, for example, such as laser energy (i.e. a light source configured to project light), as indicated above), for example); and
a vision processor configured (Par. [0422]: an object is the receiver aircraft 20 and more particularly a part AP thereof including the fuel receptacle 22, and the sensing volume 159 defines an outer envelope limit 158 in which image data of part AP can be processed… by control computer system 155; Par. [0513]: imaging system 289B is operatively connected to a control computer system 255B… an object is the receiver aircraft 20 and more particularly a part AP thereof including the fuel receptacle 22, and the sensing volume 259B defines an outer envelope limit 258B in which image data of part AP can be processed… by control computer system 255B; Par. [0541-549]: imaging system 350 is operatively connected to a control computer system 355… an object is the receiver aircraft 20 and more particularly a part AP thereof including the fuel receptacle 22 and/or another part of the receiver aircraft 20… and the sensing volume 359 defines an outer envelope limit 358 in which image data of part AP can be processed… by control computer system 355… Field Programmable Gate Array (FPGA) can be used to enable the required processing power for processing the data acquired by the LIDAR unit 351… Digital Signal Processor (DSP) can be a real time processor that can be configured to analyze the data acquired by the LIDAR unit 351 and provide the electromagnetic data and the depth data. Such data can be stored in the memory and can be transferred (e.g. transmitted) for further computations in one or more additional/alternative computers; a vision processor configured to (e.g. refueling devices, systems and methods for use in in-flight refueling system that comprises an imaging (i.e. vision, visual, optical, etc.) system for providing image data of an object, or a part thereof, for example, includes a control computer system (i.e. a processor) programmed to (i.e. a vision processor configured to) process image data acquired, as indicated above), for example):
receive a first image of the one or more images, the first image captured during projection of the light from the light source; and
determine an estimate of a location of the object based on the first image (Par. [0270-271]: data acquisition system configured for providing spatial data relating to a relative spatial disposition between a fuel delivery nozzle of the refueling device and a fuel receptacle of the receiver aircraft… system comprises an imaging system for providing said data including at least image data corresponding to a field of regard with respect to the refueling device. The data acquisition system can be in the form of an imaging system… said imaging system is configured for providing at least one of 2D images, stereoscopic images, and 3D images of a volume defined by said field of regard; Par. [0422-426]: the refueling device 100 further comprises a suitable data acquisition system for providing or enabling the calculation of spatial data relating to the relative spatial dispositions between the refueling device 100 and the receiver aircraft 20, in particular the relative spatial dispositions between the fuel delivery nozzle of the refueling device 100 and the fuel receptacle of the receiver aircraft… the data acquisition system is in the form of imaging system 150, in particular configured for providing imaging data of any object coming within a field of regard (FOR) aft of the refueling device 100… such an object is the receiver aircraft 20 and more particularly a part AP thereof including the fuel receptacle 22… FLADAR units 151 illuminate the sensing volume 159 and any object therein, in particular part AP of the receiver aircraft 20 and thereafter acquire suitable image data corresponding thereto which is sent to control computer system 155 for processing to provide the aforesaid control signals for controlling the refueling device 100… by means of the FLADAR units 151, a 3D image of the areas AP is reconstructed… the imaging system 150 can comprise any other suitable imaging system (for example, but not limited to, systems providing 2D images and/or stereoscopic images and/or 3D images of (including reconstruction of 3D data corresponding to) the sensing volume 159, in particular but not limited to images that are updated in real time, for example in the form of a video stream) that operate to provide suitable data to the control computer system 155; Par. [0576]: imaging system 150 can be configured… for providing imaging data (including spatial disposition determination enabling data such as data received from a LIDAR unit, etc.) of any object coming within a field of regard (FOR) aft of the refueling device 100 (and/or the tanker aircraft 12). The sensor control module 1810 is configured to operate sensor 1890 (or multiple sensors, mutatis mutandis) in order to acquire data that enables, inter alia, repeated determination of spatial data such as the spatial disposition of the receiver aircraft 20 with respect to an engagement area related thereto and/or determination of the spatial dispositions of the refueling device 100 with respect to an engagement enabling position; receive a first image of the one or more images, the first image captured during projection of the light from the light source; and
determine an estimate of a location of the object based on the first image (e.g. refueling devices, systems and methods for use in in-flight refueling system that comprises an imaging system for providing image data, including acquired 2D images and/or stereoscopic images and/or 3D images of an object (i.e. receive a first image of the one or more images), or a part thereof, for example, and the object includes markers that comprise a retro-reflective surface that reflects incident beams and provides a strong (i.e. high) intensity reflection when illuminated (i.e. receive a first image of the one or more images, the first image captured during projection of the light from the light source), for example, including repeated determination of spatial data (i.e. position, location, etc.) of the object (i.e. an estimate of a location of the object), or a part thereof, for example, based on spatial disposition determination enabling data, such as acquired 2D images and/or stereoscopic images and/or 3D images obtained by data received from LIDAR units, stereoscopic cameras, or conventional cameras (i.e. and determine an estimate of a location of the object based on the first image), for example, including spatial disposition of a receiver aircraft with respect to an engagement area related thereto and/or determination of the spatial dispositions of a refueling device with respect to an engagement enabling position, as indicated above), for example), but fails to teach the following as further recited in claim 1.
However, Lozano teaches and a filter positioned in a field of view of the imaging sensor and configured to pass at least a portion of the light projected by the light source and reflected by the multiple reflective markers (Par. [0032-33]: second subsystem that we will call STOF comprised by a TOF—(Time Of Flight) type camera that has the special feature of measuring the time that it takes a pulse of light generated to be reflected in the various objects… STOF subsystem also consists of electronic elements, a laser with its diffuser, a lens and a F1 narrow bandpass filter to eliminate light other than that used to light up our scenario. Here the main function of the electronic elements is to calculate the round trip time of the photons that come out of the L1 laser emitter, which is also part of this subsystem, and that bounce off the objects around the aircraft to go back to each pixel of the camera … third subsystem, which we will call SDOE, comprised of a camera that has, before reaching its sensor, a very narrow band pass filter that only lets wavelengths (λ2) pass that are very close to those generated by an L2 laser that also belongs to the subsystem; Par. [0061-62]: second subsystem comprised by a TOF-type camera that has the special feature of measuring the time that it takes a pulse of light generated to be reflected in the various objects… subsystem, which we will call STOF, has an electronic element, a focusing lens and a narrow B1 bandpass filter to eliminate the different light used to light up our scenario… calculations are performed for each pixel or point of the TOF camera sensor. It is obvious that the wavelength λ1 of the light of L1 is the same as the central wavelength of the B1 filter band of the chamber of the STOF subsystem (12)… third subsystem that we will call SDOE (10) comprised of a camera equipped with electronic elements, and an optical system that includes a narrow band pass filter (20) at a wavelength that coincides with the emission of a laser… The L2 laser of wavelength λ2, turns on and off at controlled periods to facilitate the detection of the points illuminated by the pattern generated. The DOE camera is comprised of its electronic elements, an image sensor, lenses and narrow B2 bandpass filter tuned to λ2. Once the points have been detected, the electronic elements determine the relative distances of the points illuminated and received in the pixels of the camera as the second part of the mission of this subsystem. This is done by means of triangulation, measuring the displacement generated according to the distance and with knowledge of the separation between the laser and the camera used. As already mentioned, the wavelength λ2 of the L2 light is the same as the central wavelength of the B2 filter band of the SDOE subsystem camera. The result is therefore a new point cloud corresponding to those detected in the sensor when reflected from our structured illuminator; Par. [0076-80]: Determination of the position of each light emitting point of the BD and RD elements, which are located in solidarity, respectively at the end of the pole nozzle and at the contour of the receptacle, using 3D cameras. The light emission by these emitters is uniform in the emission directions and allows 3D cameras to “see them” and thus determine the position of each of them with respect to CR. In order to facilitate this task, the emitters are alternatively flashed with certain patterns. The first are synchronised with 3D cameras, and the second are not. All are filtered temporarily with respect to the other LEDs. This avoids unnecessary overlaps between the emitters and facilitates detection by cross-correlation techniques to eliminate confusion with other points of light. Reflections can also be eliminated thanks to the synchronised alternative lighting of the emitters. This synchronism minimises the energy needed for detection. The use of a filter tuned to the light emitting wavelength of the emitters also allows an increase in the signal-to-noise ratio, again facilitating this detection… L1 laser provides a set of light pulses with wavelength λ1. The circuit that triggers the switching on of this laser is the same that governs the shooting and image acquisition of the TOF-type camera included in STOF. Considering the speed of light and the time it takes to receive the pulse generated in each pixel of the sensor of the TOF-type camera, the distance from the point of the scenario that reflects the light received can be obtained. To facilitate this task, a narrow band pass filter B1 centred in λ1 is placed in front of the TOF-type camera… L2 laser generates a pattern… of structured light… The elements of this pattern can be identified if we are able to “see” the light emitted by the laser when reflected by our environment. To facilitate this, we use a new narrow B2 band pass filter in front of the SDOE camera, tuned to L2 and that will eliminate light from other wavelengths. In addition, turning it on and off with a certain rate will also help us to distinguish the light of the laser with respect to other light from different sources… With cross-correlation techniques we will obtain the pixels that are reflected in the objects of our scenario, and from their relative intensities we will determine which pixels correspond to certain points of the pattern; and a filter positioned in a field of view of the imaging sensor and configured to pass at least a portion of the light projected by the light source and reflected by the multiple reflective markers (e.g. imaging system includes a narrow bandpass filter (i.e. a filter) to eliminate light other than that used to light up, and to be reflected in, various objects of a scenario (i.e. field of view, scene, environment, etc.), for example, including a camera (i.e. the imaging sensor) that has, before reaching its sensor (i.e. a filter positioned in a field of view of the imaging sensor), a very narrow band pass filter that only lets wavelengths (λ2) pass that are very close to those generated when reflected by various objects in an environment, including light pluses of filtered wavelengths reflected objects around the aircraft (i.e. multiple reflective markers), including points of light reflected on the environment/scenario generated as a result of structured lighting that is generated, for example, and the result is therefore a point cloud corresponding to points detected in the sensor when reflected from a structured illuminator (i.e. a filter positioned in a field of view of the imaging sensor and configured to pass at least a portion of the light projected by the light source and reflected by the multiple reflective markers), as indicated above), for example).
RIX and Lozano are considered to be analogous art because they pertain to image processing applications. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the variety of refueling devices, systems and methods for use in in-flight refueling that comprises an imaging system for providing image data of an object that includes markers that comprise a retro-reflective surface that reflects incident beams and provides a strong intensity reflection when illuminated and including repeated determination of spatial data of the object (as disclosed by RIX) with and a filter positioned in a field of view of the imaging sensor and configured to pass at least a portion of the light projected by the light source and reflected by the multiple reflective markers (as taught by Lozano, Abstract, Par. [0032-33, 61-62, 76-80]) to eliminate light other than that used to light up, and to be reflected in, various objects of a scenario (Lozano, Abstract, Par. [032-33, 61-62, 76-80]).
Regarding claim 2, claim 1 is incorporated and the combination of RIX and Lozano, as a whole, teaches the device (RIX, Par. [0001]), wherein:
the vision processor is further configured to process the first image (RIX, Par. [0270-271]: data acquisition system configured for providing spatial data relating to a relative spatial disposition between a fuel delivery nozzle of the refueling device and a fuel receptacle of the receiver aircraft… system comprises an imaging system for providing said data including at least image data corresponding to a field of regard with respect to the refueling device. The data acquisition system can be in the form of an imaging system… said imaging system is configured for providing at least one of 2D images, stereoscopic images, and 3D images of a volume defined by said field of regard; Par. [0422]: an object is the receiver aircraft 20 and more particularly a part AP thereof including the fuel receptacle 22, and the sensing volume 159 defines an outer envelope limit 158 in which image data of part AP can be processed… by control computer system 155; Par. [0513]: imaging system 289B is operatively connected to a control computer system 255B… an object is the receiver aircraft 20 and more particularly a part AP thereof including the fuel receptacle 22, and the sensing volume 259B defines an outer envelope limit 258B in which image data of part AP can be processed… by control computer system 255B; Par. [0541-549]: imaging system 350 is operatively connected to a control computer system 355… an object is the receiver aircraft 20 and more particularly a part AP thereof including the fuel receptacle 22 and/or another part of the receiver aircraft 20… and the sensing volume 359 defines an outer envelope limit 358 in which image data of part AP can be processed… by control computer system 355… Field Programmable Gate Array (FPGA) can be used to enable the required processing power for processing the data acquired by the LIDAR unit 351… Digital Signal Processor (DSP) can be a real time processor that can be configured to analyze the data acquired by the LIDAR unit 351 and provide the electromagnetic data and the depth data. Such data can be stored in the memory and can be transferred (e.g. transmitted) for further computations in one or more additional/alternative computers; wherein: the vision processor is further configured to process the first image (e.g. refueling devices, systems and methods for use in in-flight refueling system that comprises an imaging (i.e. vision, visual, optical, etc.) system for providing image data of an object, or a part thereof, for example, includes a control computer system (i.e. a processor) programmed to (i.e. a vision processor configured to) process image data acquired, as indicated above), for example); and
the imaging system, the processing of the first image by the vision processor, or both, is configured to minimize solar interference in the first image (Lozano, Par. [0032-33]: second subsystem that we will call STOF comprised by a TOF—(Time Of Flight) type camera that has the special feature of measuring the time that it takes a pulse of light generated to be reflected in the various objects… STOF subsystem also consists of electronic elements, a laser with its diffuser, a lens and a F1 narrow bandpass filter to eliminate light other than that used to light up our scenario… third subsystem, which we will call SDOE, comprised of a camera that has, before reaching its sensor, a very narrow band pass filter that only lets wavelengths (λ2) pass that are very close to those generated by an L2 laser that also belongs to the subsystem; Par. [0061-62]: second subsystem comprised by a TOF-type camera that has the special feature of measuring the time that it takes a pulse of light generated to be reflected in the various objects… subsystem, which we will call STOF, has an electronic element, a focusing lens and a narrow B1 bandpass filter to eliminate the different light used to light up our scenario… calculations are performed for each pixel or point of the TOF camera sensor. It is obvious that the wavelength λ1 of the light of L1 is the same as the central wavelength of the B1 filter band of the chamber of the STOF subsystem (12)… third subsystem that we will call SDOE (10) comprised of a camera equipped with electronic elements, and an optical system that includes a narrow band pass filter (20) at a wavelength that coincides with the emission of a laser… The L2 laser of wavelength λ2, turns on and off at controlled periods to facilitate the detection of the points illuminated by the pattern generated. The DOE camera is comprised of its electronic elements, an image sensor, lenses and narrow B2 bandpass filter tuned to λ2. Once the points have been detected, the electronic elements determine the relative distances of the points illuminated and received in the pixels of the camera as the second part of the mission of this subsystem. This is done by means of triangulation, measuring the displacement generated according to the distance and with knowledge of the separation between the laser and the camera used. As already mentioned, the wavelength λ2 of the L2 light is the same as the central wavelength of the B2 filter band of the SDOE subsystem camera. The result is therefore a new point cloud corresponding to those detected in the sensor when reflected from our structured illuminator; Par. [0076-80]: operating procedure of the automatic contact system covered by the invention comprises the following stages: [0076] Determination of the position of each light emitting point of the BD and RD elements, which are located in solidarity, respectively at the end of the pole nozzle and at the contour of the receptacle, using 3D cameras. The light emission by these emitters is uniform in the emission directions and allows 3D cameras to “see them” and thus determine the position of each of them with respect to CR. In order to facilitate this task, the emitters are alternatively flashed with certain patterns. The first are synchronised with 3D cameras, and the second are not. All are filtered temporarily with respect to the other LEDs. This avoids unnecessary overlaps between the emitters and facilitates detection by cross-correlation techniques to eliminate confusion with other points of light. Reflections can also be eliminated thanks to the synchronised alternative lighting of the emitters. This synchronism minimises the energy needed for detection. The use of a filter tuned to the light emitting wavelength of the emitters also allows an increase in the signal-to-noise ratio, again facilitating this detection… L1 laser provides a set of light pulses with wavelength λ1. The circuit that triggers the switching on of this laser is the same that governs the shooting and image acquisition of the TOF-type camera included in STOF. Considering the speed of light and the time it takes to receive the pulse generated in each pixel of the sensor of the TOF-type camera, the distance from the point of the scenario that reflects the light received can be obtained. To facilitate this task, a narrow band pass filter B1 centred in λ1 is placed in front of the TOF-type camera… L2 laser generates a pattern… of structured light… The elements of this pattern can be identified if we are able to “see” the light emitted by the laser when reflected by our environment. To facilitate this, we use a new narrow B2 band pass filter in front of the SDOE camera, tuned to L2 and that will eliminate light from other wavelengths. In addition, turning it on and off with a certain rate will also help us to distinguish the light of the laser with respect to other light from different sources… With cross-correlation techniques we will obtain the pixels that are reflected in the objects of our scenario, and from their relative intensities we will determine which pixels correspond to certain points of the pattern; the imaging system, the processing of the first image by the vision processor, or both, is configured to minimize solar interference in the first image (e.g. imaging system includes a narrow bandpass filter (i.e. a filter) to eliminate light (i.e. minimize light interference) other than that used to light up, and to be reflected in, various objects of a scenario (i.e. field of view, scene, environment, etc.), for example, including a camera (i.e. the imaging sensor) that has, before reaching its sensor, a very narrow band pass filter that only lets wavelengths (λ2) pass that are very close to those generated when reflected by various objects in an environment (i.e. the imaging system, the processing of the first image by the vision processor, or both, is configured to minimize solar (i.e. light) interference in the first image), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 6, claim 1 is incorporated and the combination of RIX and Lozano, as a whole, teaches the device (RIX, Par. [0001]), wherein:
the vision processor is further configured to:
activate the light source concurrently with initiation of a [the] first image capture operation by the imaging sensor to capture the first image (Lozano, Par. [0049]: a set of light emitters (16), that in general can consist of LEDs or laser diodes with their respective diffusers. These emitters are arranged on their surface and emit light homogeneously, at certain times, which will be detected by a set of cameras of the subsystem S3D (9), whose task will be to determine the position of these light emitters in relation to them. The electronic elements (22) consist of an adaptation of the aircraft's power supply, a set of drivers or adapters for switching on the light emitters and a communications subsystem that will receive commands from the electronics that govern the previous cameras in order to obtain a certain level of synchronisation between both subsystems (cameras and LED emitters); Par. [0062]: detect with the camera, which we will call a DOE-type camera, the points of light reflected on our scenario and generated as a result of the structured lighting that is generated. The L2 laser of wavelength λ2, turns on and off at controlled periods to facilitate the detection of the points illuminated by the pattern generated. The DOE camera is comprised of its electronic elements, an image sensor, lenses and narrow B2 bandpass filter tuned to λ2; activate the light source concurrently with initiation of the first image capture operation by the imaging sensor to capture the first image (e.g. imaging system includes a narrow bandpass filter to eliminate light other than that used to light up, and to be reflected in, various objects of a scenario (i.e. field of view, scene, environment, etc.), for example, turns on and off cameras and LED emitters at controlled periods to facilitate the detection of points illuminated to obtain a certain level of synchronization between cameras and LED emitters (i.e. activate (i.e. turn on) the light source concurrently (i.e. in synchronization) with initiation of the first image capture operation by the imaging sensor to capture the first image), as indicated above), for example);
initiate a second image capture operation by the imaging sensor to capture a second image of the one or more images, the second image captured while the light source is deactivated; and
receive the second image (Lozano, Par. [0049]: a set of light emitters (16), that in general can consist of LEDs or laser diodes with their respective diffusers. These emitters are arranged on their surface and emit light homogeneously, at certain times, which will be detected by a set of cameras of the subsystem S3D (9), whose task will be to determine the position of these light emitters in relation to them. The electronic elements (22) consist of an adaptation of the aircraft's power supply, a set of drivers or adapters for switching on the light emitters and a communications subsystem that will receive commands from the electronics that govern the previous cameras in order to obtain a certain level of synchronisation between both subsystems (cameras and LED emitters); Par. [0062]: detect with the camera, which we will call a DOE-type camera, the points of light reflected on our scenario and generated as a result of the structured lighting that is generated. The L2 laser of wavelength λ2, turns on and off at controlled periods to facilitate the detection of the points illuminated by the pattern generated. The DOE camera is comprised of its electronic elements, an image sensor, lenses and narrow B2 bandpass filter tuned to λ2; initiate a second image capture operation by the imaging sensor to capture a second image of the one or more images, the second image captured while the light source is deactivated; and receive the second image (e.g. imaging system includes a narrow bandpass filter to eliminate light other than that used to light up, and to be reflected in, various objects of a scenario (i.e. field of view, scene, environment, etc.), for example, turns on and off cameras and LED emitters at controlled periods to facilitate the detection of points illuminated to obtain a certain level of synchronization between cameras and LED emitters (i.e. initiate a second image capture operation by the imaging sensor to capture a second image of the one or more images, the second image captured while the light source is deactivated (i.e. off) and receive the second image), as indicated above), for example); and
the first image is captured prior or subsequent to the second image being captured (RIX, Par. [0648]: for example, sensor control module 1810 for repeatedly, and in some cases in real time (for example in the form of a video stream) acquiring an image of the area in which the refueling device 100 and/or the boom fuelling unit 310 is expected to be positioned; Par. [0670-688]: repeatedly (e.g. every pre-determined period) or continuously (e.g. in the form of a video stream) acquiring an image of the area aft the refueling device 100 and/or aft the tanker aircraft 12, at a predetermined Field of View… repeatedly (e.g. every pre-determined period) or continuously acquiring an image… when reference is made to an image it can also include an image based on data acquired by the LIDAR unit 351) … A series of images can be analyzed, each substantially immediately after it has been capture).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 11, is a method clam that recites broader, but similar, feature limitations as in device claim 1 above. Therefore, claim 11 is rejected as applied to claim 1 above.
Regarding claim 13, claim 11 is incorporated and recites similar feature limitations as in device claim 6 above. Therefore, claim 13 is rejected as applied to claim 6 above.
Regarding claim 17, is a device clam that recites broader, but similar, feature limitations as in device claim 1 above. Therefore, claim 17 is rejected as applied to claim 1 above.
Regarding claim 18, claim 17 is incorporated and the combination of RIX and Lozano, as a whole, teaches the device (RIX, Par. [0001]: wherein the device includes a vehicle systems and methods for refueling air vehicles, especially aircraft, in particular for refueling aircraft during flight; Par. [0368]: systems and devices for in-flight refueling of aircraft).
Regarding claim 19, claim 18 is incorporated and the combination of RIX and Lozano, as a whole, teaches the device (RIX, Par. [0001]), wherein the vehicle includes an aircraft (RIX, Par. [0001]: wherein the device includes a vehicle systems and methods for refueling air vehicles, especially aircraft, in particular for refueling aircraft during flight; Par. [0368]: systems and devices for in-flight refueling of aircraft; Par. [0705]: imaging can be provided or augmented via suitable cameras or other imaging units, provided on the tanker aircraft 12 and/or the receiver aircraft 20 and/or any other suitable air vehicle in the vicinity of the refueling device 100).
Regarding claim 20, claim 19 is incorporated and the combination of RIX and Lozano, as a whole, teaches the device (RIX, Par. [0001]), wherein the aircraft includes an autonomous air vehicle (RIX, Par. [0033]: automatic steering and the automatic engaging are performed autonomously by the refueling device; Par. [0065]: steering control module and the engagement/disengagement module are fitted within the refueling device for enabling autonomously controlling in-flight refueling of the receiver aircraft by the refueling device; Par. [0277]: controller can comprise, for example, a computer system, operatively connected to said spatial data acquisition system and/or to said spatial control system, and/or optionally configured as an automatic or autonomous system for enabling refueling device to be steered to an engagement enabling position to provide engagement of the nozzle with the fuel receptacle of the receiver aircraft, and thereafter enable refueling of the receiver aircraft).
Claims 3 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over RIX, in view of Lozano, as applied to claim 1 above, in further view of Zhao et al. (US PG Publication No. US 2021/0208283 A1), hereafter referred to as Zhao, for the same reasons as set forth in the last Office action (grounds are restated below for convenience).
Regarding claim 3, claim 1 is incorporated and the combination of RIX and Lozano, as a whole, teaches the device (RIX, Par. [0001]), wherein:
the imaging sensor includes a camera (RIX, Par. [0512-518]: the data acquisition system is in the form of imaging system 289B, in particular configured for providing imaging data of any object coming within a field of regard (FOR) aft of the refueling device 200B… imaging system 289B is for example similar to imaging system 350 as described herein… and comprises one or more Light Detection And Ranging (LIDAR) units… the LIDAR unit 351 can be replaced with any other suitable imaging system 350 that provides depth data and electromagnetic intensity data of objects within the sensing volume 359 (including, but not limited to, Flash LADAR, 3D Flash LIDAR Camera, etc.); Par. [0552-575]: the LIDAR unit 351 can be replaced with any other suitable imaging system 350 that provides depth data and electromagnetic intensity data of objects within the sensing volume 359 (including, but not limited to, Flash LADAR, 3D Flash LIDAR Camera, etc.). In still further alternative variations of this example, the LIDAR unit 351 can be replaced with other suitable imaging systems such as stereoscopic cameras, conventional cameras, various radar systems, etc… sensor control module 1810 can be operatively connected to at least one sensor 1890 and can be configured to control the operation of the sensor 1890. The sensor 1890 can be a suitable data acquisition system, for example, any image acquisition means such as a camera (e.g. a digital still camera, a digital video camera, a Flash LADAR camera, etc.). Alternatively, the sensor 1890 can be a Light Detection And Ranging (LIDAR) unit);
the light source includes a high-intensity light source (RIX, Par. [0270-274]: system comprises an imaging system for providing said data including at least image data corresponding to a field of regard with respect to the refueling device … wherein said imaging system comprises a first set of electromagnetic energy modules configured for illuminating said field of regard with electromagnetic energy (for example laser energy), and a second set of electromagnetic energy modules configured for receiving electromagnetic energy from said illuminated field of regard; Par. [0521-544]: retro-reflective surface that reflects incident beams along the same path, and thus provides a strong intensity reflection of the respective reflected beam when illuminated by a beam… control computer system 355 is further configured for operating on the depth data and the electromagnetic intensity data, as disclosed herein, mutatis mutandis, to enable identification of a second part of the depth data that corresponds to the high intensity reflection… LIDAR unit 351 comprises a fiber laser unit 352 configured for generating and radiating a laser beam B1; Par. [0555-559]: marker 340 comprises a retro-reflective surface that reflects incident beams along the same path, and thus provides a strong intensity reflection of the respective beam B2 when illuminated by beam B1… control computer system 355 is further configured for operating on the depth data and the electromagnetic intensity data to enable identification of a second part of the depth data that corresponds to the high intensity reflection; the light source includes (i.e. at least one) a high-intensity light source (e.g. refueling devices, systems and methods for use in in-flight refueling system comprises an imaging system including a first set of electromagnetic energy modules (i.e. light sources) configured for illuminating a field of regard (i.e. a field of view, scene, environment, etc.) with electromagnetic energy, for example, such as laser energy (i.e. the light source includes a high-intensity light source), for example, and including retro-reflective surfaces that reflect incident beams and provide a strong (i.e. high) intensity reflection when illuminated, as indicated above), for example), a high-speed light source, a narrow-band light source (Lozano, Par. [0032-33]: second subsystem that we will call STOF comprised by a TOF—(Time Of Flight) type camera that has the special feature of measuring the time that it takes a pulse of light generated to be reflected in the various objects… STOF subsystem also consists of electronic elements, a laser with its diffuser, a lens and a F1 narrow bandpass filter to eliminate light other than that used to light up our scenario… third subsystem, which we will call SDOE, comprised of a camera that has, before reaching its sensor, a very narrow band pass filter that only lets wavelengths (λ2) pass that are very close to those generated by an L2 laser that also belongs to the subsystem; Par. [0061-62]: second subsystem comprised by a TOF-type camera that has the special feature of measuring the time that it takes a pulse of light generated to be reflected in the various objects… subsystem, which we will call STOF, has an electronic element, a focusing lens and a narrow B1 bandpass filter to eliminate the different light used to light up our scenario… calculations are performed for each pixel or point of the TOF camera sensor. It is obvious that the wavelength λ1 of the light of L1 is the same as the central wavelength of the B1 filter band of the chamber of the STOF subsystem (12)… third subsystem that we will call SDOE (10) comprised of a camera equipped with electronic elements, and an optical system that includes a narrow band pass filter (20) at a wavelength that coincides with the emission of a laser… The L2 laser of wavelength λ2, turns on and off at controlled periods to facilitate the detection of the points illuminated by the pattern generated. The DOE camera is comprised of its electronic elements, an image sensor, lenses and narrow B2 bandpass filter tuned to λ2. Once the points have been detected, the electronic elements determine the relative distances of the points illuminated and received in the pixels of the camera as the second part of the mission of this subsystem. This is done by means of triangulation, measuring the displacement generated according to the distance and with knowledge of the separation between the laser and the camera used. As already mentioned, the wavelength λ2 of the L2 light is the same as the central wavelength of the B2 filter band of the SDOE subsystem camera. The result is therefore a new point cloud corresponding to those detected in the sensor when reflected from our structured illuminator; Par. [0076-80]: operating procedure of the automatic contact system covered by the invention comprises the following stages: [0076] Determination of the position of each light emitting point of the BD and RD elements, which are located in solidarity, respectively at the end of the pole nozzle and at the contour of the receptacle, using 3D cameras. The light emission by these emitters is uniform in the emission directions and allows 3D cameras to “see them” and thus determine the position of each of them with respect to CR. In order to facilitate this task, the emitters are alternatively flashed with certain patterns. The first are synchronised with 3D cameras, and the second are not. All are filtered temporarily with respect to the other LEDs. This avoids unnecessary overlaps between the emitters and facilitates detection by cross-correlation techniques to eliminate confusion with other points of light. Reflections can also be eliminated thanks to the synchronised alternative lighting of the emitters. This synchronism minimises the energy needed for detection. The use of a filter tuned to the light emitting wavelength of the emitters also allows an increase in the signal-to-noise ratio, again facilitating this detection… L1 laser provides a set of light pulses with wavelength λ1. The circuit that triggers the switching on of this laser is the same that governs the shooting and image acquisition of the TOF-type camera included in STOF. Considering the speed of light and the time it takes to receive the pulse generated in each pixel of the sensor of the TOF-type camera, the distance from the point of the scenario that reflects the light received can be obtained. To facilitate this task, a narrow band pass filter B1 centred in λ1 is placed in front of the TOF-type camera… L2 laser generates a pattern… of structured light… The elements of this pattern can be identified if we are able to “see” the light emitted by the laser when reflected by our environment. To facilitate this, we use a new narrow B2 band pass filter in front of the SDOE camera, tuned to L2 and that will eliminate light from other wavelengths. In addition, turning it on and off with a certain rate will also help us to distinguish the light of the laser with respect to other light from different sources… With cross-correlation techniques we will obtain the pixels that are reflected in the objects of our scenario, and from their relative intensities we will determine which pixels correspond to certain points of the pattern; a narrow-band light source (e.g. imaging system includes a narrow bandpass filter (i.e. a filter) to eliminate light other than that used to light up, and to be reflected in, various objects of a scenario (i.e. field of view, scene, environment, etc.), for example, including a camera (i.e. the imaging sensor) that has, before reaching its sensor (i.e. a filter positioned in a field of view of the imaging sensor), a very narrow band pass filter that only lets wavelengths (λ2) pass that are very close to those generated (i.e. a narrow-band light source) when reflected by various objects in an environment, including points of light reflected on the environment/scenario generated as a result of structured lighting that is generated, for example, and the result is therefore a point cloud corresponding to points detected in the sensor when reflected from a structured illuminator, as indicated above), for example), or a combination thereof.
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
The combination of RIX and Lozano, as a whole, teaches the device (RIX, Par. [0001]), but fails to teach the following as further recited in claim 3.
However, Zhao teaches the projection of the light by the light source is time-correlated with operation of a mechanical shutter or an electrical shutter of the camera (Par. [0005]: obtaining an image of a portion of the environment of the autonomous vehicle, wherein the image comprises a plurality of rows of pixels, and wherein the image was generated by a camera operating in a rolling shutter mode such that each row of pixels represents light sensed by the camera during a respective exposure time period; Par. [0093-102]: device 200 may be configured to operate cameras 208a. 208b, 208c, and/or 208d in a rolling shutter configuration to obtain a sequence of image pixels. In the scenario of FIG. 2C for example, a first row of sensing elements that includes sensing elements 228a and 228g may be configured to measure an amount of external light incident thereon during a first exposure time period… After a time delay from a start time of the first exposure time period (and optionally before the first exposure time period ends), device 200 may start exposing a second row of sensing elements that includes sensing elements 228b and 228b for a second exposure time period… Continuing with the scenario, after the second exposure time period lapses, device 200 may then similarly measure and transmit the measurements by the second row of sensing elements to the controller. This process can then be repeated until all the rows of sensing elements (i.e., a complete image frame) are scanned. For example, after a start time of the second exposure time period (and optionally before the second exposure time period lapses), device 200 may begin exposing a third row of sensing elements (adjacent to the second row) to external light 270, and so on… as LIDAR 206 rotates about axis 242, the time period in which FOVs of LIDAR 206 overlap the FOV of camera 208a may be less than the exposure time period (and readout time period) suitable for capturing an image using camera 208a… In one example scenario, where camera 208a is operated in a rolling shutter configuration (e.g., rows of sensing elements in camera 208a exposed according to different exposure start times), image sensor 226 of camera 208a may measure and output an image frame (i.e., pixel data from all the sensing elements of image sensor 226) over a period; the projection of the light by the light source is time-correlated with operation of a mechanical shutter or an electrical shutter of the camera (e.g. obtaining an image of a portion of an environment of the autonomous vehicle, wherein the image comprises a plurality of rows of pixels, and wherein the image is generated by a camera operating in a rolling shutter mode (i.e. operation of a mechanical shutter or an electrical shutter of the camera) such that each row of pixels represents light sensed by the camera during a respective exposure time period to measure an amount of external light incident thereon (i.e. projection of the light by the light source) during a first exposure time period (i.e. the projection of the light by the light source is time-correlated with operation of a mechanical shutter or an electrical shutter of the camera), as indicated above), for example).
RIX, Lozano, and Zhao are considered to be analogous art because they pertain to image processing applications. Therefore, the combined teachings of RIX, Lozano, and Zhao, as a whole, would have rendered obvious the invention recited in claim 3 with a reasonable expectation of success in order to modify the variety of refueling devices, systems and methods for use in in-flight refueling that comprises an imaging system for providing image data of an object that includes markers that comprise a retro-reflective surface that reflects incident beams and provides a strong intensity reflection when illuminated and including repeated determination of spatial data of the object (as disclosed by RIX) with the projection of the light by the light source is time-correlated with operation of a mechanical shutter or an electrical shutter of the camera (as taught by Zhao, Abstract, Par. [0005, 93-102]) to measure an amount of external light incident on image pixel, to allow for accurate localization of the world point in a rolling-shutter image via an iterative process that converges in very few iterations, and to improve the extent of matching image pixel capture times (and viewing directions) with corresponding LIDAR pulse emission times (and/or detections of corresponding reflections) (Zhao, Abstract, Par. [0005, 93-102]).
Regarding claim 10, claim 1 is incorporated and the combination of RIX and Lozano, as a whole, teaches the device (RIX, Par. [0001]), , but fails to teach the following as further recited in claim 10.
However, Zhao teaches wherein the vision processor is further configured to:
generate a score associated with the location of the object (Par. [0004]: an image of the scene can be generated, where each pixel in the image indicates one or more values (e.g., colors, etc.) based on outputs from the array of CMOS sensors; Par. [0150]: system can assign a depth value for image pixels in the image. Additionally, for instance, the system can generate (e.g., via display 140) a 3D object data model (e.g., a 3D rendering) of one or more objects in the environment (e.g., colored 3D model that indicates 3D features in the environment, etc.)… an image processing system can identify and distinguish between multiple objects in the image by comparing depth information (indicated by the associated location of the point in the environment) of the respective object; generate a score associated with the location of the object (e.g. system assigns a depth value (i.e. score, number, etc.) for image pixels in an image of one or more objects in the environment based on a 3D object data model (i.e. generate a score associated with the location of the object), as indicated above), for example); or
adjust an exposure time of the imaging sensor, a gain of the imaging sensor, an illumination intensity of the light source, a synchronization between the imaging sensor and the light source, or a combination thereof (Par. [0005]: obtaining an image of a portion of the environment of the autonomous vehicle, wherein the image comprises a plurality of rows of pixels, and wherein the image was generated by a camera operating in a rolling shutter mode such that each row of pixels represents light sensed by the camera during a respective exposure time period; Par. [0051]: system 100 may be configured to capture a sequence of images of the respective FOVs simultaneously (and/or synchronously or according to some other timing) with a scan of the environment by sensor 106 as sensor 106 rotates about the axis; Par. [0093-102]: device 200 may be configured to operate cameras 208a. 208b, 208c, and/or 208d in a rolling shutter configuration to obtain a sequence of image pixels. In the scenario of FIG. 2C for example, a first row of sensing elements that includes sensing elements 228a and 228g may be configured to measure an amount of external light incident thereon during a first exposure time period… After a time delay from a start time of the first exposure time period (and optionally before the first exposure time period ends), device 200 may start exposing a second row of sensing elements that includes sensing elements 228b and 228b for a second exposure time period… Continuing with the scenario, after the second exposure time period lapses, device 200 may then similarly measure and transmit the measurements by the second row of sensing elements to the controller. This process can then be repeated until all the rows of sensing elements (i.e., a complete image frame) are scanned. For example, after a start time of the second exposure time period (and optionally before the second exposure time period lapses), device 200 may begin exposing a third row of sensing elements (adjacent to the second row) to external light 270, and so on… as LIDAR 206 rotates about axis 242, the time period in which FOVs of LIDAR 206 overlap the FOV of camera 208a may be less than the exposure time period (and readout time period) suitable for capturing an image using camera 208a… In one example scenario, where camera 208a is operated in a rolling shutter configuration (e.g., rows of sensing elements in camera 208a exposed according to different exposure start times), image sensor 226 of camera 208a may measure and output an image frame (i.e., pixel data from all the sensing elements of image sensor 226) over a period… To account for the difference between the scanning durations of the camera and the LIDAR, in some implementations, device 200 may be configured to synchronize LIDAR light pulses emitted by LIDAR 206 with image pixels captured by some but not all the image sensing elements in a camera; or adjust an exposure time of the imaging sensor, a gain of the imaging sensor, an illumination intensity of the light source, a synchronization between the imaging sensor and the light source, or a combination thereof (e.g. obtaining an image of a portion of an environment of the autonomous vehicle, wherein the image comprises a plurality of rows of pixels, and wherein the image is generated by a camera operating in a rolling shutter mode (i.e. operation of a mechanical shutter or an electrical shutter of the camera) such that each row of pixels represents light sensed by the camera during a respective exposure time period (i.e. adjust an exposure time of the imaging sensor) to measure an amount of external light incident thereon (i.e. projection of the light by the light source) during a first exposure time period, for example, and system 100 is configured to capture a sequence of images of respective FOVs simultaneously (and/or synchronously or according to some other timing) with a scan of the environment by sensor 106 as sensor 106 rotates about the axis (i.e. a synchronization between the imaging sensor and the light source), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 3.
Claims 4-5 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over RIX, in view of Lozano, as applied to claim 1 above, in further view of Black et al. (US PG Publication No. US 2019/0272660 A1), hereafter referred to as Black, Applicant cited prior art furnished via IDS, for the same reasons as set forth in the last Office action (grounds are restated below for convenience).
Regarding claim 4, claim 1 is incorporated and the combination of RIX and Lozano, as a whole, teaches the device (RIX, Par. [0001]), wherein:
the light source includes multiple infrared light emitting diodes (LEDs) (Lozano, Par. [0028]: set or subsystem of light emitters are arranged on its surface. These emitters consist of LEDs in a preferred embodiment, along with on/off electronics; Par. [0049]: a set of light emitters (16), that in general can consist of LEDs or laser diodes with their respective diffusers… obtain a certain level of synchronisation between both subsystems (cameras and LED emitters));
the filter is configured block light other than the light projected by the light source and to pass at least a portion of the light projected by the light source (Lozano, Par. [0032-33]: second subsystem that we will call STOF comprised by a TOF—(Time Of Flight) type camera that has the special feature of measuring the time that it takes a pulse of light generated to be reflected in the various objects… STOF subsystem also consists of electronic elements, a laser with its diffuser, a lens and a F1 narrow bandpass filter to eliminate light other than that used to light up our scenario… third subsystem, which we will call SDOE, comprised of a camera that has, before reaching its sensor, a very narrow band pass filter that only lets wavelengths (λ2) pass that are very close to those generated by an L2 laser that also belongs to the subsystem; Par. [0061-62]: second subsystem comprised by a TOF-type camera that has the special feature of measuring the time that it takes a pulse of light generated to be reflected in the various objects… subsystem, which we will call STOF, has an electronic element, a focusing lens and a narrow B1 bandpass filter to eliminate the different light used to light up our scenario… calculations are performed for each pixel or point of the TOF camera sensor. It is obvious that the wavelength λ1 of the light of L1 is the same as the central wavelength of the B1 filter band of the chamber of the STOF subsystem (12)… third subsystem that we will call SDOE (10) comprised of a camera equipped with electronic elements, and an optical system that includes a narrow band pass filter (20) at a wavelength that coincides with the emission of a laser… The L2 laser of wavelength λ2, turns on and off at controlled periods to facilitate the detection of the points illuminated by the pattern generated. The DOE camera is comprised of its electronic elements, an image sensor, lenses and narrow B2 bandpass filter tuned to λ2. Once the points have been detected, the electronic elements determine the relative distances of the points illuminated and received in the pixels of the camera as the second part of the mission of this subsystem. This is done by means of triangulation, measuring the displacement generated according to the distance and with knowledge of the separation between the laser and the camera used. As already mentioned, the wavelength λ2 of the L2 light is the same as the central wavelength of the B2 filter band of the SDOE subsystem camera. The result is therefore a new point cloud corresponding to those detected in the sensor when reflected from our structured illuminator; Par. [0076-80]: Determination of the position of each light emitting point of the BD and RD elements, which are located in solidarity, respectively at the end of the pole nozzle and at the contour of the receptacle, using 3D cameras. The light emission by these emitters is uniform in the emission directions and allows 3D cameras to “see them” and thus determine the position of each of them with respect to CR. In order to facilitate this task, the emitters are alternatively flashed with certain patterns. The first are synchronised with 3D cameras, and the second are not. All are filtered temporarily with respect to the other LEDs. This avoids unnecessary overlaps between the emitters and facilitates detection by cross-correlation techniques to eliminate confusion with other points of light. Reflections can also be eliminated thanks to the synchronised alternative lighting of the emitters. This synchronism minimises the energy needed for detection. The use of a filter tuned to the light emitting wavelength of the emitters also allows an increase in the signal-to-noise ratio, again facilitating this detection… L1 laser provides a set of light pulses with wavelength λ1. The circuit that triggers the switching on of this laser is the same that governs the shooting and image acquisition of the TOF-type camera included in STOF. Considering the speed of light and the time it takes to receive the pulse generated in each pixel of the sensor of the TOF-type camera, the distance from the point of the scenario that reflects the light received can be obtained. To facilitate this task, a narrow band pass filter B1 centred in λ1 is placed in front of the TOF-type camera… L2 laser generates a pattern… of structured light… The elements of this pattern can be identified if we are able to “see” the light emitted by the laser when reflected by our environment. To facilitate this, we use a new narrow B2 band pass filter in front of the SDOE camera, tuned to L2 and that will eliminate light from other wavelengths. In addition, turning it on and off with a certain rate will also help us to distinguish the light of the laser with respect to other light from different sources… With cross-correlation techniques we will obtain the pixels that are reflected in the objects of our scenario, and from their relative intensities we will determine which pixels correspond to certain points of the pattern; the filter is configured block light other than the light projected by the light source and to pass at least a portion of the light projected by the light source (e.g. imaging system includes a narrow bandpass filter (i.e. a filter) to eliminate light (i.e. minimize light interference) other than that used to light up, and to be reflected in, various objects of a scenario (i.e. field of view, scene, environment, etc.), for example, including a camera (i.e. the imaging sensor) that has, before reaching its sensor, a very narrow band pass filter that only lets wavelengths (λ2) pass that are very close to those generated when reflected by various objects in an environment (i.e. the filter is configured block light other than the light projected by the light source and to pass at least a portion of the light projected by the light source), as indicated above), for example);
the multiple reflective markers are positioned on a canopy of the object (RIX, Par. [0520]: the data acquisition system 299B further includes fuel receptacle marker 342, comprised on the receiver aircraft 20 in a pre-determined location with respect to the fuel receptacle 22 thereof. The fuel receptacle marker 342 is at a fixed and known geometrical relationship with respect to the fuel receptacle 22, and is electromagnetically visible to the imaging system 350; Par. [0556]: the data acquisition system 360 further includes fuel receptacle marker 342, comprised on the receiver aircraft 20 in a pre-determined location with respect to the fuel receptacle 22 thereof. The fuel receptacle marker 342 is at a fixed and known geometrical relationship with respect to the fuel receptacle 22, and is electromagnetically visible to the imaging system 350; the multiple reflective markers are positioned on a canopy of the object (e.g. refueling devices, systems and methods for use in in-flight refueling system that comprises an imaging system for providing image data, including acquired 2D images and/or stereoscopic images and/or 3D images (i.e. generate one or more images) of an object, or a part (i.e. portion, section, region, etc.) thereof, for example, including fuel receptacle (i.e. a canopy of the object) markers at a fixed and known geometrical relationship with respect to a fuel receptacle (i.e. the multiple reflective markers are positioned on a canopy of the object), as indicated above), for example); and
at least one reflective marker of the multiple reflective markers includes a retroreflector (RIX, Par. [0521]: the fuel receptacle marker 342 comprises a retro-reflective surface that reflects incident beams along the same path, and thus provides a strong intensity reflection of the respective reflected beam when illuminated by a beam; Par. [0557]: the fuel receptacle marker 342 comprises a retro-reflective surface that reflects incident beams along the same path, and thus provides a strong intensity reflection of the respective beam B2 when illuminated by beam B1).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
RIX and Lozano, as a whole, teach the device, as indicated above, but fails to teach that the claimed “object” recited in the claim “includes a drogue” as now recited in claim 4.
However, Black teaches the object includes a drogue (Par. [0002-3]: systems and methods for vehicle refueling, and more particularly, to refueling systems and methods for providing indications to a receiver vehicle during refueling operations… During refueling operations, such as aerial refueling operations for example, a tanker aircraft supplies fuel to a receiver aircraft while in-flight. One approach to aerial refueling is known as probe-and-drogue refueling. In a probe-and-drogue system, the tanker aircraft extends a flexible hose having a refueling coupler (commonly referred to as a drogue) disposed at a distal end of the hose. The receiver aircraft has a probe, which the receiver aircraft maneuvers into engagement with the drogue; Par. [0043]: hose 122 includes (i) a first end 122A in the housing 124 for receiving the fuel from a fuel tank in the tanker aircraft 110 and (ii) a second end 122B. The second end 122B includes a refueling coupler 130 (which may be referred to as a drogue) that has a conical shape to assist in stabilizing the hose 122 during flight and coupling the hose 122 with a probe 128 of the receiver aircraft 112… receiver aircraft 112 also includes a vision system 134 for collecting or capturing image information indicative of the hose 122 and the refueling coupler 130; the object includes a drogue (e.g. systems and methods for vehicle refueling, and more particularly, to refueling systems and methods for providing indications to a receiver vehicle during refueling operations, include a vision system for collecting or capturing image information indicative of a hose and a refueling coupler, commonly referred to as a drogue (i.e. the object includes a drogue), disposed at a distal end of the hose, as indicated above), for example).
RIX, Lozano, and Black are considered to be analogous art because they pertain to image processing applications. Therefore, the combined teachings of RIX, Lozano, and Black, as a whole, would have rendered obvious the invention recited in claim 5 with a reasonable expectation of success in order to modify the variety of refueling devices, systems and methods for use in in-flight refueling that comprises an imaging system for providing image data of an object that includes markers that comprise a retro-reflective surface that reflects incident beams and provides a strong intensity reflection when illuminated and including repeated determination of spatial data of the object (as disclosed by RIX) with the object includes a drogue (as taught by Black, Abstract, Par. [0002-3, 43]) that has a conical shape to assist in stabilizing a hose during flight and coupling of the hose with a probe of a receiver aircraft during vehicle refueling operations (Black, Abstract, Par. [0002-3, 43]).
Regarding claim 5, claim 1 is incorporated and the combination of RIX and Lozano, as a whole, teaches the device (RIX, Par. [0001]), as indicated above, but fails to teach the following as further recited in claim 5.
However, Black teaches wherein the vision processor is further configured to:
perform a thresholding operation based on the first image to generate a thresholded image (Par. [0065]: FIG. 5 is a block diagram illustrating an example method of image processing for generation of the visual representation 172…Initially, the camera 160 and/or LIDAR 162 capture images of the refueling coupler 130, the hose 122, and portions of both the tanker aircraft 110 and the receiver aircraft 112, and the captured images are received by the vision system processor(s) 154 for processing, as shown at block 184. Initial processing can include processing the image for conversion to a binary image for feature extraction, or conversion of a color image to grayscale using a thresholding technique making the image a binary image).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 4.
Regarding claim 12, claim 11 is incorporated and the combination of RIX and Lozano, as a whole, teaches the method (RIX, Par. [0001]), as indicated above, but fails to teach the following as further recited in claim 12.
However, Black teaches further comprising performing a thresholding operation based on the image to generate a thresholded image (Par. [0065]: FIG. 5 is a block diagram illustrating an example method of image processing for generation of the visual representation 172…Initially, the camera 160 and/or LIDAR 162 capture images of the refueling coupler 130, the hose 122, and portions of both the tanker aircraft 110 and the receiver aircraft 112, and the captured images are received by the vision system processor(s) 154 for processing, as shown at block 184. Initial processing can include processing the image for conversion to a binary image for feature extraction, or conversion of a color image to grayscale using a thresholding technique making the image a binary image).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 4.
Claims 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over RIX, in view of Lozano, as applied to claim 1 above, in further view of Rutkiewicz et al. (US PG Publication No. US 2018/0261111 A1), hereafter referred to as Rutkiewicz, for the same reasons as set forth in the last Office action (grounds are restated below for convenience).
Regarding claim 7, claim 6 is incorporated and the combination of RIX and Lozano, as a whole, teaches the device (RIX, Par. [0001]), but fails to teach the following as further recited in claim 7.
However, Rutkiewicz teaches wherein the vision processor is further configured to:
perform a correlation operation based on the first image and the second image to generate a difference image (Par. [0033]: generate a difference image based on a difference between the first and second images of each image pair. This technique can reduce the pixel response of light reflected from the objects that is projected thereon from these other sources; Par. [0040]: Detector 54 generates a difference image by subtracting the first image from the second image. Detector 54 then performs structured light analysis on the difference image so generated); and
perform a thresholding operation on the difference image to generate a thresholded image (Par. [0040]: Detector 54 generates a difference image by subtracting the first image from the second image. Detector 54 then performs structured light analysis on the difference image so generated… structured light analysis can include determining range values for each pixel of the difference image that has an intensity value greater than a predetermined threshold).
RIX, Lozano, and Rutkiewicz are considered to be analogous art because they pertain to image processing applications. Therefore, the combined teachings of RIX, Lozano, and Rutkiewicz, as a whole, would have rendered obvious the invention recited in claim 7 with a reasonable expectation of success in order to modify the variety of refueling devices, systems and methods for use in in-flight refueling that comprises an imaging system for providing image data of an object that includes markers that comprise a retro-reflective surface that reflects incident beams and provides a strong intensity reflection when illuminated and including repeated determination of spatial data of the object (as disclosed by RIX) with wherein the vision processor is further configured to: perform a correlation operation based on the first image and the second image to generate a difference image and perform a thresholding operation on the difference image to generate a thresholded image (as taught by Rutkiewicz, Abstract, Par. [0033, 40]) to determine range values for each pixel of a difference image that has an intensity value greater than a predetermined threshold (Rutkiewicz, Abstract, Par. [0033, 40]).
Regarding claim 14, claim 13 is incorporated and the combination of RIX and Lozano, as a whole, teaches the method (RIX, Par. [0001]) but fails to teach the following as further recited in claim 14.
However, Rutkiewicz teaches further comprising:
performing a correlation operation based on the first image and the second image to generate a difference image (Par. [0033]: generate a difference image based on a difference between the first and second images of each image pair. This technique can reduce the pixel response of light reflected from the objects that is projected thereon from these other sources; Par. [0040]: Detector 54 generates a difference image by subtracting the first image from the second image. Detector 54 then performs structured light analysis on the difference image so generated); and
perform a thresholding operation on the difference image to generate a thresholded image (Par. [0040]: Detector 54 generates a difference image by subtracting the first image from the second image. Detector 54 then performs structured light analysis on the difference image so generated… structured light analysis can include determining range values for each pixel of the difference image that has an intensity value greater than a predetermined threshold).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 7.
Claims 8-9 and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over RIX, in view of Lozano, as applied to claim 1 above, in further view of Rutkiewicz, as applied to claim 7 above, and in further view of Black, for the same reasons as set forth in the last Office action (grounds are restated below for convenience).
Regarding claim 8, claim 7 is incorporated and the combination of RIX, Lozano, and Rutkiewicz, as a whole, teaches the device (RIX, Par. [0001]), but fails to teach the following as further recited in claim 8.
However, Black teaches wherein the vision processor is further configured to:
perform a blob segregation operation on the thresholded image to identify multiple blobs (i.e. regions, etc.) (Par. [0065]: processing can include processing the image for conversion to a binary image for feature extraction, or conversion of a color image to grayscale using a thresholding technique making the image a binary image. Feature extraction then can include an edge detection technique to identify objects in the binary image (e.g., as the pixels in the image change from white to black). The initial processing can also include segmentation of foreground target features from the background, which can be achieved through edge/contour detection of the target feature to identify changes in intensity at a boundary of the feature).
RIX, Lozano, Rutkiewicz, and Black are considered to be analogous art because they pertain to image processing applications. Therefore, the combined teachings of RIX, Lozano, Rutkiewicz, and Black, as a whole, would have rendered obvious the invention recited in claim 8 with a reasonable expectation of success in order to modify the variety of refueling devices, systems and methods for use in in-flight refueling that comprises an imaging system for providing image data of an object that includes markers that comprise a retro-reflective surface that reflects incident beams and provides a strong intensity reflection when illuminated and including repeated determination of spatial data of the object (as disclosed by RIX) with wherein the vision processor is further configured to: perform a blob segregation operation on the thresholded image to identify multiple blobs (as taught by Black, Abstract, Par. [0065]) to convert captured images into binary images for feature extraction (Black, Abstract, Par. [0065]).
Regarding claim 15, claim 14 is incorporated and the combination of RIX, Lozano, and Rutkiewicz, as a whole, teaches the method (RIX, Par. [0001]), but fails to teach the following as further recited in claim 15.
However, Black teaches further comprising performing a blob segregation operation on the thresholded image to identify multiple blobs (i.e. regions, etc.) (Par. [0065]: processing can include processing the image for conversion to a binary image for feature extraction, or conversion of a color image to grayscale using a thresholding technique making the image a binary image. Feature extraction then can include an edge detection technique to identify objects in the binary image (e.g., as the pixels in the image change from white to black). The initial processing can also include segmentation of foreground target features from the background, which can be achieved through edge/contour detection of the target feature to identify changes in intensity at a boundary of the feature).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 8.
Regarding claim 9, claim 8 is incorporated and the combination of RIX, Lozano, Rutkiewicz, and Black, as a whole, teaches the device (RIX, Par. [0001]), wherein:
the vision processor is further configured to, based on the multiple blobs:
identify a center of a set of blobs of the multiple blobs (Par. [0061-65]: image 170 is processed by the display processor(s) 152 to generate various visual representations. On the right in FIG. 3 is an example visual representation 172 that includes multiple graphics. A first graphic includes four arrows 174, 176, 178, and 180 that represent a location of the refueling coupler 130 with respect to the probe 128. A center of the four arrows 174, 176, 178, and 180 is a location of the probe 128 (as denoted by the square graphic in FIG. 3)… processing can include processing the image for conversion to a binary image for feature extraction, or conversion of a color image to grayscale using a thresholding technique making the image a binary image. Feature extraction then can include an edge detection technique to identify objects in the binary image (e.g., as the pixels in the image change from white to black). The initial processing can also include segmentation of foreground target features from the background, which can be achieved through edge/contour detection of the target feature to identify changes in intensity at a boundary of the feature; based on the multiple blobs: identify a center of a set of blobs of the multiple blobs (e.g. processing the image for conversion to a binary image for feature (i.e. blob) extraction, or conversion of a color image to grayscale using a thresholding technique making the image a binary image, for example, including extracted features that represent a location of a refueling coupler with respect to a probe, for example, and a center that is a location of the probe (i.e. identify a center of a set of blobs of the multiple blobs), as indicated above), for example);
fit a model to the set of blobs (Par. [0074-76]: tanker aircraft 110 can transmit a modeled hose catenary 192 to the receiver aircraft 112 (e.g., the second transceiver 138 sends the modeled hose catenary 192 to the first transceiver 136) to be displayed on the HUD 156 as a dashed line. The hose 122 is shown extending from the housing 124 of the tanker aircraft 110, and the model is based on altitude and air speed (of the tanker aircraft 110) during the refueling operation. The model can be retrieved from memory and a number of models… a relative position of the receiver aircraft 112 between the instantaneous position 194 of the receiver aircraft 112 and where the receiver aircraft 112 would be according to the modeled hose catenary 192 can be determined… A position deviation 196 can be determined to show how variation of the actual vs. modeled behavior of the receiver aircraft 112 and a position correction advisory 198 can be determined and illustrated; fit a model to the set of blobs (e.g. processing the image for conversion to a binary image for feature (i.e. blob) extraction, or conversion of a color image to grayscale using a thresholding technique making the image a binary image, for example, including extracted features that represent a location of a refueling coupler with respect to a probe, for example, such as a modeled hose (i.e. fit a model to the set of blobs), as indicated above), for example); or
a combination thereof ; and
the location of the object is based on the center of the set of blobs (Par. [0061-65]: image 170 is processed by the display processor(s) 152 to generate various visual representations. On the right in FIG. 3 is an example visual representation 172 that includes multiple graphics. A first graphic includes four arrows 174, 176, 178, and 180 that represent a location of the refueling coupler 130 with respect to the probe 128. A center of the four arrows 174, 176, 178, and 180 is a location of the probe 128 (as denoted by the square graphic in FIG. 3)… processing can include processing the image for conversion to a binary image for feature extraction, or conversion of a color image to grayscale using a thresholding technique making the image a binary image. Feature extraction then can include an edge detection technique to identify objects in the binary image (e.g., as the pixels in the image change from white to black). The initial processing can also include segmentation of foreground target features from the background, which can be achieved through edge/contour detection of the target feature to identify changes in intensity at a boundary of the feature; based on the multiple blobs: identify a center of a set of blobs of the multiple blobs (e.g. processing the image for conversion to a binary image for feature (i.e. blob) extraction, or conversion of a color image to grayscale using a thresholding technique making the image a binary image, for example, including extracted features that represent a location of a refueling coupler with respect to a probe, for example, and a center that is a location of the probe (i.e. identify a center of a set of blobs of the multiple blobs), as indicated above), for example), the model (Par. [0074-76]: tanker aircraft 110 can transmit a modeled hose catenary 192 to the receiver aircraft 112 (e.g., the second transceiver 138 sends the modeled hose catenary 192 to the first transceiver 136) to be displayed on the HUD 156 as a dashed line. The hose 122 is shown extending from the housing 124 of the tanker aircraft 110, and the model is based on altitude and air speed (of the tanker aircraft 110) during the refueling operation. The model can be retrieved from memory and a number of models… a relative position of the receiver aircraft 112 between the instantaneous position 194 of the receiver aircraft 112 and where the receiver aircraft 112 would be according to the modeled hose catenary 192 can be determined… A position deviation 196 can be determined to show how variation of the actual vs. modeled behavior of the receiver aircraft 112 and a position correction advisory 198 can be determined and illustrated; fit a model to the set of blobs (e.g. processing the image for conversion to a binary image for feature (i.e. blob) extraction, or conversion of a color image to grayscale using a thresholding technique making the image a binary image, for example, including extracted features that represent a location of a refueling coupler with respect to a probe, for example, such as a modeled hose (i.e. fit a model to the set of blobs), as indicated above), for example), or a combination thereof.
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 8.
Regarding claim 16, claim 15 is incorporated and the combination of RIX, Lozano, Rutkiewicz, and Black, as a whole, teaches the device (RIX, Par. [0001]), but fails to teach the following as further recited in claim 16.
However, Black teaches further comprising:
identifying a center of a set of blobs of the multiple blobs (Par. [0061-65]: image 170 is processed by the display processor(s) 152 to generate various visual representations. On the right in FIG. 3 is an example visual representation 172 that includes multiple graphics. A first graphic includes four arrows 174, 176, 178, and 180 that represent a location of the refueling coupler 130 with respect to the probe 128. A center of the four arrows 174, 176, 178, and 180 is a location of the probe 128 (as denoted by the square graphic in FIG. 3)… processing can include processing the image for conversion to a binary image for feature extraction, or conversion of a color image to grayscale using a thresholding technique making the image a binary image. Feature extraction then can include an edge detection technique to identify objects in the binary image (e.g., as the pixels in the image change from white to black). The initial processing can also include segmentation of foreground target features from the background, which can be achieved through edge/contour detection of the target feature to identify changes in intensity at a boundary of the feature; based on the multiple blobs: identify a center of a set of blobs of the multiple blobs (e.g. processing the image for conversion to a binary image for feature (i.e. blob) extraction, or conversion of a color image to grayscale using a thresholding technique making the image a binary image, for example, including extracted features that represent a location of a refueling coupler with respect to a probe, for example, and a center that is a location of the probe (i.e. identify a center of a set of blobs of the multiple blobs), as indicated above), for example), fitting a model to the set of blobs (Par. [0074-76]: tanker aircraft 110 can transmit a modeled hose catenary 192 to the receiver aircraft 112 (e.g., the second transceiver 138 sends the modeled hose catenary 192 to the first transceiver 136) to be displayed on the HUD 156 as a dashed line. The hose 122 is shown extending from the housing 124 of the tanker aircraft 110, and the model is based on altitude and air speed (of the tanker aircraft 110) during the refueling operation. The model can be retrieved from memory and a number of models… a relative position of the receiver aircraft 112 between the instantaneous position 194 of the receiver aircraft 112 and where the receiver aircraft 112 would be according to the modeled hose catenary 192 can be determined… A position deviation 196 can be determined to show how variation of the actual vs. modeled behavior of the receiver aircraft 112 and a position correction advisory 198 can be determined and illustrated; fit a model to the set of blobs (e.g. processing the image for conversion to a binary image for feature (i.e. blob) extraction, or conversion of a color image to grayscale using a thresholding technique making the image a binary image, for example, including extracted features that represent a location of a refueling coupler with respect to a probe, for example, such as a modeled hose (i.e. fit a model to the set of blobs), as indicated above), for example), or a combination thereof; and
wherein the location of the object is based on the center of the set of blobs (Par. [0061-65]: image 170 is processed by the display processor(s) 152 to generate various visual representations. On the right in FIG. 3 is an example visual representation 172 that includes multiple graphics. A first graphic includes four arrows 174, 176, 178, and 180 that represent a location of the refueling coupler 130 with respect to the probe 128. A center of the four arrows 174, 176, 178, and 180 is a location of the probe 128 (as denoted by the square graphic in FIG. 3)… processing can include processing the image for conversion to a binary image for feature extraction, or conversion of a color image to grayscale using a thresholding technique making the image a binary image. Feature extraction then can include an edge detection technique to identify objects in the binary image (e.g., as the pixels in the image change from white to black). The initial processing can also include segmentation of foreground target features from the background, which can be achieved through edge/contour detection of the target feature to identify changes in intensity at a boundary of the feature; based on the multiple blobs: identify a center of a set of blobs of the multiple blobs (e.g. processing the image for conversion to a binary image for feature (i.e. blob) extraction, or conversion of a color image to grayscale using a thresholding technique making the image a binary image, for example, including extracted features that represent a location of a refueling coupler with respect to a probe, for example, and a center that is a location of the probe (i.e. identify a center of a set of blobs of the multiple blobs), as indicated above), for example), the model (Par. [0074-76]: tanker aircraft 110 can transmit a modeled hose catenary 192 to the receiver aircraft 112 (e.g., the second transceiver 138 sends the modeled hose catenary 192 to the first transceiver 136) to be displayed on the HUD 156 as a dashed line. The hose 122 is shown extending from the housing 124 of the tanker aircraft 110, and the model is based on altitude and air speed (of the tanker aircraft 110) during the refueling operation. The model can be retrieved from memory and a number of models… a relative position of the receiver aircraft 112 between the instantaneous position 194 of the receiver aircraft 112 and where the receiver aircraft 112 would be according to the modeled hose catenary 192 can be determined… A position deviation 196 can be determined to show how variation of the actual vs. modeled behavior of the receiver aircraft 112 and a position correction advisory 198 can be determined and illustrated; fit a model to the set of blobs (e.g. processing the image for conversion to a binary image for feature (i.e. blob) extraction, or conversion of a color image to grayscale using a thresholding technique making the image a binary image, for example, including extracted features that represent a location of a refueling coupler with respect to a probe, for example, such as a modeled hose (i.e. fit a model to the set of blobs), as indicated above), for example), or a combination thereof.
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 8.
Conclusion
Applicant’s amendment of claim 4 necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GUILLERMO RIVERA-MARTINEZ whose telephone number is 571-272-4979. The examiner can normally be reached on Monday-Friday (8am - 5pm Eastern Time). If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached on 571-270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GUILLERMO M RIVERA-MARTINEZ/ Primary Examiner, Art Unit 2677
Prosecution Insights