DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on 01/30/2025 and 05/13/2025 is/are being considered by the Examiner.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-4, 8-14, 18, and 19 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Mishra et al, U.S. Patent No. 11,170,524.
Regarding claim 11, Mishara teaches a system comprising:
one or more processors, wherein the one or more processors are configured to perform operations (see Mishara claim 1) comprising:
selecting a first image frame and a second image frame of a sequence of image frames of a scene (see Figure 3, step 320 and 340) captured by an image capture device (see Figure 3, step 310 and column 21, “In still other embodiments, the second image may have been captured by the first imaging device, prior to the capture of the first image. The portion of the second image may be selected based on a difference in time between the capture of the first image and the capture of the second image”); and
forming an output image frame based on the first image frame and the second image frame (see Figure 3, step 360), wherein forming the output image frame based on the first image frame and the second image frame comprises:
selecting a plurality of portions of the first image frame, wherein each selected portion of the first image frame is associated with respective image data from the first image frame (see Figures 3, step 330 and column 20, “At box 330, an affected area within the first image is identified. For example, the first image may include pixels that are unreadable or are inconsistent with colors or textures of surrounding pixels within the first image, or color or textures of corresponding pixels within other images previously captured by the first imaging device or one or more other imaging devices”);
determining a bounding region within the second image frame (see Figure 3, step 340 and column 20, “At box 340, a portion of a second image is selected based on the affected area and aerial vehicle dynamics according to a geometric transform function. For example, a portion of the second image depicting background features, foreground features or other aspects that would also have been present within the affected area of the first image had the first imaging device not experienced the anomaly, or are depicted within or near pixels of the affected area, may be identified in any manner”. This is also exemplified in Figure 4C with region 44-2C);
retrieving, based on the bounding region, image data from the second image frame (see Figure 3, step 350 and column 21, “At box 350, the portion of the second image selected at box 340 is extracted therefrom, e.g., by copying the portion of the second image, by determining values of colors or intensities of pixels or groups of pixels in the portion of the second image, such as according to the RGB or a hexadecimal model, or in any other manner”);
for each selected portion of the first image frame, identifying corresponding image data in the retrieved image data from the second image frame; and combining the image data of each selected portion of the first image frame with the corresponding image data in the retrieved image data from the second image frame (see Figure 3, step 360 and column 21, “At box 360, a third image is generated by patching the portion of the second image extracted at box 350 onto the affected area of the first image. The third image may be generated in any manner, such as by inpainting, patching or superimposing the portion of the second image onto the affected area of the third image”).
Method claim 1 recites similar limitations as claim 11, and is rejected under similar rationale.
Regarding claim 12, Mishara teaches all the limitations of claim 11, and further teaches wherein the selected portions of the first image frame have a square or rectangular shape (see Mishara column 20, “At box 330, an affected area within the first image is identified. For example, the first image may include pixels that are unreadable or are inconsistent with colors or textures of surrounding pixels within the first image, or color or textures of corresponding pixels within other images previously captured by the first imaging device or one or more other imaging devices” wherein pixels are well-known to either be square or rectangular).
Method claim 2 recites similar limitations as claim 12, and is rejected under similar rationale.
Regarding claim 13, Mishara teaches all the limitations of claim 11, and further teaches wherein the selected portions of the first image frame are non-overlapping (see Mishara column 20, “At box 330, an affected area within the first image is identified. For example, the first image may include pixels that are unreadable or are inconsistent with colors or textures of surrounding pixels within the first image, or color or textures of corresponding pixels within other images previously captured by the first imaging device or one or more other imaging devices” wherein pixels are well-known to not overlap).
Method claim 3 recites similar limitations as claim 13, and is rejected under similar rationale.
Regarding claim 14, Mishara teaches all the limitations of claim 11, and further teaches wherein, for each selected portion of the first image frame, identifying corresponding image data in the retrieved image data from the second image frame comprises:
for each selected portion of the first image frame, searching the retrieved image data to find image data that best matches the image data of that selected portion of the first image frame (see Mishara column 20, “At box 340, a portion of a second image is selected based on the affected area and aerial vehicle dynamics according to a geometric transform function. For example, a portion of the second image depicting background features, foreground features or other aspects that would also have been present within the affected area of the first image had the first imaging device not experienced the anomaly, or are depicted within or near pixels of the affected area, may be identified in any manner. In some embodiments, a geometric transform function may be generated for each pixel of an image plane of an imaging device, and used to identify pixels of other images, e.g., counterpart images, depicting content similar or identical to that which is depicted by the imaging device at such pixels”).
Method claim 4 recites similar limitations as claim 14, and is rejected under similar rationale.
Regarding claim 18, Mishara teaches all the limitations of claim 11, and further teaches wherein determining the bounding region within the second image frame comprises:
determining the bounding region based on a predicted change between the first image frame and the second image frame, wherein the predicted change is related to movement of one or more objects in the scene and/or movement of the image capture device (see Mishara column 20, “At box 340, a portion of a second image is selected based on the affected area and aerial vehicle dynamics according to a geometric transform function…The geometric transform function may consider a velocity or an altitude of the aerial vehicle, as well as one or more attributes of a pose of the first imaging device (e.g., an axis of orientation, a focal length, an aperture width, a depth of field or a field of view), or any other factors. Moreover, the portion of the second image may be selected using one or more masks or other image processing settings or features”).
Method claim 8 recites similar limitations as claim 18, and is rejected under similar rationale.
Regarding claim 19, Mishara teaches all the limitations of claim 11, and further teaches wherein the output image frame has an improved signal-to-noise (SNR) ratio relative to the first image frame (see Mishara column 18, “Such anomalies may adversely impact the quality of the images or other imaging data that may be captured thereby, and may occur at random or as a result of intentional or unintentional events occurring within a vicinity of the first imaging device. For example, an improperly installed or aligned device may capture images that are blurred or out-of-focus, e.g., a failure to properly monitor or adjust the focus of the camera during or following installation. Moreover, even a properly installed or aligned imaging device may drift out-of-focus due to any mechanical vibrations that may be encountered during operation” wherein correcting such anomalies implies improving SNR).
Method claim 9 recites similar limitations as claim 19, and is rejected under similar rationale.
Regarding claim 10, Mishra teaches all the limitations of claim 1, and further teaches wherein retrieving, by the processor and based on the bounding region, image data from the second image frame comprises retrieving the image data (see Mishra Figure 4C, image 42-2C taken at a prior time and column 6, “A digital image is a collection of pixels, typically arranged in an array, which defines an optically formed reproduction of one or more objects, backgrounds or other features of a scene and may be stored in a data file”) from a memory that is external to the processor (see Figure 2, imaging device 250 in aerial vehicle 210 with distinct processor 252 and memory 254 and column 6, “Such sensors may generate data files including such information, e.g., digital images, and store such data files in one or more onboard or accessible data stores (e.g., a hard drive or other like component), as well as one or more removable data stores (e.g., flash memory devices)”. It is implied that to use data from a prior image that the prior image would be stored in a memory).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mishra et al, U.S. Patent No. 11,170,524.
Regarding claim 20, Mishra teaches all the limitations of claim 11, but does not expressively teach wherein the output image frame has a higher dynamic range than the first image frame. However, one of ordinary skill in the art before the effective filing date of the invention would have found it obvious as a matter of design choice to correct the anomalies to have an improved dynamic range thereby improving the quality and reliability of the output image.
Claim(s) 5 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mishra et al, U.S. Patent No. 11,170,524 in view of Ermilios et al, U.S. Publication No. 2019/0080476.
Regarding claim 15, Mishra teaches all the limitations of claim 11, but does not expressively teach wherein each selected portion of the first image frame is associated with a respective displacement vector that aligns the selected portion of the first image frame with a corresponding portion of the second image frame.
However, Mishra does teach that each selected portion of the first image frame is associated a corresponding portion of the second image frame due to motion of the image capture device (see Mishra column 20, “n some embodiments, a geometric transform function may be generated for each pixel of an image plane of an imaging device, and used to identify pixels of other images, e.g., counterpart images, depicting content similar or identical to that which is depicted by the imaging device at such pixels. The geometric transform function may consider a velocity or an altitude of the aerial vehicle, as well as one or more attributes of a pose of the first imaging device”).
In addition, Ermilios in a similar invention in the same field of endeavor teaches a system configured to associate selection portions of a first image with a second image due to motion of an image capture device (see Ermilios Figure 1 and paragraph [0046]) as taught in Mishra wherein
each selected portion of the first image frame is associated with a respective displacement vector that aligns the selected portion of the first image frame with a corresponding portion of the second image frame (see paragraph [0046]).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious as a matter of simple substation to replace the use of general motion to determine associations between images taught in Mishra with a displacement vector as taught in Ermilios to yield the predictable results of successfully associating the portions of the two images.
Allowable Subject Matter
Claim 6, 7, 16, and 17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CASEY L KRETZER whose telephone number is (571)272-5639. The examiner can normally be reached M-F 10:00-7:00 PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Payne can be reached at (571)272-3024. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CASEY L KRETZER/Primary Examiner, Art Unit 2635
Prosecution Insights