DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on October 28, 2024 and December 9, 2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 8 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent No. 7,606,417 to Steinberg et al. (hereinafter Steinberg), and further in view of U.S. Patent No. 11,017,511 to Lukac et al. (hereinafter Lukac).
Regarding independent claim 1, Steinberg discloses A method (abstract, “A digital segmentation method and apparatus determines foreground and/or background within at least one portion of a captured image”) comprising:
obtaining multiple images via a multi-frame capture operation (Figure 2(a), element 522, “Generate a single image from multiple images”);
processing the multiple images using a multi-frame processing pipeline to generate a single frame image (Figure 2(a), element 522, “Generate a single image from multiple images”);
processing the multiple images to generate a backlit segmentation map that identifies regions in a scene of the single frame image that are illuminated by at least one background light source (column 1, line 50, “While available ambient light such as sunlight ” column 7, line 36, “After the thresholds VH, VL are determined, the image is processed via a segmenting tool, 590, to designate pixels or regions as background or foreground. In one embodiment, pixels whose values change less than a threshold amount, say VH−VL (or some other empirically determined value) between flash I′(x,y) and non-flash versions P″(x,y) of the image represent pixels in areas of a flash-image forming a boundary between background and foreground objects. When such individual pixels are linked, then segments of the image I′(x,y) substantially enclosed by linked boundary pixels and having pixel values on average brighter than in the corresponding segment of the non-flash image P″(x,y) are designated as foreground, whereas segments of the image substantially enclosed by boundary pixels and having pixel values on average darker than in the corresponding segment of the non-flash image are designated as background.”).
Steinberg fails to explicitly disclose as further recited. However, Lukac discloses generating a dehazed single frame image based on modifying a local contrast of the single frame image using the backlit segmentation map (column 10, line 3, “Process 300 may include “provide LUT as the correction factors” 342. The resulting LUT forms, or is, the correction factors representing a haze correction curve or tone mapping-type curve that is applied to the original image after refinement into correction parameters as described below.”).
Steinberg is directed toward, “A digital segmentation method and apparatus determines foreground and/or background within at least one portion of a captured image (abstract).” Lukac is directed toward, “A method, system, and article are directed to haze reduction for image processing (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Steinberg and Lukac are directed toward similar methods of endeavor of image processing. Further, one of ordinary skill in the art before the effective filing date of the claimed invention would easily understand that when generating images and further processing images, it is ideal for an image to be as clear as possible to retain all details of interest. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Lukac in order to ensure the image output is as clear and accurate for future reviewers or processing as possible.
Regarding independent claim 8, the rejection of claim 1 applies directly. Additionally, Steinberg discloses An electronic device (abstract, “A digital segmentation method and apparatus determines foreground and/or background within at least one portion of a captured image.”) comprising:
at least one processing device (column 2, line 43, “ includes a processor 120”) configured to:
obtain multiple images via a multi-frame capture operation (Figure 2(a), element 522, “Generate a single image from multiple images”);
process the multiple images using a multi-frame processing pipeline to generate a single frame image (Figure 2(a), element 522, “Generate a single image from multiple images”);
process the multiple images to generate a backlit segmentation map that identifies regions in a scene of the single frame image that are illuminated by at least one background light source (column 1, line 50, “While available ambient light such as sunlight ” column 7, line 36, “After the thresholds VH, VL are determined, the image is processed via a segmenting tool, 590, to designate pixels or regions as background or foreground. In one embodiment, pixels whose values change less than a threshold amount, say VH−VL (or some other empirically determined value) between flash I′(x,y) and non-flash versions P″(x,y) of the image represent pixels in areas of a flash-image forming a boundary between background and foreground objects. When such individual pixels are linked, then segments of the image I′(x,y) substantially enclosed by linked boundary pixels and having pixel values on average brighter than in the corresponding segment of the non-flash image P″(x,y) are designated as foreground, whereas segments of the image substantially enclosed by boundary pixels and having pixel values on average darker than in the corresponding segment of the non-flash image are designated as background.”); and
Steinberg fails to explicitly disclose as further recited. However, Lukac discloses generate a dehazed single frame image based on modifying a local contrast of the single frame image using the backlit segmentation map (column 10, line 3, “Process 300 may include “provide LUT as the correction factors” 342. The resulting LUT forms, or is, the correction factors representing a haze correction curve or tone mapping-type curve that is applied to the original image after refinement into correction parameters as described below.”).
Steinberg is directed toward, “A digital segmentation method and apparatus determines foreground and/or background within at least one portion of a captured image (abstract).” Lukac is directed toward, “A method, system, and article are directed to haze reduction for image processing (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Steinberg and Lukac are directed toward similar methods of endeavor of image processing. Further, one of ordinary skill in the art before the effective filing date of the claimed invention would easily understand that when generating images and further processing images, it is ideal for an image to be as clear as possible to retain all details of interest. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Lukac in order to ensure the image output is as clear and accurate for future reviewers or processing as possible.
Regarding independent claim 15, the rejection of claim 1 applies directly. Additionally, Steinberg discloses A non-transitory machine readable medium comprising instructions that when executed cause at least one processor of an electronic device (column 2, line 43, “ a processor 120. It can be appreciated that many of the processes implemented in the digital camera may be implemented in or controlled by software operating in a microprocessor, central processing unit, controller, digital signal processor and/or an application specific integrated circuit, collectively depicted as block 120 labelled “processor”.”) to:
obtain multiple images via a multi-frame capture operation (Figure 2(a), element 522, “Generate a single image from multiple images”);
process the multiple images using a multi-frame processing pipeline to generate a single frame image (Figure 2(a), element 522, “Generate a single image from multiple images”);
process the multiple images to generate a backlit segmentation map that identifies regions in a scene of the single frame image that are illuminated by at least one background light source (column 1, line 50, “While available ambient light such as sunlight ” column 7, line 36, “After the thresholds VH, VL are determined, the image is processed via a segmenting tool, 590, to designate pixels or regions as background or foreground. In one embodiment, pixels whose values change less than a threshold amount, say VH−VL (or some other empirically determined value) between flash I′(x,y) and non-flash versions P″(x,y) of the image represent pixels in areas of a flash-image forming a boundary between background and foreground objects. When such individual pixels are linked, then segments of the image I′(x,y) substantially enclosed by linked boundary pixels and having pixel values on average brighter than in the corresponding segment of the non-flash image P″(x,y) are designated as foreground, whereas segments of the image substantially enclosed by boundary pixels and having pixel values on average darker than in the corresponding segment of the non-flash image are designated as background.”); and
Steinberg fails to explicitly disclose as further recited. However, Lukac discloses generate a dehazed single frame image based on modifying a local contrast of the single frame image using the backlit segmentation map (column 10, line 3, “Process 300 may include “provide LUT as the correction factors” 342. The resulting LUT forms, or is, the correction factors representing a haze correction curve or tone mapping-type curve that is applied to the original image after refinement into correction parameters as described below.”).
Steinberg is directed toward, “A digital segmentation method and apparatus determines foreground and/or background within at least one portion of a captured image (abstract).” Lukac is directed toward, “A method, system, and article are directed to haze reduction for image processing (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Steinberg and Lukac are directed toward similar methods of endeavor of image processing. Further, one of ordinary skill in the art before the effective filing date of the claimed invention would easily understand that when generating images and further processing images, it is ideal for an image to be as clear as possible to retain all details of interest. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Lukac in order to ensure the image output is as clear and accurate for future reviewers or processing as possible.
Claim(s) 2, 9 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Steinberg further in view of Lukac as applied to claims 1, 8 and 15 respectively above, and further in view of U.S. Patent No. 11,431,918 to Price et al. (hereinafter Price).
Regarding dependent claim 2, the rejection of claim 1 is incorporated herein. Additionally, Steinberg discloses wherein processing the multiple images to generate the backlit segmentation map includes:
determining saturation of regions in the (column 7, line 36, “After the thresholds VH, VL are determined, the image is processed via a segmenting tool, 590, to designate pixels or regions as background or foreground. In one embodiment, pixels whose values change less than a threshold amount, say VH−VL (or some other empirically determined value) between flash I′(x,y) and non-flash versions P″(x,y) of the image represent pixels in areas of a flash-image forming a boundary between background and foreground objects.” Saturation is read as a value of a pixel to be analyzed ); and
generating the backlit segmentation map based on the saturation of the regions in the short frame (column 7, line 36, “After the thresholds VH, VL are determined, the image is processed via a segmenting tool, 590, to designate pixels or regions as background or foreground. In one embodiment, pixels whose values change less than a threshold amount, say VH−VL (or some other empirically determined value) between flash I′(x,y) and non-flash versions P″(x,y) of the image represent pixels in areas of a flash-image forming a boundary between background and foreground objects.”).
Steinberg and Lukac in the combination fails to explicitly disclose as further recited. However, Price discloses selecting a short frame from the multiple images (abstract, “ The system is configurable to perform a short exposure operation by applying a set of short exposure shutter operations to configure each SPAD pixel of the SPAD array to enable photon detection. The short exposure operation occurs during the time period that intervenes between the first and second sets of long exposure shutter operations.” Column 3, line 59, “ a short-exposure operation may be performed and read out to form a short-exposure image to capture brighter objects in the environment.”)
As noted above, Steinberg and Lukac are directed toward similar methods of endeavor of image processing. Further, Price discloses, “A system for HDR image capture (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Steinberg, Lukac and Price are all directed toward similar methods of endeavor of image processing. Further, one of ordinary skill in the art would easily understand HDR images are useful for obtaining accurate and dynamic images especially in scenes containing brightness differences, which is further exemplified in Price, column 1, line 7+. Thus, being that Steinberg and Lukac allow for image processing of foreground and background segmentation with different exposure, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Price in order to obtain an image most suitable for capturing a scene of different brightness, to then limit processing needs of adjusting exposure in the image itself.
Regarding dependent claim 9, the rejection of claim 8 is incorporated herein. Additionally, Steinberg discloses wherein, to process the multiple images to generate the backlit segmentation map, the at least one processing device is further configured to:
determine saturation of regions in the (column 7, line 36, “After the thresholds VH, VL are determined, the image is processed via a segmenting tool, 590, to designate pixels or regions as background or foreground. In one embodiment, pixels whose values change less than a threshold amount, say VH−VL (or some other empirically determined value) between flash I′(x,y) and non-flash versions P″(x,y) of the image represent pixels in areas of a flash-image forming a boundary between background and foreground objects.” Saturation is read as a value of a pixel to be analyzed ); and
generate the backlit segmentation map based on the saturation of the regions in the short frame (column 7, line 36, “After the thresholds VH, VL are determined, the image is processed via a segmenting tool, 590, to designate pixels or regions as background or foreground. In one embodiment, pixels whose values change less than a threshold amount, say VH−VL (or some other empirically determined value) between flash I′(x,y) and non-flash versions P″(x,y) of the image represent pixels in areas of a flash-image forming a boundary between background and foreground objects.”).
Steinberg and Lukac in the combination fails to explicitly disclose as further recited. However, Price discloses select a short frame from the multiple images (abstract, “ The system is configurable to perform a short exposure operation by applying a set of short exposure shutter operations to configure each SPAD pixel of the SPAD array to enable photon detection. The short exposure operation occurs during the time period that intervenes between the first and second sets of long exposure shutter operations.” Column 3, line 59, “ a short-exposure operation may be performed and read out to form a short-exposure image to capture brighter objects in the environment.”)
As noted above, Steinberg and Lukac are directed toward similar methods of endeavor of image processing. Further, Price discloses, “A system for HDR image capture (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Steinberg, Lukac and Price are all directed toward similar methods of endeavor of image processing. Further, one of ordinary skill in the art would easily understand HDR images are useful for obtaining accurate and dynamic images especially in scenes containing brightness differences, which is further exemplified in Price, column 1, line 7+. Thus, being that Steinberg and Lukac allow for image processing of foreground and background segmentation with different exposure, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Price in order to obtain an image most suitable for capturing a scene of different brightness, to then limit processing needs of adjusting exposure in the image itself.
Regarding dependent claim 16, the rejection of claim 15 is incorporated herein. Additionally, Steinberg discloses wherein the instructions that when executed cause the at least one processor of the electronic device to process the multiple images to generate the backlit segmentation map further include instructions that when executed cause the at least one processor of the electronic device to:
determine saturation of regions in the (column 7, line 36, “After the thresholds VH, VL are determined, the image is processed via a segmenting tool, 590, to designate pixels or regions as background or foreground. In one embodiment, pixels whose values change less than a threshold amount, say VH−VL (or some other empirically determined value) between flash I′(x,y) and non-flash versions P″(x,y) of the image represent pixels in areas of a flash-image forming a boundary between background and foreground objects.” Saturation is read as a value of a pixel to be analyzed ); and
generate the backlit segmentation map based on the saturation of the regions in the short frame (column 7, line 36, “After the thresholds VH, VL are determined, the image is processed via a segmenting tool, 590, to designate pixels or regions as background or foreground. In one embodiment, pixels whose values change less than a threshold amount, say VH−VL (or some other empirically determined value) between flash I′(x,y) and non-flash versions P″(x,y) of the image represent pixels in areas of a flash-image forming a boundary between background and foreground objects.”).
Steinberg and Lukac in the combination fails to explicitly disclose as further recited. However, Price discloses select a short frame from the multiple images (abstract, “ The system is configurable to perform a short exposure operation by applying a set of short exposure shutter operations to configure each SPAD pixel of the SPAD array to enable photon detection. The short exposure operation occurs during the time period that intervenes between the first and second sets of long exposure shutter operations.” Column 3, line 59, “ a short-exposure operation may be performed and read out to form a short-exposure image to capture brighter objects in the environment.”)
As noted above, Steinberg and Lukac are directed toward similar methods of endeavor of image processing. Further, Price discloses, “A system for HDR image capture (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Steinberg, Lukac and Price are all directed toward similar methods of endeavor of image processing. Further, one of ordinary skill in the art would easily understand HDR images are useful for obtaining accurate and dynamic images especially in scenes containing brightness differences, which is further exemplified in Price, column 1, line 7+. Thus, being that Steinberg and Lukac allow for image processing of foreground and background segmentation with different exposure, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Price in order to obtain an image most suitable for capturing a scene of different brightness, to then limit processing needs of adjusting exposure in the image itself.
Allowable Subject Matter
Claims 3-7, 10-14 and 17-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: the closest prior arts of record teach methods of performing haze reduction in images for scenes with varying lighting. However, none of them alone or in any combination teaches determining a map of the level of dehazing based on weights applied to a backlight segmentation map, then performing dehazing based on the dehazing map.
The closest prior art Steinberg discloses, “A digital segmentation method and apparatus determines foreground and/or background within at least one portion of a captured image. The determining includes comparing a captured image to a pre-captured or post captured reference image of nominally the same scene. One of the images is taken with flash and the other without (abstract).” Steinberg generates a backlight segmentation map (column 7, line 36, “After the thresholds VH, VL are determined, the image is processed via a segmenting tool, 590, to designate pixels or regions as background or foreground.”) but doesn’t generate a dehazing level based on weights that were applied to the backlight segmentation map.
However, Steinberg fails to disclose determining a map of the level of dehazing based on weights applied to a backlight segmentation map, then performing dehazing based on the dehazing map.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
U.S. Publication No. 2021/0201452 to Treibitz et al. discloses, “Methods for dehazing a digital image and for restoring an underwater digital image (abstract).”
U.S. Publication No. 2024/0007600 to Zhu et al. discloses, “Methods, systems, devices, and tangible non-transitory computer readable media for haze reduction are provided (abstract).”
U.S. Patent No. 9,288,458 to Chen et al. discloses, “system for image de-hazing for real-time video processing (abstract).”
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Courtney J. Windsor whose telephone number is (571)272-3956. The examiner can normally be reached Monday - Friday 8:00 - 4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/COURTNEY JOAN NELSON/Primary Examiner, Art Unit 2661
Prosecution Insights