DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 3, 4, 7, 9-13, 17, 18, 20, 21, 24 and 26-30 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Zhen et al. (US 20200396370).
Regarding claim 1, Zhen et al. discloses a method of processing one or more images, comprising:
obtaining a first image captured using an image sensor, the first image being associated with a first exposure (“image frame 204 is captured using a shorter exposure (compared to the automatic or longer exposure)” at paragraph 0049, line 8);
obtaining a second image captured using the image sensor, the second image being associated with a second exposure that is longer than the first exposure (“one image frame 202 is captured using an automatic exposure (referred to here as an “auto-exposure”) or other longer exposure” at paragraph 0049, line 6);
modifying a first region of the first image based on a first transformation and a second region of the first image based on a second transformation to generate a modified first image (“For example, the image registration operation 206 may modify the non-reference image frame so that particular features in the non-reference image frame align with corresponding features in the reference image frame” at paragraph 0052, line 4; “The aligned image frame 210 (the non-reference image frame) is provided to a histogram matching operation 212. The histogram matching operation 212 generally operates to match a histogram of the non-reference image frame to a histogram of the reference image frame, such as by applying a suitable transfer function to the aligned image frame 210” at paragraph 0053, line 1; “A tone mapping operation 220 generally operates to apply a global tone mapping curve to the aligned image frame 210 in order to brighten darker areas and increase image contrast in the aligned image frame 210” at paragraph 0055, line 1; different areas of the non-reference frame are modified for alignment and brightness depending on the comparison with the reference frame); and
generating a combined image at least in part by combining the modified first image and the second image (“A blending operation 224 blends or otherwise combines the pixels from the image frames 208 and 222 based on the label map(s) 218 in order to produce at least one final image 226 of a scene” at paragraph 0056, line 1).
Regarding claim 18, Zhen et al. discloses an apparatus for processing one or more images, the apparatus comprising:
at least one memory (“The memory 130 can include a volatile and/or non-volatile memory. For example, the memory 130 can store commands or data related to at least one other component of the electronic device 101” at paragraph 0036, line 1); and
at least one processor coupled with the at least one memory (“The processor 120 includes one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor 120 is able to perform control on at least one of the other components of the electronic device 101 and/or perform an operation or data processing relating to communication’ at paragraph 0035, line 1), wherein the at least one processor is configured to:
obtain a first image captured using an image sensor, the first image being associated with a first exposure (“image frame 204 is captured using a shorter exposure (compared to the automatic or longer exposure)” at paragraph 0049, line 8);
obtain a second image captured using the image sensor, the second image being associated with a second exposure that is longer than the first exposure (“one image frame 202 is captured using an automatic exposure (referred to here as an “auto-exposure”) or other longer exposure” at paragraph 0049, line 6);
modify a first region of the first image based on a first transformation and a second region of the first image based on a second transformation to generate a modified first image (“For example, the image registration operation 206 may modify the non-reference image frame so that particular features in the non-reference image frame align with corresponding features in the reference image frame” at paragraph 0052, line 4; “The aligned image frame 210 (the non-reference image frame) is provided to a histogram matching operation 212. The histogram matching operation 212 generally operates to match a histogram of the non-reference image frame to a histogram of the reference image frame, such as by applying a suitable transfer function to the aligned image frame 210” at paragraph 0053, line 1; “A tone mapping operation 220 generally operates to apply a global tone mapping curve to the aligned image frame 210 in order to brighten darker areas and increase image contrast in the aligned image frame 210” at paragraph 0055, line 1; different areas of the non-reference frame are modified for alignment and brightness depending on the comparison with the reference frame); and
generate a combined image at least in part by combining the modified first image and the second image (“A blending operation 224 blends or otherwise combines the pixels from the image frames 208 and 222 based on the label map(s) 218 in order to produce at least one final image 226 of a scene” at paragraph 0056, line 1).
Regarding claims 3 and 20, Zhen et al. discloses a method and apparatus wherein the first region is associated with an object at a first depth in a scene relative to the image sensor, and wherein the second region includes a background region at a second depth in the scene relative to the image sensor (“In this particular scene, a person is standing inside a structure in the image foreground, while people are walking in the image background (which is outside and bright)” at paragraph 0075, second to last sentence; brightness areas of adjustment include underexposed areas and the alignment of the two frames causes image data in both foreground and background areas to be adjusted).
Regarding claims 4 and 21, Zhen et al. discloses a method and apparatus wherein the at least one processor is configured to:
generate a first matrix for performing the first transformation (“The aligned image frame 210 (the non-reference image frame) is provided to a histogram matching operation 212. The histogram matching operation 212 generally operates to match a histogram of the non-reference image frame to a histogram of the reference image frame, such as by applying a suitable transfer function to the aligned image frame 210” at paragraph 0053, line 1; “A tone mapping operation 220 generally operates to apply a global tone mapping curve to the aligned image frame 210 in order to brighten darker areas and increase image contrast in the aligned image frame 210” at paragraph 0055, line 1; areas of underexposure are particularly of interest for adjustment and produces a matrix of changed pixels values for brightness adjustment); and
generate a second matrix for performing the second transformation (“For example, the image registration operation 206 may modify the non-reference image frame so that particular features in the non-reference image frame align with corresponding features in the reference image frame” at paragraph 0052, line 4; alignment is performed for various areas of the non-reference frame and produces a matrix of changed pixel values for the realigned areas).
Regarding claims 7 and 24, Zhen et al. discloses a method and apparatus wherein the at least one processor is configured to:
extract first feature points from the first image (“For example, the image registration operation 206 may modify the non-reference image frame so that particular features in the non-reference image frame align with corresponding features in the reference image frame” at paragraph 0052, line 4; features are extracted for the non-reference frame); and
extract second feature points from the second image (corresponding features are extracted for the reference frame for comparison).
Regarding claims 9 and 26, Zhen et al. discloses a method and apparatus wherein the at least one processor is configured to:
detect an object in the second image (“As can be seen in FIG. 3C, the white pixels in the label map 306 generally include the foreground objects and other relatively-close objects, such as the various natural objects and artificial structures in the scene. Given this particular scene, this is generally to be expected since the image frame 302 contains more image details than the image frame 304, at least with respect to these objects” at paragraph 0072, line 1); and
determine a bounding region associated with a location of the object in the second image (“In addition, as can be seen in FIG. 3C, the labels in the label map 306 here have clear discontinuities along edges or boundaries of objects within the scene, such as the various natural objects and artificial structures. As a result, most or substantially all of the pixels in the final image showing the various natural objects and artificial structures will be extracted from a single image frame (the image frame 302 in this example), helping to avoid the creation of artifacts within these objects in the final image” at paragraph 0073; “The smoothness cost function generally considers how each pixel's neighbors are labeled so that cuts in the label map tend to naturally follow object boundaries in the image frames” at paragraph 0054, last sentence).
Regarding claims 10 and 27, Zhen et al. discloses a method and apparatus wherein the at least one processor is configured to:
identify a subset of the first feature points within the bounding region (“As can be seen in FIG. 3C, the white pixels in the label map 306 generally include the foreground objects and other relatively-close objects, such as the various natural objects and artificial structures in the scene. Given this particular scene, this is generally to be expected since the image frame 302 contains more image details than the image frame 304, at least with respect to these objects” at paragraph 0072, line 1; pixels belonging to the natural objects constitute one subset);
identify a subset of the second feature points within the bounding region (pixels belonging to the artificial structure constitute another subset); and
generate the first matrix based on the subset of the first feature points and the subset of the second feature points (“The aligned image frame 210 (the non-reference image frame) is provided to a histogram matching operation 212. The histogram matching operation 212 generally operates to match a histogram of the non-reference image frame to a histogram of the reference image frame, such as by applying a suitable transfer function to the aligned image frame 210” at paragraph 0053, line 1; “A tone mapping operation 220 generally operates to apply a global tone mapping curve to the aligned image frame 210 in order to brighten darker areas and increase image contrast in the aligned image frame 210” at paragraph 0055, line 1; areas corresponding to the foreground areas in the non-reference frame are adjusted for brightness depending on the brightness of the corresponding foreground areas in the reference frame).
Regarding claims 11 and 28, Zhen et al. discloses a method and apparatus wherein the at least one processor is configured to:
generate, based on the first matrix and the second matrix, a hybrid transformation matrix for modifying the first region of the first image and the second region of the first image (“The output of the tone mapping operation 220 is a tone-matched aligned image frame 222, which (ideally) has the same or substantially similar tone as the aligned image frame 208” at paragraph 0055, last sentence; the tone matched aligned image is therefore an amalgamation of the previous brightness adjustments and the alignment).
Regarding claims 12 and 29, Zhen et al. discloses a method and apparatus wherein the at least one processor is configured to:
add values from the first matrix to the hybrid transformation matrix that at least correspond to the first region (“The aligned image frame 210 (the non-reference image frame) is provided to a histogram matching operation 212. The histogram matching operation 212 generally operates to match a histogram of the non-reference image frame to a histogram of the reference image frame, such as by applying a suitable transfer function to the aligned image frame 210” at paragraph 0053, line 1; “A tone mapping operation 220 generally operates to apply a global tone mapping curve to the aligned image frame 210 in order to brighten darker areas and increase image contrast in the aligned image frame 210” at paragraph 0055, line 1; areas of underexposure are particularly of interest for adjustment and produces a matrix of changed pixels values for brightness adjustment; the tone matched aligned image is therefore an amalgamation of the previous brightness adjustments and the alignment); and
add values from the second matrix to the hybrid transformation matrix that at least correspond to the second region (“For example, the image registration operation 206 may modify the non-reference image frame so that particular features in the non-reference image frame align with corresponding features in the reference image frame” at paragraph 0052, line 4; alignment is performed for various areas of the non-reference frame and produces a matrix of changed pixel values for the realigned areas which is subject to further brightness processing).
Regarding claims 13 and 30, Zhen et al. discloses a method and apparatus wherein the at least one processor is configured to:
determine a transition region between the first region and the second region based on a size of a bounding region associated with a location of an object in at least one of the first image or the second image (“The smoothness cost function generally considers how each pixel's neighbors are labeled so that cuts in the label map tend to naturally follow object boundaries in the image frames” at paragraph 0054, last sentence);
determine values associated with the transition region based on a representation of the first matrix and the second matrix (“As can be seen in FIG. 3C, the white pixels in the label map 306 generally include the foreground objects and other relatively-close objects, such as the various natural objects and artificial structures in the scene. Given this particular scene, this is generally to be expected since the image frame 302 contains more image details than the image frame 304, at least with respect to these objects. The black pixels in the label map 306 generally include the sky in the scene's background. Again, given this particular scene, this is generally to be expected since the image frame 302 is over-exposed in the areas showing the sky, while the image frame 304 is well-exposed in the areas showing the sky. The final image of the scene will therefore include the pixels defining the foreground objects and other relatively-close objects from the pre-processed image frame 302 and the pixels defining the sky from the pre-processed image frame 304 based on the label map 306 shown here’ at paragraph 0072); and
add the values associated with the transition region to the hybrid transformation matrix (“An edge map in the smoothness cost aims to make any cut seams along object boundaries, and the values β and γ regularize the contribution of each edge map” at paragraph 0064, last sentence; “As a result, most or substantially all of the pixels in the final image showing the various natural objects and artificial structures will be extracted from a single image frame (the image frame 302 in this example), helping to avoid the creation of artifacts within these objects in the final image” at paragraph 0073, line 4).
Regarding claim 17, Zhen et al. discloses a method wherein the combined image is a high dynamic range image (“The label map(s) can therefore be used to combine different input image frames into a composite or final image having high dynamic range (HDR)” at paragraph 0030, line 10).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Zhen et al.
Zhen et al. discloses the elements of claims 1 and 18 above.
Zhen et al. does not explicitly disclose that the image sensor is oriented in a same direction as a display for displaying preview images captured by the image sensor.
However, Zhen et al. further discloses that the apparatus is a smartphone or tablet (“Examples of an “electronic device” according to embodiments of this disclosure may include at least one of a smartphone, a tablet personal computer (PC)” at paragraph 0015, line 1). These devices are known to have self-facing cameras with functionality to allow the user to view the captured image. As such, this is an obvious extension of the explicitly disclosed description for purposes of letting the user preview the intended captured HDR image.
Claim(s) 5, 6, 16, 22 and 23 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Zhen et al. and Yuan et al. (US 20210337101).
Regarding claims 5 and 22, Zhen et al. discloses a method and apparatus as described in claims 4 and 21 above.
Zhen et al. does not explicitly disclose that the at least one processor is configured to generate the second matrix based on movement detected by a motion sensor between a first time when the first image is captured and a second time when the second image is captured.
Yuan et al. teaches a method and apparatus in the same field of endeavor of HDR imaging, wherein the at least one processor is configured to generate the second matrix based on movement detected by a motion sensor between a first time when the first image is captured and a second time when the second image is captured (“The poses of the first 130 and the second 140 motion data may comprise angular orientation data indicating a yaw, a pitch and optionally a roll of the rolling shutter image sensor when reading out pixel data from the corresponding pixel region when capturing the first/second image 100, 110. Alternative or additionally, the poses of the first 130 and the second 140 motion data may comprise position data indicating to a position in space of the rolling shutter image sensor when reading out pixel data from the corresponding pixel region when capturing the first/second image 100, 110. The position may be indicated by at least one of a x-value, y-value and z-value of the rolling shutter image sensor” at paragraph 0093; “the at least one motion sensor 404 comprises a gyroscope to determine an orientation of the rolling shutter image sensor” at paragraph 0119, second to last sentence).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize motion sensor data during HDR generation as taught by Yuan et al. in the system of Zhen et al. “to stabilize the final HDR on a block basis based on the corresponding pose of the rolling shutter image sensor when capturing that block of pixels. This may in turn improve the perceived quality of the final HDR image” (Yuan et al. at paragraph 0028, second to last sentence).
Regarding claims 6 and 23, Yuan et al. discloses a method and apparatus wherein the motion sensor comprises a gyroscope sensor, and wherein the second transformation comprises a rotational transformation (“the at least one motion sensor 404 comprises a gyroscope to determine an orientation of the rolling shutter image sensor” at paragraph 0119, second to last sentence; “Each pose may thus represent the position and/or rotation of the rolling shutter image sensor 402, where the position and/or rotation may be represented by one degree of freedom to six degrees of freedom in space” at paragraph 0121).
Regarding claim 16, Zhen et al. discloses a method as described in claim 1 above.
Zhen et al. does not explicitly disclose that the first transformation comprises a translational matrix associated with movement of the image sensor during the obtaining of the first image and the obtaining of the second image.
Yuan et al. teaches a method in the same field of endeavor of HDR imaging, wherein the first transformation comprises a translational matrix associated with movement of the image sensor during the obtaining of the first image and the obtaining of the second image (“The poses of the first 130 and the second 140 motion data may comprise angular orientation data indicating a yaw, a pitch and optionally a roll of the rolling shutter image sensor when reading out pixel data from the corresponding pixel region when capturing the first/second image 100, 110. Alternative or additionally, the poses of the first 130 and the second 140 motion data may comprise position data indicating to a position in space of the rolling shutter image sensor when reading out pixel data from the corresponding pixel region when capturing the first/second image 100, 110. The position may be indicated by at least one of a x-value, y-value and z-value of the rolling shutter image sensor” at paragraph 0093).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize motion sensor data during HDR generation as taught by Yuan et al. in the system of Zhen et al. “to stabilize the final HDR on a block basis based on the corresponding pose of the rolling shutter image sensor when capturing that block of pixels. This may in turn improve the perceived quality of the final HDR image” (Yuan et al. at paragraph 0028, second to last sentence).
Claim(s) 8 and 25 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Zhen et al. and Kaji (US 20230319407).
Zhen et al. discloses a method and apparatus as described in claims 7 and 24 above.
Zhen et al. does not explicitly disclose that the at least one processor is configured to: increase a brightness of the first image based on an exposure ratio difference between the first image and the second image.
Kaji teaches a method and apparatus in the same field of endeavor of HDR image processing, wherein the at least one processor is configured to:
increase a brightness of the first image based on an exposure ratio difference between the first image and the second image (“More specifically, in the low-luminance region, based on the generated α map, images are combined in units of pixels in a manner that the blending ratio of the enlarged binning image which is the long-exposure image is higher than the blending ratio of the blur-reduced image which is the short-exposure image” at paragraph 0135).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to blend the images of Zhen et al. using the blending ratio of Kaji as “gradation correction is performed in a manner that the gradation change becomes natural, that is, the gradation change becomes gentle” (Kaji at paragraph 0137, line 2).
Claim(s) 14 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Zhen et al. and Sharma et al. (US 20140307960).
Zhen et al. discloses a method as described in claim 13 above.
Zhen et al. does not explicitly disclose that the representation of the first matrix and the second matrix includes a weighted average of the first matrix and the second matrix.
Sharma et al. teaches a method in the same field of endeavor of HDR image processing, wherein the representation of the first matrix and the second matrix includes a weighted average of the first matrix and the second matrix (“At block 615, the embodiment can merge the set of images using the weighted average of the set of images, the weighted average of the images based upon the respective ghost-free weight for each of the images” at paragraph 0087).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize a weighted average as taught by Sharma et al. in creating the HDR image of Zhen et al. to minimize ghosting in the merged image (Sharma et al. at paragraph 0008).
Allowable Subject Matter
Claim 15 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: the prior art does not teach or disclose that the representation of the first matrix and the second matrix is based on a proportional distance from an inner edge of the transition region to an outer edge of the transition region as required by claim 15.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Kang et al., Park, Levoy et al., Narasimha et al., Chuang et al., Yao et al, Han et al. and Ravirala et al. teach various disclosures associated with HDR image processing and formation by utilizing multiple images.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATRINA R FUJITA whose telephone number is (571)270-1574. The examiner can normally be reached Monday - Friday 9:30-5:30 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 5712723638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATRINA R FUJITA/Primary Examiner, Art Unit 2672
Prosecution Insights