DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-5 and 9-15 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Condorovici et al. (U.S. Patent App. Pub No. 2023/0066958 A1, hereafter referred as Condorovici).
Regarding Claim 1:
Condorovici teaches an apparatus for generating image data, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to (Condorovici: Par. [0010]; a computer program product comprising a computer readable medium on which instructions are stored which when executed on a computing device): obtain a first image captured according to first image-capture settings (Condorovici: Par. [0037]; the camera 50 acquires successive images which are then processed by the ISP 40 and then written to memory 70); determine second image-capture settings related to the first image-capture settings; obtain a second image captured according to the second image-capture settings (Condorovici: Par. [0040]; the foreground and background statistics produced at steps 220 and 240 are used to determine any adjustment which may be required to the camera and/or ISP parameters for acquisition of a subsequent image from the camera 50); determine a mask indicative of pixels of the first image that are overexposed or underexposed (Condorovici: Par. [0055]; a simple measure of the illumination level of the foreground or face region determines whether the foreground region is overexposed or not; in order to detect if a face region of an image acquired at step 205 is overexposed, the number of pixels of the face region with a luminance higher than 130, on a scale of 0 . . . 255, is counted—these are considered highlight pixels; the number of pixels with a luminance value between 80 and 180 is also counted—these pixels are considered to be (probably) normally illuminated pixels; if a ratio of highlight pixels to normally illuminated pixels is higher than 1, the face region is considered to be overexposed); adjust luma values of the pixels of the first image based on luma values of corresponding pixels of the second image (Condorovici: Par. [0059]; if the background is very dark or very bright, step 250 will adjust the face exposure against that direction to make the face region of a subsequently acquired image either brighter or darker respectively); and adjust chroma values of the pixels of the first image based on chroma values of the corresponding pixels of the second image (Condorovici: Par. [0062]; for the current frame, any distance between the two temperatures is compensated in step 260 by modifying the color temperature of the face region; in order to allow for appropriate compensation of subsequently acquired images, if the difference is greater than a certain threshold, for example 4000K, then in step 250, the processing of the subsequently acquired image by the ISP 40 is adjusted).
In regards to Claim 2, Condorovici further teaches the apparatus of claim 1, wherein the first image-capture settings comprise at least one of an auto-focus setting, an auto-exposure setting, or an auto-white-balance setting (Condorovici: Par. [0018] and [0033]; semi-automatic white balancing process, rather than automatic white balancing (AWB) often employed within an ISP; automatic exposure (AE) enables the ISP 40 to set camera gain and/or exposure time to achieve maximum contrast and minimum saturation within a given acquired image or for a specific region of an image).
In regards to Claim 3, Condorovici further teaches the apparatus of claim 1, wherein, to determine the second image-capture settings, the at least one processor is configured to determine that the pixels of the first image are overexposed and determine the second image-capture settings to decrease an exposure of the second image (Condorovici: Par. [0055]; a simple measure of the illumination level of the foreground or face region determines whether the foreground region is overexposed or not; in order to detect if a face region of an image acquired at step 205 is overexposed, the number of pixels of the face region with a luminance higher than 130, on a scale of 0 . . . 255, is counted—these are considered highlight pixels; the number of pixels with a luminance value between 80 and 180 is also counted—these pixels are considered to be (probably) normally illuminated pixels; if a ratio of highlight pixels to normally illuminated pixels is higher than 1, the face region is considered to be overexposed; in step 250, a command may be sent to the camera 50 to reduce the exposure level by a minimum amount when acquiring the next image).
In regards to Claim 4, Condorovici further teaches the apparatus of claim 1, wherein, to determine the second image-capture settings, the at least one processor is configured to determine that the pixels of the first image are underexposed and determine the second image-capture settings to increase an exposure of the second image (Condorovici: Par. [0056]; in order to detect if a face region of an acquired image is underexposed, the number of pixels of the face region having the luminance value lower than 120 is counted—these pixels are considered to be (potentially) shadow pixels; if the ratio of shadow pixels to normal pixels is higher than 1, the image is considered to be underexposed, and in this case in step 250, a command may be sent to the camera 50 to increase the exposure level by the minimum amount when acquiring the next image).
In regards to Claim 5, Condorovici further teaches the apparatus of claim 1, wherein, to determine the second image-capture settings, the at least one processor is configured to: obtain a third image captured prior to the first image; and determine the first image-capture settings and the second image-capture settings based on an exposure of the third image (Condorovici: Par. [0037] and [0044]; the camera 50 acquires successive images which are then processed by the ISP 40 and then written to memory 70; the analysis of step 220 can be performed either on the first frame of the video, a certain relevant frame or a set of frames (every n.sup.th frame); the statistics produced by the analysis comprise a color temperature for the background and, in some cases, the illumination level of the background).
In regards to Claim 9, Condorovici further teaches the apparatus of claim 1, wherein the at least one processor is configured to write the corresponding pixels of the second image to a memory based on the mask (Condorovici: Par. [0037]; the camera 50 acquires successive images which are then processed by the ISP 40 and then written to memory 70 from which each image including its spatial and meta data can then be acquired by the application 20 running on the processor, step 205).
In regards to Claim 10, Condorovici further teaches the apparatus of claim 1, wherein, to adjust the luma values of the pixels of the first image, the at least one processor is configured to replace the luma values of the pixels of the first image with the luma values of corresponding pixels of the second image (Condorovici: Par. [0069]; in step 260, a transformation is generated to change the color temperature of the foreground region pixels of the image acquired at step 205 towards a target temperature, equivalent to the inferred for the background image).
In regards to Claim 11, Condorovici further teaches the apparatus of claim 1, wherein, to adjust the luma values of the pixels of the first image, the at least one processor is configured to blend the luma values of the pixels of the first image with the luma values of corresponding pixels of the second image (Condorovici: Par. [0041-0042]; in step 260, the foreground region of the current acquired image is adjusted, again based on the statistics produced at steps 220 and 240; once this foreground adjustment has been made, the foreground portion of the current acquired image can now be combined with the remaining portion of the selected background image to produce a composite image, step 270, in any one of a number of conventional manners including using suitable blending functions at the interface between the foreground and background regions).
In regards to Claim 12, Condorovici further teaches the apparatus of claim 1, wherein the at least one processor is configured to determine for which of the pixels of the first image to adjust the chroma values based on a comparison between a saturation of the pixels of the first image and a saturation of the corresponding pixels of the second image (Condorovici: Par. [0069]; in step 260, a transformation is generated to change the color temperature of the foreground region pixels of the image acquired at step 205 towards a target temperature, equivalent to the inferred for the background image).
In regards to Claim 13, Condorovici further teaches the apparatus of claim 1, wherein, to adjust the chroma values of the pixels of the first image, the at least one processor is configured to replace the chroma values of the pixels of the first image with the chroma values of corresponding pixels of the second image (Condorovici: Par. [0069]; in step 260, a transformation is generated to change the color temperature of the foreground region pixels of the image acquired at step 205 towards a target temperature, equivalent to the inferred for the background image).
In regards to Claim 14, Condorovici further teaches the apparatus of claim 1, wherein, to adjust the chroma values of the pixels of the first image, the at least one processor is configured to blend the chroma values of the pixels of the first image with the chroma values of corresponding pixels of the second image (Condorovici: Par. [0041-0042]; in step 260, the foreground region of the current acquired image is adjusted, again based on the statistics produced at steps 220 and 240; once this foreground adjustment has been made, the foreground portion of the current acquired image can now be combined with the remaining portion of the selected background image to produce a composite image, step 270, in any one of a number of conventional manners including using suitable blending functions at the interface between the foreground and background regions).
In regards to Claim 15, Condorovici further teaches the apparatus of claim 1, wherein the at least one processor is configured to: determine third image-capture settings related to the first image-capture settings; and obtain a third image captured according to the third image-capture settings (Condorovici: Par. [0037] and [0044]; the camera 50 acquires successive images which are then processed by the ISP 40 and then written to memory 70; the analysis of step 220 can be performed either on the first frame of the video, a certain relevant frame or a set of frames (every n.sup.th frame); the statistics produced by the analysis comprise a color temperature for the background and, in some cases, the illumination level of the background); wherein: at least some pixels of the second image are underexposed; at least some pixels of the third image are overexposed; to determine the mask, the at least one processor is configured to determine underexposed pixels of the first image (Condorovici: Par. [0056]; in order to detect if a face region of an acquired image is underexposed, the number of pixels of the face region having the luminance value lower than 120 is counted—these pixels are considered to be (potentially) shadow pixels; if the ratio of shadow pixels to normal pixels is higher than 1, the image is considered to be underexposed, and in this case in step 250, a command may be sent to the camera 50 to increase the exposure level by the minimum amount when acquiring the next image) and determine overexposed pixels of the first image (Condorovici: Par. [0055]; a simple measure of the illumination level of the foreground or face region determines whether the foreground region is overexposed or not; in order to detect if a face region of an image acquired at step 205 is overexposed, the number of pixels of the face region with a luminance higher than 130, on a scale of 0 . . . 255, is counted—these are considered highlight pixels; the number of pixels with a luminance value between 80 and 180 is also counted—these pixels are considered to be (probably) normally illuminated pixels; if a ratio of highlight pixels to normally illuminated pixels is higher than 1, the face region is considered to be overexposed; in step 250, a command may be sent to the camera 50 to reduce the exposure level by a minimum amount when acquiring the next image); to adjust the luma values of the pixels of the first image (Condorovici: Par. [0055]; in step 250, a command may be sent to the camera 50 to reduce the exposure level by a minimum amount when acquiring the next image), the at least one processor is configured to adjust the luma values of the overexposed pixels of the first image based on luma values of corresponding pixels of the second image (Condorovici: Par. [0059]; if the background is very dark or very bright, step 250 will adjust the face exposure against that direction to make the face region of a subsequently acquired image either brighter or darker respectively); and to adjust chroma values of the pixels of the first image, the at least one processor is configured to adjust the chroma values of the overexposed pixels of the first image based on chroma values of the corresponding pixels of the second image (Condorovici: Par. [0062]; for the current frame, any distance between the two temperatures is compensated in step 260 by modifying the color temperature of the face region; in order to allow for appropriate compensation of subsequently acquired images, if the difference is greater than a certain threshold, for example 4000K, then in step 250, the processing of the subsequently acquired image by the ISP 40 is adjusted); and the at least one processor is configured to: (Condorovici: Par. [0056]; and in this case in step 250, a command may be sent to the camera 50 to increase the exposure level by the minimum amount when acquiring the next image) adjust the luma values of the underexposed pixels of the first image based on luma values of corresponding pixels of the third image (Condorovici: Par. [0059]; if the background is very dark or very bright, step 250 will adjust the face exposure against that direction to make the face region of a subsequently acquired image either brighter or darker respectively); and adjust the chroma values of the underexposed pixels of the first image based on chroma values of the corresponding pixels of the third image (Condorovici: Par. [0062]; for the current frame, any distance between the two temperatures is compensated in step 260 by modifying the color temperature of the face region; in order to allow for appropriate compensation of subsequently acquired images, if the difference is greater than a certain threshold, for example 4000K, then in step 250, the processing of the subsequently acquired image by the ISP 40 is adjusted).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 6-8 are rejected under 35 U.S.C. 103 as being unpatentable over Condorovici et al. (U.S. Patent App. Pub No. 2023/0066958 A1, hereafter referred as Condorovici) in view of the knowledge of a person of ordinary skill in the art.
In regards to Claim 6, Condorovici fails to explicitly teach the apparatus of claim 1, wherein the second image has a lower resolution than the first image.
However, it would have been an obvious design optimization to a person of ordinary skill in the art to use a lower resolution second image since reduced resolution lowers computational cost and memory bandwidth.
In regards to Claim 7, Condorovici fails to explicitly teach the apparatus of claim 1, wherein the at least one processor is configured to downscale the second image based on a scaling factor.
However, it would have been an obvious preprocessing operation to a person of ordinary skill in the art to downscale the second image as a routine implementation step.
In regards to Claim 8, Condorovici fails to explicitly teach the apparatus of claim 1, wherein the at least one processor is configured to downscale the second image based on the mask.
However, it would have been an obvious extension of downscaling the second image to a person of ordinary skill in the art to process the images by their regions so as only to downscale the regions that need to be in order to increase efficiency.
Allowable Subject Matter
Claims 16-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Claim 16 recites, wherein the at least one processor is configured to: downscale the second image to generate a downscaled second image; write the downscaled second image to memory; read the downscaled second image from the memory; upscale the downscaled second image to generate a upscaled downscaled second image; generate a texture image; and modify the upscaled downscaled second image based on the texture image to generate a modified second image, wherein: the luma values of the pixels of the first image are adjusted based on luma values of corresponding pixels of the modified second image; and the chroma values of the pixels of the first image are adjusted based on chroma values of the corresponding pixels of the modified second image. The cited art of record does not teach or suggest such a combination of features.
Claims 17-20 are allowable subject matter by virtue of their dependency on Claim 16.
Because the cited art of record, alone or in combination, does not teach or suggest each and every feature of dependent Claims 16-20, these claims would be allowable.
Pertinent Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Lowe et al. (U.S. Patent App. Pub No. 2015/0035857 A1) teaches apparatuses and methods for correcting position information of captured images received by position sensors based on alignment of overlapping images.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RENAE BITOR whose telephone number is (703)756-5563. The examiner can normally be reached Monday to Friday: 8:00 - 5:30 but off the 1st Friday of the biweek.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, GREG MORSE can be reached on (571)272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RENAE A BITOR/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights