DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1, 5, 7-9, 12, 14-16 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Lukac U.S. Patent Application 20250193339 in view of Toizumi U.S. Patent Application 20230059499.
Regarding claim 1, Lukac discloses an apparatus comprising:
at least one imaging sensor (sensor 102) configured to capture a sequence of image frames (paragraph [0023]: the image video/data 110 includes the scene image 103 in the form of raw image data captured by the sensor 102); and
at least one processing device (IPU 108) configured, for each of at least some of the captured image frames, to:
generate a model associated with the captured image frame, the model based on (i) one or more previous visually-enhanced image frames associated with one or more previous image frames in the sequence (paragraph [0041]: At block 404, the IPU 108 generates the face model and/or the scene model 112 based on the image/video data 110; paragraph [0050]: At block 508, the process continues with enhancement of the face model and/or the scene model 112 to further improve performance of model merging 510... super-resolution imaging enhances the resolution of the face/scene model to produce the best possible image quality and user experience) and
(ii) one or more previous models associated with the one or more previous visually-enhanced image frames (paragraph [0051]: block 510, the method 500 completes with merging the face model and/or the scene model 112 of the image/video data 110 with the current face/scene model baseline. Thus, the face/scene model is updated in each iteration and/or frame to increase its accuracy and information contents);
apply the model to the captured image frame in order to generate a modified captured image frame (paragraph [0045]: At block 412, the image capturing device 100 performs scene editing and/or image blending using a blend module... The IPU 108 applies scene editing directly on the image 110 itself and/or within separate area of memory designated for editing operations, including color space conversion, luminance-chrominance decomposition, and frequency decomposition); and
render an image for display based on the modified captured image frame (paragraph [0046]: Finally, at step 414, the image capturing device 100 performs postprocessing and applies any necessary filters desired by the user... and color and lighting effects; see fig. 4 provide output image/video 114; paragraph [0031]: The adjusted image 224 will be a high-quality image that is displayed during the videoconference event and is output as the output image 114).
Lukac discloses all the features with respect to claim 1 as outlined above. However, Lukac fails to disclose generating a lightness model explicitly.
Toizumi discloses generating a lightness model (paragraph [0088]: in S202, the pixel value correction unit 320 corrects the luminance value of the monochrome image M0 by using the correction model including the acquired correction parameters; paragraph [0081]: The luminance prediction model predicts correction parameters related to the luminance distribution of the monochrome image M... the correction parameter optimization unit 350 receives a processed monochrome image MLN that has been subjected to a predetermined luminance processing, and the luminance prediction model of the previous stage of the colorization prediction model is trained by machine learning using training data in which a color image CL corresponding to the aforementioned monochrome image MLN is used as an output).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lukac’s to use lightness model as taught by Toizumi, to generate correct luminance value for each pixel.
Regarding claim 5, Lukac as modified by Toizumi discloses the apparatus of claim 1, wherein:
the at least one processing device is further configured to apply a transformation to the modified captured image frame in order to generate a transformed image frame (Toizumi’s paragraph [0002]: image processing system which converts a monochrome image into a colorized image using a trained prediction model based on colors (color hints) designated by a user using a general-purpose color palette; Lukac’s paragraph [0012]: the blending is performed directly on the input image, or alternatively is performed in a transformed domain obtained through color space conversion, luminance-chrominance decomposition, frequency decomposition, and the like); and
to render the image for display, the at least one processing device is configured to render the transformed image frame (Lukac’s paragraph [0031]: The adjusted image 224 will be a high-quality image that is displayed during the videoconference event and is output as the output image 114).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lukac’s to use lightness model as taught by Toizumi, to generate correct luminance value for each pixel.
Regarding claim 7, Lukac as modified by Toizumi discloses the apparatus of claim 1, wherein the at least one processing device is further configured to:
save the modified captured image frame as part of a sequence of previous visually-enhanced image frames; save the lightness model as part of a sequence of previous lightness models (Toizumi’s paragraph [0053]: The storage unit 210 is a storage medium for storing various types of data and the like related to a pixel value correction (image) process and a colorization prediction model (model) learning process); and
process an additional image frame in the sequence of image frames based on the sequence of previous visually-enhanced image frames and the sequence of previous lightness models (Lukac’s paragraph [0050]: At block 508, the process continues with enhancement of the face model and/or the scene model 112 to further improve performance of model merging 510... super-resolution imaging enhances the resolution of the face/scene model to produce the best possible image quality and user experience; paragraph [0051]: block 510, the method 500 completes with merging the face model and/or the scene model 112 of the image/video data 110 with the current face/scene model baseline. Thus, the face/scene model is updated in each iteration and/or frame to increase its accuracy and information contents; paragraph [0054]: the iterative process of enhancing the baseline model continues for each subsequent frame until the scene change requires establishing a new baseline model).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lukac’s to use lightness model as taught by Toizumi, to generate correct luminance value for each pixel.
Regarding claim 8, Lukac as modified by Toizumi discloses the apparatus of claim 1, wherein the at least one processing device is further configured to:
apply at least one predefined lightness model to one or more initial captured image frames in the sequence in order to generate at least one of the one or more previous visually-enhanced image frames (Toizumi’s paragraph [0081]: the correction parameter optimization unit 350 receives a processed monochrome image MLN that has been subjected to a predetermined luminance processing, and the luminance prediction model of the previous stage of the colorization prediction model is trained by machine learning using training data in which a color image CL corresponding to the aforementioned monochrome image MLN is used as an output); and
use the at least one predefined lightness model as at least one of the one or more previous lightness models (Lukac’s paragraph [0054]: the iterative process of enhancing the baseline model continues for each subsequent frame until the scene change requires establishing a new baseline model; paragraph [0051]: block 510, the method 500 completes with merging the face model and/or the scene model 112 of the image/video data 110 with the current face/scene model baseline. Thus, the face/scene model is updated in each iteration and/or frame to increase its accuracy and information contents).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lukac’s to use lightness model as taught by Toizumi, to generate correct luminance value for each pixel.
Claim 9 recites the functions of the apparatus recited in claim 1 as method steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 1 applies to the method steps of claim 9.
Claim 12 recites the functions of the apparatus recited in claim 5 as method steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 5 applies to the method steps of claim 12.
Claim 14 recites the functions of the apparatus recited in claim 7 as method steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 7 applies to the method steps of claim 14.
Claim 15 recites the functions of the apparatus recited in claim 8 as method steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 8 applies to the method steps of claim 15.
Claim 16 recites the functions of the apparatus recited in claim 1 as medium steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 1 applies to the medium steps of claim 16.
Claim 20 recites the functions of the apparatus recited in claim 8 as medium steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 8 applies to the medium steps of claim 20.
Claim 2-4, 10-11 and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Lukac U.S. Patent Application 20250193339 in view of Toizumi U.S. Patent Application 20230059499, and further in view of Li U.S. Patent Application 20190333200.
Regarding claim 2, Lukac as modified by Toizumi discloses generating the lightness model for each of at least some of the captured image frames (Lukac’s paragraph [0041]: At block 404, the IPU 108 generates the face model and/or the scene model 112 based on the image/video data 110; Toizumi’s paragraph [0088]: in S202, the pixel value correction unit 320 corrects the luminance value of the monochrome image M0 by using the correction model including the acquired correction parameters). However, Lukac as modified by Toizumi fails to disclose identifying (i) one or more parameters of a lightness transform function based on one or more properties of the at least one imaging sensor and (ii) an exposure ratio map associated with the captured image frame.
Li discloses identifying (i) one or more parameters of a lightness transform function based on one or more properties of the at least one imaging sensor and (ii) an exposure ratio map associated with the captured image frame (paragraph [0041]: Step 2, determination of the parameters a and b of the camera response equation model: for a specific camera, its model parameters need to be determined; paragraph [0047]: Step 4, estimation of the exposure ratio K: different multi-exposure image sequences will have a great impact on performance of the image fusion algorithm... an exposure ratio set K between each image and the inputted image in the multi-exposure image sequence needs to be reasonably selected).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lukac and Toizumi’s to determine sensor parameters as taught by Li, to enhance a low-illumination image by using a camera response characteristic.
Regarding claim 3, Lukac as modified by Toizumi and Li discloses the apparatus of claim 1, wherein, for each of at least some of the captured image frames, the at least one processing device is configured to apply both the lightness model and a response model in order to generate the modified captured image frame, the response model associated with the at least one imaging sensor (Li’s paragraph [0007]: selecting the exposure model and obtaining the camera response equation model corresponding to the exposure model, estimating the exposure ratio of the multi-exposure image sequence to be generated to the original image and generating the multi-exposure image sequence by using the exposure model; Toizumi’s paragraph [0088]: in S202, the pixel value correction unit 320 corrects the luminance value of the monochrome image M0 by using the correction model including the acquired correction parameters and thereby generates the monochrome image M1).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lukac and Toizumi’s to determine sensor parameters as taught by Li, to enhance a low-illumination image by using a camera response characteristic.
Regarding claim 4, Lukac as modified by Toizumi and Li discloses the apparatus of claim 3, wherein,
the at least one processing device is further configured to generate the response model; and to generate the response model (Li’s paragraph [0007]: selecting the exposure model and obtaining the camera response equation model corresponding to the exposure model), the at least one processing device is configured to:
obtain color image frames captured using the at least one imaging sensor, different color image frames captured using different exposure settings (Lukac’s paragraph [0023]: the image video/data 110 includes the scene image 103 in the form of raw image data captured by the sensor 102; paragraph [0046]: depending on settings in the CCM 106, the IPU 108 and/or the ISP 104 will apply skin filters, face and/or eye enhancements, edge and texture enhancement, noise adders, dithering, background blur, and color and lighting effects); and
identify one or more parameters of the response model based on the color image frames (Li’s paragraph [0041]: Step 2, determination of the parameters a and b of the camera response equation model: for a specific camera, its model parameters need to be determined; paragraph [0047]: Step 4, estimation of the exposure ratio K: different multi-exposure image sequences will have a great impact on performance of the image fusion algorithm... an exposure ratio set K between each image and the inputted image in the multi-exposure image sequence needs to be reasonably selected; Toizumi’s paragraph [0191]: a parameter indicating how close a color is made to the “preferred color” is defined as t(−1≤t≤1)).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lukac and Toizumi’s to determine sensor parameters as taught by Li, to enhance a low-illumination image by using a camera response characteristic.
Claim 10 recites the functions of the apparatus recited in claim 2 as method steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 2 applies to the method steps of claim 10.
Regarding claim 11, Lukac as modified by Toizumi and Li discloses the method of claim 9, wherein:
for each of at least some of the captured image frames, both the lightness model and a response model are applied in order to generate the modified captured image frame, the response model associated with the at least one imaging sensor (Li’s paragraph [0007]: selecting the exposure model and obtaining the camera response equation model corresponding to the exposure model, estimating the exposure ratio of the multi-exposure image sequence to be generated to the original image and generating the multi-exposure image sequence by using the exposure model; Toizumi’s paragraph [0088]: in S202, the pixel value correction unit 320 corrects the luminance value of the monochrome image M0 by using the correction model including the acquired correction parameters and thereby generates the monochrome image M1); and
the response model is generated by:
obtaining color image frames captured using the at least one imaging sensor, different color image frames captured using different exposure settings (Lukac’s paragraph [0023]: the image video/data 110 includes the scene image 103 in the form of raw image data captured by the sensor 102; paragraph [0046]: depending on settings in the CCM 106, the IPU 108 and/or the ISP 104 will apply skin filters, face and/or eye enhancements, edge and texture enhancement, noise adders, dithering, background blur, and color and lighting effects); and
identifying one or more parameters of the response model based on the color image frames (Li’s paragraph [0041]: Step 2, determination of the parameters a and b of the camera response equation model: for a specific camera, its model parameters need to be determined; paragraph [0047]: Step 4, estimation of the exposure ratio K: different multi-exposure image sequences will have a great impact on performance of the image fusion algorithm... an exposure ratio set K between each image and the inputted image in the multi-exposure image sequence needs to be reasonably selected; Toizumi’s paragraph [0191]: a parameter indicating how close a color is made to the “preferred color” is defined as t(−1≤t≤1)).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lukac and Toizumi’s to determine sensor parameters as taught by Li, to enhance a low-illumination image by using a camera response characteristic.
Claim 17 recites the functions of the apparatus recited in claim 2 as medium steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 2 applies to the medium steps of claim 17.
Claim 18 recites the functions of the method recited in claim 11 as medium steps. Accordingly, the mapping of the prior art to the corresponding functions of the method in claim 11 applies to the medium steps of claim 18.
Claim 6, 13 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Lukac U.S. Patent Application 20250193339 in view of Toizumi U.S. Patent Application 20230059499, and further in view of Ghanem U.S. Patent Application 20130335635.
Regarding claim 6, Lukac as modified by Toizumi discloses captured image frame and the one or more previous visually-enhanced image frames (Lukac's paragraph [0023]: the image video/data 110 includes the scene image 103 in the form of raw image data captured by the sensor 102; paragraph [0050]: super-resolution imaging enhances the resolution of the face/scene model to produce the best possible image quality and user experience), and determining one or more parameters of the lightness model (Toizumi’s paragraph [0081]: The luminance prediction model predicts correction parameters related to the luminance distribution of the monochrome image M... the correction parameter optimization unit 350 receives a processed monochrome image MLN that has been subjected to a predetermined luminance processing, and the luminance prediction model of the previous stage of the colorization prediction model is trained by machine learning using training data in which a color image CL corresponding to the aforementioned monochrome image MLN is used as an output). However, Lukac as modified by Toizumi fails to disclose performing registration between the image frames to obtain pixel correspondences between the image frames.
Ghanem discloses performing registration between the image frames to obtain pixel correspondences between the image frames (paragraph [0041]: Labeling module 216 identifies pixel pairs between a frame of each video sequence and the reference image, and can identify these pixel pairs in various manners such as automatically based on various rules or criteria... Labeling module 216 can provide these pixel correspondences to frame to reference image registration module 218 in various manners, such as passing the pixel correspondences as a parameter; paragraph [0081]: The multiple frames are registered to spatially align each of the multiple frames to a reference image (act 304)).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lukac and Toizumi’s to perform registration as taught by Ghanem, to track and generate object and trajectories.
Claim 13 recites the functions of the apparatus recited in claim 6 as method steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 6 applies to the method steps of claim 13.
Claim 19 recites the functions of the apparatus recited in claim 6 as medium steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 6 applies to the medium steps of claim 19.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Yi Yang whose telephone number is (571)272-9589. The examiner can normally be reached on Monday-Friday 9:00 AM-6:00 PM EST.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached on 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/YI YANG/
Primary Examiner, Art Unit 2616
