DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Prior arts cited in this office action:
Wang et al. (CN 113066017 A, hereinafter “Wang”)
Hui Zeng et al. (Learning Image-Adaptive 3D Lookup Tables for High Performance Photo Enhancement in Real-time, Aug. 2015, hereinafter “Zeng”)
Mu et al. (CN 118608386 A, hereinafter “Mu”)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-16, 18-21 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (CN 113066017 A, hereinafter “Wang”) in view of Hui Zeng et al. (Learning Image-Adaptive 3D Lookup Tables for High Performance Photo Enhancement in Real-time, Aug. 2015, hereinafter “Zeng”) and in view of Mu et al. (CN 118608386 A, hereinafter “Mu”).
Regarding claims 1, 16 and 18:
Wang teaches a model training method for image processing (Wang [0005], where Wang teaches a model training method and device, for obtaining the target lookup table according to the pixel type of each pixel point in the input image and the constructed space sensing three-dimensional lookup table, and realizing the enhancement of the input image based on the target lookup table), comprising:
obtaining a picture training sample, and obtaining a ground truth picture corresponding to
the picture training sample (Wang [0023]-[0025] , [0183]-[0185], where Wang teaches for example, supposing the training image is a blurred image, then the real image is a real image corresponding to the shot, each training image has a corresponding real image, for comparing with the output enhanced image, and adjusting each parameter of the model through the loss function until the model is converged);
inputting the picture training sample into a three-dimensional color look-up model to obtain a model predicted picture, and performing loss calculation on the model predicted picture and the ground truth picture to obtain a loss calculation result (Wang [0023]-[0025], [034]-[0142], [0183]-0188], where Wang teaches further based on the first classification information and space sensing three-dimensional lookup table to obtain a target lookup table; the spatial perceptual three-dimensional lookup table is used for representing input pixel information (can be pixel information on any one image, is a mapping) and output pixel information between the mapping relationship, the input pixel information comprises the pixel value of the input pixel point; the position of the input pixel point in the image and the pixel type of the input pixel point (common three-dimensional sensing lookup table only comprises the pixel value of the input pixel point); and
adjusting the three-dimensional color look-up model according to the loss calculation result to obtain a target image processing model, wherein the target image processing model is
configured to perform image processing on an image to be processed to obtain an image
processing result corresponding to the image to be processed (Wang [0023]-[0025], [0084]-86, [0134]-[0160], [0175]-[0189], where Wang teaches For example, supposing the training image is a blurred image, then the real image is a real image corresponding to the shot, each training image has a corresponding real image, for comparing with the output enhanced image, and adjusting each parameter of the model through the loss function until the model is converged).
Wang did not use the word obtaining a ground truth picture corresponding to
the picture training sample. However, Wang teaches each training image has a corresponding real image, for comparing with the output enhanced image, and adjusting each parameter of the model. Furthermore, Zang teaches a learning image-adaptive lookup tables for high performance photo enhancement in real-time wherein multiple basis 3D LUTs and a small convolutional neural network (CNN) are used simultaneously in an end-to-end manner. The small CNN works on the down-sampled version of the input image to predict content-dependent weights to fuse the multiple basis 3D LUTs into an image-adaptive one, which is employed to transform the color and tone of source images efficiently. The training sample could contains, for example, 5,000 image pairs with human-retouched groundtruth, fig. 1. Given the input/output image pairs, they extracted a set of handcrafted features from them, and trained a regression model to predict the user’s adjustments (Zheng page 1 left paragraph, section 3, 3D LUT and trilinear interpolation). Mu further teaches The embodiment is illustrated by combining with FIG. 1, in the figure, the solid line orange frame represents the parameter needing network model training, the dotted line gray frame represents the numerical calculation, IGLM is the image-guided learnable network module, DREN is the shared depth multi-scale enhanced network, LGF is the learnable guide filter network, Upsampling network represents upper sampling network, interpolation operation adopts three-dimensional interpolation calculation mode, Guided Filter adopts the guide filtering calculation mode in step three, The linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear-linear The ultra-high definition conversion method based on deep learning is based on 3DLUT and neural network theory, the input image with 3DLUT table parameter interpolation operation to obtain three-channel characteristic image, and through IGLM model obtaining three-channel parameter; respectively obtaining the characteristic enhanced image by the three-channel characteristic image through DREN network, and then adding the characteristic enhanced image with the three-channel parameter multiplicative property to obtain the fusion image; finally, obtaining the final super-high definition image after the fusion image and the input image pass through the learnable guide filter network. (Mu [0030]-[0035], fig. 1)
Therefore, taking the teachings of Wang and Zeng and Mu as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the application to use a plurality of basic models to obtain basic lookup tables and weight model to obtain the final weight for the enhanced model using training image including the real images or annotated images in this context is considered the ground truth picture that corresponds to the input training sample images, in order to obtain a better image enhancement model able to generate better enhanced image from low quality images.
Regarding claims 2, 7 and 20:
Wang further discloses (see citation above): according to the first classification information (equivalent to the weight values), performing weighted summation on the M sub-three-dimension lookup tables (equivalent to the plurality of basic lookup tables) to obtain the target lookup table, wherein, the target lookup table may be obtained specifically on the basis of the first classification information, the second classification information, and a spatial-aware three-dimension lookup table, and the input pixel information also includes the image category of the image to which the input pixel information belongs; and the spatial- aware three-dimension lookup table is constructed by TxM sub-three-dimension lookup tables (equivalent
to the plurality of second derivative models), wherein T is the number of the image categories, M is the number of the pixel categories, and a sub-three-dimensional lookup table corresponds to an image category and a pixel category (Wang [0005], [0023]-[0025], [0084]-86, [0134]-[0160], [0175]-[0189]; see also Zheng page 1 left paragraph, section 3, 3D LUT and trilinear interpolation and Mu [0030]-[0035], fig. 1).
Regarding claim 3 and 21:
Wang in view of Zeng and in view of Mu teaches wherein a linear combination relationship exists between the one basic look-up model and the first derived model in the first optimization model; wherein adjusting the three-dimensional color look-up model according to the loss calculation result to obtain the target image processing model, comprises: in response to that the linear combination relationship exists between the one basic look-up model and the first derived model in the first optimization model, according to the loss calculation result, adjusting the weight values which correspond to the plurality of basic look-up models in the first derived model, respectively, and the basic color mapping relationships corresponding to the basic look-up models in the first derived model, to obtain the target image processing model (Wang [0252]-[0266]; Zheng page 1 left paragraph, section 3; Mu [0030]-[0035], fig.).
Regarding claim 4:
Wang in view of Zeng and in view of Mu teaches wherein a product combination relationship exists between the one basic look-up model and the first derived model in the first optimization model;
wherein adjusting the three-dimensional color look-up model according to the loss
calculation result to obtain the target image processing model, comprises:
in response to that the product combination relationship exists between the one basic
look-up model and the first derived model in the first optimization model, adjusting the weight
values which correspond to the plurality of basic look-up models in the first derived model,
respectively, and a plurality of basic color mapping relationships corresponding to the plurality
of basic look-up models in the first derived model, to obtain a training result corresponding to
the first derived model; and when the training result meets a training end condition, training the first optimization model according to the loss calculation result to obtain the target image processing model (Wang [0252]-[0266]; Zheng page 1 left paragraph, section 3; Mu [0030]-[0035], fig.1).
Regarding claim 5:
Wang in view of Zeng and in view of Mu teaches Wherein the weight model comprises a picture size fixing layer, a plurality of sampling layers and an output layer which are connected in order, the picture size fixing layer is used to fix a size of the picture training sample, the sampling layers are used to extract the picture feature corresponding to the picture training sample, and the output layer is used to determine, according to the picture feature, the weight values which correspond to the plurality of basic look-up models in the first derived model, respectively (Wang [0100]-[0103], [0252]-[0266]; Zheng page 1 left paragraph, section 3; Mu [0030]-[0035], fig.1).
Regarding claim 6:
Wang in view of Zeng and in view of Mu teaches Wherein the three-dimensional color look-up model comprises a combined model, and the combined model comprises two first optimization models; wherein a combination relationship of one of the first optimization models in the combined model is a linear combination relationship, and a combination relationship of the other one of the first optimization models in the combined model is a product combination relationship (Wang [0100]-[0103], [0168]-[0174], [0252]-[0266]; Zheng page 1 left paragraph, section).
Regarding claim 8:
Wang in view of Zeng and in view of Mu teaches Wherein the three-dimensional color look-up model comprises a combined model, and the combined model comprises two second optimization models; wherein one of the second optimization models in the combined model has a linear combination relationship, and the other one of the second optimization models in the combined model has a product combination relationship (Wang [0100]-[0103], [0168]-[0174], [0252]-[0266]; Zheng page 1 left paragraph, section; Mu [0030]-[0035], fig.1).
Regarding claim 9:
Wang in view of Zeng and in view of Mu teaches wherein the combined model comprises a combination of one first optimization model
and one second optimization model, the first optimization model comprises one basic look-up
model and a first derived model, and the first derived model comprises a weight model and a
plurality of basic look-up models;
in response to that the first optimization model is a linear combined model of the one basic
look-up model and the first derived model, the second optimization model being a product
combined model of the one basic look-up model and a plurality of second derived models; and
in response to that the first optimization model is a product combined model of the one
basic look-up model and the first derived model, the second optimization model being a linear
combined model of the one basic look-up model and the plurality of second derived models (Wang [0100]-[0103], [0168]-[0174], [0252]-[0266]; Zheng page 1 left paragraph, section wherein the combination can be done either by linearly/addition or multiplication/product, etc. by one of ordinary skill in the art; Mu [0030]-[0035], fig.1).
Regarding claim 10:
Wang in view of Zeng and in view of Mu teaches wherein a linear combination relationship exists between the one basic look-up model and the plurality of second derived models in the second optimization model;
wherein adjusting the three-dimensional color look-up model according to the loss
calculation result to obtain the target image processing model, comprises: in response to that the linear combination relationship exists between the one basic look-up model and the plurality of second derived models in the second optimization model, adjusting the second derived color mapping relationships according to the loss calculation result to obtain the target image processing model (Wang [0100]-[0103], [0168]-[0174], [0252]-[0266]; Zheng page 1 left paragraph, section; Mu [0030]-[0035], fig.1).
Regarding claim 11:
Wang in view of Zeng and in view of Mu teaches wherein a product combination relationship exists between the one basic look-up model and the second derived models in the second optimization model;
wherein adjusting the three-dimensional color look-up model according to the loss
calculation result to obtain the target image processing model, comprises:
in response to that the product combination relationship exists between the one basic
look-up model and the second derived models in the second optimization model,
adjusting the second derived color mapping relationships according to the loss calculation result
to obtain a training result corresponding to the second derived models; and
in response to that the training result meets a training end condition, training the second
optimization model according to the loss calculation result to obtain the target image processing
model (Wang [0100]-[0103], [0168]-[0174], [0252]-[0266]; Zheng page 1 left paragraph, section; Mu [0030]-[0035], fig.1).
Regarding claim 12:
Wang in view of Zeng and in view of Mu teaches wherein one of the second derived models comprises a picture size fixing layer, a matrix transformation layer, a plurality of sampling layers and an output layer which are connected in order, the picture size fixing layer is used to fix a size of the picture training sample, the matrix transformation layer is used to transform a matrix output by the picture size fixing layer, the sampling layers are used to extract the picture feature corresponding to the picture training sample, and the output layer is used to output one of second derived color mapping relationships corresponding to the picture feature (Wang [0100]-[0103], [0168]-[0174], [0252]-[0266]; Zheng page 1 left paragraph, section; Mu [0030]-[0035], fig.1).
Regarding claim 13:
Wang in view of Zeng and in view of Mu teaches wherein the three-dimensional color look-up model comprises one third optimization model or a plurality of third optimization models, and the third optimization models are any one of the basic look-up model, the first optimization model, a second optimization model and the combined model, the second optimization model comprises a combination of a plurality of second derived models and one basic look-up model;
wherein inputting the picture training sample into the three-dimensional color look-up
model to obtain the model predicted picture, comprises: obtain a plurality of down-sampling factors, and sampling the picture training sample according to the plurality of down-sampling factors to obtain a plurality of down-sampling results, wherein the down-sampling factors comprise integer factors; determining at least one up-sampling factor according to a picture processing requirement, and sampling the picture training sample according to the at least one up-sampling factor to obtain at least one up-sampling result, wherein the at least one up-sampling factor comprises a decimal factor;
inputting the down-sampling results into the one third optimization model or the plurality of
third optimization models to obtain first model output results corresponding to the
down-sampling results;
inputting the at least one up-sampling result into the one third optimization model or the
plurality of third optimization models to obtain a second model output corresponding to the at
least one up-sampling result;
comparing a magnitude of the at least one up-sampling factor to obtain a first factor
comparison result, and comparing magnitudes of the down-sampling factors to obtain a second
factor comparison result; and
based on the first factor comparison result and the second factor comparison result,
determining an input-output relationship between the first model output results and the second
model output result, so as to obtain the model predicted picture based on the input-output
relationship (Wang [0021]-[0022] [0100]-[0103], [0168]-[0174], [0252]-[0266]; Zheng page 1 left paragraph, section; Mu [0030]-[0035], fig.1).
Regarding claim 14:
Wang in view of Zeng and in view of Mu teaches wherein the method further includes:
inputting the picture training sample into a model to be learned to obtain a ground truth
picture corresponding to the picture training sample, wherein the model to be learned comprises
any one of the basic look-up model, the first optimization model, the second optimization model,
the third optimization model and an open source model;
inputting the picture training sample into a target optimization model to obtain the model
predicted picture, wherein the target optimization model comprises any one of the basic look-up
model, the first optimization model, the second optimization model and the combined model;
and performing loss calculation on the model predicted picture and the ground truth picture to
adjust the target optimization model according to a loss calculation result to obtain the target
optimization model with a same function as the model to be learned (Wang [0021]-[0025] [0100]-[0103], [0168]-[0174], [0183]-[0185], [0252]-[0266]; Zheng page 1 left paragraph, section).
Regarding claim 15:
Wang in view of Zeng and in view of Mu teaches wherein the three-dimensional color look-up model comprises a basic look-up model;
wherein inputting the picture training sample into the three-dimensional color look-up
model to obtain the model predicted picture, comprises:
inputting the picture training sample into the basic look-up model, wherein the basic
look-up model is used to determine a pixel color value in the picture training sample, and
determine a target pixel color value that has a basic color mapping relationship with the pixel
color value; and
determining a target pixel corresponding to the target pixel color value, and determining a
picture formed by the target pixel as the model predicted picture (Wang [0021]-[0025] [0100]-[0103], [0168]-[0174], [0183]-[0185], [0252]-[0266]; Zheng page 1 left paragraph, section).
Regarding claim 19:
Wang in view of Zeng and in view of Mu teaches an image processing method, comprising:
obtaining an image to be processed and an image processing requirement; and
inputting the image to be processed and the image processing requirement into the target
image processing model according to claim 1 to obtain an image processing result (Wang [0021]-[0025] [0100]-[0103], [0134]-[0137], [0168]-[0174]; Zheng page 1 left paragraph, section; Mu [0030]-[0035], fig.1).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEDNEL CADEAU whose telephone number is (571)270-7843. The examiner can normally be reached Mon-Fri 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chieh Fan can be reached at 571-272-3042. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEDNEL CADEAU/Primary Examiner, Art Unit 2632 February 1, 2026