DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4, 7, 10-12, 14, 17, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al. (US20220004798A1), hereinafter Zhu, in view of Hasegawa et al. (US20230169761A1), hereinafter Hasegawa.
Regarding claim 1, Zhu teaches a processor-implemented method with image restoration, the method comprising: obtaining an input image (Para. 5 see "The method comprises: using at least one processor to perform: obtaining an input image comprising pixels of a first bit depth;"; Para. 15 see "The method comprises: using at least one processor to perform: obtaining an input image to be enhanced;"; Para. 59 see "Process 200 includes the system obtaining (202) an input image with pixels of a first bit-depth. For example, the system may receive an image from an image capture device (e.g., a digital camera).") generating a correction image by converting the input image based on a conversion parameter that converts a pixel value distribution of the input image (Para. 4 see "The techniques described herein provide for transforming image intensity values of an image (e.g., pixel values) using nonlinear techniques. The transformed images can be used for image enhancement (e.g., as a preprocessing step prior to performing image enhancement)."; Para. 15 see "applying a nonlinear transform to pixel intensities of the input image to obtain a transformed input image;"; Para. 48 see "In some embodiments, the image preprocessing system 102 may be configured to apply the nonlinear transform to the image with the surjective mapping such that discrimination among low pixel intensities in the quantized image is greater than discrimination among high pixel intensities. In applying the nonlinear transform, the image preprocessing system 102 may dedicate a larger portion of the range of the second bit depth to low pixel intensities than to high pixel intensities to maintain discrimination among the low pixel intensities.") generating an output image by inputting the correction image to an image restoration model (Para. 11 see "In one embodiment, the image processing pipeline comprises a machine learning model trained using a plurality of quantized images comprising pixels of the second bit depth; and providing the quantized input image to the image processing pipeline comprises providing the quantized input image to the machine learning model to obtain an enhanced output image."; Para. 15 see "generating, using the transformed input image, an input be provided to a trained machine learning model; and providing the generated input to the trained machine learning model to obtain an enhanced output image."; Para. 61 see "In some embodiments, the processor may be configured to execute a machine learning model to enhance the input image. ... In some embodiments, the processor may be a neural processor configured to execute a neural network. In some embodiments, the neural network may be trained to enhance the image. In some embodiments, the neural network may be trained to enhance the image by denoising the image.") and generating an enhanced image by inversely converting the output image based on an inverse conversion parameter corresponding to the conversion parameter (Para. 8 see "In one embodiment, the method further comprises: obtaining, from the image processing pipeline, an output image comprising pixels of the second bit depth; and de-quantizing the output image at least in part by applying a second nonlinear transform to pixel intensities of the output image to generate a de-quantized output image comprising pixels of the first bit depth. In one embodiment, the second nonlinear transform comprises an inverse of the first nonlinear transform."; Para. 51 see "For example, the system may have previously applied nonlinear transform 102 A to an input image, and provided the transformed image to the image processing system 106. The system may then obtain a processed version of the image from the image processing system 106 and apply a nonlinear transform to the processed image (e.g., by applying an inverse of the nonlinear transform 102 A)."; Para. 63 see "The system de-quantizes (210) the output image to generate a de-quantized output image with pixels of the first bit depth. ... In some embodiments, the system may be configured to map the pixel intensities of the second bit depth to pixel intensities of the first bit depth by applying a non-linear transform (e.g., an inverse of the transform used for quantizing the input image) to the pixel intensities of the second bit depth.").
While Zhu teaches generating an output image by inputting the correction image to an image restoration model, Zhu does not teach generating an output image by inputting the correction image to a quantized image restoration model.
However, Hasegawa teaches generating an output image by inputting the correction image to a quantized image restoration model (Para. 3 see "there is also proposed a scheme in which a precision of an arithmetic operation of the CNN is set to be INT8 (8-bit precision)."; Para. 50 see "the CPU 106 controls the image processing unit 104 to input the third image into the first neural network. The first neural network mentioned here is one that is optimally learned to perform noise elimination"; Para. 78 see "the CPU 106 performs quantization in order to convert, into INT8, the parameters of the neural network learned with FP32. ... or quantization may be performed into INT4.").
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhu to incorporate the teachings of Hasegawa to generate the output image by sending the correction image through an image restoration model that uses quantized network parameters. Doing so would predictably reduce memory use and speed up processing by letting the noise removal network run with 8-bit or 4-bit weights instead of full precision weights.
Regarding claim 2, Zhu in view of Hasegawa teaches the method of claim 1.
While Zhu teaches applying a nonlinear intensity map to the input before the enhancement network and applying the inverse of that map to the network output, Zhu does not teach wherein the generating of the correction image comprises generating the correction image by converting the input image with a first gamma correction value and the generating of the enhanced image comprises generating the enhanced image by inversely converting the output image with a second gamma correction value.
However, Hasegawa teaches wherein the generating of the correction image comprises generating the correction image by converting the input image with a first gamma correction value (Para. 116 see "The offset mentioned here is the black level added by the image capturing device 102. In addition, the tone compression process mentioned here may employ various methods. For example, gamma correction may be used."; Para. 132 see "At S 1806, the CPU 106 controls the image processing unit 104 to generate a 49th image by performing a tone compression process on each pixel value of the 48th image. The tone compression process mentioned here may employ various methods. For example, gamma correction may be used.") and the generating of the enhanced image comprises generating the enhanced image by inversely converting the output image with a second gamma correction value (Para. 117 see "At S 1608, the CPU 106 controls the image processing unit 104 to generate a 46th image by performing a tone expansion process on each pixel value of a 45th image output from the neural network. The tone expansion process mentioned here may employ various methods. For example, degamma correction may be used.").
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhu in view of Hasegawa to incorporate the teachings of Hasegawa to convert the input image with a first gamma correction value to make the correction image and convert the network output with a second gamma correction value to make the enhanced image. Doing so would predictably keep more detail in dark areas after processing by compressing pixel values before restoration and expanding them back afterward.
Regarding claim 4, Zhu in view of Hasegawa teaches the method of claim 1.
While Zhu teaches applying a chosen nonlinear intensity map to the pixels of the obtained input image, Zhu does not teach wherein the generating of the correction image further comprises determining the conversion parameter corresponding to the input image.
However, Hasegawa teaches wherein the generating of the correction image further comprises determining the conversion parameter corresponding to the input image (Para. 84 see "Next, at S 805, the CPU 106 controls the image processing unit 104 to calculate the mean of the pixel values for each predetermined area. The predetermined area mentioned here refers to, for example, each of 9 areas of an image divided by 3×3 as illustrated in FIG. 9, with the mean of the pixel values being calculated for each area."; Para. 86 see "At S 808, the CPU 106 controls the image processing unit 104 to calculate the mean over the I-th area, and determines whether or not the value is equal to or larger than 4096. When the mean over the I-th area is equal to or larger than 4096, the CPU 106 advances the process to S 809. At S 809, the CPU 106 sets the parameters of the first neural network recorded in the ROM 105 to the neural network in the image processing unit 104."; Para. 93 see "At S 1007 (FIG. 10 B), CPU 106 determines whether or not the mean of the 27th image is equal to or larger than 4096. Upon determining that the mean of the 27th image is equal to or larger than 4096, the CPU 106 advances the process to S 1008. At step S 1008, the CPU 106 sets the parameters of the first neural network recorded in the ROM 105 to the neural network in the image processing unit 104. Subsequently, at S 1009, the CPU 106 controls the image processing unit 104 to generate a 28th image by clipping an 8-bit segment from a first position of each pixel value of the 27th image.").
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhu in view of Hasegawa to incorporate the teachings of Hasegawa to determine the conversion parameter from the input image when generating the correction image. Doing so would predictably improve restoration of images by matching the intensity conversion to the pixel values of that image instead of using one fixed setting.
Regarding claim 7, Zhu in view of Hasegawa teaches the method of claim 1.
While Zhu teaches obtaining a depth image from a capture device, Zhu does not teach wherein the input image comprises any one or any combination of any two or more of a color filter array (CFA) raw image, an output image, and an intermediate image of an image signal processor (ISP).
However, Hasegawa teaches wherein the input image comprises any one or any combination of any two or more of a color filter array (CFA) raw image, an output image, and an intermediate image of an image signal processor (ISP) (Para. 43 see "the training images to be Bayer array images ... Not only for the Bayer array, the same goes for other arrangements (such as honeycomb structure, X-Trans CMOS sensor filter array, or the like). Bayer array images may be kept in one channel of the Bayer array, or may be separated into each color channel for training images."; Para. 81 see "For demosaicing, a training image may be prepared by preparing a correct image using a three-plate type image capturing device or the like, and re-sampling the correct image using a Bayer array or the like.").
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhu in view of Hasegawa to incorporate the teachings of Hasegawa to use a color filter array raw image as the input image. Doing so would predictably improve image quality by restoring the raw mosaic before it is turned into a full color picture.
Regarding claim 10, Zhu in view of Hasegawa teaches the method of claim 1.
In addition, Zhu teaches a non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform the method of claim 1 (Para. 102 see "The memory 1212 stores programs (e.g., sequences of instructions coded to be executable by the processor 1210) and data during operation of the computer system 1202. Thus, the memory 1212 may be a relatively high performance, volatile, random access memory such as a dynamic random access memory (“DRAM”) or static memory (“SRAM”).").
Regarding claim 20, Zhu teaches a processor-implemented method with image restoration, the method comprising (Para. 5 see "The method comprises: using at least one processor to perform: obtaining an input image comprising pixels of a first bit depth;"; Para. 15 see "The method comprises: using at least one processor to perform: obtaining an input image to be enhanced;"; Para. 59 see "Process 200 includes the system obtaining (202) an input image with pixels of a first bit-depth. For example, the system may receive an image from an image capture device (e.g., a digital camera).") generating, using a conversion parameter, a correction image by modifying pixel values of an input image such that a difference between low pixel values is increased a difference between high pixel values is decreased (Para. 4 see "The techniques described herein provide for transforming image intensity values of an image (e.g., pixel values) using nonlinear techniques. The transformed images can be used for image enhancement (e.g., as a preprocessing step prior to performing image enhancement)."; Para. 15 see "applying a nonlinear transform to pixel intensities of the input image to obtain a transformed input image;"; Para. 48 see "In some embodiments, the image preprocessing system 102 may be configured to apply the nonlinear transform to the image with the surjective mapping such that discrimination among low pixel intensities in the quantized image is greater than discrimination among high pixel intensities. In applying the nonlinear transform, the image preprocessing system 102 may dedicate a larger portion of the range of the second bit depth to low pixel intensities than to high pixel intensities to maintain discrimination among the low pixel intensities.") generating an output image by inputting the correction image to an image restoration model (Para. 11 see "In one embodiment, the image processing pipeline comprises a machine learning model trained using a plurality of quantized images comprising pixels of the second bit depth; and providing the quantized input image to the image processing pipeline comprises providing the quantized input image to the machine learning model to obtain an enhanced output image."; Para. 15 see "generating, using the transformed input image, an input be provided to a trained machine learning model; and providing the generated input to the trained machine learning model to obtain an enhanced output image."; Para. 61 see "In some embodiments, the processor may be configured to execute a machine learning model to enhance the input image. ... In some embodiments, the processor may be a neural processor configured to execute a neural network. In some embodiments, the neural network may be trained to enhance the image. In some embodiments, the neural network may be trained to enhance the image by denoising the image.") and generating an enhanced image by inversely converting the output image based on an inverse conversion parameter corresponding to the conversion parameter (Para. 8 see "In one embodiment, the method further comprises: obtaining, from the image processing pipeline, an output image comprising pixels of the second bit depth; and de-quantizing the output image at least in part by applying a second nonlinear transform to pixel intensities of the output image to generate a de-quantized output image comprising pixels of the first bit depth. In one embodiment, the second nonlinear transform comprises an inverse of the first nonlinear transform."; Para. 51 see "For example, the system may have previously applied nonlinear transform 102 A to an input image, and provided the transformed image to the image processing system 106. The system may then obtain a processed version of the image from the image processing system 106 and apply a nonlinear transform to the processed image (e.g., by applying an inverse of the nonlinear transform 102 A)."; Para. 63 see "The system de-quantizes (210) the output image to generate a de-quantized output image with pixels of the first bit depth. ... In some embodiments, the system may be configured to map the pixel intensities of the second bit depth to pixel intensities of the first bit depth by applying a non-linear transform (e.g., an inverse of the transform used for quantizing the input image) to the pixel intensities of the second bit depth.").
While Zhu teaches generating an output image by inputting the correction image to an image restoration model, Zhu does not teach generating an output image by inputting the correction image to a quantized image restoration model.
However, Hasegawa teaches generating an output image by inputting the correction image to a quantized image restoration model (Para. 3 see "there is also proposed a scheme in which a precision of an arithmetic operation of the CNN is set to be INT8 (8-bit precision)."; Para. 50 see "the CPU 106 controls the image processing unit 104 to input the third image into the first neural network. The first neural network mentioned here is one that is optimally learned to perform noise elimination"; Para. 78 see "the CPU 106 performs quantization in order to convert, into INT8, the parameters of the neural network learned with FP32. ... or quantization may be performed into INT4.").
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhu to incorporate the teachings of Hasegawa to generate the output image by sending the correction image through an image restoration model that uses quantized network parameters. Doing so would predictably reduce memory use and speed up processing by letting the noise removal network run with 8-bit or 4-bit weights instead of full precision weights.
Claim 11 is rejected under the same analysis as claim 1 above.
Claim 12 is rejected under the same analysis as claim 2 above.
Claim 14 is rejected under the same analysis as claim 4 above.
Claim 17 is rejected under the same analysis as claim 7 above.
Claims 3, 13 are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al. (US20220004798A1), hereinafter Zhu, in view of Hasegawa et al. (US20230169761A1), hereinafter Hasegawa, and further in view of Okisu et al. (US20020141005A1), hereinafter Okisu.
Regarding claim 3, Zhu in view of Hasegawa teaches the method of claim 2.
While Zhu teaches applying a nonlinear intensity map to the input before the enhancement network and applying the inverse of that map to the network output, Zhu does not teach wherein the first gamma correction value and the second gamma correction value have a reciprocal relationship.
However, Okisu teaches wherein the first gamma correction value and the second gamma correction value have a reciprocal relationship (Para. 101 see "in the case where the assumed γ value is 2.2, a γ correction process of γ=1/2.2=0.455 is effected on the image data in the digital camera 1. Therefore, by subjecting the image data that has been subjected to such a γ correction process to the inverse γ correction process of γ=2.2, the linearity of the image data is secured.").
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhu in view of Hasegawa to incorporate the teachings of Okisu to use a first and second gamma value with a reciprocal relationship. Doing so would predictably restore brightness after processing by undoing the same gamma conversion that was applied to the input.
Claim 13 is rejected under the same analysis as claim 3 above.
Claims 5, 15 are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al. (US20220004798A1), hereinafter Zhu, in view of Hasegawa et al. (US20230169761A1), hereinafter Hasegawa, and further in view of Chino (US20170213322A1), hereinafter Chino.
Regarding claim 5, Zhu in view of Hasegawa teaches the method of claim 1.
While Zhu teaches applying a chosen nonlinear intensity map to the pixels of the obtained image, Zhu does not teach wherein the generating of the correction image further comprises determining the conversion parameter based on illuminance of the input image.
However, Chino teaches wherein the generating of the correction image further comprises determining the conversion parameter based on illuminance of the input image (Para. 29 see "The illuminance calculation unit 65 calculates the illuminance level of the object and the like being captured, based on at least one of the illuminance data from the illuminance meter 5, the diaphragm value of the diaphragm 2, the gain set by the gain setting unit 61, the shutter speed set by the shutter speed setting unit 62, and the information about the luminance Y from the luminance information generation unit 63. The illuminance information calculated by the illuminance calculation unit 65 is sent to a gamma correction amount calculation unit 67."; Para. 30 see "the gamma correction amount calculation unit 67 calculates the correction amount with respect to the gamma characteristic, in accordance with the illuminance information calculated by the illuminance calculation unit 65, and causes the gamma setting unit 66 to set the gamma characteristic as a result of correction with the correction amount."; Para. 40 see "In step S 1001, the illuminance calculation unit 65 calculates the illuminance of the object and the like in the captured image, based on at least one of the illuminance data, the diaphragm value, the set gain, the set shutter speed, and the luminance value Y as described above").
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhu in view of Hasegawa to incorporate the teachings of Chino to determine the conversion parameter from the illuminance of the input image when generating the correction image. Doing so would predictably improve dark and bright images by matching the intensity conversion to how light or dark that image is instead of using one fixed setting.
Claim 15 is rejected under the same analysis as claim 5 above.
Claims 6, 16 are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al. (US20220004798A1), hereinafter Zhu, in view of Hasegawa et al. (US20230169761A1), hereinafter Hasegawa, and further in view of Mantiuk et al. (WO2021093980A1), hereinafter Mantiuk.
Regarding claim 6, Zhu in view of Hasegawa teaches the method of claim 1.
While Zhu teaches applying a chosen nonlinear intensity map to the pixels of the obtained image, Zhu does not teach wherein the generating of the correction image further comprises determining the conversion parameter based on a specific task target to which the method of image restoration is used.
However, Mantiuk teaches wherein the generating of the correction image further comprises determining the conversion parameter based on a specific task target to which the method of image restoration is used (Para. 38 see "the SNR threshold is further obtained based on a type of the computer vision application and/or a gain parameter of the image sensor."; Para. 45 see "the device is further configured to apply an inverse logarithmic TF to the transferred image data for obtaining image data or video data in a determined format according to the computer vision application."; Para. 74 see "The TF 105 may be a noise-adaptive transfer function determined for a specific computer vision application 140.").
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhu in view of Hasegawa to incorporate the teachings of Mantiuk to determine the conversion parameter from the task the restored image is used for. Doing so would predictably improve the desired result by matching the conversion parameter to the task instead of using one conversion for every use.
Claim 16 is rejected under the same analysis as claim 6 above.
Claims 8-9, 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al. (US20220004798A1), hereinafter Zhu, in view of Hasegawa et al. (US20230169761A1), hereinafter Hasegawa, and further in view of Campbell (US20030185457A1), hereinafter Campbell.
Regarding claim 8, Zhu in view of Hasegawa teaches the method of claim 1.
While Zhu teaches applying a chosen nonlinear intensity map to the pixels of the obtained image, Zhu does not teach wherein the generating of the correction image comprises: obtaining a look-up table (LUT) and generating the correction image corresponding to the input image based on the LUT.
However, Campbell teaches wherein the generating of the correction image comprises: obtaining a look-up table (LUT) (Para. 40 see "This enhancement curve is then used to construct a 1D LUT 48, which may be used to perform image tone enhancement and adjustment 36."; Para. 83 see "the neural net processes the histogram data 114 and creates an enhancement curve Look-Up Table (LUT) with point data that approximates an enhancement curve.") and generating the correction image corresponding to the input image based on the LUT (Para. 34 see "Once luminance has been separated, the luminance channel may be modified 36 in many ways. In the illustrated embodiments, one-dimensional look-up table (1D LUT) is used. The 1D LUT is a discrete representation of the enhancement curve."; Para. 83 see "The LUT may then be applied to an image 116 to enhance the image to more closely conform to observer luminance preferences."; Para. 85 see "the neural net may create an enhancement curve LUT 130 that can be used to enhance the subject image so that it more closely complies with observer luminance preferences 132. The end result is an image 134 that is preferred by an observer.").
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhu in view of Hasegawa to incorporate the teachings of Campbell to obtain a look-up table and generate the correction image from that table. Doing so would predictably improve image quality by mapping each input brightness to a preferred output brightness stored for that image.
Regarding claim 9, Zhu in view of Hasegawa and further in view of Campbell teaches the method of claim 8.
While Zhu teaches applying a chosen nonlinear intensity map to the pixels of the obtained image, Zhu does not teach wherein the obtaining of the LUT comprises generating the LUT corresponding to the input image by inputting the input image to an LUT generative model.
However, Campbell teaches wherein the obtaining of the LUT comprises generating the LUT corresponding to the input image by inputting the input image to an LUT generative model (Para. 40 see "The luminance channel histogram, in the illustrated embodiments of FIG. 3, serves as the input to a neural network 46. The neural network 46 analyzes the histogram data and produces an enhancement curve shape. This enhancement curve is then used to construct a 1D LUT 48, which may be used to perform image tone enhancement and adjustment 36."; Para. 83 see "the neural net processes the histogram data 114 and creates an enhancement curve Look-Up Table (LUT)"; Para. 85 see "A subject image luminance histogram is then created 126 and input to the neural net 128 for analysis. Once analysis is complete, the neural net may create an enhancement curve LUT 130 that can be used to enhance the subject image").
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhu in view of Hasegawa and further in view of Campbell to incorporate the teachings of Campbell to generate the look-up table for the input image by feeding that image to a look-up table generative model. Doing so would predictably improve tone on both dark and bright images by building a table that matches this image instead of using one stored table for every photo.
Claim 18 is rejected under the same analysis as claim 8 above.
Claim 19 is rejected under the same analysis as claim 9 above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Lannes et al. (US20180158177A1) discloses a system to process images for facial recognition using a neural network that carries out a nonlinear transformation such as gamma correction.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEXANDER VAUGHN whose telephone number is (571) 272-5253. The examiner can normally be reached M-F 11am-7pm (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JENNIFER MEHMOOD can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALEXANDER VAUGHN/
Examiner, Art Unit 2675
/ATIBA O FITZPATRICK/Primary Examiner, Art Unit 2677
Prosecution Insights