Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12/2/2025 has been entered.
Response to Amendment
The Amendment filed December 2 2025 has been entered and considered. Claims 1, 9, 14, 23 and 25 have been amended. Claims 2-4, 6, 11, 22, and 24 were previously canceled. In light of the amendment the prior art rejections of claims 1, 23, and 25 are withdrawn as moot.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 5, 7-10, 12-21, 23, and 25 are rejected under 35 U.S.C. 103 as being unpatentable over Afra (previously cited) in view of Thomas et al. (NPL, previously cited), further in view of Woods et al. (NPL, previously cited), as evidenced by Reinhard et al. (NPL, previously cited), further in view of Su et al. (US Patent Pub. No. 20210150812 A1, published 5/20/2021).
Regarding claim 1, Afra teaches an image processing apparatus comprising: one or more processors; and a memory storing instructions which, when the instructions are executed by the one or more processors, cause the image processing apparatus to function (Para. 35, “In one embodiment the memory device 120 can operate as system memory for the system 100, to store data 122 and instructions 121 for use when the one or more processors 102 executes an application or process.”) as a plurality of units: (1) an obtainment unit configured to obtain a brightness of first image data; (2) a tone compression unit configured to, in accordance with the brightness obtained by the obtainment unit, compress tones of the first image data using a selected characteristic among a plurality of characteristics in which the lower the brightness, more tones are allocated (Para. 234, “The present design significantly downsamples (e.g., change resolution, downsampling by an integer factor) the image and computes an auto-exposure value from an average log luminance. The image is then pre-scaled with this auto-exposure value and then tone mapped with a gamma-corrected log function (e.g., (log.sub.2(x+1)/16).sup.1/2.2)” and Para. 239 “At operation 1906, the method includes applying a tone mapping operator (e.g., log function) to the image and scaling the log function to generate a tone mapped image. The log function compresses a range of values significantly for the image”, wherein 2.2 gamma correction allocates more tones to lower brightness levels); (3) a processing unit configured to, by applying a neural network that performs predetermined image processing on image data whose tones have been compressed by the tone compression unit, output image data on which the predetermined image processing has been performed (Para. 239, “At operation 1912, the neural network processes (e.g., denoising algorithm, anti-aliasing algorithm, super resolution, demosaicing, etc.) the tone mapped image to generate an output of the neural network”); (4) a tone decompression unit configured to decompress the tones of the image data on which the predetermined image processing has been performed, wherein the image data is decompressed by using a characteristic that corresponds to the characteristic used in the tone compression unit (Para. 239, “. At operation 1914, the method applies the inverse tone mapping operator and then the inverse exposure scale to this output of the neural network to generate the final output image.”); and (5), in accordance with the characteristic used in the tone compression unit of brightness obtained by the obtainment unit, select a set of parameters of the neural network that performs the predetermined image processing from among a plurality of sets of parameters of the neural network that has been trained in advance (Para. 244, “At operation 2008, the method includes applying a gamma correction to the tone mapped image to generate a gamma corrected tone mapped image and generates a tone mapping curve that is more perceptually linear. At operation 2010, the method includes providing the tone mapped image as input for a neural network (e.g., CNN). At operation 2012, the neural network processes (e.g., denoising algorithm, anti-aliasing algorithm, super resolution, demosaicing, etc.) the tone mapped image to generate an output of the neural network.”).
Afra also teaches the use of different bit precision depending on the implementation of the neural network (Para. 67), however the reference does not explicitly teach wherein the number of bits that represent a pixel value in the neural network is smaller than the number of bits that represent a pixel value of the first image data. They do not explicitly teach the use of a selection unit to select one neural network from among a plurality of neural networks.
Thomas teaches wherein the number of bits that represent a pixel value in the neural network is smaller than the number of bits that represent a pixel value of the first image data (Pg. 3, Section 2.3 “Quantized Networks”, goes in depth about different techniques for quantizing a neural network. Quantizing a neural network is done by representing pixel values with a lower number of bits within the neural network than the number of bits used to represent the input image).
Woods teaches to use a selected characteristic among a plurality of characteristics in which the lower the brightness, the more tones are allocated (Fig. 3.6, Woods discloses the well-known method of using different steps for gamma correction values)
PNG
media_image1.png
292
502
media_image1.png
Greyscale
.
Su teaches to use a selection unit configured to select one neural network from among a plurality of neural networks, each of which has been trained in advance to perform the predetermined image processing on the image data (Para. 28, “The encoder selects a neural network model from the variety of NN models to determine an output image which approximates the second image based on the first image and the second image.”; Para. 115, “Given these inputs, in step 510, the mapping processor decides on which neural network (NN) model to select. As described before, the mapping processor may select among a variety of NN models, including (but not necessarily limited to): a global mapping model, a local mapping model, a mapping using multiple grades, or a combination of the above.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Afra to incorporate the teachings of Thomas, Woods, and Su to include that the number of bits that represent a pixel value in the neural network is smaller than the number of bits that represent a pixel value of the first image data as well as a selection unit to select one neural network from among a plurality of neural networks which have been trained in advance for a different respective characteristic. Neural networks are often quantized to use reduced-precision arithmetic, as it greatly improves their storage and computational costs, as acknowledged by Thomas (Pg. 1, Col. 1). Afra discloses the ability to change the bit precision of the neural network depending on the workload (Para. 67). Implementing the teachings of Thomas to the system of Afra would allow it to perform the same high quality tone mapping while using a lower precision on a wider range of workloads, improving efficiency and cutting computation costs. It would also be obvious to extend Afra’s teaching of training a neural network for 2.2 gamma correction to additionally account for other gamma values in view of Woods teachings that different gamma values yield systematically different image characteristics. In view of Su’s teaching of using a selection unit to swap between specialized neural networks to handle different dynamic range mappings, it would be a predictable application of this analogous architecture to swap networks based on a specific gamma characteristic. A skilled person would have recognized that training a network specifically for each value would improve image quality across a broader set of inputs, avoid the performance degradation that occurs when a single model is forced to process distinct curves, and ensure that the most appropriate model is used for the specific compression characteristic of the input data.
Regarding claim 5, Afra as modified teaches all of the elements of claim 1, as stated above, as well as wherein the obtainment unit obtains the brightness of the first image data that has been captured at a first time and a brightness of second image data that has been captured at a second time that is after the first time, and the tone compression unit compresses tones of the second image data (Para. 235, “The image is then pre-scaled with this auto-exposure value and then tone mapped with a gamma-corrected log function (e.g., (log.sub.2(x+1)/16).sup.1/2.2). This transforms the image into a roughly perceptually linear and normalized (between [0, 1]) color space, without destroying the dynamic range of the original image.”).
Afra as modified does not explicitly disclose using, among the plurality of characteristics that correspond to a brightness of image data and differ stepwise, a second characteristic that is adjacent to a first characteristic that corresponds to the brightness of the first image data.
Woods teaches to use, among the plurality of characteristics that correspond to a brightness of image data and differ stepwise, a second characteristic that is adjacent to a first characteristic that corresponds to the brightness of the first image data (Fig. 3.6, Woods discloses the well-known method of using different steps for gamma correction values.
PNG
media_image1.png
292
502
media_image1.png
Greyscale
Utilizing different steps of gamma values is a well-known technique in the art as disclosed by Woods. Afra as modified also includes a tone mapping method disclosed by Reinhard which teaches dynamically modifying tone mapping characteristics depending on local luminance regions (See evidentiary reference Reinhard, Section 3.2), reinforcing that applying different tone mapping levels based on image brightness is an obvious modification.)
Regarding claim 7, Afra as modified teaches all of the elements of claim 5, as stated above, as well as wherein the tone decompression unit decompresses tones of image data using a characteristic that corresponds to the second characteristic among a plurality of characteristics that are for decompressing tones and differ stepwise. (Para. 234, “The neural network receives this tone mapped image as input, and to determine the final result, the inverse tone mapping operator and the inverse scale are applied to the output of the network.”).
Regarding claim 8, Afra as modified teaches all of the elements of claim 2, as stated above, as well as wherein the obtainment unit obtains a brightness of a selected region among a plurality of regions of the first image data, and wherein the tone compression unit compresses the tones of the first image data (Para. 232, “Another, more recent approach is to apply the well-known Reinhard tone mapping operator with gamma correction”, See evidentiary reference Reinhard, Section 3.2 discusses performing tone mapping on local regions) using a third characteristic that corresponds to the brightness of the selected region among the plurality of characteristics (Woods; Fig. 3.6, Woods discloses the well-known method of using different steps for gamma correction values)
PNG
media_image1.png
292
502
media_image1.png
Greyscale
Regarding claim 9, Afra as modified teaches all of the elements of claim 8, as stated above, as well as wherein the selection unit selects the neural network in accordance with the third characteristic (Su; Para. 115, “Given these inputs, in step 510, the mapping processor decides on which neural network (NN) model to select. As described before, the mapping processor may select among a variety of NN models, including (but not necessarily limited to): a global mapping model, a local mapping model, a mapping using multiple grades, or a combination of the above.”), and wherein the processing unit applies the neural network using a set of parameters that corresponds to the third characteristic that corresponds to the brightness of the selected region among a plurality of sets of parameters of the neural network that are associated with the plurality of characteristics (Para. 200, “For example, the input to a convolution layer can be a multidimensional array of data that defines the various color components of an input image. The convolution kernel can be a multidimensional array of parameters, where the parameters are adapted by the training process for the neural network.”, many different parameters are used which depend on the input as well as the tone compression).
Regarding claim 10, Afra as modified teaches all of the elements of claim 8, as stated above, as well as wherein the selected region is a region in which a brightness for a respective region is lower than a predetermined threshold among a plurality of regions of the first image data (Para. 232, “Another, more recent approach is to apply the well-known Reinhard tone mapping operator with gamma correction”, (See evidentiary reference Reinhard, Section 3.2 which discloses contrast thresholding of regions, which is very similar to brightness); (Woods; Fig. 10.1, Pg. 702, “Region-based segmentation approaches in the second category are based on partitioning an image into regions that are similar according to a set of predefined criteria”,
PNG
media_image2.png
327
600
media_image2.png
Greyscale
It is well-known in the art to perform region-based segmentation which relies on a specific characteristic, with Woods also disclosing brightness (intensity)).
Regarding claim 12, Afra as modified teaches all of the elements of claim 1, as stated above, as well as wherein in accordance with a predetermined setting, the tone compression unit compresses the tones of the first image data using a selected characteristic among the plurality of characteristics in which the lower the brightness, more tones are allocated (Para. 232, “Another, more recent approach is to apply the well-known Reinhard tone mapping operator with gamma correction ((x/(x+1)).sup.1/2.2)”, The gamma correction value set is predetermined, Woods teaches using different gamma values, and the reasons for combining the references are the same as those discussed above in conjunction with claim 1).
Regarding claim 13, Afra as modified teaches all of the elements of claim 12, as stated above, as well as wherein for the predetermined setting whose number of bits that represent a pixel value of image data to be processed is greater, the tone compression unit uses a characteristic that more tones are allocated in a predetermined low luminance region (Para. 232, “Another, more recent approach is to apply the well-known Reinhard tone mapping operator with gamma correction ((x/(x+1)).sup.1/2.2)”, The gamma correction disclosed by Afra allocates more tones to low luminance regions. For an image with greater bit depth, the low luminance regions would likewise be allocated more tones ).
Regarding claim 14, Afra as modified teaches all of the elements of claim 12, as stated above, as well as wherein in accordance with the predetermined setting, the selection unit selects the neural network in accordance with the predetermined setting (Su; Para. 115, “Given these inputs, in step 510, the mapping processor decides on which neural network (NN) model to select. As described before, the mapping processor may select among a variety of NN models, including (but not necessarily limited to): a global mapping model, a local mapping model, a mapping using multiple grades, or a combination of the above.”), and wherein the processing unit applies the neural network using a selected set of parameters among the plurality of sets of parameters of the neural network that has been trained in advance (Para. 200, “For example, the input to a convolution layer can be a multidimensional array of data that defines the various color components of an input image. The convolution kernel can be a multidimensional array of parameters, where the parameters are adapted by the training process for the neural network.”, Many different parameters are used which depend on the input as well as the tone compression. This indicates a predetermined setting in which the neural network training is based. Also see analysis of claim 1.).
Regarding claim 15, Afra as modified teaches all of the elements of claim 12, as stated above, as well as wherein in accordance with the predetermined setting, the tone decompression unit decompresses tones of image data using a selected characteristics among a plurality of characteristics for decompressing tones (Para. 234, “The neural network receives this tone mapped image as input, and to determine the final result, the inverse tone mapping operator and the inverse scale are applied to the output of the network.”, A predetermined setting is used to perform compression, the inverse is done according to that setting as well).
Regarding claim 16, Afra as modified teaches all of the elements of claim 12, as stated above, as well as wherein the predetermined setting is a setting for image data to be outputted from the image processing apparatus (Para. 234, “The image is then pre-scaled with this auto-exposure value and then tone mapped with a gamma-corrected log function (e.g., (log.sub.2(x+1)/16).sup.1/2.2)… the inverse tone mapping operator and the inverse scale are applied to the output of the network.”, One example of a predetermined setting, a gamma-corrected log function).
Regarding claim 17, Afra as modified teaches all of the elements of claim 16, as stated above, as well as wherein the setting for image data to be outputted from the image processing apparatus includes any of a characteristic to be used for tone compression, a characteristic to be used for tone decompression, the number of tones of the image data to be outputted, and the number of bits that represent a pixel value of the image data to be outputted (Para. 234, “The image is then pre-scaled with this auto-exposure value and then tone mapped with a gamma-corrected log function (e.g., (log.sub.2(x+1)/16).sup.1/2.2).” The Examiner notes that only one of the claimed alternatives is required to be taught).
Regarding claim 18, Afra as modified teaches all of the elements of claim 12, as stated above, as well as wherein the predetermined setting is a setting for image data to be inputted to the neural network (Para. 234, “The image is then pre-scaled with this auto-exposure value and then tone mapped with a gamma-corrected log function (e.g., (log.sub.2(x+1)/16).sup.1/2.2).”).
Regarding claim 19, Afra as modified teaches all of the elements of claim 18, as stated above, as well as wherein the setting for image data to be inputted to the neural network includes any of an upper limit value of a pixel value of the image data to be inputted to the neural network, the number of tones of the image data, and the number of bits that represent a pixel value of the image data (Para. 200, “For example, the input to a convolution layer can be a multidimensional array of data that defines the various color components of an input image.”, Number of tones falls under various color components. The Examiner notes that only one of the claimed alternatives is required to be taught).
Regarding claim 20, Afra as modified teaches all of the elements of claim 12, as stated above, as well as to obtain first image data and the result of tone decompression. Afra does not explicitly teach to composite image data.
PNG
media_image3.png
463
570
media_image3.png
Greyscale
Woods teaches having a composite unit configured to composite image data (Fig. 6.46, Pg. 454, Discloses a compositing technique).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Afra and Thomas to incorporate the teachings of Woods to include a composite unit configured to composite image data. Making a composite image from multiple different images is a well-known technique in the art. HDR images are normally created by combining multiple images at different exposure levels (compositing), so utilizing composites to merge image data from different stages of processing is an obvious modification that allows for a comparison at those stages.
Regarding claim 21, Afra as modified teaches all of the elements of claim 20, as stated above, as well as wherein the first image data includes image data that has been clipped using a predetermined upper limit value of a pixel value (Para. 115, “Arithmetic operations on the texture data and the input geometry data compute pixel color data for each geometric fragment, or discards one or more pixels from further processing.”, Although not explicitly stated, discarding pixels indicates clipping based on a threshold; Woods further discloses clipping when image values exceed an allowed range (Pg. 92)).
Regarding claim 23, the image processing method recites the same function as that of claim 1, it is rejected under the same analysis.
Regarding claim 25, the non-transitory computer-readable storage medium comprising instructions for performing an image processing method (Afra, Para. 169, “One or more aspects of at least one embodiment may be implemented by representative code stored on a machine-readable medium which represents and/or defines logic within an integrated circuit such as a processor”) recites the same function as that of claim 1, it is rejected under the same analysis.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID A WAMBST whose telephone number is (703)756-1750. The examiner can normally be reached M-F 9-6:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571)272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID ALEXANDER WAMBST/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698