DETAILED ACTION
This action is in response to the application filed on November 3rd, 2023. Claims 1-20 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on November 3rd, 2023 is being considered by the examiner.
Claim Objections
Claims 1, 2, and 10-11 are objected to because of the following informalities:
Claim 1, lines 10-11: “the corresponding pixel in the output image” lacks antecedent basis, this should read “a corresponding pixel in the output image”
Claim 2, lines 5-7 and claim 10, lines 3-5: “the respective number of pixels of the input image associated with different luminance ranges” lacks antecedent basis, this should read “a respective number of pixels of the input image associated with different luminance ranges”
Claim 11, lines 4-5: “the position of the corresponding pixels of the input image” lacks antecedent basis, this should read “a position of corresponding pixels of the input image”
Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-10 and 12-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by US20190370946 (herein after referred to by its primary author, Samadani).
In regards to claim 1, Samadani teaches an image processing device for converting an input image having a first dynamic range into an output image having a second dynamic range distinct from the first dynamic range (Samadani Paragraph [0007] “The tone mapping parameters may be used to increase the dynamic range of the image to take advantage of the capabilities of a high dynamic range display”), the image processing device comprising: at least one processor (Samadani Paragraph [0014] “Processing circuitry in control circuitry 12 may be used to control the operation of device 10. The processing circuitry may be based on one or more microprocessors, microcontrollers, digital signal processors, baseband processors, power management units, audio chips, application-specific integrated circuits, graphics processing units, display driver circuitry such as timing controller integrated circuits and other display driver integrated circuits, and other control circuitry.”) configured to transform, using a conversion parameter , an input luminance value associated with a pixel of the input image into an output luminance value associated with the corresponding pixel in the output image (Samadani Figure 6 Steps 108/110; Paragraph [0047] “At step 108, control circuitry 12 (e.g., tone mapping engine 24 and/or other code running on device 10) may apply a tone mapping process to the image using the tone mapping parameters determined in step 104. The tone mapping process may, for example, map content luminance values to display luminance values according to a tone mapping curve defined by the tone mapping parameters, as described in connection with FIG. 4.”), the at least one processor being configured to perform processing based on a neural network (Samadani Figure 5; Paragraph [0037] “FIG. 5 is a flow chart of illustrative steps involved in building a mapping algorithm for mapping image metadata to the desired tone mapping parameters. This process may be achieved using calibration computing equipment during manufacturing of device 10.”; Paragraph [0040] “At step 84, the calibration computing equipment may build a mapping algorithm that maps metadata to tone mapping parameters. This may include, for example, applying a linear regression model, a machine learning model, or any other suitable technique for identifying trends in the training data so that predictions can be made based on new data.”) that is configured to receive, as an input, a statistical representation depending on a plurality of input luminance values respectively associated with a plurality of pixels of the input image (Samadani Figure 6 Step 102; Paragraph [0044] “At step 102, control circuitry 12 (e.g., tone mapping engine 24 and/or other code running on device 10) may extract metadata from the image captured in step 100. Metadata that may be extracted in step 102 may include camera settings (e.g., lens aperture, focal length, shutter speed, ISO sensitivity, whether flash was used or not, etc.), image content information (e.g., color and luminance histograms, whether a face is detected in the image, etc.), image classification information (e.g., snow, concert, landscape, etc.), and/or other metadata.”), the neural network being configured to provide, as an output, the conversion parameter (Samadani Figure 6 Step 104 Paragraph [0045] “At step 104, control circuitry 12 (e.g., tone mapping engine 24 and/or other code running on device 10) may determine appropriate tone mapping parameters for the image based on the extracted metadata. This may include, for example, applying the mapping algorithm built in step 84 of FIG. 5, which maps metadata to tone mapping parameters based on user preference data”).
In regards to claim 2, Samadani teaches the image processing device according to claim 1, wherein the at least one processor is configured to perform pre-processing to determine the statistical representation based on counting the respective numbers of pixels of the input image associated with different luminance ranges (Samadani Paragraph [0031] “For example, images captured by camera 34 such as high dynamic range images and/or standard dynamic range images may have associated metadata embedded therein. Metadata that may be embedded in images captured by camera 34 include camera settings (e.g., lens aperture, focal length, shutter speed, ISO sensitivity, whether flash was used or not), image content information (e.g., color and luminance histograms, whether a face is detected in the image, etc.), image classification information (e.g., snow, concert, landscape, etc.) and/or other metadata.” Examiner note: This reference discloses a luminance histogram, which is a graph type where each pixel’s luminance value is counted depending on the corresponding range which it falls into).
In regards to claim 3, Samadani teaches the image processing device according to claim 1, wherein the conversion parameter is an exponent, the at least one processor being configured to perform exponentiating using said exponent (Samadani Figure 4; Paragraph [0033] “The use of tone mapping parameters to define content-luminance-to-display-luminance mapping curves is shown in FIG. 4. The content luminance and display luminance axes of the graph of FIG. 4 have logarithmic scales.” Examiner note: This reference teaches that the output display luminance can be derived from the content luminance based on a logarithmic relationship. A logarithm is defined as “the exponent (or power) to which a base number must be raised to produce a given number.” This is analogous to exponentiating, as they are both forms of conversion which deal with raising a number to a power).
In regards to claim 4, Samadani teaches the image processing device according to claim 3, wherein the neural network is configured to provide as an additional output, a peak luminance value (Samadani Figure 4 “SW1”; Paragraph [0033] “In the example of FIG. 4, there are three illustrative mapping curves: curve 50, 52, and 54. Each of these curves may be identified using a set of tone mapping parameters such as a black (BL), reference white level (RW), and specular white level (SW). During operation, tone mapping engine 24 may determine which tone mapping parameters are appropriate for a given image based on the metadata associated with that image and the capabilities of display 14.” Examiner note: The specular white level is analogous to the peak luminance value).
In regards to claim 5, Samadani teaches the image processing device according to claim 1, wherein the neural network is configured to provide, as the conversion parameter, a peak luminance value (Samadani Figure 4 “SW1”; Paragraph [0033] “In the example of FIG. 4, there are three illustrative mapping curves: curve 50, 52, and 54. Each of these curves may be identified using a set of tone mapping parameters such as a black (BL), reference white level (RW), and specular white level (SW). During operation, tone mapping engine 24 may determine which tone mapping parameters are appropriate for a given image based on the metadata associated with that image and the capabilities of display 14.” Examiner note: The specular white level (SW) is analogous to the peak luminance value.).
In regards to claim 6, Samadani teaches the image processing device according to claim 4, wherein the at least one processor is configured to map a first interval of luminance values into a second interval of luminance values, whereof the peak luminance value is a supremum (Samadani Figure 4; Paragraph [0033] “The use of tone mapping parameters to define content-luminance-to-display-luminance mapping curves is shown in FIG. 4. The content luminance and display luminance axes of the graph of FIG. 4 have logarithmic scales. In the example of FIG. 4, there are three illustrative mapping curves: curve 50, 52, and 54. Each of these curves may be identified using a set of tone mapping parameters such as a black (BL), reference white level (RW), and specular white level (SW). During operation, tone mapping engine 24 may determine which tone mapping parameters are appropriate for a given image based on the metadata associated with that image and the capabilities of display 14.” Examiner note: This figure and corresponding paragraph shows that the input content luminance range is mapped to the output display luminance range. The specular white level is analogous to the peak luminance value, which is the supremum of the set of luminance values).
In regards to claim 7, Samadani teaches the image processing device according to claim 1, wherein pixel values of at least one component of the input image are provided as an additional input of the neural network (Samadani Paragraph [0044] “At step 102, control circuitry 12 (e.g., tone mapping engine 24 and/or other code running on device 10) may extract metadata from the image captured in step 100. Metadata that may be extracted in step 102 may include camera settings (e.g., lens aperture, focal length, shutter speed, ISO sensitivity, whether flash was used or not, etc.), image content information (e.g., color and luminance histograms, whether a face is detected in the image, etc.), image classification information (e.g., snow, concert, landscape, etc.), and/or other metadata.” Examiner note: The metadata which is provided as input may include image content information, which can include for example whether a face is detected in an image. The detect face is analogous to a component of the image).
In regards to claim 8, Samadani teaches the image processing device according to claim 1, wherein the at least one processor is configured to train the neural network by using predetermined statistical representations associated with predetermined input images and corresponding conversion parameters (Samadani Figure 5; Paragraph [0038] “At step 80, the calibration computing equipment may gather training data from a given population of users. This may include, for example, displaying different images (e.g., on a display such as display 14 and/or a display similar to display 14) for different users and gathering input from the user to determine the user's preferred image. The images may have different image metadata and different tone mapping parameters. The metadata associated with the user-preferred images and the tone mapping parameters that were used to tone map the user-preferred images may be used as training data.”).
In regards to claim 9, Samadani teaches a method for converting an input image having a first dynamic range into an output image having a second dynamic range distinct from the first dynamic range (Samadani Paragraph [0007] “The tone mapping parameters may be used to increase the dynamic range of the image to take advantage of the capabilities of a high dynamic range display”), the method comprising: determining a conversion parameter using a neural network (Samadani Figure 6 Step 104; Paragraph [0045] “At step 104, control circuitry 12 (e.g., tone mapping engine 24 and/or other code running on device 10) may determine appropriate tone mapping parameters for the image based on the extracted metadata. This may include, for example, applying the mapping algorithm built in step 84 of FIG. 5, which maps metadata to tone mapping parameters based on user preference data.”) that is configured to receive, as an input, a statistical representation depending on a plurality of input luminance values respectively associated with a plurality of pixels of the input image (Samadani Figure 6 Step 102; Paragraph [0044] “At step 102, control circuitry 12 (e.g., tone mapping engine 24 and/or other code running on device 10) may extract metadata from the image captured in step 100. Metadata that may be extracted in step 102 may include camera settings (e.g., lens aperture, focal length, shutter speed, ISO sensitivity, whether flash was used or not, etc.), image content information (e.g., color and luminance histograms, whether a face is detected in the image, etc.), image classification information (e.g., snow, concert, landscape, etc.), and/or other metadata.”), the neural network being configured to provide, as an output, the conversion parameter (Samadani Figure 6 Step 104); and transforming, using the conversion parameter, an input luminance value associated with a pixel of the input image into an output luminance value associated with the corresponding pixel in the output image (Samadani Figure 6 Step 108/110; Paragraph [0047] “At step 108, control circuitry 12 (e.g., tone mapping engine 24 and/or other code running on device 10) may apply a tone mapping process to the image using the tone mapping parameters determined in step 104.”).
In regards to claim 10, Samadani anticipates the claim language as in the consideration of claims 2 and 9.
In regards to claim 12, Samadani anticipates the claim language as in the consideration of claims 3 and 9.
In regards to claim 13, Samadani anticipates the claim language as in the consideration of claims 4 and 12.
In regards to claim 14, Samadani anticipates the claim language as in the consideration of claims 5 and 9.
In regards to claim 15, Samadani anticipates the claim language as in the consideration of claims 6 and 13.
In regards to claim 16, Samadani anticipates the claim language as in the consideration of claims 7 and 9.
In regards to claim 17, Samadani anticipates the claim language as in the consideration of claims 8 and 9.
In regards to claim 18, Samadani teaches the image processing device according to claim 2, wherein the conversion parameter is an exponent, the at least one processor being configured to perform exponentiating using said exponent (Samadani Figure 4; Paragraph [0033] “The use of tone mapping parameters to define content-luminance-to-display-luminance mapping curves is shown in FIG. 4. The content luminance and display luminance axes of the graph of FIG. 4 have logarithmic scales.” Examiner note: This reference teaches that the output display luminance can be derived from the content luminance based on a logarithmic relationship. A logarithm is defined as “the exponent (or power) to which a base number must be raised to produce a given number.” This is analogous to exponentiating, as they are both forms of conversion which deal with raising a number to a power).
In regards to claim 19, Samadani teaches the image processing device according to claim 2, wherein the neural network is configured to provide, as the conversion parameter, a peak luminance value (Samadani Figure 4 “SW1”; Paragraph [0033] “In the example of FIG. 4, there are three illustrative mapping curves: curve 50, 52, and 54. Each of these curves may be identified using a set of tone mapping parameters such as a black (BL), reference white level (RW), and specular white level (SW). During operation, tone mapping engine 24 may determine which tone mapping parameters are appropriate for a given image based on the metadata associated with that image and the capabilities of display 14.” Examiner note: The specular white level is analogous to the peak luminance value).
In regards to claim 20, Samadani teaches the image processing device according to claim 5, wherein the at least one processor is configured to map a first interval of luminance values into a second interval of luminance values, whereof the peak luminance value is a supremum (Samadani Figure 4; Paragraph [0033] “The use of tone mapping parameters to define content-luminance-to-display-luminance mapping curves is shown in FIG. 4. The content luminance and display luminance axes of the graph of FIG. 4 have logarithmic scales. In the example of FIG. 4, there are three illustrative mapping curves: curve 50, 52, and 54. Each of these curves may be identified using a set of tone mapping parameters such as a black (BL), reference white level (RW), and specular white level (SW). During operation, tone mapping engine 24 may determine which tone mapping parameters are appropriate for a given image based on the metadata associated with that image and the capabilities of display 14.” Examiner note: This figure and corresponding paragraph shows that the input content luminance range is mapped to the output display luminance range. The specular white level is analogous to the peak luminance value, which is the supremum of the set of luminance values).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Samadani in view of “A framework for inverse tone mapping” (herein after referred to by its primary author, Banterle)
In regards to claim 11, Samadani teaches the method according to claim 9, but fails to teach determining the statistical representation based on processing the input image by analyzing the input luminance values depending on a position of corresponding pixels of the input image.
However, Banterle teaches determining the statistical representation based on processing the input image by analyzing the input luminance values depending on a position of corresponding pixels of the input image (Banterle Section 2 “1. An initial HDRI is generated by applying an iTMO. Many monotonically increasing TMOs can be used for this purpose. We show how our framework works with two: one based on a simple exponential TMO [21] and the other on the photographic tone reproduction TMO [20]. 2. The areas of high luminance are identified using methods commonly associated with importance sampling of light sources. We demonstrate this approach using median cut [5]. 3. An expand map is created from density estimation of the areas of high luminance found using the results from the previous process.” Examiner note: The statistical representation of luminance values is analogous to the expand map of this disclosure. The expand map is created by first identifying areas with high luminance. These high luminance areas are identified and their position is used in creating the expand map).
Banterle is considered to be analogous to the claimed invention because they are both in the same field of increasing the dynamic range of an input image. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Samadani to include the teachings of Banterle, to provide the advantage of extending the range of high luminance areas (Banterle Abstract “Our framework uses importance sampling of light sources to find the areas considered to be of high luminance and subsequently applies density estimation to generate an expand map in order to extend the range in the high luminance areas using an inverse tone mapping operator”)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US20090015690 teaches a method of generating a synthesized image where the luminance of the image is distributed in such a way to better suit the subject of the image. This is done with a luminance histogram.
“A Visibility Matching Tone Reproduction Operator for High Dynamic Range Scenes” teaches a method of tone reproduction which uses a histogram.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CALEB LOGAN ESQUINO whose telephone number is (703)756-1462. The examiner can normally be reached M-Fr 8:00AM-4:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CALEB L ESQUINO/ Examiner, Art Unit 2677
/JAYESH A PATEL/ Primary Examiner, Art Unit 2677