DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In a preliminary amendment submitted on February 10, 2025; applicant has amended claims 1-4, 8, 10, 12, 15, 17, 19, 21, 23-25, 27, 28, and 31, and canceled claims 5-7, 11, 13, 14, 16, 22, 26, 29, and 30 to expedite prosecution.
Claims 1-4, 8-10, 12, 15, 17-21, 23-25, 27-28, and 31 are pending in this application.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-4, 8-10, 12, 15, 17-21, 23-25, 27-28, and 31 are rejected on the ground of nonstatutory double patenting as being unpatentable over Claims 1, 5-9, 11-12, 14-16, 18, 21-24, 26-28, and 30 of U.S. Application No. 18/860,510. Although the claims at issue are not identical, they are not patentably distinct from each other because they are both directed towards the generation of a recovery map for image data that have differing dynamic ranges.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1-2, 8-10, 12, 15, 17-19, 21, 23-25, 27-28, and 31 are rejected under 35 U.S.C. 103 as being unpatentable over Ward et al. (US 2012/0230597) in view of Wu et al. (US 10148907) and Lim et al. (US 2020/0152111).
Consider Claims 1-2, 19 and 31.
Ward teaches:
1. (Currently Amended) A computer-implemented method comprising: / 19. (Currently Amended) A computer-implemented method comprising: / 31. (Currently Amended) A system comprising: a processor; and a memory coupled to the processor, with instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising: (Ward: abstract, A data structure defining a high dynamic range image comprises a tone map having a reduced dynamic range and HDR information. The high dynamic range image can be reconstructed from the tone map and the HDR information. The data structure can be backwards compatible with legacy hardware or software viewers. The data structure may comprise a JFIF file having the tone map encoded as a JPEG image with the HDR information in an application extension or comment field of the JFIF file, or a MPEG file having the tone map encoded as a MPEG image with the HDR information in a video or audio channel of the MPEG file. Apparatus and methods for encoding or decoding the data structure may apply pre- or post correction to compensate for lossy encoding of the high dynamic range information.[0020] One aspect of this invention provides data structures for representing HDR images (HDR data structures). In preferred embodiments, the HDR data structures permit images to be viewed in a standard dynamic range mode using standard image viewing software and permits high dynamic range versions of the same images to be viewed in a high dynamic range mode using a HDR viewer and appropriate HDR display.)
1. obtaining a first video including a plurality of first frames, wherein each of the first frames depicts a respective scene, the first frames having a first dynamic range; / 19. obtaining portions of a video container, the portions including: a first video including a plurality of first frames, wherein each of the first frames depicts a respective scene, the first frames having a first dynamic range; / 31. 31. obtaining a video container that includes: a first frame video including a plurality of first frames, the first frames having a first dynamic range; (Ward: [0021] FIG. 1 shows a system 10 according to the invention for creating HDR data structures 16 and for viewing images represented by HDR data structures 16. FIG. 2 shows a method 30 performed by system 10 for creating HDR data structures and alternative methods 31A and 31B for displaying images from data in the HDR data structures 16. [0022] System 10 comprises an encoder 14 for creating a HDR image data structure 16 based on original HDR image data 12. Data structure 16 may be decoded by a standard decoder 18 to provide a standard dynamic range image 19. In some embodiments of the invention Standard decoder 18 comprises a “legacy” hardware decoder or software-based decoder such as suitable image viewer software. Data structure 16 may be decoded by a HDR decoder 20 to yield a reconstructed HDR image 21.)
1. obtaining a second video including a plurality of second frames, wherein each of the second frames depicts the respective scene of a corresponding frame of the first frames, the second frames having a second dynamic range that is lower than the first dynamic range; (Ward [0023] "acquiring HDR image data 12"; [0023] Method 30 begins in block 32 by acquiring HDR image data 12. HDR image data 12 includes information that directly or indirectly specifies the luminance of pixels in an image. HDR image data 12 may be in any suitable format and may be acquired through the use of a suitable HDR camera (possibly by combining multiple exposures) or rendered directly in a computer. The source of HDR image data 12 is not important to the practice of the invention. [0024] Method 30 also obtains (block 34) tone map data 15 corresponding to HDR image data 12. Tone map data 15 represents a likeness of the image of HDR image 12, but has a lower dynamic range than HDR image data 12. Tone map data 15 may be generated from HDR image data 12, as indicated by line 13, or in some other manner derived from data having a common source with HDR image data 12. If tone map data 15 is not derived from HDR image data 12 then the order in which tone map data 15 and HDR image data 12 are obtained (i.e. the order of blocks 32 and 34) is not important.)
1. generating a recovery map track, wherein generating the recovery map track includes, for each first frame of the first frames and corresponding second frame of the second frames;
19. and a recovery map track that includes a plurality of recovery map frames, wherein each recovery map frame corresponds to a respective frame of the first frames, each recovery map frame encoding luminance gains of pixels of the corresponding first frame lower than the first dynamic range; / 31. and a recovery map track that includes a plurality of recovery map frames, wherein each recovery map frame corresponds to a respective frame of the first frames, each recovery map frame encoding luminance gains of pixels of the corresponding first frame; (Ward: [0025] Encoder 14 generates a data structure 16. Data structure 16 includes a tone map portion 16A, which is based on tone map data 15, and an HDR information portion 16B, which contains information that may be combined by HDR decoder 20 with the data from tone map portion 16A to reconstruct HDR image data 12 or a close approximation thereto. Method 30 generates (block 36) HDR information portion by comparing tone map data 15 (or, equivalently, tone map data reconstructed from tone map portion 16A of data structure 16) and HDR image data 12. In block 38, method 30 stores tone map portion 16A and HDR information portion 16B in data structure 16. [0026] In preferred embodiments of the invention, data structure 16 has a format that can be read by a standard decoder to produce a lower dynamic range (LDR) image. A standard decoder 18 may implement decoding method 31A. Standard decoder 18 generates a standard LDR image 19 by retrieving tone map portion 16A and displaying an image represented by tone map portion 16A (block 39). The standard decoder can ignore HDR information portion 16B. [0026]-[0029], [0027] Data structure 16 can also be read by an HDR decoder 20. HDR decoder 20 implements decoding method 31B and generates an HDR image 21 based upon information from both tone map portion 16A and HDR information portion 16B. Method 31B retrieves data from tone map portion 16A and HDR information portion 16B of data structure 16 in block 40. In block 42, a reconstructed HDR image is created by modifying a tone map extracted from tone map portion 16A according to HDR information from HDR information portion 16B. The reconstructed HDR image is displayed in block 44.)
19. determining whether to display one of: the first video or a derived video that includes a plurality of derived frames having a dynamic range lower than the first dynamic range; / 31. determining whether to display one of: the first video or a derived video that includes a plurality of derived frames corresponding to the first frames in depicted subject matter and having a second dynamic range that is lower than the first dynamic range; (Ward: [0029]-[0031] In some embodiments of the invention, data structure 16 comprises a MPEG formatted file. In such embodiments, tone map portion 16A may be contained in the image portion of a MPEG file and HDR information portion 16B may be stored in one or more application extensions the MPEG file and/or in one or more comment portions of the MPEG file. In such embodiments any standard MPEG viewer can open data structure 16 and display the image provided in tone map portion 16A at a dynamic range lower than that of the original HDR data 12 or reconstructed HDR image 21. One HDR information portion 16B may be associated with each frame of a MPEG video file, or for versions of MPEG which employ keyframes, HDR information portion 16B may be associated only with the keyframes. Conventional MPEG keyframe interpolation techniques may be used to create the inter-keyframe frames (i.e., the frames which are between the keyframes).)
19. in response to determining to display the first video: causing at least a portion of the first video to be displayed by a display device; and in response to determining to display the derived video: / 31. in response to determining to display the first video: causing at least a portion of the first video to be displayed by a first display device; (Ward: [0033]-[0038] Tone map portion 16A may be created from tone map data 15 in any suitable manner. For example, tone map portion 16A may be generated using a suitable tone mapping operator. The tone mapping operator preferably has the properties that: an original HDR input (i.e. original HDR image data 12) is mapped smoothly into a standard dynamic resolution (typically 24-bit) output domain; no components of the output of the tone mapping operator are clamped at values of 0 or 255; hue is maintained for each pixel; and, if the tone mapping operator changes saturation values, it makes only mild changes that may be described by invertible functions. The inventors have found that the bilateral filter described in Durand and Dorsey, Fast bilateral filtering for the display of high dynamic range images, ACM Transactions on Graphics, 21, 3, 249-256 (2002) provides a suitable tone mapping operator. Tone map portion 16A may be encoded using a suitable encoder, such as a JPEG encoder or a MPEG encoder. [0038] Tone map portion 16A may represent pixel color values in any suitable manner. For example, pixel color values may be represented as RGB (red, green and blue) values, CMYK (cyan, magenta, yellow and black) values, YCbCr (luminance and chrominance) values, or the like. The data in tone map portion 16A may be compressed using any suitable compression scheme. For example, the data in tone map portion 16A may be compressed in a manner compatible with JPEG or MPEG standards.)
1. generating a recovery map frame based on the first frame and the corresponding second frame, wherein the recovery map frame encodes gains in luminances between portions of the first frame and corresponding portions of the corresponding second frame; and / 19. applying the gains of the recovery map frames of the recovery map track to luminances of pixels of the corresponding first frames to determine respective corresponding pixel values of corresponding derived frames of the derived video; and / 31. And in response to determining to display the derived video: applying the gains of the recovery map frames of the recovery map track to luminances of pixels of the corresponding first frames to determine respective corresponding pixel values of corresponding derived frames of the derived video, wherein applying the gains includes scaling the luminances of the corresponding first frames based on a particular luminance output of the second display device and based on the recovery map frames; (Ward: [0052] FIG. 4 is a flow chart that illustrates the operation of methods 60 which apply corrections to the data in tone map portion 16A or HDR information portion 16B to reduce artefacts resulting from lossy encoding of tone map portion 16A and/or HDR information portion 16B. Methods 60 acquire HDR image data 90 and tone map data 91 in blocks 62 and 64. HDR image data 90 and tone map data 91 may be obtained in any suitable manner including those manners described above. In some embodiments, tone map data 91 is extracted from HDR image data 90 as indicated by arrow 65. [0053] In block 66, tone map data 91 is encoded to yield encoded tone map data 92. In some embodiments of the invention encoding block 66 comprises JPEG or MPEG encoding. Subsequently, in block 68, encoded tone map data 92 is decoded to yield reconstructed tone map data 94. Reconstruction block 68 may comprise passing encoded tone map data 92 to a suitable decoder, such as a JPEG or MPEG decoder in the case that block 66 comprises JPEG or MPEG encoding. [0054] Block 70 generates ratio data 96 by applying a function which takes as inputs values from HDR image data 90 (first values) and corresponding values from reconstructed tone map data 94 (second values). The function includes dividing the first values by the second values or vice versa. In a simple embodiment of the invention, ratio data 96 includes a value RI for each pixel in an image given by:
PNG
media_image1.png
49
314
media_image1.png
Greyscale
where: (x, y) are coordinates identifying a pixel; L is a function which returns the luminance of the pixel from data for the pixel; HDR(x, y) is the pixel data in HDR image data 90 at coordinates (x, y); and, TM(x, y) is the pixel data in reconstructed tone map data 94 (or, tone map data 91) for the pixel at coordinates (x, y). In some embodiments, the ratio data stores the logarithm of RI, the square root of RI or another function of RI. [0055] Blocks 72 and 74 encode ratio data 96. In this example embodiment, the encoding includes downsampling ratio data 96 in block 72 to yield downsampled ratio data 98 and then compressing downsampled ratio data 98 to yield encoded ratio data 100. The amount of downsampling performed in block 72 may be selected based upon the competing goals of making HDR image portion 16B small and making a HDR image reconstructed from HDR data structure 16 reproduce HDR image data 90 with the highest fidelity. In some embodiments of the invention, ratio data 96 is downsampled sufficiently that downsampled ratio data 98 has fewer pixels than ratio data 96 by a factor in the range of 4 to 15.)
1. providing the first video and the recovery map track in a video container, wherein the video container is readable to display a derived video based on applying the recovery map track to the first video, wherein the derived video includes a plurality of derived frames that have a dynamic range that is lower than the first dynamic range. / 19. causing at least a portion of the derived video to be displayed by the display device. / 31. and causing at least a portion of the derived video to be displayed by a second display device. (Ward: [0045] Encoded tone map data 15A is decoded by decoder 54 to yield reconstructed tone map data 55. HDR image data 12 is divided by reconstructed tone map data 55 by divider 56 to yield ratio data 57. Ratio data 57 is optionally compressed by data compressor 58 to yield HDR information 16B. Data compressor 58 may conveniently comprise a JPEG or MPEG encoder. In some embodiments of the invention, the same JPEG or MPEG encoder is used to encode both tone map portion 16A and HDR information portion 16B of HDR data structure 16. [0046] In some embodiments of the invention, ratio data 57 comprises some function of the ratio of values of HDR image data 12 to corresponding values specified by tone map data 15 (or tone map portion 16A). For example, ratio data 57 may comprise information specifying a logarithm of such a ratio. [0047] In some alternative embodiments of the invention, tone map data 15 is provided directly to divider 56 as indicated by line 53. In such embodiments, decoder 54 is not required. Where tone map portion 16A is encoded using a lossy algorithm, such as JPEG or MPEG encoding, it is preferable to base HDR information portion 16B on reconstructed tone map data 55 instead of on tone map data 15. Basing HDR information portion 16B on reconstructed tone map data 55 permits a more accurate reconstruction of HDR image data 12 from HDR data structure 16 in cases where tone map information portion 16A is encoded by a lossy encoding process. Tone map information portion 16A, rather than tone map data 15 will be used to reconstruct HDR image 21 (FIG. 1).)
Even if Ward does not specifically teach:
1. generating a recovery map frame based on the first frame and the corresponding second frame, wherein the recovery map frame encodes gains in luminances between portions of the first frame and corresponding portions of the corresponding second frame; and / 19. applying the gains of the recovery map frames of the recovery map track to luminances of pixels of the corresponding first frames to determine respective corresponding pixel values of corresponding derived frames of the derived video; and / 31. And in response to determining to display the derived video: applying the gains of the recovery map frames of the recovery map track to luminances of pixels of the corresponding first frames to determine respective corresponding pixel values of corresponding derived frames of the derived video, wherein applying the gains includes scaling the luminances of the corresponding first frames based on a particular luminance output of the second display device and based on the recovery map frames;
Wu teaches:
1. (Currently Amended) A computer-implemented method comprising: / 19. (Currently Amended) A computer-implemented method comprising: / 31. (Currently Amended) A system comprising: a processor; and a memory coupled to the processor, with instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising: (Wu: abstract, In some aspects, the disclosure is directed to methods and systems for transformation between media formats, such as between standard dynamic range (SDR) and high dynamic range (HDR) media or between HDR media formats, without undesired hue shifting, via one or both of a luminance mapping ratio technique and a direct color component mapping technique.)
1. obtaining a first video including a plurality of first frames, wherein each of the first frames depicts a respective scene, the first frames having a first dynamic range; / 19. obtaining portions of a video container, the portions including: a first video including a plurality of first frames, wherein each of the first frames depicts a respective scene, the first frames having a first dynamic range; / 31. 31. obtaining a video container that includes: a first frame video including a plurality of first frames, the first frames having a first dynamic range; (Wu: column 2 lines 65-67, column 3 lines 1-24; For example, referring first to FIG. 1A, illustrated is a chart illustrating an example of differences between SDR 104 and HDR 106 luminance ranges and mappings. As shown, the chart displays graphs of signal value 100 compared to input light 102. In the chart, the input light 102 and signal values 100 are normalized and unitless. However, other values for luminance may be used, such as 100 candela/m2; and signal values may represent a signal having all bits on (e.g. an 8-bit signal of 1111 1111, for some implementations). As shown, SDR media 104 has a signal/input luminance curve that may increase relatively steadily from 0 to 1, and then clipping or limited at a full scale value 100 (as additional bits are not provided and the signal value cannot increase beyond the all-bits-on representation). Accordingly, brighter input signals 102 cannot be encoded within the SDR signal 104. To allow these brighter input light values, in some implementations, the SDR luminance curve may be shifted to the right (frequently referred to as increasing the “gamma” offset of the signal). However, the curve region does not grow: increasing the brightness to reflect higher level input signals results in corresponding clipping or limiting of darker signals. Thus, in typical implementations, an SDR display may be calibrated to display a “black” input signal as a dark grey, so as to display a “white” input signal as a bright grey.)
1. obtaining a second video including a plurality of second frames, wherein each of the second frames depicts the respective scene of a corresponding frame of the first frames, the second frames having a second dynamic range that is lower than the first dynamic range; (Wu: column 3 lines 8-67; Due to its additional bit-depth and better perceptual luminance quantizer which is closer to human visual system, the HDR luminance curve 106 is significantly broader. In the example shown, with the same black level as the SDR luminance curve 104, the HDR luminance curve can provide different signal values 100 for up to three times the same input light values 102. In other implementations, with different bit-depths and/or other nonlinear encoding functions, the curve may be broader or narrower. Furthermore, as discussed above, the SDR luminance curve 104 may be gamma shifted to the right to display differences between brighter areas, at the expense of black values. The HDR luminance curve 104 may be similarly shifted, or may be left unshifted (or shifted by a different amount), with the result that an HDR display may display brighter whites and darker blacks than an SDR display.
Converting between SDR and HDR media formats may be important, particularly during the transition between these technologies, and also for preserving legacy recordings and images generated and encoded in SDR formats. Furthermore, because SDR media uses less data, downconverting of media formats from HDR to SDR may be useful for bandwidth or storage limited environments (e.g. mobile devices). However, such inter-format conversions may be difficult due to non-linear relationships between values in the SDR format luminance range and values in the HDR format luminance range. Similarly, it may be difficult to convert between different HDR formats (e.g. HDR formats with different bit depths or color spaces). In particular, many implementations of SDR/HDR or HDR/HDR format conversions result in substantial chroma hue shift that may be quite noticeable to the viewer.
For example, in the example illustrated in FIG. 1A, the HDR luminance curve 106 is not a linear multiple of the SDR luminance curve 104. This may be further exacerbated in some implementations of SDR luminance, such as where a “knee” is provided within the curve, changing the slope of the curve near the extreme signal values 100 of 0 and/or 1 to cover a broader range of input light values 102. In such implementations, a mapping between the SDR luminance curve and HDR luminance curve may require a piecewise linear function (PWL), a non-linear function, or other such function.)
1. generating a recovery map track, wherein generating the recovery map track includes, for each first frame of the first frames and corresponding second frame of the second frames;
19. and a recovery map track that includes a plurality of recovery map frames, wherein each recovery map frame corresponds to a respective frame of the first frames, each recovery map frame encoding luminance gains of pixels of the corresponding first frame that are scaled by an associated range scaling factor that includes isa ratio of a maximum luminance of a corresponding second frame to a maximum luminance of the corresponding first frame, wherein the corresponding second frame depicts the respective scene of the first frame has a second dynamic range that is lower than the first dynamic range; / 31. and a recovery map track that includes a plurality of recovery map frames, wherein each recovery map frame corresponds to a respective frame of the first frames, each recovery map frame encoding luminance gains of pixels of the corresponding first frame; (Wu: column 3 lines 1-24; For example, referring first to FIG. 1A, illustrated is a chart illustrating an example of differences between SDR 104 and HDR 106 luminance ranges and mappings. As shown, the chart displays graphs of signal value 100 compared to input light 102. In the chart, the input light 102 and signal values 100 are normalized and unitless. However, other values for luminance may be used, such as 100 candela/m2; and signal values may represent a signal having all bits on (e.g. an 8-bit signal of 1111 1111, for some implementations). As shown, SDR media 104 has a signal/input luminance curve that may increase relatively steadily from 0 to 1, and then clipping or limited at a full scale value 100 (as additional bits are not provided and the signal value cannot increase beyond the all-bits-on representation). Accordingly, brighter input signals 102 cannot be encoded within the SDR signal 104. To allow these brighter input light values, in some implementations, the SDR luminance curve may be shifted to the right (frequently referred to as increasing the “gamma” offset of the signal). However, the curve region does not grow: increasing the brightness to reflect higher level input signals results in corresponding clipping or limiting of darker signals. Thus, in typical implementations, an SDR display may be calibrated to display a “black” input signal as a dark grey, so as to display a “white” input signal as a bright grey. Column 4 lines 28-64, In a first technique, referred to herein as luminance mapping ratio 112, each output linear light R, G, B component of a second format signal 116 may be derived by a linear equation from each input linear light R, G, B component of a first format signal 110, multiplied by a luminance ratio (LMR[Yin]) of output and input pixels (e.g. given an input SDR pixel at 100 nits of brightness, and an output HDR pixel to be at 300 nits, the luminance ratio LMR[Yin] is 3). This scaled value may be further multiplied by a multiplicand scale factor (S), providing adjustment of peak luminance values; and may be added to an additive offset (A), providing black level adjustment. For example, given a mapping between an SDR luminance curve and HDR luminance curve that both start at 0 signal value/input light value, as shown in the example of FIG. 1A, the multiplicand scale value S may be 1, and the additive offset A may be 0. However, in a different mapping in which the SDR curve may be gamma shifted to represent brighter input light levels, there may be a corresponding negative additive offset A. Similarly, in a mapping in which the SDR curve is broadened by a knee at high or low signal values, the multiplicand scale value S may be increased or decreased. Thus, as an equation, the luminance mapping ratio technique may be represented in some implementations as:
Output(color channel value)=Input(color channel value)*LMR[Y in])*S+A;
for each color channel of the media (e.g. Output(R) based on Input (R); Output(G) based on input (G); and Output(B) based on Input(B)). The calculation may be performed on each color component value for the pixel iteratively, in any order, or in parallel, or in a combination of parallel and serial (e.g. performing calculations for red and blue channels together, and then for a green channel, or in any other order). In some implementations, input and output signals may be normalized (e.g. to values between 0 and 1), so the luminance ratio value may refer to pre-renormalization ratio.)
19. determining whether to display one of: the first video or a derived video that includes a plurality of derived frames having a dynamic range lower than the first dynamic range; / 31. determining whether to display one of: the first video or a derived video that includes a plurality of derived frames corresponding to the first frames in depicted subject matter and having a second dynamic range that is lower than the first dynamic range; (Wu: column 14 lines 34-64; In some embodiments, the computing device 400 may include or be connected to one or more display devices 424 a-424 n. As such, any of the I/O devices 430 a-430 n and/or the I/O controller 423 may include any type and/or form of suitable hardware, software, or combination of hardware and software to support, enable or provide for the connection and use of the display device(s) 424 a-424 n by the computing device 400. For example, the computing device 400 may include any type and/or form of video adapter, video card, driver, and/or library to interface, communicate, connect or otherwise use the display device(s) 424 a-424 n. In one embodiment, a video adapter may include multiple connectors to interface to the display device(s) 424 a-424 n. In other embodiments, the computing device 400 may include multiple video adapters, with each video adapter connected to the display device(s) 424 a-424 n. In some embodiments, any portion of the operating system of the computing device 400 may be configured for using multiple displays 424 a-424 n. One ordinarily skilled in the art will recognize and appreciate the various ways and embodiments that a computing device 400 may be configured to have one or more display devices 424 a-424 n.)
1. generating a recovery map frame based on the first frame and the corresponding second frame, wherein the recovery map frame encodes gains in luminances between portions of the first frame and corresponding portions of the corresponding second frame; and / 19. in response to determining to display the first video: causing at least a portion of the first video to be displayed by a display device; and in response to determining to display the derived video: applying the gains of the recovery map frames of the recovery map track to luminances of pixels of the corresponding first frames to determine respective corresponding pixel values of corresponding derived frames of the derived video; and / 31. in response to determining to display the first video: causing at least a portion of the first video to be displayed by a first display device; and in response to determining to display the derived video: applying the gains of the recovery map frames of the recovery map track to luminances of pixels of the corresponding first frames to determine respective corresponding pixel values of corresponding derived frames of the derived video, wherein applying the gains includes scaling the luminances of the corresponding first frames based on a particular luminance output of the second display device and based on the recovery map frames; (Wu: column 4 lines 65-67, column 5, column 6 lines 1-5, Figures 1-2; The luminance ratio LMR[Yin] can be a linear or non-linear function implemented in a lookup table (LUT) or a piecewise linear function (PWL). In some implementations, it may be more efficient to use an LUT with a high number of reference points than a PWL, because the luminance ratio function may tend to have a faster transition curve than direct R,G,B component mapping 114. Accordingly, looking up luminance ratio values LMR[Yin] in a LUT may be computationally faster and reduce latency in the processing system. In other implementations, the luminance ratio linear or non-linear function may be dynamically calculated, rather than using a precomputed LUT or PWL approximation.
The luminance mapping ratio technique 112 can be used for dynamic range mapping (e.g. SDR to HDR and HDR to SDR), nonlinear format conversions, system gamma adjustments, or for any other type and form of conversion. For an example of SDR to HDR conversion, as discussed above, luminance ratio function LMR[Yin] is the luminance ratio of HDR versus SDR referenced to each input SDR luminance value. If an input SDR pixel is at 100 nits, and the corresponding output HDR pixel is mapped to 300 nits, the luminance ratio LMR[Yin] is 3. As discussed above, the ratio function can be implemented in a lookup table. In some implementations, the multiplicand scale factor may be 1.0 and the additive coefficient may be 0.0 for a typical black level or unadjusted gamma offset. The output for each channel is therefore R_out=3*R_in, G_out=3*G_in, B_out=3*B_in.
The luminance mapping ratio technique 112 only adjusts luminance and does not cause any undesired hue shift, even when the luminance transfer function is highly non-linear and the R,G,B, component values are quite different within the same pixel.
The second technique, direct R,G,B component mapping 114, may use a LUT or PWL to directly map an input color component value of a pixel to an output color component value for the pixel in a different format. In some implementations, each component output of R,G,B linear light components may be directly mapped from each input R,G,B linear light component separately through the same linear or non-linear function. The non-linear function can be implemented through a LUT or PWL. Direct R,G,B component mapping 114 can also be used for dynamic range mapping (e.g. SDR to HDR and HDR to SDR), nonlinear format conversions, system gamma adjustments, or any other such format change. For example, in one implementation, given an input SDR pixel with input values of R_in =50, G_in=100, B_in=20, and SDR mapped to HDR in a linear scale of 3, R_out=150, G_out=300, and B_out=60. The same scaling or mapping may be used for each channel, or different mapping per channel. Furthermore, mappings may be non-linear across the pixel range. For example, in one implementation, R_out/R_in may be 3 as discussed above for an input of R_in<100, but may be 2 for an input of R_in>100. The non-linear function may have as many distinct points or mapping regions as desired.
In some implementations, unlike luminance mapping ratio 112, direct R,G,B mapping 114 may result in saturation and hue variations, especially for high luminance level inputs. However, saturation variation can be beneficial in some cases. Additionally, direct R,G,B component mapping 114 tends to be smoother in implementation. It is particularly suitable for PWL-based implementations that utilize fewer reference points than LUT-based implementations for luminance mapping ratio techniques 112, and may accordingly reduce hardware expenses for implementation. In further implementations, luminance mapping and component mapping techniques may be combined to reduce hue variations.
FIG. 2 is a block diagram of an implementation of a computing device for luminance processing during conversions between media formats, including SDR and HDR media formats. A device 200 may comprise a video processor, a set top box, a video router or distribution amplifier, a splitter, an X-Y interface, a switcher, a compositor, or any other type and form of appliance or computing device.)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify Ward’s method and system for encoding, decoding and representing HDR image data with Wu’s method and system for luminance processing in HDR and SDR image data. The determination of obviousness is predicated upon the following findings: Both references are in the same field of endeavor for image processing and analysis of HDR data and the references lend themselves for combination. One skilled in the art would have been motivated to modify Ward in order to improve and account for luminance processing in a more accurate manner as disclosed by Wu during the overall image encoding and decoding process. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of modify Ward’s HDR image processing to use the overall tone map for luminance representation as suggested by Wu for HDR to SDR conversion, while the teaching of Wu continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of ensuring that luminance values are more accurately accounted for in the overall image mapping process from HDR to SDR image data conversion (Wu col. 5 II. 10-25 "The luminance mapping ratio technique 112 can be used for dynamic range mapping (e.g. SDR to HDR and HDR to SDR)”). It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Even if Wu and Ward does not teach:
2. (Currently Amended) The method of claim 1, wherein the gains in luminances are scaled by a range scaling factor that is a ratio of a maximum luminance of the corresponding second frame to a maximum luminance of the first frame./ 19. that are scaled by an associated range scaling factor that includes a ratio of a maximum luminance of a corresponding second frame to a maximum luminance of the corresponding first frame,
Lim teaches:
2. (Currently Amended) The method of claim 1, wherein the gains in luminances are scaled by a range scaling factor that is a ratio of a maximum luminance of the corresponding second frame to a maximum luminance of the first frame./ 19. that are scaled by an associated range scaling factor that includes a ratio of a maximum luminance of a corresponding second frame to a maximum luminance of the corresponding first frame, (Lim [0019] "The second image quality compensation data may be the first image quality compensation data adjusted based on a ratio between the first maximum luminance value and the second maximum luminance value"). [0067]-[0068], [0078] According to an example, the first image quality compensation data may include image quality compensation data corrected based on the first maximum luminance (l1) and the second maximum luminance (l2). [0079] The first image quality compensation data may include image quality compensation data in which image quality compensation data corresponding to the driving frequency of the display panel is reduced based on a ratio between the first maximum luminance (l1) and the second maximum luminance (l2).)
It would have been obvious to one of ordinary skill in the art to modify the combination of Ward and Wu with Lim’s representation of luminance ratio to ensure the image quality compensation values. The determination of obviousness is predicated upon the following findings: One skilled in the art would have been motivated to modify the data structure of the combination of Ward and Wu in order to include the range scale factor of Lim for adjustment. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of the combination of Ward and Wu, while the teaching of Lim continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of ensure that the luminance values is more accurately corrected using a scaling measure. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claims 8-9 and 17.
The combination of Ward, Wu and Lim teaches:
8. (Currently Amended) The method of claim 1, wherein generating each recovery map frame includes encoding the gains in luminances such that applying the gains to luminances of individual pixels of the first frame results in pixels corresponding to the corresponding second frame. / 9. (Original) The method of claim 8, wherein generating each recovery map frame is based on: recovery(x, y)= log(pixel_gain(x, y)) / log(range scaling factor),wherein recovery(x,y) is the recovery map frame for pixel position (x, y) of the corresponding second frame, and pixel_gain(x,y) is a ratio of luminances at the position (x, y) of the corresponding second frame to the first frame. / 17. (Currently Amended) The method of claim 1, wherein generating the recovery map track includes generating multiple recovery map tracks and providing the recovery map track in the video container includes providing the multiple recovery map tracks in the video container, wherein each of the multiple recovery map tracks encodes gains in luminances between portions of the first frame and corresponding portions of the corresponding second frame, and wherein each of the multiple recovery map tracks is readable from the video container to cause a respective derived video to be displayed, each respective derived video having one or more characteristics that differ from each other. (Ward: [0052] FIG. 4 is a flow chart that illustrates the operation of methods 60 which apply corrections to the data in tone map portion 16A or HDR information portion 16B to reduce artefacts resulting from lossy encoding of tone map portion 16A and/or HDR information portion 16B. Methods 60 acquire HDR image data 90 and tone map data 91 in blocks 62 and 64. HDR image data 90 and tone map data 91 may be obtained in any suitable manner including those manners described above. In some embodiments, tone map data 91 is extracted from HDR image data 90 as indicated by arrow 65. [0053] In block 66, tone map data 91 is encoded to yield encoded tone map data 92. In some embodiments of the invention encoding block 66 comprises JPEG or MPEG encoding. Subsequently, in block 68, encoded tone map data 92 is decoded to yield reconstructed tone map data 94. Reconstruction block 68 may comprise passing encoded tone map data 92 to a suitable decoder, such as a JPEG or MPEG decoder in the case that block 66 comprises JPEG or MPEG encoding. [0054] Block 70 generates ratio data 96 by applying a function which takes as inputs values from HDR image data 90 (first values) and corresponding values from reconstructed tone map data 94 (second values). The function includes dividing the first values by the second values or vice versa. In a simple embodiment of the invention, ratio data 96 includes a value RI for each pixel in an image given by:
PNG
media_image1.png
49
314
media_image1.png
Greyscale
where: (x, y) are coordinates identifying a pixel; L is a function which returns the luminance of the pixel from data for the pixel; HDR(x, y) is the pixel data in HDR image data 90 at coordinates (x, y); and, TM(x, y) is the pixel data in reconstructed tone map data 94 (or, tone map data 91) for the pixel at coordinates (x, y). In some embodiments, the ratio data stores the logarithm of RI, the square root of RI or another function of RI. [0055] Blocks 72 and 74 encode ratio data 96. In this example embodiment, the encoding includes downsampling ratio data 96 in block 72 to yield downsampled ratio data 98 and then compressing downsampled ratio data 98 to yield encoded ratio data 100. The amount of downsampling performed in block 72 may be selected based upon the competing goals of making HDR image portion 16B small and making a HDR image reconstructed from HDR data structure 16 reproduce HDR image data 90 with the highest fidelity. In some embodiments of the invention, ratio data 96 is downsampled sufficiently that downsampled ratio data 98 has fewer pixels than ratio data 96 by a factor in the range of 4 to 15. Wu: column 4 lines 65-67, column 5, column 6 lines 1-5, Figures 1-2; The luminance ratio LMR[Yin] can be a linear or non-linear function implemented in a lookup table (LUT) or a piecewise linear function (PWL). In some implementations, it may be more efficient to use an LUT with a high number of reference points than a PWL, because the luminance ratio function may tend to have a faster transition curve than direct R,G,B component mapping 114. Accordingly, looking up luminance ratio values LMR[Yin] in a LUT may be computationally faster and reduce latency in the processing system. In other implementations, the luminance ratio linear or non-linear function may be dynamically calculated, rather than using a precomputed LUT or PWL approximation.
The luminance mapping ratio technique 112 can be used for dynamic range mapping (e.g. SDR to HDR and HDR to SDR), nonlinear format conversions, system gamma adjustments, or for any other type and form of conversion. For an example of SDR to HDR conversion, as discussed above, luminance ratio function LMR[Yin] is the luminance ratio of HDR versus SDR referenced to each input SDR luminance value. If an input SDR pixel is at 100 nits, and the corresponding output HDR pixel is mapped to 300 nits, the luminance ratio LMR[Yin] is 3. As discussed above, the ratio function can be implemented in a lookup table. In some implementations, the multiplicand scale factor may be 1.0 and the additive coefficient may be 0.0 for a typical black level or unadjusted gamma offset. The output for each channel is therefore R_out=3*R_in, G_out=3*G_in, B_out=3*B_in.
The luminance mapping ratio technique 112 only adjusts luminance and does not cause any undesired hue shift, even when the luminance transfer function is highly non-linear and the R,G,B, component values are quite different within the same pixel.
The second technique, direct R,G,B component mapping 114, may use a LUT or PWL to directly map an input color component value of a pixel to an output color component value for the pixel in a different format. In some implementations, each component output of R,G,B linear light components may be directly mapped from each input R,G,B linear light component separately through the same linear or non-linear function. The non-linear function can be implemented through a LUT or PWL. Direct R,G,B component mapping 114 can also be used for dynamic range mapping (e.g. SDR to HDR and HDR to SDR), nonlinear format conversions, system gamma adjustments, or any other such format change. For example, in one implementation, given an input SDR pixel with input values of R_in =50, G_in=100, B_in=20, and SDR mapped to HDR in a linear scale of 3, R_out=150, G_out=300, and B_out=60. The same scaling or mapping may be used for each channel, or different mapping per channel. Furthermore, mappings may be non-linear across the pixel range. For example, in one implementation, R_out/R_in may be 3 as discussed above for an input of R_in<100, but may be 2 for an input of R_in>100. The non-linear function may have as many distinct points or mapping regions as desired.
In some implementations, unlike luminance mapping ratio 112, direct R,G,B mapping 114 may result in saturation and hue variations, especially for high luminance level inputs. However, saturation variation can be beneficial in some cases. Additionally, direct R,G,B component mapping 114 tends to be smoother in implementation. It is particularly suitable for PWL-based implementations that utilize fewer reference points than LUT-based implementations for luminance mapping ratio techniques 112, and may accordingly reduce hardware expenses for implementation. In further implementations, luminance mapping and component mapping techniques may be combined to reduce hue variations.
FIG. 2 is a block diagram of an implementation of a computing device for luminance processing during conversions between media formats, including SDR and HDR media formats. A device 200 may comprise a video processor, a set top box, a video router or distribution amplifier, a splitter, an X-Y interface, a switcher, a compositor, or any other type and form of appliance or computing device. Lim [0019] "The second image quality compensation data may be the first image quality compensation data adjusted based on a ratio between the first maximum luminance value and the second maximum luminance value"). [0067]-[0068], [0078] According to an example, the first image quality compensation data may include image quality compensation data corrected based on the first maximum luminance (l1) and the second maximum luminance (l2). [0079] The first image quality compensation data may include image quality compensation data in which image quality compensation data corresponding to the driving frequency of the display panel is reduced based on a ratio between the first maximum luminance (l1) and the second maximum luminance (l2).)
Consider Claims 10 and 25.
The combination of Ward, Wu and Lim teaches:
10. (Currently Amended) The method of claim 1, wherein generating each recovery map frame includes encoding the recovery map frame into a bilateral grid. / 25. (Currently Amended) The method of claim 19, wherein each recovery map frame is encoded in a bilateral grid, and further comprising decoding the recovery map frames from the bilateral grid. (Examiner Note: the representation of data in a JFIF or matrix form is analogous in scope to a bilateral grid; Ward: [0029] In some embodiments of the invention, data structure 16 comprises a JPEG File Interchange Format (JFIF) formatted file. In such embodiments, tone map portion 16A may be contained in the image portion of a JFIF file and HDR information portion 16B may be stored in one or more application extension portions of the JFIF file and/or in one or more comment portions of the JFIF file. In such embodiments any standard JPEG viewer can open data structure 16 and display the image provided in tone map portion 16A at a dynamic range lower than that of the original HDR data 12 or reconstructed HDR image 21. [0042] FIG. 3 shows an HDR encoder 50 according to an embodiment of the invention wherein the HDR information used to make HDR portion 16B of data structure 16 comprises ratios of pixel values in HDR image 12 to the corresponding values specified by tone map portion 16A. Encoder 50 receives HDR image data 12. Encoder 50 obtains tone map data 15 either by extracting tone map data 15 from HDR image data 12, as indicated by dashed line 13 and tone mapper 17, or by receiving tone map data 15 from some other source, as indicated by dashed line 13A. Tone mapper 17 preferably does not clip colour or luminance values, and maintains colour and luminance ratios for each pixel in tone map data 15. Lim: [0057] The display panel 110 may be embodied with a liquid crystal display panel. In this case, the display panel 110 may include a plurality of data lines DL1 to DLm, a plurality of gate lines GL1 to GLn, and a plurality of pixels in which the plurality of data lines DL1 to DLm cross plurality of gate lines GL1 to GLn. For example, the plurality of data lines DL1 to DLm and the plurality of gate lines GL1 to GLn are arranged in a matrix form, Each pixel may include a liquid crystal (or, a liquid crystal cell), a Thin Film Transistor (TFT), a liquid crystal capacitor Clc connected to the transistor, and a storage capacitor Cst.)
Consider Claims 11, 13-14, 16 and 22.
11. (Cancelled)
13. (Cancelled)
14. (Cancelled)
16. (Cancelled)
22. (Cancelled)
Consider Claims 15.
The combination of Ward, Wu and Lim teaches:
15. (Currently Amended) The method of claim 1,wherein the video container is readable to: display the first video by a first display device capable of displaying the first dynamic range; and display the derived video by a second display device only capable of displaying a dynamic range lower than the first dynamic range. (Ward [0022] System 10 comprises an encoder 14 for creating a HDR image data structure 16 based on original HDR image data 12. Data structure 16 may be decoded by a standard decoder 18 to provide a standard dynamic range image 19. In some embodiments of the invention Standard decoder 18 comprises a “legacy” hardware decoder or software-based decoder such as suitable image viewer software. Data structure 16 may be decoded by a HDR decoder 20 to yield a reconstructed HDR image 21. [0023] "acquiring HDR image data 12"; [0023] Method 30 begins in block 32 by acquiring HDR image data 12. HDR image data 12 includes information that directly or indirectly specifies the luminance of pixels in an image. HDR image data 12 may be in any suitable format and may be acquired through the use of a suitable HDR camera (possibly by combining multiple exposures) or rendered directly in a computer. The source of HDR image data 12 is not important to the practice of the invention. [0024] Method 30 also obtains (block 34) tone map data 15 corresponding to HDR image data 12. Tone map data 15 represents a likeness of the image of HDR image 12, but has a lower dynamic range than HDR image data 12. Tone map data 15 may be generated from HDR image data 12, as indicated by line 13, or in some other manner derived from data having a common source with HDR image data 12. If tone map data 15 is not derived from HDR image data 12 then the order in which tone map data 15 and HDR image data 12 are obtained (i.e. the order of blocks 32 and 34) is not important.) (Wu: column 4 lines 65-67, column 5, column 6 lines 1-5, Figures 1-2; The luminance ratio LMR[Yin] can be a linear or non-linear function implemented in a lookup table (LUT) or a piecewise linear function (PWL). In some implementations, it may be more efficient to use an LUT with a high number of reference points than a PWL, because the luminance ratio function may tend to have a faster transition curve than direct R,G,B component mapping 114. Accordingly, looking up luminance ratio values LMR[Yin] in a LUT may be computationally faster and reduce latency in the processing system. In other implementations, the luminance ratio linear or non-linear function may be dynamically calculated, rather than using a precomputed LUT or PWL approximation.
The luminance mapping ratio technique 112 can be used for dynamic range mapping (e.g. SDR to HDR and HDR to SDR), nonlinear format conversions, system gamma adjustments, or for any other type and form of conversion. For an example of SDR to HDR conversion, as discussed above, luminance ratio function LMR[Yin] is the luminance ratio of HDR versus SDR referenced to each input SDR luminance value. If an input SDR pixel is at 100 nits, and the corresponding output HDR pixel is mapped to 300 nits, the luminance ratio LMR[Yin] is 3. As discussed above, the ratio function can be implemented in a lookup table. In some implementations, the multiplicand scale factor may be 1.0 and the additive coefficient may be 0.0 for a typical black level or unadjusted gamma offset. The output for each channel is therefore R_out=3*R_in, G_out=3*G_in, B_out=3*B_in.
The luminance mapping ratio technique 112 only adjusts luminance and does not cause any undesired hue shift, even when the luminance transfer function is highly non-linear and the R,G,B, component values are quite different within the same pixel.
The second technique, direct R,G,B component mapping 114, may use a LUT or PWL to directly map an input color component value of a pixel to an output color component value for the pixel in a different format. In some implementations, each component output of R,G,B linear light components may be directly mapped from each input R,G,B linear light component separately through the same linear or non-linear function. The non-linear function can be implemented through a LUT or PWL. Direct R,G,B component mapping 114 can also be used for dynamic range mapping (e.g. SDR to HDR and HDR to SDR), nonlinear format conversions, system gamma adjustments, or any other such format change. For example, in one implementation, given an input SDR pixel with input values of R_in =50, G_in=100, B_in=20, and SDR mapped to HDR in a linear scale of 3, R_out=150, G_out=300, and B_out=60. The same scaling or mapping may be used for each channel, or different mapping per channel. Furthermore, mappings may be non-linear across the pixel range. For example, in one implementation, R_out/R_in may be 3 as discussed above for an input of R_in<100, but may be 2 for an input of R_in>100. The non-linear function may have as many distinct points or mapping regions as desired.
In some implementations, unlike luminance mapping ratio 112, direct R,G,B mapping 114 may result in saturation and hue variations, especially for high luminance level inputs. However, saturation variation can be beneficial in some cases. Additionally, direct R,G,B component mapping 114 tends to be smoother in implementation. It is particularly suitable for PWL-based implementations that utilize fewer reference points than LUT-based implementations for luminance mapping ratio techniques 112, and may accordingly reduce hardware expenses for implementation. In further implementations, luminance mapping and component mapping techniques may be combined to reduce hue variations.
FIG. 2 is a block diagram of an implementation of a computing device for luminance processing during conversions between media formats, including SDR and HDR media formats. A device 200 may comprise a video processor, a set top box, a video router or distribution amplifier, a splitter, an X-Y interface, a switcher, a compositor, or any other type and form of appliance or computing device.)
Consider Claims 18.
The combination of Ward, Wu and Lim teaches:
18. (Original) The method of claim 17, wherein the one or more characteristics of the respective derived videos that differ from each other include a dynamic range, wherein a greater dynamic range is provided from applying a first recovery map track of the multiple recovery map tracks, and a lower dynamic range is provided from applying a second recovery map track of the multiple recovery map tracks. (Ward [0022] System 10 comprises an encoder 14 for creating a HDR image data structure 16 based on original HDR image data 12. Data structure 16 may be decoded by a standard decoder 18 to provide a standard dynamic range image 19. In some embodiments of the invention Standard decoder 18 comprises a “legacy” hardware decoder or software-based decoder such as suitable image viewer software. Data structure 16 may be decoded by a HDR decoder 20 to yield a reconstructed HDR image 21. [0023] "acquiring HDR image data 12"; [0023] Method 30 begins in block 32 by acquiring HDR image data 12. HDR image data 12 includes information that directly or indirectly specifies the luminance of pixels in an image. HDR image data 12 may be in any suitable format and may be acquired through the use of a suitable HDR camera (possibly by combining multiple exposures) or rendered directly in a computer. The source of HDR image data 12 is not important to the practice of the invention. [0024] Method 30 also obtains (block 34) tone map data 15 corresponding to HDR image data 12. Tone map data 15 represents a likeness of the image of HDR image 12, but has a lower dynamic range than HDR image data 12. Tone map data 15 may be generated from HDR image data 12, as indicated by line 13, or in some other manner derived from data having a common source with HDR image data 12. If tone map data 15 is not derived from HDR image data 12 then the order in which tone map data 15 and HDR image data 12 are obtained (i.e. the order of blocks 32 and 34) is not important.) (Wu: column 4 lines 65-67, column 5, column 6 lines 1-5, Figures 1-2; The luminance ratio LMR[Yin] can be a linear or non-linear function implemented in a lookup table (LUT) or a piecewise linear function (PWL). In some implementations, it may be more efficient to use an LUT with a high number of reference points than a PWL, because the luminance ratio function may tend to have a faster transition curve than direct R,G,B component mapping 114. Accordingly, looking up luminance ratio values LMR[Yin] in a LUT may be computationally faster and reduce latency in the processing system. In other implementations, the luminance ratio linear or non-linear function may be dynamically calculated, rather than using a precomputed LUT or PWL approximation.
The luminance mapping ratio technique 112 can be used for dynamic range mapping (e.g. SDR to HDR and HDR to SDR), nonlinear format conversions, system gamma adjustments, or for any other type and form of conversion. For an example of SDR to HDR conversion, as discussed above, luminance ratio function LMR[Yin] is the luminance ratio of HDR versus SDR referenced to each input SDR luminance value. If an input SDR pixel is at 100 nits, and the corresponding output HDR pixel is mapped to 300 nits, the luminance ratio LMR[Yin] is 3. As discussed above, the ratio function can be implemented in a lookup table. In some implementations, the multiplicand scale factor may be 1.0 and the additive coefficient may be 0.0 for a typical black level or unadjusted gamma offset. The output for each channel is therefore R_out=3*R_in, G_out=3*G_in, B_out=3*B_in.
The luminance mapping ratio technique 112 only adjusts luminance and does not cause any undesired hue shift, even when the luminance transfer function is highly non-linear and the R,G,B, component values are quite different within the same pixel.
The second technique, direct R,G,B component mapping 114, may use a LUT or PWL to directly map an input color component value of a pixel to an output color component value for the pixel in a different format. In some implementations, each component output of R,G,B linear light components may be directly mapped from each input R,G,B linear light component separately through the same linear or non-linear function. The non-linear function can be implemented through a LUT or PWL. Direct R,G,B component mapping 114 can also be used for dynamic range mapping (e.g. SDR to HDR and HDR to SDR), nonlinear format conversions, system gamma adjustments, or any other such format change. For example, in one implementation, given an input SDR pixel with input values of R_in =50, G_in=100, B_in=20, and SDR mapped to HDR in a linear scale of 3, R_out=150, G_out=300, and B_out=60. The same scaling or mapping may be used for each channel, or different mapping per channel. Furthermore, mappings may be non-linear across the pixel range. For example, in one implementation, R_out/R_in may be 3 as discussed above for an input of R_in<100, but may be 2 for an input of R_in>100. The non-linear function may have as many distinct points or mapping regions as desired.
In some implementations, unlike luminance mapping ratio 112, direct R,G,B mapping 114 may result in saturation and hue variations, especially for high luminance level inputs. However, saturation variation can be beneficial in some cases. Additionally, direct R,G,B component mapping 114 tends to be smoother in implementation. It is particularly suitable for PWL-based implementations that utilize fewer reference points than LUT-based implementations for luminance mapping ratio techniques 112, and may accordingly reduce hardware expenses for implementation. In further implementations, luminance mapping and component mapping techniques may be combined to reduce hue variations.
FIG. 2 is a block diagram of an implementation of a computing device for luminance processing during conversions between media formats, including SDR and HDR media formats. A device 200 may comprise a video processor, a set top box, a video router or distribution amplifier, a splitter, an X-Y interface, a switcher, a compositor, or any other type and form of appliance or computing device.)
Consider Claims 12, 21 and 23.
The combination of Ward, Wu and Lim teaches:
12. (Currently Amended) The method of claim 1, further comprising: obtaining the video container; determining to display the second video by the second display device; scaling a plurality of pixel luminances of the first frames of [[the ]]a first video in the [[frame ]]video container based on a particular luminance output of the second display device and based on the corresponding recovery map frames to obtain the derived frames; and after the scaling, causing the derived frames to be displayed by the second display device as output frames that have a different dynamic range than the first frame. / 21. (Currently Amended) The method of claim 19, wherein the display device is capable of displaying a display dynamic range that is different than the first dynamic range, and wherein applying the gains of the recovery map frames includes adapting the luminances of the pixel values of the corresponding derived frames to the display dynamic range of the display device. / 23. (Currently Amended) The method of claim 19, wherein in response to determining to display the derived video, applying the gains of the recovery map frames includes: scaling the luminances of the first frames based on a particular luminance output of the display device and based on the corresponding recovery map frames. (Ward [0022] System 10 comprises an encoder 14 for creating a HDR image data structure 16 based on original HDR image data 12. Data structure 16 may be decoded by a standard decoder 18 to provide a standard dynamic range image 19. In some embodiments of the invention Standard decoder 18 comprises a “legacy” hardware decoder or software-based decoder such as suitable image viewer software. Data structure 16 may be decoded by a HDR decoder 20 to yield a reconstructed HDR image 21. [0023] "acquiring HDR image data 12"; [0023] Method 30 begins in block 32 by acquiring HDR image data 12. HDR image data 12 includes information that directly or indirectly specifies the luminance of pixels in an image. HDR image data 12 may be in any suitable format and may be acquired through the use of a suitable HDR camera (possibly by combining multiple exposures) or rendered directly in a computer. The source of HDR image data 12 is not important to the practice of the invention. [0024] Method 30 also obtains (block 34) tone map data 15 corresponding to HDR image data 12. Tone map data 15 represents a likeness of the image of HDR image 12, but has a lower dynamic range than HDR image data 12. Tone map data 15 may be generated from HDR image data 12, as indicated by line 13, or in some other manner derived from data having a common source with HDR image data 12. If tone map data 15 is not derived from HDR image data 12 then the order in which tone map data 15 and HDR image data 12 are obtained (i.e. the order of blocks 32 and 34) is not important.) (Wu: column 4 lines 65-67, column 5, column 6 lines 1-5, Figures 1-2; The luminance ratio LMR[Yin] can be a linear or non-linear function implemented in a lookup table (LUT) or a piecewise linear function (PWL). In some implementations, it may be more efficient to use an LUT with a high number of reference points than a PWL, because the luminance ratio function may tend to have a faster transition curve than direct R,G,B component mapping 114. Accordingly, looking up luminance ratio values LMR[Yin] in a LUT may be computationally faster and reduce latency in the processing system. In other implementations, the luminance ratio linear or non-linear function may be dynamically calculated, rather than using a precomputed LUT or PWL approximation.
The luminance mapping ratio technique 112 can be used for dynamic range mapping (e.g. SDR to HDR and HDR to SDR), nonlinear format conversions, system gamma adjustments, or for any other type and form of conversion. For an example of SDR to HDR conversion, as discussed above, luminance ratio function LMR[Yin] is the luminance ratio of HDR versus SDR referenced to each input SDR luminance value. If an input SDR pixel is at 100 nits, and the corresponding output HDR pixel is mapped to 300 nits, the luminance ratio LMR[Yin] is 3. As discussed above, the ratio function can be implemented in a lookup table. In some implementations, the multiplicand scale factor may be 1.0 and the additive coefficient may be 0.0 for a typical black level or unadjusted gamma offset. The output for each channel is therefore R_out=3*R_in, G_out=3*G_in, B_out=3*B_in.
The luminance mapping ratio technique 112 only adjusts luminance and does not cause any undesired hue shift, even when the luminance transfer function is highly non-linear and the R,G,B, component values are quite different within the same pixel.
The second technique, direct R,G,B component mapping 114, may use a LUT or PWL to directly map an input color component value of a pixel to an output color component value for the pixel in a different format. In some implementations, each component output of R,G,B linear light components may be directly mapped from each input R,G,B linear light component separately through the same linear or non-linear function. The non-linear function can be implemented through a LUT or PWL. Direct R,G,B component mapping 114 can also be used for dynamic range mapping (e.g. SDR to HDR and HDR to SDR), nonlinear format conversions, system gamma adjustments, or any other such format change. For example, in one implementation, given an input SDR pixel with input values of R_in =50, G_in=100, B_in=20, and SDR mapped to HDR in a linear scale of 3, R_out=150, G_out=300, and B_out=60. The same scaling or mapping may be used for each channel, or different mapping per channel. Furthermore, mappings may be non-linear across the pixel range. For example, in one implementation, R_out/R_in may be 3 as discussed above for an input of R_in<100, but may be 2 for an input of R_in>100. The non-linear function may have as many distinct points or mapping regions as desired.
In some implementations, unlike luminance mapping ratio 112, direct R,G,B mapping 114 may result in saturation and hue variations, especially for high luminance level inputs. However, saturation variation can be beneficial in some cases. Additionally, direct R,G,B component mapping 114 tends to be smoother in implementation. It is particularly suitable for PWL-based implementations that utilize fewer reference points than LUT-based implementations for luminance mapping ratio techniques 112, and may accordingly reduce hardware expenses for implementation. In further implementations, luminance mapping and component mapping techniques may be combined to reduce hue variations.
FIG. 2 is a block diagram of an implementation of a computing device for luminance processing during conversions between media formats, including SDR and HDR media formats. A device 200 may comprise a video processor, a set top box, a video router or distribution amplifier, a splitter, an X-Y interface, a switcher, a compositor, or any other type and form of appliance or computing device.)
Consider Claims 24.
The combination of Ward, Wu and Lim teaches:
24. (Currently Amended) The method of claim 23, wherein the scaling of the luminances of the first frames is performed based on:derivedframe(x, y) = firstframe(x, y) + log(display_factor) * recovery(x,y) wherein derived_frame(x,y) is a logarithmic space version of the corresponding derived frame, first_frame(x,y) is a logarithmic space version of the first frame recovered from the frame video container, display factor is a minimum of a range scaling factor and a maximum display luminance of the second display device, recovery(x,y) is the recovery map for pixel position (x, y) of the first frame, and the range scaling factor is a ratio of a maximum luminance of the corresponding second frame to a maximum luminance of the first frame. (Ward: [0025] Encoder 14 generates a data structure 16. Data structure 16 includes a tone map portion 16A, which is based on tone map data 15, and an HDR information portion 16B, which contains information that may be combined by HDR decoder 20 with the data from tone map portion 16A to reconstruct HDR image data 12 or a close approximation thereto. Method 30 generates (block 36) HDR information portion by comparing tone map data 15 (or, equivalently, tone map data reconstructed from tone map portion 16A of data structure 16) and HDR image data 12. In block 38, method 30 stores tone map portion 16A and HDR information portion 16B in data structure 16. [0026] In preferred embodiments of the invention, data structure 16 has a format that can be read by a standard decoder to produce a lower dynamic range (LDR) image. A standard decoder 18 may implement decoding method 31A. Standard decoder 18 generates a standard LDR image 19 by retrieving tone map portion 16A and displaying an image represented by tone map portion 16A (block 39). The standard decoder can ignore HDR information portion 16B. [0046] In some embodiments of the invention, ratio data 57 comprises some function of the ratio of values of HDR image data 12 to corresponding values specified by tone map data 15 (or tone map portion 16A). For example, ratio data 57 may comprise information specifying a logarithm of such a ratio. [0054] Block 70 generates ratio data 96 by applying a function which takes as inputs values from HDR image data 90 (first values) and corresponding values from reconstructed tone map data 94 (second values). The function includes dividing the first values by the second values or vice versa. In a simple embodiment of the invention, ratio data 96 includes a value RI for each pixel in an image given by:
PNG
media_image1.png
49
314
media_image1.png
Greyscale
where: (x, y) are coordinates identifying a pixel; L is a function which returns the luminance of the pixel from data for the pixel; HDR(x, y) is the pixel data in HDR image data 90 at coordinates (x, y); and, TM(x, y) is the pixel data in reconstructed tone map data 94 (or, tone map data 91) for the pixel at coordinates (x, y). In some embodiments, the ratio data stores the logarithm of RI, the square root of RI or another function of RI. Lim: [0129] The operator 500 may be embodied with a touch screen, a touch pad, a key button, a key pad, etc., to provide user operation of the display apparatus 1000. According to an embodiment, it will be exemplified that a control command is received through the operator 500 included in the display apparatus 1000, but the operator 500 may receive a user operation from an external control device (e.g., a remote controller). [0130] The audio outputter 600 may perform signal processing such as decoding on audio data input from the receiver 300 and the storage 400, and output the audio data. The audio outputter 600 may be embodied as a speaker, etc.)
Consider Claims 27 and 28.
The combination of Ward, Wu and Lim teaches:
27. (Currently Amended) The method of claim 19,further comprising, in response to determining to display the derived video, decoding additional information from the video container, the additional information including the range scaling factor. / 28. (Currently Amended) The method of claim 19, further comprising: decoding the recovery map track from a block included in the video container that is separate from the first video. (Ward: [0025] Encoder 14 generates a data structure 16. Data structure 16 includes a tone map portion 16A, which is based on tone map data 15, and an HDR information portion 16B, which contains information that may be combined by HDR decoder 20 with the data from tone map portion 16A to reconstruct HDR image data 12 or a close approximation thereto. Method 30 generates (block 36) HDR information portion by comparing tone map data 15 (or, equivalently, tone map data reconstructed from tone map portion 16A of data structure 16) and HDR image data 12. In block 38, method 30 stores tone map portion 16A and HDR information portion 16B in data structure 16. [0026] In preferred embodiments of the invention, data structure 16 has a format that can be read by a standard decoder to produce a lower dynamic range (LDR) image. A standard decoder 18 may implement decoding method 31A. Standard decoder 18 generates a standard LDR image 19 by retrieving tone map portion 16A and displaying an image represented by tone map portion 16A (block 39). The standard decoder can ignore HDR information portion 16B. [0026]-[0029], [0027] Data structure 16 can also be read by an HDR decoder 20. HDR decoder 20 implements decoding method 31B and generates an HDR image 21 based upon information from both tone map portion 16A and HDR information portion 16B. Method 31B retrieves data from tone map portion 16A and HDR information portion 16B of data structure 16 in block 40. In block 42, a reconstructed HDR image is created by modifying a tone map extracted from tone map portion 16A according to HDR information from HDR information portion 16B. The reconstructed HDR image is displayed in block 44. Lim: [0129] The operator 500 may be embodied with a touch screen, a touch pad, a key button, a key pad, etc., to provide user operation of the display apparatus 1000. According to an embodiment, it will be exemplified that a control command is received through the operator 500 included in the display apparatus 1000, but the operator 500 may receive a user operation from an external control device (e.g., a remote controller). [0130] The audio outputter 600 may perform signal processing such as decoding on audio data input from the receiver 300 and the storage 400, and output the audio data. The audio outputter 600 may be embodied as a speaker, etc.)
Consider Claims 26 and 29-30.
26. (Cancelled)
29. (Cancelled)
30. (Cancelled)
Claims 3-4 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Ward et al. (US 2012/0230597) in view of Wu et al. (US 10148907) and Lim et al. (US 2020/0152111) further in view of Piramanagayam et. al. (US PGPub US 20180115777 A1, hereby referred to as “Piramanayagam”).
Consider Claims 3 and 20.
The combination of Ward, Wu and Lim teaches The method of Claim 1, and The method of Claim 19.
20. (Original) The method of claim 19, wherein the video container (Lim [0019] "The second image quality compensation data may be the first image quality compensation data adjusted based on a ratio between the first maximum luminance value and the second maximum luminance value"). [0067]-[0068], [0078] According to an example, the first image quality compensation data may include image quality compensation data corrected based on the first maximum luminance (l1) and the second maximum luminance (l2). [0079] The first image quality compensation data may include image quality compensation data in which image quality compensation data corresponding to the driving frequency of the display panel is reduced based on a ratio between the first maximum luminance (l1) and the second maximum luminance (l2).)
The combination of Ward, Wu and Lim does not teach: “a metadata track” as required by claims 3 & 20.
Piramanayagam teaches:
1. (Currently Amended) A computer-implemented method comprising: / 19. (Currently Amended) A computer-implemented method comprising: (Piramanayagam: Methods for screen-adaptive decoding of video with high dynamic range (HDR) are described. The methods combine the traditional compositing and display management steps into one screen-adaptive compositing step. Given decoded standard dynamic range (SDR) input data, metadata related to the prediction of output HDR data in a reference dynamic range, and the dynamic range of a target display, new output luma and chroma prediction functions are generated that map directly the input SDR data to output HDR data in the target dynamic range, thus eliminating the display management step. [0020]-[0021] Example Video Delivery Processing Pipeline [0022] FIG. 1 depicts an example process of a conventional video delivery pipeline (100) showing various stages from video capture to video content display. A sequence of video frames (102) is captured or generated using image generation block (105). Video frames (102) may be digitally captured (e.g. by a digital camera) or generated by a computer (e.g. using computer animation) to provide video data (107). Alternatively, video frames (102) may be captured on film by a film camera. The film is converted to a digital format to provide video data (107). In a production phase (110), video data (107) is edited to provide a video production stream (112).)
3. (Currently Amended) The method of claim 1, further comprising generating a metadata track that includes respective metadata frames related to corresponding recovery map frames, wherein providing the first video and the recovery map track in the video container includes providing the metadata track in the video container. (Piramanayagam: [0024] Following post-production (115), video data of final production (117) may be delivered to encoding block (120) for delivering downstream to decoding and playback devices such as television sets, set-top boxes, movie theaters, and the like. In some embodiments, coding block (120) may include audio and video encoders, such as those defined by ATSC, DVB, DVD, Blu-Ray, and other delivery formats, to generate coded bit stream (122). Coded bitstream (122) may be represented by a single layer video coded bitstream or by a multi-layer bitstream. For example, in a multi-layer bitstream, signal (122) may include a base layer (say, an SDR layer or a 10-bit HDR (HDR10) layer) and an enhancement layer, which when combined with the base layer yields a HDR bitstream with higher dynamic range than the base layer alone (e.g., a 12-bit HDR signal). In addition to the signal (122), the output bitstream from the encoder may also include metadata (124). Metadata (124) may include prediction parameters and other data to assist a decoder to better reconstruct an HDR signal. For example, in an embodiment, metadata (124) may include parameters related to a backward reshaping function that needs to be applied to the decoded signal (e.g., 132) after decompression (e.g., 130). [0025] In a receiver (100D), the coded bit stream (122) is decoded by decoding unit (130) to generate a decoded signal (132). Given decoded signal (132) (which may be represented by a single layer or multiple layers) and metadata (124), composer (135) generates signal (137) representing an identical or close approximation of signal (117) at either a standard dynamic range or a high dynamic range, depending on the capabilities of target (or receiver) display (150). For example, in an embodiment with a dual-layer coded signal, the composer may use metadata to derive a prediction function, generate HDR predicted data based on the base layer, and then combine the HDR predicted data with the enhancement layer to generate the output HDR signal. In an embodiment with a reshaped single layer signal, the composer may use the metadata to reconstruct a reverse (or backward) reshaping function which it can apply to the received signal (132) to generate an output HDR signal.)
20. (Original) The method of claim 19, wherein the video container includes a metadata track that includes respective range scaling factors that are associated with the recovery map frames of the recovery map track, wherein each of the respective range scaling factors is used to scale the luminance gains of pixels of the corresponding first frame, the luminance gains provided by the associated recovery map frame. (Piramanayagam: [0045] In equations (3) and (4), i represents a pixel index value, j represents a frame index value, y denotes luma (e.g., Y) and u, and v denote chroma (e.g., Cb, Cr). mj,k u,100→4k denotes the k-th MMR coefficient to convert reference SDR values (e.g., at 100 nits) to reference HDR (e.g., at 4k nits). While the values 100 and 4k are quite common when grading HDR content, here they are given without limitation, simply as notation examples. Typically, the MMR coefficients are generated at the encoder for each frame or scene of the video by some optimization techniques, such as least square minimization between the input HDR values and predicted HDR values using a reference SDR stream. The coefficients are then transmitted as a metadata alongside with the SDR image. At the decoder, these coefficients are used to convert SDR chroma to HDR chroma. Note that the number of coefficients depends on the order of luma and chroma being used. As an example and without limitation, for the two channel MMR scheme, the chroma channel has 10 MMR coefficients for the Cb channel and 10 MMR coefficients for the Cr channel (e.g., see equation (3)). For the three channel MMR model, 22 MMR coefficients are being used (e.g., see equation 4)). In the rest of this discussion, embodiments will use the two-channel MMR model; however, the methods described here can easily be adapted to the three-channel MMR model or any other linear or non-linear prediction model.)
It would have been obvious to one of ordinary skill in the art to modify the combination of Ward, Wu and Lim with the teachings of Piramanayagam for representation of HDR dynamic range data in the metadata for display management. The determination of obviousness is predicated upon the following findings: One skilled in the art would have been motivated to modify the data structure of the combination of Ward, Wu and Lim in order to include additional luma prediction function parameters in the metadata as disclosed by Piramanayagam. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of the combination of Ward, Wu and Lim, while the teaching of Piramanayagam continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of ensure that the luminance values is more accurately corrected using parameters associated with the luma prediction function in the metadata as disclosed by Piramanayagam (Piramanayagam: [0020]). It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claim 4.
The combination of Ward, Wu, Lim and Piramanayagam teaches:
4. (Currently Amended) The method of claim 3, wherein the gains in luminances are scaled by a range scaling factor that includes a ratio of a maximum luminance of the corresponding second frame to a maximum luminance of the first frame, wherein the respective metadata frames in the [[timed ]]metadata track include a respective range scaling factor associated with an associated recovery map frame of the recovery map track. (Lim [0019] "The second image quality compensation data may be the first image quality compensation data adjusted based on a ratio between the first maximum luminance value and the second maximum luminance value"). [0067]-[0068], [0078] According to an example, the first image quality compensation data may include image quality compensation data corrected based on the first maximum luminance (l1) and the second maximum luminance (l2). [0079] The first image quality compensation data may include image quality compensation data in which image quality compensation data corresponding to the driving frequency of the display panel is reduced based on a ratio between the first maximum luminance (l1) and the second maximum luminance (l2). Piramanayagam: [0045] In equations (3) and (4), i represents a pixel index value, j represents a frame index value, y denotes luma (e.g., Y) and u, and v denote chroma (e.g., Cb, Cr). mj,k u,100→4k denotes the k-th MMR coefficient to convert reference SDR values (e.g., at 100 nits) to reference HDR (e.g., at 4k nits). While the values 100 and 4k are quite common when grading HDR content, here they are given without limitation, simply as notation examples. Typically, the MMR coefficients are generated at the encoder for each frame or scene of the video by some optimization techniques, such as least square minimization between the input HDR values and predicted HDR values using a reference SDR stream. The coefficients are then transmitted as a metadata alongside with the SDR image. At the decoder, these coefficients are used to convert SDR chroma to HDR chroma. Note that the number of coefficients depends on the order of luma and chroma being used. As an example and without limitation, for the two channel MMR scheme, the chroma channel has 10 MMR coefficients for the Cb channel and 10 MMR coefficients for the Cr channel (e.g., see equation (3)). For the three channel MMR model, 22 MMR coefficients are being used (e.g., see equation 4)). In the rest of this discussion, embodiments will use the two-channel MMR model; however, the methods described here can easily be adapted to the three-channel MMR model or any other linear or non-linear prediction model. Piramanayagam: [0045] In equations (3) and (4), i represents a pixel index value, j represents a frame index value, y denotes luma (e.g., Y) and u, and v denote chroma (e.g., Cb, Cr). mj,k u,100→4k denotes the k-th MMR coefficient to convert reference SDR values (e.g., at 100 nits) to reference HDR (e.g., at 4k nits). While the values 100 and 4k are quite common when grading HDR content, here they are given without limitation, simply as notation examples. Typically, the MMR coefficients are generated at the encoder for each frame or scene of the video by some optimization techniques, such as least square minimization between the input HDR values and predicted HDR values using a reference SDR stream. The coefficients are then transmitted as a metadata alongside with the SDR image. At the decoder, these coefficients are used to convert SDR chroma to HDR chroma. Note that the number of coefficients depends on the order of luma and chroma being used. As an example and without limitation, for the two channel MMR scheme, the chroma channel has 10 MMR coefficients for the Cb channel and 10 MMR coefficients for the Cr channel (e.g., see equation (3)). For the three channel MMR model, 22 MMR coefficients are being used (e.g., see equation 4)). In the rest of this discussion, embodiments will use the two-channel MMR model; however, the methods described here can easily be adapted to the three-channel MMR model or any other linear or non-linear prediction model.)
Consider Claims 5-7.
5. (Cancelled)
6. (Cancelled)
7. (Cancelled)
Conclusion
The prior art made of record in form PTO-892 and not relied upon is considered pertinent to applicant's disclosure.
PNG
media_image2.png
195
910
media_image2.png
Greyscale
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAHMINA ANSARI whose telephone number is 571-270-3379. The examiner can normally be reached on IFP Flex - Monday through Friday 9 to 5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’NEAL MISTRY can be reached on 313-446-4912. The fax phone numbers for the organization where this application or proceeding is assigned are 571-273-8300 for regular communications and 571-273-8300 for After Final communications. TC 2600’s customer service number is 571-272-2600.
Any inquiry of a general nature or relating to the status of this application or proceeding should be directed to the receptionist whose telephone number is 571-272-2600.
2674
/Tahmina Ansari/
September 23, 2026
/TAHMINA N ANSARI/Primary Examiner, Art Unit 2674
