DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA ..
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claim 1 currently examined application (19/015987) rejected on the ground of nonstatutory obviousness-type double patenting as being unpatentable over claim 1 of U.S. Patent No. 19/015992. Although the claims at issue are not identical, they are not patentably distinct from each other because it would have been obvious to one of ordinary skill in the art that the claim language provided in claim 1 of the current application 19/015987: “An image enhancement system comprising: an ambient light sensor operational to measure an ambient light level; a circuit operational to: generate a histogram based on an input video signal; export the histogram; receive a gray shade look up table; and generate an output video signal by converting a plurality of gray shades in the input video signal based on the gray shade look up table; and a processor operational to: receive the histogram from the circuit; develop the gray shade look up table based on the histogram and the ambient light level; and transfer the gray shade look up table to the circuit.”, with a slight variation of from claim 1 of application 19/015992: “An image enhancement system comprising: an ambient light sensor operational to measure an ambient light level; a circuit operational to: generate a histogram based on an input video signal; export the histogram; receive a gray shade look up table; and generate an output video signal by converting a plurality of gray shades in the input video signal based on the gray scale look up table; and a processor operational to: receive the histogram from the circuit; develop the gray shade look up table based on the histogram and the ambient light level; dynamically remap the plurality of gray shades in the gray shade look up table in response to the ambient light level to compress a first region of the plurality of gray shades and stretch a second region of the plurality of gray shades; and transfer the gray shade look up table to the circuit”, is similar in scope and would provide analogous generation of stylized painting effects. As indicated below, one of which discloses claims from the current application that are also rejected on the ground of nonstatutory obviousness-type double patenting and the second table indicating a mapping between the current and the discovered application.
Current Application: 19/015987
Claims 1-20
Application: 19/015992
Claims 1-20
Current Application: 19/015987
Application: 19/015992
An image enhancement system comprising:
an ambient light sensor operational to measure an ambient light level;
a circuit operational to: generate a histogram based on an input video signal;
export the histogram; receive a gray shade look up table;
and generate an output video signal by converting a plurality of gray shades in the input video signal based on the gray shade look up table;
and a processor operational to: receive the histogram from the circuit;
develop the gray shade look up table based on the histogram and the ambient light level;
and transfer the gray shade look up table to the circuit.
An image enhancement system comprising:
an ambient light sensor operational to measure an ambient light level;
a circuit operational to: generate a histogram based on an input video signal;
export the histogram; receive a gray shade look up table;
and generate an output video signal by converting a plurality of gray shades in the input video signal based on the gray scale look up table;
and a processor operational to: receive the histogram from the circuit;
develop the gray shade look up table based on the histogram and the ambient light level;
dynamically remap the plurality of gray shades in the gray shade look up table in response to the ambient light level to compress a first region of the plurality of gray shades and stretch a second region of the plurality of gray shades;
and transfer the gray shade look up table to the circuit
Claim 1 currently examined application (19/015987) rejected on the ground of nonstatutory obviousness-type double patenting as being unpatentable over claim 1 of U.S. Patent No. 19/016057. Although the claims at issue are not identical, they are not patentably distinct from each other because it would have been obvious to one of ordinary skill in the art that the claim language provided in claim 1 of the current application 19/015987: “An image enhancement system comprising: an ambient light sensor operational to measure an ambient light level; a circuit operational to: generate a histogram based on an input video signal; export the histogram; receive a gray shade look up table; and generate an output video signal by converting a plurality of gray shades in the input video signal based on the gray shade look up table; and a processor operational to: receive the histogram from the circuit; develop the gray shade look up table based on the histogram and the ambient light level; and transfer the gray shade look up table to the circuit.”, with a slight variation of from claim 1 of application 19/016057: “A true color image enhancement compensation system comprising: an ambient light sensor operational to measure an ambient light level; a circuit operational to: generate a histogram based on an input video signal; export the histogram; receive a gray shade look up table; and generate an output video signal by converting a plurality of gray shades in the input video signal based on a plurality of input video luminance ratios and the gray shade look up table to determine a plurality of new output gray shade values to maintain input video color coordinates; and a processor operational to: receive the histogram from the circuit; develop the gray shade look up table based on the histogram and the ambient light level; dynamically remap the plurality of gray shades in the gray shade look up table in response to the ambient light signal to compress a first region of the plurality of gray shades and stretch a second region of the plurality of gray shades; dynamically remap the plurality of gray shades in the gray shade look up table to correct for ambient lighting conditions; and transfer the gray shade look up table to the circuit.”, is similar in scope and would provide analogous generation of stylized painting effects. As indicated below, one of which discloses claims from the current application that are also rejected on the ground of nonstatutory obviousness-type double patenting and the second table indicating a mapping between the current and the discovered application.
Current Application: 19/015987
Claims 1-20
Application: 19/016057
Claims 1-20
Current Application: 19/015987
Application: 19/016057
An image enhancement system comprising:
an ambient light sensor operational to measure an ambient light level;
a circuit operational to: generate a histogram based on an input video signal;
export the histogram; receive a gray shade look up table;
and generate an output video signal by converting a plurality of gray shades in the input video signal based on the gray shade look up table;
and a processor operational to: receive the histogram from the circuit;
develop the gray shade look up table based on the histogram and the ambient light level;
and transfer the gray shade look up table to the circuit.
A true color image enhancement compensation system comprising:
an ambient light sensor operational to measure an ambient light level;
a circuit operational to: generate a histogram based on an input video signal;
export the histogram; receive a gray shade look up table;
and generate an output video signal by converting a plurality of gray shades in the input video signal based on a plurality of input video luminance ratios and the gray shade look up table to determine a plurality of new output gray shade values to maintain input video color coordinates;
and a processor operational to: receive the histogram from the circuit;
develop the gray shade look up table based on the histogram and the ambient light level; dynamically remap the plurality of gray shades in the gray shade look up table in response to the ambient light signal to compress a first region of the plurality of gray shades and stretch a second region of the plurality of gray shades; dynamically remap the plurality of gray shades in the gray shade look up table to correct for ambient lighting conditions;
and transfer the gray shade look up table to the circuit.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 10, and 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Deepala et al. (U.S. Pub. No. 20190362686) in view of Jung et al (U.S. Pub. No. 20180108327) and Intel, Intel® Iris® Plus Graphics and UHD Graphics Open Source Programmer's Reference Manual Jan 2020, Vol 12.
Regarding claim 1, Deepala discloses an image enhancement system comprising (para 66, “FIG. 7 shows a diagram of a system 700 including a device 705 that supports histogram reads for efficient display post processing and improved power gains in accordance with aspects of the present disclosure.”; also, para 27, “CABL may, for example, include backlight reduction and/or pixel boosting (e.g., by applying a pixel mapping look-up table (LUT)) while FOSS may include pixel modification.”; also, Backlight reduction, pixel boosting, and modifying pixels indicates enhancements of an image. CABL, LUT, and FOSS also contribute to enhancing an image): processor operational to: receive the histogram from the circuit (para 48, “Aspects of the present disclosure relate to limiting the number of histogram read and analysis operations performed by host processor 305 for various use cases.”; also, para 28, “Such features (e.g., CABL, FOSS, etc.) may be based on reading a content histogram from hardware (e.g., a display pipeline) whenever there is a frame update. The histogram data may be passed through post-processing algorithms (e.g., proprietary algorithms), which algorithms may analyze the scene changes and identify transitions. In the case of a transition, the algorithms may program new pixel adjustment parameters (e.g., backlight values, LUT values, pixel boosting values, etc.) to the hardware drivers.”; also, Since when post processor is being limited by the number of histogram reads and analysis operations, the processor inherently receives histograms from the circuit (hardware)); develop the gray shade look up table based on the histogram (para 48, “In the case that a scene change is detected (e.g., based on a comparison), the processing algorithms may generate one or more pixel adjustment parameters 335, which may then be passed from compositor 315 to display driver 310 (e.g., in the form of a LUT write or backlight scale update 340).”; also, para 28, “Such features (e.g., CABL, FOSS, etc.) may be based on reading a content histogram from hardware (e.g., a display pipeline) whenever there is a frame update. The histogram data may be passed through post-processing algorithms (e.g., proprietary algorithms), which algorithms may analyze the scene changes and identify transitions. In the case of a transition, the algorithms may program new pixel adjustment parameters (e.g., backlight values, LUT values, pixel boosting values, etc.) to the hardware drivers.”; also, since when a scene change is being detected, the algorithm with pick this up and adjust the pixel parameter and pass that by means of a LUT) gray shade look up table to the circuit (para 48, “the case that a scene change is detected (e.g., based on a comparison), the processing algorithms may generate one or more pixel adjustment parameters 335, which may then be passed from compositor 315 to display driver 310 (e.g., in the form of a LUT write or backlight scale update 340).”; also, algorithm generate the pixel adjustments and transfer the resulting parameter in the form of an LUT to the display driver which is a form of a circuit). Deepala does not disclose an ambient light sensor operational to measure an ambient light level; a circuit operational to: generate a histogram based on an input video signal; export the histogram ;receive a gray shade look up table; and generate an output video signal by converting a plurality of gray shades in the input video signal based on the gray shade look up table and the ambient light level.
However, in a similar field of endeavor, Intel PRM discloses a circuit operational to: generate a histogram based on an input video signal (page 225, “The hardware histogram block generates image statistics based on the pixel stream input. These statistics are used by the Processing block to determine how much the backlight level can be reduced”; also, page 226, “The histogram is composed of 32 bins with each bin covering a range of 8 values for 8 bit pixel component values. The first bin covers the values 0 thru 7.”; also, input video signal can be referred to as the pixel stream input since when a display is collection of pixels); export the histogram (page 226, “Each bin of the saved histogram is readable by software.”); receive a gray shade look up table (page 227, “Find the enhancement factor from a Look Up Table (LUT) with 33 entries”; also, page 227, “Each entry of the LUT is programmable by software.”); and generate an output video signal by converting a plurality of gray shades in the input video signal based on the gray shade look up table (page 226, “The hardware enhancement block adjusts the pixel values sent to the display, compensating for the brightness loss due to lowering of the display backlight level.”; also, page 227, “LUT is addressed by the 6 most significant 6 bits of either HSV max(RGBin), or the Y channel after converting RGBin to YUV.”; also, page 227, “The final enhancement factor is derived by interpolating between the addressed LUT entry and the next entry, using the lower bits of the input.”; also, page 227, “The enhancement factor modifies each input pixel component value with the method selected by DPST_CTL Enhancement Mode.”; also, page 227, “Direct lookup mode replaces the input pixel value with the enhancement factor.”);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Deepala's invention of a system supporting histogram reads for display post processing in which pixel boosting is carried out by applying a pixel mapping look up table, a host processor performs the histogram read and analysis operations by reading a content histogram from hardware whenever there is a frame update, post processing algorithms develop new look up table values from that histogram data, and those values are passed as a look up table write to the display driver, with the features of the Intel PRM's invention of a hardware histogram block that generates image statistics from the pixel stream input into a histogram of thirty two bins each covering a range of eight values, of each bin of the saved histogram being readable by software, of a look up table of thirty three entries each programmable by software that is addressed by the most significant bits of each input pixel and from which an enhancement factor is found by interpolating between the addressed entry and the next, and of a hardware enhancement block that adjusts the pixel values sent to the display, that enhancement factor modifying each input pixel component value so that the input pixel value is replaced by the enhancement factor. The combination would have been obvious because Deepala and the Intel PRM describe the same arrangement of a display side hardware block that collects a per frame histogram and host software that reads that histogram and programs a pixel mapping look up table back into the display hardware, Deepala reading the content histogram from the display pipeline whenever there is a frame update and writing look up table values to the display driver, and the Intel PRM supplying the hardware side of that same arrangement in the form of a histogram block generating image statistics from the pixel stream input, bins readable by software, and a software programmable look up table addressed by each input pixel whose enhancement factor replaces that pixel value in the pixel values the enhancement block sends to the display, so that the two describe complementary halves of one practice and each supplies what the other leaves unstated. The result would have been the predictable one of Deepala's host side histogram read and look up table write operating upon the Intel PRM's hardware histogram block and software programmable look up table.
Jung discloses an ambient light sensor operational to measure an ambient light level (para 31, “In one embodiment, s.sub.g.sup.min and s.sub.g.sup.max may be derived by a non-linear mapping from an ambient light sensor's input. In one or more other embodiments, input 960 may be provided by an ambient light sensor”; also, para 23, “Values for s.sub.g.sup.min and s.sub.g.sup.max may depend upon the captured scene's overall brightness as determined by an ambient light sensor”) and the ambient light level (para 31, “An image's normalized histograms 935 may be used by LTM unit 940 to find initial local tone mapping curves 945 for each region as described (see FIGS. 4-6). An image's collection of LTMs may be represented by LUTs 950; one LUT for each region's LTM”; also, para 23, “Values for s.sub.g.sup.min and s.sub.g.sup.max may depend upon the captured scene's overall brightness as determined by an ambient light sensor”; also, para 31, “In one embodiment, s.sub.g.sup.min and s.sub.g.sup.max may be derived by a non-linear mapping from an ambient light sensor's input.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Deepala in view of the Intel PRM, in which a hardware histogram block generates image statistics from the pixel stream input, each bin of the saved histogram is readable by software, and host post processing algorithms develop and write back a software programmable look up table whose entries convert the input pixel values, with the features of Jung's invention of normalized histograms used to find local tone mapping curves that are represented by look up tables, and of the global minimum and maximum tone mapping slope values that shape those curves being derived by a non-linear mapping from an ambient light sensor's input and depending upon the overall brightness determined by that sensor. The combination would have been obvious because Jung develops the very same kind of table from the very same kind of input, a per-frame histogram, and adds the one input the combination lacks, teaching that the values controlling the shape of the tone mapping curve depend upon the overall brightness as determined by an ambient light sensor. Deepala and the Intel PRM already compute and write back a look up table from the frame histogram but take no account of the light in which the frame is viewed. A person of ordinary skill would therefore have supplied Jung's ambient light sensor input to the post processing algorithms that already develop the look up table values from the histogram, with the predictable result of a look up table whose gray shade assignments respond both to the content of the current frame and to the ambient light in which that frame is actually viewed.
Regarding claim 10, Deepala as modified by Intel and Jung discloses the image enhancement system according to claim 1, wherein Deepala further comprising: a display panel operational to generate a visible image based on the video output signal (para 26, “The refresh operation may include a host processor transferring a refreshed array of pixels to a display panel, which may subsequently display the refreshed array of pixels.”; also, para 43, “Display panel controller 215 may update display 220 (e.g., by communicating pixel array 105 over link 230). In some cases, display panel controller 215 may generate multiple frames for display 220 based at least in part on the set of pixel adjustment parameters.”).
Regarding claim 11, Deepala discloses a method for image enhancement comprising (para 74, “FIG. 8 shows a flowchart illustrating a method 800 that supports histogram reads for efficient display post processing and improved power gains in accordance with aspects of the present disclosure. The operations of method 800 may be implemented by a device or its components as described herein. For example, the operations of method 800 may be performed by a display panel controller as described with reference to FIGS. 6 and 7.”; also, para 25, “FIGS. 8 through 10 show flowcharts illustrating methods that support histogram reads for efficient display post processing and improved power gains in accordance with aspects of the present disclosure.”; also, para 27, “CABL may, for example, include backlight reduction and/or pixel boosting (e.g., by applying a pixel mapping look-up table (LUT)) while FOSS may include pixel modification.”): histogram at the processor from the circuit (para 28, “Aspects of the present disclosure relate to limiting the number of histogram read and analysis operations performed by host processor 305 for various use cases.”; also, para 48, “Such features (e.g., CABL, FOSS, etc.) may be based on reading a content histogram from hardware (e.g., a display pipeline) whenever there is a frame update. The histogram data may be passed through post-processing algorithms (e.g., proprietary algorithms), which algorithms may analyze the scene changes and identify transitions. In the case of a transition, the algorithms may program new pixel adjustment parameters (e.g., backlight values, LUT values, pixel boosting values, etc.) to the hardware drivers.”); developing a gray shade look up table with the processor based on the histogram (para 28, “In the case that a scene change is detected (e.g., based on a comparison), the processing algorithms may generate one or more pixel adjustment parameters 335, which may then be passed from compositor 315 to display driver 310 (e.g., in the form of a LUT write or backlight scale update 340).”; also, para 48, “Such features (e.g., CABL, FOSS, etc.) may be based on reading a content histogram from hardware (e.g., a display pipeline) whenever there is a frame update. The histogram data may be passed through post-processing algorithms (e.g., proprietary algorithms), which algorithms may analyze the scene changes and identify transitions. In the case of a transition, the algorithms may program new pixel adjustment parameters (e.g., backlight values, LUT values, pixel boosting values, etc.) to the hardware drivers.”) gray shade look up table from the processor to the circuit (para 48, “In the case that a scene change is detected (e.g., based on a comparison), the processing algorithms may generate one or more pixel adjustment parameters 335, which may then be passed from compositor 315 to display driver 310 (e.g., in the form of a LUT write or backlight scale update 340).”); generating a visible image based on the video output signal (para 26, “The refresh operation may include a host processor transferring a refreshed array of pixels to a display panel, which may subsequently display the refreshed array of pixels.”; also, para 43, “Display panel controller 215 may update display 220 (e.g., by communicating pixel array 105 over link 230). In some cases, display panel controller 215 may generate multiple frames for display 220 based at least in part on the set of pixel adjustment parameters.”). Deepala does not disclose measuring an ambient light level with an ambient light sensor; generating a histogram with a circuit based on an input video signal; exporting the histogram from the circuit to a processor and the ambient light level receiving the gray shade look up table at the circuit from the processor; generating an output video signal with the circuit by converting a plurality of gray shades in the input video signal based on the gray shade look up table.
However, in a similar field of endeavor, Intel discloses generating a histogram with a circuit based on an input video signal (page 225, “The hardware histogram block generates image statistics based on the pixel stream input. These statistics are used by the Processing block to determine how much the backlight level can be reduced.”; also, para 226, “The histogram is composed of 32 bins with each bin covering a range of 8 values for 8 bit pixel component values. The first bin covers the values 0 thru 7.”); exporting the histogram from the circuit to a processor (page 226, “Each bin of the saved histogram is readable by software.”), receiving the gray shade look up table at the circuit from the processor (page 227, “Find the enhancement factor from a Look Up Table (LUT) with 33 entries”; also, page 227, “Each entry of the LUT is programmable by software.”); generating an output video signal with the circuit by converting a plurality of gray shades in the input video signal based on the gray shade look up table (page 226, “The hardware enhancement block adjusts the pixel values sent to the display, compensating for the brightness loss due to lowering of the display backlight level.”; also, page 227, “LUT is addressed by the 6 most significant 6 bits of either HSV max(RGBin), or the Y channel after converting RGBin to YUV.”; also, page 227, “The final enhancement factor is derived by interpolating between the addressed LUT entry and the next entry, using the lower bits of the input.”; also, page 227, “The enhancement factor modifies each input pixel component value with the method selected by DPST_CTL Enhancement Mode.”; also, page 227, “Direct lookup mode replaces the input pixel value with the enhancement factor.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Deepala's invention of histogram reads for display post processing in which pixel boosting is carried out by applying a pixel mapping look up table, a host processor performs the histogram read and analysis operations by reading a content histogram from hardware whenever there is a frame update, post processing algorithms develop new look up table values from that histogram data, those values are passed as a look up table write to the display driver, and a display panel controller drives a pixel array that forms the images of a video stream a device is displaying, with the features of the Intel PRM's invention of a hardware histogram block that generates image statistics from the pixel stream input into a histogram of thirty two bins, of each bin of the saved histogram being readable by software, of a look up table of thirty three entries each programmable by software that is addressed by the most significant bits of each input pixel and from which an enhancement factor is found by interpolating between the addressed entry and the next, and of a hardware enhancement block that adjusts the pixel values sent to the display, that enhancement factor modifying each input pixel component value so that the input pixel value is replaced by the enhancement factor. The combination would have been obvious because Deepala and the Intel PRM describe the same arrangement of a display side hardware block that collects a per frame histogram and host software that reads that histogram and programs a pixel mapping look up table back into the display hardware, Deepala reading the content histogram from the display pipeline whenever there is a frame update and writing look up table values to the display driver, and the Intel PRM supplying the hardware side of that same arrangement in the form of a histogram block generating image statistics from the pixel stream input, bins readable by software, and a software programmable look up table addressed by each input pixel whose enhancement factor replaces that pixel value in the pixel values the enhancement block sends to the display, so that the two describe complementary halves of one practice. The result would have been the predictable one of Deepala's host side histogram read and look up table write operating upon the Intel PRM's hardware histogram block and software programmable look up table.
Jung discloses measuring an ambient light level with an ambient light sensor (para 31, “In one embodiment, s.sub.g.sup.min and s.sub.g.sup.max may be derived by a non-linear mapping from an ambient light sensor's input. In one or more other embodiments, input 960 may be provided by an ambient light sensor”; also, para 23, “Values for s.sub.g.sup.min and s.sub.g.sup.max may depend upon the captured scene's overall brightness as determined by an ambient light sensor”) and the ambient light level (para 31, “An image's normalized histograms 935 may be used by LTM unit 940 to find initial local tone mapping curves 945 for each region as described (see FIGS. 4-6). An image's collection of LTMs may be represented by LUTs 950; one LUT for each region's LTM”; also, para 23, “Values for s.sub.g.sup.min and s.sub.g.sup.max may depend upon the captured scene's overall brightness as determined by an ambient light sensor”; also, para 31, “In one embodiment, s.sub.g.sup.min and s.sub.g.sup.max may be derived by a non-linear mapping from an ambient light sensor's input.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Deepala in view of the Intel PRM, in which a hardware histogram block generates image statistics from the pixel stream input, each bin of the saved histogram is readable by software, and host post processing algorithms develop and write back a software programmable look up table whose entries convert the input pixel values, with the features of Jung's invention of normalized histograms used to find local tone mapping curves that are represented by look up tables, and of the global minimum and maximum tone mapping slope values that shape those curves being derived by a non-linear mapping from an ambient light sensor's input and depending upon the overall brightness determined by that sensor. The combination would have been obvious because Jung develops the very same kind of table from the very same kind of input, a per-frame histogram, and adds the one input the combination lacks, teaching that the values controlling the shape of the tone mapping curve depend upon the overall brightness as determined by an ambient light sensor. Deepala and the Intel PRM already compute and write back a look up table from the frame histogram but take no account of the light in which the frame is viewed. A person of ordinary skill would therefore have supplied Jung's ambient light sensor input to the post processing algorithms that already develop the look up table values from the histogram, with the predictable result of a look up table whose gray shade assignments respond both to the content of the current frame and to the ambient light in which that frame is actually viewed.
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Deepala et al. (U.S. Pub. No. 20190362686) in view of Jung et al (U.S. Pub. No. 20180108327), Intel, Intel® Iris® Plus Graphics and UHD Graphics Open Source Programmer's Reference Manual Jan 2020, Vol 12, and Kawaguchi (U.S. Pub. No. 20180040282).
Regarding claim 20, Deepala discloses generate an input video signal (para 31, “Frame analysis operation 100 includes pixel array 105, which may be generated in a variety of ways in accordance with the present disclosure. In some cases, pixel array 105 may be generated by a graphics processing unit (GPU) of a device. For example, the GPU may generate (e.g., or be involved in generating) a pixel array 105 for each frame in a sequence of frames (e.g., by performing one or more rendering operations to generate set of layers).”; also, para 32, “Examples are provided below in the context of a display panel controller, which may serve to complement the operations of the GPU and/or host processor.”);gray shade look up table based on the histogram (para 28, “Such features (e.g., CABL, FOSS, etc.) may be based on reading a content histogram from hardware (e.g., a display pipeline) whenever there is a frame update. The histogram data may be passed through post-processing algorithms (e.g., proprietary algorithms), which algorithms may analyze the scene changes and identify transitions. In the case of a transition, the algorithms may program new pixel adjustment parameters (e.g., backlight values, LUT values, pixel boosting values, etc.) to the hardware drivers.”; also, para 48, “In the case that a scene change is detected (e.g., based on a comparison), the processing algorithms may generate one or more pixel adjustment parameters 335, which may then be passed from compositor 315 to display driver 310 (e.g., in the form of a LUT write or backlight scale update 340).”) display panel coupled to the control unit and operational to generate a visible image based on the video output signal (para 26, “The refresh operation may include a host processor transferring a refreshed array of pixels to a display panel, which may subsequently display the refreshed array of pixels.”; also, para 43, “Display panel controller 215 may update display 220 (e.g., by communicating pixel array 105 over link 230). In some cases, display panel controller 215 may generate multiple frames for display 220 based at least in part on the set of pixel adjustment parameters.”). Deepala does not disclose a vehicle comprising: an ambient light sensor operational to measure an ambient light level; a control unit coupled to the ambient light sensor and operational to: generate a histogram based on the input video signal and the ambient light level and generate an output video signal by converting a plurality of gray shades in the input video signal based on the gray shade look up table.
However, in a similar field of endeavor, Intel discloses generate a histogram based on the input video signal (page 225, “The hardware histogram block generates image statistics based on the pixel stream input. These statistics are used by the Processing block to determine how much the backlight level can be reduced.”; also, para 226, “The histogram is composed of 32 bins with each bin covering a range of 8 values for 8 bit pixel component values. The first bin covers the values 0 thru 7.”) and generate an output video signal by converting a plurality of gray shades in the input video signal based on the gray shade look up table (page 227, “Find the enhancement factor from a Look Up Table (LUT) with 33 entries”; page 226, “The hardware enhancement block adjusts the pixel values sent to the display, compensating for the brightness loss due to lowering of the display backlight level.”; also, page 227, “LUT is addressed by the 6 most significant 6 bits of either HSV max(RGBin), or the Y channel after converting RGBin to YUV.”; also, page 227, “The final enhancement factor is derived by interpolating between the addressed LUT entry and the next entry, using the lower bits of the input.”; also, page 227, “The enhancement factor modifies each input pixel component value with the method selected by DPST_CTL Enhancement Mode.”; also, page 227, “Direct lookup mode replaces the input pixel value with the enhancement factor.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Deepala's invention of a device displaying a video stream whose pixel array forms the images of successive frames and is associated with a histogram whose bins each correspond to a brightness value, of post processing algorithms that develop new look up table values from the content histogram read from the display pipeline and pass them to the display driver as a look up table write, and of a display panel controller that complements the operations of the graphics processing unit and the host processor, with the features of the Intel PRM's invention of a hardware histogram block that generates image statistics from the pixel stream input into a histogram of thirty two bins, of a look up table of thirty three entries that is addressed by the most significant bits of each input pixel and from which an enhancement factor is found by interpolating between the addressed entry and the next, and of a hardware enhancement block that adjusts the pixel values sent to the display, that enhancement factor modifying each input pixel component value so that the input pixel value is replaced by the enhancement factor. The combination would have been obvious because Deepala and the Intel PRM describe the same arrangement of a display side hardware block that collects a per frame histogram and host software that reads that histogram and programs a pixel mapping look up table back into the display hardware, Deepala reading the content histogram from the display pipeline and writing look up table values to the display driver and the Intel PRM supplying the hardware side of that arrangement in the form of a histogram block generating image statistics from the pixel stream input and a look up table addressed by each input pixel whose enhancement factor replaces that pixel value in the pixel values the enhancement block sends to the display, so that each reference supplies what the other leaves unstated. The result would have been the predictable one of Deepala's histogram read and look up table write operating upon the Intel PRM's hardware histogram block and look up table.
Jung discloses an ambient light sensor operational to measure an ambient light level (para 31, “In one embodiment, s.sub.g.sup.min and s.sub.g.sup.max may be derived by a non-linear mapping from an ambient light sensor's input. In one or more other embodiments, input 960 may be provided by an ambient light sensor”; also, para 23, “Values for s.sub.g.sup.min and s.sub.g.sup.max may depend upon the captured scene's overall brightness as determined by an ambient light sensor”) and the ambient light level (para 31, “An image's normalized histograms 935 may be used by LTM unit 940 to find initial local tone mapping curves 945 for each region as described (see FIGS. 4-6). An image's collection of LTMs may be represented by LUTs 950; one LUT for each region's LTM”; also, para 23, “Values for s.sub.g.sup.min and s.sub.g.sup.max may depend upon the captured scene's overall brightness as determined by an ambient light sensor”; also, para 31, “In one embodiment, s.sub.g.sup.min and s.sub.g.sup.max may be derived by a non-linear mapping from an ambient light sensor's input.”), and a control unit coupled to the ambient light sensor (para 31, “In one embodiment, s.sub.g.sup.min and s.sub.g.sup.max may be derived by a non-linear mapping from an ambient light sensor's input. In one or more other embodiments, input 960 may be provided by an ambient light sensor”; also, para 23, “Values for s.sub.g.sup.min and s.sub.g.sup.max may depend upon the captured scene's overall brightness as determined by an ambient light sensor”; also, para 31, “An image's normalized histograms 935 may be used by LTM unit 940 to find initial local tone mapping curves 945 for each region as described (see FIGS. 4-6). An image's collection of LTMs may be represented by LUTs 950; one LUT for each region's LTM”; also, para 23, “Values for s.sub.g.sup.min and s.sub.g.sup.max may depend upon the captured scene's overall brightness as determined by an ambient light sensor”; also, para 31, “In one embodiment, s.sub.g.sup.min and s.sub.g.sup.max may be derived by a non-linear mapping from an ambient light sensor's input.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Deepala in view of the Intel PRM, in which a hardware histogram block generates image statistics from the pixel stream input, each bin of the saved histogram is readable by software, and host post processing algorithms develop and write back a software programmable look up table whose entries convert the input pixel values, with the features of Jung's invention of normalized histograms used to find local tone mapping curves that are represented by look up tables, and of the global minimum and maximum tone mapping slope values that shape those curves being derived by a non-linear mapping from an ambient light sensor's input and depending upon the overall brightness determined by that sensor. The combination would have been obvious because Jung develops the very same kind of table from the very same kind of input, a per-frame histogram, and adds the one input the combination lacks, teaching that the values controlling the shape of the tone mapping curve depend upon the overall brightness as determined by an ambient light sensor. Deepala and the Intel PRM already compute and write back a look up table from the frame histogram but take no account of the light in which the frame is viewed. A person of ordinary skill would therefore have supplied Jung's ambient light sensor input to the post processing algorithms that already develop the look up table values from the histogram, with the predictable result of a look up table whose gray shade assignments respond both to the content of the current frame and to the ambient light in which that frame is actually viewed.
Kawaguchi discloses a vehicle comprising (para 172, “In particular, Embodiment 2 manifests its effect when applied to an in-vehicle apparatus. An in-vehicle apparatus is viewed under the bright external light environment in daytime or outdoor where the picture visibility is poor; therefore, it requires the enhancement effect of the luminance from the viewpoint of visual recognition.”; also, para 156, “To the image processing device 100, a display panel 90 such as a liquid crystal to which a backlight controller 91 is attached and video equipment 93 which supplies a video signal are coupled. In addition, an external light sensor 92 may be coupled”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Deepala in view of the Intel PRM and Jung, in which a control unit coupled to an ambient light sensor generates an input video signal, develops a gray shade look up table from a histogram of that signal and from the ambient light level, and converts the gray shades of that signal for display, with the features of Kawaguchi's invention of an image processing device applied to an in-vehicle apparatus, having a display panel and video equipment coupled to it and an external light sensor coupled to it. The combination would have been obvious because Kawaguchi identifies the vehicle as the environment in which this very enhancement is needed, stating that an in-vehicle apparatus is viewed under the bright external light environment in daytime or outdoor where the picture visibility is poor and therefore requires the enhancement effect of the luminance from the viewpoint of visual recognition. A person of ordinary skill seeking to improve visibility under bright ambient light would therefore have deployed the ambient adapted, histogram driven gray shade look up table in the vehicle Kawaguchi describes, with the predictable result of a vehicle display whose gray shades remain visible to the driver across the range of on road lighting conditions.
Claim(s) 9 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Deepala et al. (U.S. Pub. No. 20190362686) as modified by Jung et al (U.S. Pub. No. 20180108327) and Intel, Intel® Iris® Plus Graphics and UHD Graphics Open Source Programmer's Reference Manual Jan 2020, Vol 12, further in view of Tsubokura et al. (U.S. Pub. No. 20210241718).
Regarding claim 9, Deepala as modified by Intel and Jung discloses the image enhancement system according to claim 1, generation of the output video signal includes: convert a 10-bit video signal or an 11-bit video signal to an 8-bit output video signal with a frame rate control function.
However, in a similar field of endeavor, Tsubokura discloses wherein the generation of the output video signal includes: convert a 10-bit video signal or an 11-bit video signal to an 8-bit output video signal with a frame rate control function (para 56, “Specifically, TCON 21b obtains a 10-bit output gradation corresponding to the 8-bit image data transmitted from terminal body 30 by referring to the LUT data for GSDF held in the internal memory and having been held in EEPROM 21a. TCON 21b then calculates 8-bit image data corresponding to the acquired 10-bit output gradation by a frame rate control (FRC) method and outputs the image data to display panel 22. As illustrated in FIG. 8, FRC is a technology for improving a gradation number in which a plurality of patterns are switched for each frame corresponding to LSB2 bit of 10-bit image data, to generate three or more pseudo gradations between two gradations in 8-bit image data.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Deepala in view of the Intel PRM and Jung, in which a hardware histogram block generates the histogram from the input video signal, post processing algorithms develop from that histogram and from the ambient light level a look up table, and that table converts the gray shades of the input video signal into the output video signal, with the features of Tsubokura's invention of obtaining a ten bit output gradation by reference to look up table data and then calculating eight bit image data corresponding to that gradation by a frame rate control method before the image data is output to the display panel. The combination would have been obvious because Deepala performs its pixel boosting by applying a pixel mapping look up table to the pixels of a video stream a device is displaying, so the mapped values carry finer resolution than a conventional panel interface can accept, and Tsubokura addresses that same situation by switching a plurality of patterns for each frame so that three or more pseudo gradations appear between two adjacent eight bit gradations. A person of ordinary skill delivering the mapped pixel values to a conventional eight bit display panel would therefore have applied Tsubokura's frame rate control conversion, with the predictable result that the additional gray shade resolution the look up table was computed to provide survives the reduction to the panel's native bit depth without contouring.
Regarding claim 19, Deepala as modified by Intel and Jung discloses the method according to claim 11, generating of the output video signal includes: converting a 10-bit video signal or an 11-bit video signal to an 8-bit output video signal with a frame rate control function.
However, in a similar field of endeavor, Tsubokura discloses wherein the generating of the output video signal includes: converting a 10-bit video signal or an 11-bit video signal to an 8-bit output video signal with a frame rate control function (para 56, “Specifically, TCON 21b obtains a 10-bit output gradation corresponding to the 8-bit image data transmitted from terminal body 30 by referring to the LUT data for GSDF held in the internal memory and having been held in EEPROM 21a. TCON 21b then calculates 8-bit image data corresponding to the acquired 10-bit output gradation by a frame rate control (FRC) method and outputs the image data to display panel 22. As illustrated in FIG. 8, FRC is a technology for improving a gradation number in which a plurality of patterns are switched for each frame corresponding to LSB2 bit of 10-bit image data, to generate three or more pseudo gradations between two gradations in 8-bit image data.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Deepala in view of the Intel PRM and Jung, in which a hardware histogram block generates the histogram from the input video signal, post processing algorithms develop from that histogram and from the ambient light level a look up table, and that table converts the gray shades of the input video signal into the output video signal, with the features of Tsubokura's invention of obtaining a ten bit output gradation by reference to look up table data and then calculating eight bit image data corresponding to that gradation by a frame rate control method before the image data is output to the display panel. The combination would have been obvious because Deepala performs its pixel boosting by applying a pixel mapping look up table to the pixels of a video stream a device is displaying, so the mapped values carry finer resolution than a conventional panel interface can accept, and Tsubokura resolves that same situation by switching a plurality of patterns for each frame so that three or more pseudo gradations appear between two adjacent eight bit gradations. A person of ordinary skill generating the output video signal for a conventional eight bit display panel would therefore have applied Tsubokura's frame rate control conversion, with the predictable result that the additional gray shade resolution survives the reduction to the panel's native bit depth without contouring.
Allowable Subject Matter
Claim 2-8 and 12-18 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
The following is an examiner’s statement of reasons for allowance:
Claims 2 and 12 are novel in developing the gray shade look up table from three separately defined functions rather than as one mapping, and in the inputs assigned to each. The low region is shaped by the measured ambient level, the intermediate region is stretched to keep intermediate gray shades separated from one another, and the high region is shaped by the white shade content of the image itself. The art develops the table as a single transfer characteristic computed from frame content, or shapes one curve from an ambient reading, and it derives nothing from the white shade content of the image.
Claims 3 and 13 are novel in computing both an output value and a slope at the boundary between the low region and the intermediate region, and then reshaping the low region function from those two quantities so that it joins the intermediate function without a discontinuity. The art produces table entries directly from frame content and identifies no boundary between regions, so no junction value or slope is computed anywhere.
Claims 4 and 14 are novel in fixing the starting output gray shade by a power relation of the measured background luminance with an offset and a slope constant, which sets the black end of the table to the level at which the darkest shades stay distinguishable under the measured ambient. The ambient dependence in the art is an unspecified non-linear mapping used to bound the slope of a curve, not a specified relation that fixes an entry of the table.
Claims 5 and 15 are novel in filtering the light sensor value on time constants matched to the adaptation of the eye, so that the gray shade correction tracks real changes in the lighting environment rather than transient shading at the sensor. The art applies the ambient reading as taken, with no temporal filtering.
Claims 6 and 16 are novel in the form prescribed for the intermediate region: successive gray shades either hold a constant contrast ratio with the reflected ambient taken into account, or follow a gamma relation to which an additive term is applied. The art prescribes no form for the middle of the transfer characteristic.
Claims 7 and 17 are novel in locating the end point of the transfer characteristic at the gray shade value returned by a percentage of the histogram, so that a few isolated bright pixels do not set the mapping applied to the whole frame. The art fixes no point of the characteristic from a histogram percentage.
Claims 8 and 18 are novel in matching the gamma slope at that end point, so that the high region joins the region below it without a visible discontinuity in the highlights.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jai Li whose telephone number is (571)272-1170. The examiner can normally be reached Mon-Thu between 06:00-16:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571)272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAI W LI/Junior Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights