Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This action is in response to the amendment filed on 1/22/2026. Claims 1, 15, and 20 have been amended. The amended claims have been fully considered but are not persuasive. Claims 1-20 remain rejected in the application.
Response to Arguments
In response to applicant’s arguments regarding Kunkel and Brunner failing to teach displaying a view of an XR environment. Argument is fully considered but is not persuasive because the limitations are explicitly disclosed [Brunner: 0033 “sent out over interface 112 to the device's graphics display hardware 148 for displaying (step 360).”] (presenting in an XR view using their respective mappings is a straightforward display/compositing step that would have been obvious to a person of skill in the art). Claims 1-20 remain rejected in the application.
In response to applicant’s arguments regarding the dependent claims being allowable, since the
rejection for independent claims are maintained, rejections for these dependent claims are maintained. Claims 1-20 remain rejected in the application.
Claim Rejections - 35 USC § 103
Claims 1, 2, 8, 9, 10, 11, 12, 13, 15, 16, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Kunkel et al. (U.S. Patent Publication No. 2015/0256860), in view of Brunner et. al (U.S. Patent Publication No. 2010/0309346), in further view of Fortin-Deschenes et al. (U.S. Patent Publication No. 2021/0011289).
Regarding claim 1, Kunkel discloses a method comprising: obtaining virtual content, the virtual content associated with a virtual content tone map relating pixel luminance values of the virtual content to display space luminance values (interpreted as virtual/graphics content uses a tone map to map pixel luminance to display luminance)[Kunkel: 0008 “’display management’ denotes the processing (e.g., tone and gamut mapping) required to map an input video signal of a first dynamic range (e.g., 1000 nits) to a display of a second dynamic range (e.g., 500 nits).”][Kunkel: 0017 “The remapped video signal is blended with the graphics data to generate a video composite signal.”](teaches mapping content to display space and a graphics compositing path like a ton aping relationship for virtual/graphics); determining an adjustment adjusting at least a portion of the virtual content tone map based on the ISP tone map (adjusting/aligning the virtual content tone map based on the ISP curve is routine and an obvious limitation); but fails to explicitly disclose at a head mounted device (HMD) having a processor and one or more sensors, obtaining a pass-through video signal from an image sensor of the one or more sensors, the pass-through video signal comprising pass-through video depicting a physical environment, the pass-through video associated with an image signal processing (ISP) tone map relating pixel luminance values of the pass-through video signal to display space luminance values, and displaying a view of an extended reality (XR) environment, the view comprising the pass-through video displayed using the ISP tone map and the virtual content displayed using the virtual content tone map with the adjustment.
However, Brunner discloses having a processor and one or more sensors [Brunner: 0019 “The ISP 100 may include a digital image sensor 102.”], obtaining a pass-through video signal from an image sensor of the one or more sensors, the pass-through video signal comprising pass-through video depicting a physical environment, the pass-through video associated with an image signal processing (ISP) tone map relating pixel luminance values of the pass-through video signal to display space luminance values (interpreted as obtaining a camera passthrough video that has an ISP tone map that maps pixel to display luminance values)[Brunner: 0016 “performing tone-mapping in the camera's image sensor package (ISP) is computationally “free,” that is, the data that is to be mapped through the use of tone mapping curves already gets mapped through a table to perform gamma correction. Thus, loading a different table which combines gamma correction and tone curve mapping”][Brunner: 0035 “a table of values representative of the appropriate tone mapping curve may be generated 145, as was described above. The tone curve mapping data 145 may then be combined with data representative of the appropriate RGB gamma boost correction curve, and the image may be boosted according to the combined table of values 111. Finally, a noise reduction filter 110 may be applied to the image data. As mentioned above, as an alternative to generating a table of values representative of the generated tone mapping curve, ISP 150 may instead by given the necessary information, e.g., the coefficients, to generate the tone map ping curve data itself. ISP 150 could then individually calculate the boost amount for each pixel in the image and then pass on the boosted data to combined boost module 111. Finally, the image data may be encoded into the YCbCr family of color spaces and passed over interface 112 for any further processing and/or display.”](teaches ISP tone mapping curve/table associated with the camera output which corresponds to the passthrough) and displaying a view of an extended reality (XR) environment, the view comprising the pass-through video displayed using the ISP tone map and the virtual content displayed using the virtual content tone map with the adjustment [Brunner: 0033 “sent out over interface 112 to the device's graphics display hardware 148 for displaying (step 360).”] (presenting in an XR view using their respective mappings is a straightforward display/compositing step that would have been obvious to a person of skill in the art).
However, Fortin-Deschenes discloses at a head mounted device (HMD) (Fortin-Deschenes: Abstract “A Head - Mounted Display system”).
Kunkel, Brunner, and Fortin-Deschenes are considered analogous to the claimed invention because they are in the same field of tone mapping and compositing of camera video. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kunkel to incorporate Brunner’s and Fortin-Deschenes teachings of using the cameras ISP generated tone mapping curve for the captured video as the basis to adjust the virtual or graphics tone map and utilizing HMD. The motivation for such a combination would provide the benefit of consistent appearance between pass through and virtual content, reduced artifacts, and straightforward implementation using already available ISP data.
Regarding claim 2, Kunkel and Fortin-Deschenes disclose the method of claim 1, wherein the view further comprises the virtual content displayed using the virtual content tone map with the second adjustment (interpreted as the view shows virtual content using that adjusted virtual tone map) [Kunkel: 0017 “The remapped video signal is blended with the graphics data to generate a video composite signal.”][Kunkel: 0022 “TV display management processor (130) adjust the luminance and color of the input data (122) to match the dynamic range and color gamut of the display panel (135).”](teaches display of a composite where adjusted graphics are shown on the display), but fail to explicitly disclose further comprising: determining a second adjustment adjusting a second portion of the virtual content tone map comprising a range differing from a range represented in said ISP tone map.
However, Brunner discloses further comprising: determining a second adjustment adjusting a second portion of the virtual content tone map comprising a range differing from a range represented in said ISP tone map (interpreted as make an additional adjustment to the virtual tone map for a portion whose luminance range differs from the ISP maps range)[Brunner: 0016 “performing tone-mapping in the camera's image sensor package (ISP) is computationally “free,” that is, the data that is to be mapped through the use of tone mapping curves already gets mapped through a table to perform gamma correction. Thus, loading a different table which combines gamma correction and tone curve mapping”][Brunner: Abstract “a tone mapping curve can automatically be generated within the sensor and adjusted appropriately for the scene based on predetermined parameters. Further, it has been determined that independently varying the slopes of the tone mapping curve at the low end (S) and high end (S) of the curve results in more visually appealing images. By dynamically and independently selecting So and S values based on image metadata”](teaches that the ISP provides a tone map with its own range (predetermined parameters)).
Kunkel, Brunner, and Fortin-Deschenes are considered analogous to the claimed invention because they are in the same field of tone mapping and compositing of camera video. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kunkel and Fortin-Deschenes to incorporate Brunner’s teachings of using the cameras ISP generated tone mapping curve for the captured video as the basis to adjust the virtual or graphics tone map. The motivation for such a combination would provide the benefit of consistent appearance between pass through and virtual content, reduced artifacts, and straightforward implementation using already available ISP data.
Regarding claim 8, Kunkel, Brunner, and Fortin-Deschenes disclose the method of claim 1, Kunkel further discloses wherein the virtual content tone map is a filmic tone map [Kankul: 0008 “(e.g., tone and gamut mapping)”][Kankul: 0029 “tone-mapping and inverse tone-mapping”][Kakul: 0031 “temporal correction to the input metadata to alleviate perceptual jumps in tone-mapping changes.”](teaches different tone mapping and the ability to change the tone map, the claimed limitation of filmic tone map selection is a routine choice among well-known tone maps to achieve natural highlights and midtones, a person of skill in the art would pick a filmic curve as a predictable substitution to meet aesthetic consistency goals).
Regarding claim 9, Kunkel and Fortin-Deschenes disclose the method of claim 1, further comprising: periodically updating the virtual content [Kunkel: 0026 “Content metadata may be adjusted on a per frame or a per scene basis, or whenever there is a change.”](teaches that the display management mapping is driven by per frame/scene metadata), but fail to explicitly disclose tone map with respect to ISP tone map modifications occurring while a user is viewing bright or dark portions of the pass-through video.
However, Brunner discloses tone map with respect to ISP tone map modifications [Brunner: 0005 “possessing the digital camera. To further increase efficiency, light-product information in the current image frame can be used to determine the tone curve for the next frame.”] (teaches the ISP tone curve is modified frame to frame in response to capture conditions) occurring while a user is viewing bright or dark portions of the pass-through video [Brunner: 0006 “the exposure parameters in a typical digital camera or digital video camera are designed to change gradually, the tone curves can be changed gradually as well, producing no visually jarring changes for the user when the camera enters a different lighting condition.”](teaches ISP tone curve changes occur during different lighting conditions in the camera feed).
Kunkel, Brunner, and Fortin-Deschenes are considered analogous to the claimed invention because they are in the same field of tone mapping and compositing of camera video. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kunkel and Fortin-Deschenes to incorporate Brunner’s teachings of frame-to-frame ISP tone curve updates and to periodically update the virtual content in response to those ISP modifications while the user is viewing bright or dark portions of the pass through. The motivation for such a combination would provide the benefit of consistent appearance under changing scenes.
Regarding claim 10, Kunkel, Brunner, and Fortin-Deschenes disclose the method of claim 9, Kunkel further discloses wherein said periodically updating occurs during every frame of the pass-through video [Kunkel: 0026 “Content metadata may be adjusted on a per frame or a per scene basis, or whenever there is a change.”](teaches that the display management mapping is driven by per frame/scene metadata).
Regarding claim 11, Kunkel, Brunner, and Fortin-Deschenes disclose the method of claim 1, Kunkel further discloses wherein said adjusting comprises aligning mapping values of at least a portion of the virtual content tone map with respect to mapping values of the ISP tone map such that the virtual content tone map matches the ISP tone map for at least a portion of a tone range (interpreted as the adjustment aligns the virtual tone maps numeric mapping over some range to match the ISP tone maps mapping over that same range) [Kunkel: 0028 “In an embodiment, the receiver's display manager processor DM-R (110) maps the incoming dynamic range (e.g., 1000 nits) to the desired graphics or compositing dynamic range (e.g., 100 nits)”] [Kunkel: 0017 “The remapped video signal is blended with the graphics data to generate a video composite signal”](teaches mapping one signal curve into the same compositing range used by graphics before blending and aligning mapping values of the virtual tone map to a reference curve over an overlapping range is inherent to achieve seamless blend in a single compositing domain. Once the ISP tone map exists, aligning the virtual map to match the ISP map over a shared portion is a routine curve matching step to avoid seams and luminance discontinuities).
Regarding claim 12, Kunkel, Brunner, and Fortin-Deschenes disclose the method of claim 11, Kunkel further discloses wherein said aligning comprises matching midpoint values of the mapping values of at least a portion of the virtual content tone map with respect to midpoint values of the mapping values of the ISP tone map (interpreted as set the virtual tone map so that over a selected range, its midpoint (the mid luminance pivot value) maps to the same output as the ISP tone maps midpoint) [Kunkel: 0017 “a display management process uses input content-based metadata to map input EDR Video data from a first dynamic range into the dynamic range of the available graphics data. The remapped video signal is blended with the graphics data to generate a video composite signal.”](given Kunkel’s mapping into a common range and the existence of an ISP tone curve from the camera, matching midpoint values of the virtual map to the ISP map over a shared tone range is a routine curve alignment step a person of skill in the art would perform to avoid luminance seams).
Regarding claim 13, Kunkel and Fortin-Deschenes disclose the method of claim 1, but fail to explicitly disclose wherein the image sensor is an outward facing camera of the electronic device.
However, Brunner discloses wherein the image sensor is an outward facing camera of the electronic device [Brunner: 0034 “a digital camera comprising a camera sensor unit 141 configured for taking still images or Video may be integrated into a multimedia device, such as a mobile phone 149.”](teaches on device camera that can be a phone and phones have outward facing cameras).
Kunkel, Brunner, and Fortin-Deschenes are considered analogous to the claimed invention because they are in the same field of tone mapping and compositing of camera video. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kunkel and Fortin-Deschenes to incorporate Brunner’s teachings of utilizing an outward facing camera on a device. The motivation for such a combination would provide the benefit of reducing latency and integration complexity by capturing the physical environment directly on device.
Claims 15 and 20 are electronic device and non-transitory computer readable storage medium claims corresponding to claim 1 without any additional limitations. Thus, claims 15 and 20 are rejected for the same reasons as claim 1 above.
Claim 16 is an electronic device claim corresponding to claim 2 without any additional limitations. Thus, claim 16 is rejected for the same reasons as claim 2 above.
Claims 3, 4, 17, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Kunkel et al. (U.S. Patent Publication No. 2015/0256860), in view of Brunner et. al (U.S. Patent Publication No. 2010/0309346), in view of Fortin-Deschenes et al. (U.S. Patent Publication No. 2021/0011289), in further view of Nikkanen et al. (U.S. Patent Publication No. 2015/0054980).
Regarding claim 3, Kunkel, Brunner, and Fortin-Deschenes disclose the method of claim 2, Kunkel further discloses wherein the second portion comprises an extended dynamic range (EDR) portion [Kunkel: 0005 “As used herein, the terms enhanced dynamic range (EDR) or visual dynamic range (VDR) may individually or interchangeably”], but fail to explicitly disclose and wherein the ISP tone map comprises a standard dynamic range (SDR) tone map.
However, Nikkanen discloses and wherein the ISP tone map comprises a standard dynamic range (SDR) tone map [Nikkanen: 0051 “RGB to sRGB”](teaches sRGB which can be the claimed SDR. The definition of sRGB corresponding to SDR can be found at the Standard Dynamic Range Video article on Wikipedia “In some cases the term SDR is also used with a meaning including the standard color gamut (i.e. Rec. 709 / sRGB color primaries)”, article is attached in this office action).
Kunkel, Brunner, Fortin-Deschenes, and Nikkanen are considered analogous to the claimed invention because they are in the same field of tone mapping and compositing of camera video. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kunkel, Brunner, and Fortin-Deschenes to incorporate Nikkanen’s teachings of utilizing sRGB. The motivation for such a combination would provide the benefit of consistent across heterogeneous dynamic ranges, enabling reliable blending and predictable visual results.
Regarding claim 4, Kunkel, Brunner, Fortin-Deschenes, and Nikkanen disclose the method of claim 3, Kunkel further discloses wherein the second adjustment comprises using an extrapolation technique (extrapolation is interpreted as estimate unknown values) [Kunkel: 0028 “the output (112) of this process is not clipped and therefore may contain illegal display values, such as negative values or values larger than a maximum allowable threshold.”] (teaches estimating values which corresponds to extrapolation) [Kunkel: 0029 “an inverse display management process (120) converts the SDR composite video signal (117) back to an EDR output signal”](teaches mapping an SDR composite back to EDR which necessarily employs extrapolation for values outside the SDR domain).
Claim 17 is an electronic device claim corresponding to claim 3 without any additional limitations. Thus, claim 17 is rejected for the same reasons as claim 3 above.
Claim 18 is an electronic device claim corresponding to claim 4 without any additional limitations. Thus, claim 18 is rejected for the same reasons as claim 4 above.
Claims 5, 6, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Kunkel et al. (U.S. Patent Publication No. 2015/0256860), in view of Brunner et. al (U.S. Patent Publication No. 2010/0309346), in view of Fortin-Deschenes et al. (U.S. Patent Publication No. 2021/0011289), in further view of Molholm (U.S. Patent Publication No. 2021/0099632).
Regarding claim 5, Kunkel, Brunner and Fortin-Deschenes disclose the method of claim 1, but fail to explicitly disclose further comprising: determining a second adjustment adjusting pixel luminance of the virtual content based on exposure values of the image sensor before applying the virtual content tone map such that a modified view of virtual content is displayed.
However, Molholm discloses further comprising: determining a second adjustment adjusting pixel luminance of the virtual content based on exposure values of the image sensor before applying the virtual content tone map such that a modified view of virtual content is displayed (interpreted as compute a second adjustment that changes the virtual pixels luminance using the cameras exposure values, do this before the virtual content tone map and show the modified virtual view) [Molholm: 0052 “exposure compensation may be calculated from the auto - exposure information of the camera and the ambient lighting information . For example : Exposure compensation = Scene exposure - Camera exposure”][Molholm: 0048 “virtual content 271 may be rendered into an image to be blended with the image captured by the camera 250 in the display pipeline 280 . Exposure compensation 272 is applied so that the rendered virtual content has the same scene exposure as the exposure compensated image in the display pipeline 280 .”](teaches camera exposure values which drive an exposure compensation that adjusts virtual content luminance).
Kunkel, Brunner, Fortin-Deschenes, and Molholm are considered analogous to the claimed invention because they are in the same field of tone mapping and compositing of camera video. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kunkel, Brunner, and Fortin-Deschenes to incorporate Molholm’s teachings of adjusting luminance based on exposure value. The motivation for such a combination would provide the benefit of consistent scene exposure and reduced luminance mismatch.
Regard claim 6, Kunkel, Brunner, and Fortin-Deschenes disclose the method of claim 5, but fail to explicitly disclose the second adjustment comprises a rescaling adjustment.
However, Molholm discloses the second adjustment comprises a rescaling adjustment [Molholm: 0005 “The camera image is compensated to scale its RGB values based on the scene exposure”][Molholm: 0046 “Exposure compensation is calculated at 266”](explicitly teaches scaling values based on exposure).
Kunkel, Brunner, Fortin-Deschenes, and Molholm are considered analogous to the claimed invention because they are in the same field of tone mapping and compositing of camera video. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kunkel, Brunner, and Fortin-Deschenes to incorporate Molholm’s teachings of rescaling adjustment to the virtual content prior to tone mapping. The motivation for such a combination would provide the benefit of consistent luminance alignment between pass through and virtual content.
Claim 19 is an electronic device claim corresponding to claim 5 without any additional limitations. Thus, claim 19 is rejected for the same reasons as claim 5 above.
Claims 7 is rejected under 35 U.S.C. 103 as being unpatentable over Kunkel et al. (U.S. Patent Publication No. 2015/0256860), in view of Brunner et. al (U.S. Patent Publication No. 2010/0309346), in view of Fortin-Deschenes et al. (U.S. Patent Publication No. 2021/0011289), in view of Molholm (U.S. Patent Publication No. 2021/0099632), in further view of McDougal et al. (U.S. Patent No. 8,417,046).
Regarding claim 7, Kunkel, Brunner, Fortin-Deschenes, and Molholm disclose the method of claim 6, but fail to explicitly disclose wherein the exposure values comprise a brightest pixel being used as a reference value for the rescaling adjustment.
However, McDougal discloses wherein the exposure values comprise a brightest pixel being used as a reference value for the rescaling adjustment (McDougal: Col. 7, Lines 1-3 “Each pixel is proportionally scaled such that the brightest pixel in the original image is assigned the highest value in the numerical representation of pixel intensity.”)(McDougal: Col. 8, Lines 34-38 “the original image may be normalized, for example, by Scaling or stretching in terms of light intensity, so that the brightest pixel in the image is assigned the maximum value possible in a binary representation of a pixel.”).
Kunkel, Brunner, Fortin-Deschenes, Molholm, and McDougal are considered analogous to the claimed invention because they are in the same field of tone mapping and compositing of camera video. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kunkel, Brunner, Fortin-Deschenes, Molholm and McDougal to incorporate McDougal’s teachings of assigning the scaling value the brightest pixels value. The motivation for such a combination would provide the benefit of consistent luminance alignment between pass through and virtual content.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Kunkel et al. (U.S. Patent Publication No. 2015/0256860), in view of Brunner et. al (U.S. Patent Publication No. 2010/0309346), in view of Fortin-Deschenes et al. (U.S. Patent Publication No. 2021/0011289), in further view of Lalonde et al. (U.S. Patent No. 2017/0287215).
Regarding claim 14, Kunkel, Brunner, and Fortin-Deschenes disclose the method of claim 1, but fail to explicitly disclose wherein the electronic device is an HMD.
However, Lalonde discloses wherein the electronic device is an HMD (interpreted as head mounted display) [Lalonde: 0029 “Such optical components can include pass - through cameras mounted to ( or incorporated into ) a display associated with an HMD device . Image content captured by the pass - through cameras can be combined with virtual content in a display of the HMD device”].
Kunkel, Brunner, Fortin-Deschenes, and Lalonde are considered analogous to the claimed invention because they are in the same field of tone mapping and compositing of camera video. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kunkel, Brunner, and Fortin-Deschenes to incorporate Lalonde’s teachings of utilizing a HMD. The motivation for such a combination would provide the benefit of straightforward integration.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to AHMED TAHA whose telephone number is (571)272-6805. The examiner can normally be reached 8:30 am - 5 pm, Mon - Fri. Examiner interviews are available via telephone, in person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, XIAO WU can be reached at (571)272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786- 9199 (IN USA OR CANADA) or 571-272-1000.
/AHMED TAHA/Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights