DETAILED ACTION
Response to Amendment
Applicant’s amendments filed on 02 March 2026 have been entered. Claims 1, 3-5, 7, 9, 10, 12- 14, and 20 have been amended. Claims 6 and 17 have been canceled. Claims 1-5, 7-16, and 18-20 are still pending in this application, with claims 1 and 14 being independent.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 7, 9, 11-16 and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tavakoli et al. (US 20200066024 A1), referred herein as Tavakoli in view of Schmidt et al. (US 20180227630 A1), referred herein as Schmidt, and Young et al. (US 20170287446 A1), referred herein as Young (from IDS).
Regarding Claim 1, Tavakoli in view of Schmidt and Young teaches an apparatus for image processing, comprising (Tavakoli Abst: An apparatus is configured to render graphics content to reduce latency of the graphics content; [0008] a method of operation of a device includes generating an indication of an area of interest of graphics content at the device based on scene information related to the graphics content; [0010] a computer-readable medium stores instructions executable by a processor to cause the processor to generate an indication of an area of interest of graphics content based on scene information related to the graphics content):
at least one memory; and at least one processor coupled to the at least one memory and, based at least in part on information stored in the at least one memory, the at least one processor is configured to (Tavakoli [0089] The memory 932 may store instructions 960. The instructions 960 may be executable by the processor 910, by the GPU 996, or by both; FIG. 9):
select or generate a weight map for a periphery region of a display (Tavakoli [0028] The fovea estimation engine 104 may be configured to determine an evaluation metric associated with one or more parts of the graphics content 102. For example, the fovea estimation engine 104 may determine an evaluation metric (e.g., a ranking) on a per-bin basis, as described further with reference to FIG. 2; [0043] The fovea estimation engine 104 may be configured to “rank” each bin associated with the first frame 114 based on the scene information 110. For example, the fovea estimation engine 104 may be configured to determine a first result 232 of an evaluation metric for the first bin 202 and a second result 234 of the evaluation metric for the second bin 204 based on the scene information 110. The fovea estimation engine 104 may identify the first portion 108 based on the results 232, 234).
calculate, based on the weight map, a weighted sum of Tavakoli [0027] the fovea estimation engine 104 is configured to generate an indication 106 of an area of interest (e.g., the first portion 108) of the graphics content 102. The fovea estimation engine 104 is configured to identify the first portion 108 based on scene information 110 related to the graphics content 102; [0031] the rendering engine 112 may be configured to render the graphics content 102 by decreasing one or more of a fidelity of the second portion 118, an amount of detail of the second portion 118, or a resolution of the second portion 118; FIG. 6); second portion 118 reads on the periphery region;
Tavakoli teaches estimate a foveated region, but does not teach the detail mechanism of the estimation. However, Schmidt teaches
a weighted sum of at least one of scene statistics or saliency to obtain a value indicating a detail level specified by the periphery region of the display (Schmidt [0033] Also a prediction of a user action can be used to predict the point of regard, e. g. based on gaze and user behavior, and also on history data, statistics of other users, etc., for example to start rendering hidden entities revealed by an action the user is likely to make. Also one can predict the likeliness for this action and therefore determine a detail of the (pre)rendering; [0036] Moreover, if at least one of the saliency points is found and is visible in the at least one of the images presenting the scene when displayed, the point of regard is predicted for the certain future point of time in dependency on the at least one found saliency point).
Tavakoli in view of Schmidt further teaches
determine one of a first Tavakoli [0031] The rendering engine 112 is configured to render the graphics content 102 by performing a comparison of a first result of an evaluation metric on part of the area of interest with a second result of the evaluation metric with another part of the area of interest, and the rendering engine 112 is configured to render the graphics content 102 to reduce latency based on the comparison),
Tavakoli in view of Schmidt teaches a first and second level, but does not teach a set of foveation levels.
However, Young teaches a first/second set of foveation levels (Young Abst: The foveated images are characterized by level of detail within the one or more regions of interest and lower level of detail outside the one or more regions of interest; [0081] the density of vertices and/or pixel resolution is highest in the subsection or subsections of the screen containing a portion of interest 480 and progressively lower in subsections further away from the foveal portion, such as the transition area(s) 482 and the peripheral areas 600 described above and additionally shown in FIG. 4B; [0082] The transition area(s) 482 may have a pixel density distribution determined by, for example, a mathematical relationship between the pixel density distributions of the area of interest 480 and the peripheral region 483 (See FIGS. 6C-6D)).
Tavakoli in view of Schmidt and Young further teaches
wherein the first set of foveation levels is applied if the detail level is below a predetermined threshold or a first threshold, wherein the second set of foveation levels is applied if the detail level is above the predetermined threshold or a second threshold (Tavakoli [0043] The fovea estimation engine 104 may be configured to “rank” each bin associated with the first frame 114 based on the scene information 110. For example, the fovea estimation engine 104 may be configured to determine a first result 232 of an evaluation metric for the first bin 202 and a second result 234 of the evaluation metric for the second bin 204 based on the scene information 110. The fovea estimation engine 104 may identify the first portion 108 based on the results 232, 234. For example, if the first result 232 satisfies a threshold value (e.g., is greater than the threshold value), the fovea estimation engine 104 may select the first bin 202 for foveating by the rendering engine 112. In this example, the first bin 202 may correspond to the first portion 108);
switch, based on the determination, to the first set of foveation levels or the second set of foveation levels if the display is applying a different set of foveation levels (Tavakoli [0070] Although the binning and ranking process 600 of FIG. 6 is described with reference to two portions 108, 118, it should be appreciated that the binning and ranking process 600 may be performed using a different number of areas of the graphics content 102… a “spectrum” of foveating may be applied to the graphics content 102 so that differently ranked areas are foveated based on different amounts. In some cases, the binning and ranking process 600 may enable foveating of multiple areas of interest. For example, although user gaze may be focused on the soccer ball illustrated in FIG. 6, in some cases both the soccer ball and the soccer player of FIG. 6 may be foveated); and
output a set of images or videos via the display based on the first set of foveation levels or the second set of foveation levels (Schmidt [0013] The at least one parameter, which is correlated with an image characteristic of the first image, may relate to a resolution. For example in the first area the image can be displayed with higher resolution and in at least one second area the image can be displayed with lower resolution).
Schmidt discloses a system and method for subsequently displaying a stream of images in a display area of a display device, which is analogous to the present patent application.
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tavakoli to incorporate the teachings of Schmidt, and apply the statistical and history data, as well as saliency points in the scene into the fovea estimation engine method of Tavakoli.
Doing so can provide an improvement with regard to the displaying of areas with different image characteristics in dependency of the point of regard.
Young discloses a real-time user adaptive foveated rendering method, which is analogous to the present patent application.
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tavakoli to incorporate the teachings of Young, and apply the method of rendering the transition regions 482 to establish, for example, a mathematical relationship between the pixel density distributions of the area of interest 480 and the peripheral region 483 into the rendering engine for foveated area of interest of Tavakoli.
Doing so can reduce the computational load on graphics processing hardware by concentrating computational resources on rendering more important parts of an image on a display.
Regarding Claim 2, Tavakoli in view of Schmidt and Young teaches the apparatus of claim 1, and further teaches wherein the first set of foveation levels includes two levels of foveation and the second set of foveation levels includes three levels of foveation (Young [0087] FIGS. 6B, 6C, and 6D illustrate examples of “falloff” in vertex density or pixel resolution between the foveal portions and remaining portions of the screen space. FIG. 6B illustrates an example of a step function transition between Max and Min density with respect to distance from a foveal screen space region. In this case, the vertex density has the maximum value within the foveal region and the minimum value elsewhere; [0088] FIG. 6C illustrates an example of a linear function transition between Max and Min density with respect to distance from foveal screen space region. In a transition region 482 the density depends linearly on distance from the region of maximum or minimum density, depending on how the coordinate system is defined).
Regarding Claim 3, Tavakoli in view of Schmidt and Young teaches the apparatus of claim 1, and further teaches wherein to switch to the first set of foveation levels or the second set of foveation levels if the display is applying the different set of foveation levels, the at least one processor is configured to: switch to the first set of foveation levels if the display is currently applying the second set of foveation levels or a third set of foveation levels, or switch to the second set of foveation levels if the display is currently applying the first set of foveation levels or the third set of foveation levels (Tavakoli [0074] The drawcall pattern 430 includes a drawcall pattern change 702 associated with one or more drawcalls of the second frame 414. To illustrate, the third drawcall 406 may be associated with the drawcall pattern change 702. The drawcall pattern change 702 may correspond to a scene change (e.g., movement of the first portion 108). For example, if the soccer ball illustrated in FIG. 6 is kicked, the third drawcall 406 may be used to depict movement of the soccer ball.).
Regarding Claim 4, Tavakoli in view of Schmidt and Young teaches the apparatus of claim 1, and further teaches wherein the detail level specified by the periphery region of the display is based on at least one of: auxiliary camera data for eye tracking, inertial measurement unit (IMU) data for head tracking, or a fovea location (Tavakoli [0083] the first portion 108 may be identified using eye position information, such as the eye position information 504. The method 800 may include receiving the eye position information 504 (e.g., from the camera 550 of FIG. 5) and identifying the first portion 108 further based on the eye position information 504).
Regarding Claim 5, Tavakoli in view of Schmidt and Young teaches the apparatus of claim 1, and further teaches wherein the detail level specified by the periphery region of the display is based on at least one of contrast, sharpness, or brightness statistics (Schmidt [0013] the at least one parameter can also relate to a brightness, a color saturation, or especially with regard to computer-generated graphics a level of detail; [0037] Also if a moving object appears in the peripheral view of a user, this object has a higher chance of attracting visual attention than others, further elevated by parameters like size, color, and contrast to the environment).
Regarding Claim 7, Tavakoli in view of Schmidt and Young teaches the apparatus of claim 1, and further teaches wherein the at least one processor is further configured to: configure or modify the weight map based on at least one of a lighting condition, a head movement, an eye movement, or a focus state (Tavakoli [0062] the eye position analyzer 510 may be configured to receive the eye position information 504 from the camera 550 (e.g., a sensor device) that tracks eye position of a user. For example, in some applications, a headset may be worn by a user, and the headset may include the camera 550. The eye position analyzer 510 may be configured to determine an actual eye position 520 based on the eye position information 504; [0064] The fovea estimation engine 104 may be configured to determine an estimation error 522 (e.g., a difference between the estimated eye position 516 and the actual eye position 520). The fovea estimation engine 104 may be configured to use the estimation error 522 in connection with one or more subsequent operations. For example, the fovea estimation engine 104 may be configured to use the estimation error 522 to “predict” subsequent eye positions of a user).
Regarding Claim 9, Tavakoli in view of Schmidt and Young teaches the apparatus of claim 1, and further teaches wherein to estimate the detail level specified by the periphery region of the display the at least one processor is further configured to estimate the detail level specified by the periphery region of the display in a number of consecutive frames (Tavakoli [0056] Referring to FIG. 4, another example of the device 100 of FIG. 1 is depicted and generally designated 400. In the example of FIG. 4, the graphics content 102 may include the first frame 114 and a second frame 414 (e.g., consecutive frames of the graphics content 102). The first frame 114 may include a set of drawcalls, such as a first drawcall 402 and a second drawcall 404. The second frame 414 may include the first drawcall 402, the second drawcall 404, and a third drawcall 406),
wherein to determine one of the first set of foveation levels or the second set of foveation levels to be applied to the display based on the detail level the at least one processor is further configured to determine one of the first set of foveation levels or the second set of foveation levels to be applied to the display based on the detail level being above or below the predetermined threshold for the number of consecutive frames (Tavakoli [0043] The fovea estimation engine 104 may identify the first portion 108 based on the results 232, 234. For example, if the first result 232 satisfies a threshold value (e.g., is greater than the threshold value), the fovea estimation engine 104 may select the first bin 202 for foveating by the rendering engine 112. In this example, the first bin 202 may correspond to the first portion 108).
Regarding Claim 11, Tavakoli in view of Schmidt and Young teaches the apparatus of claim 1, and further teaches wherein the apparatus is a head-mounted display that is capable of providing virtual reality (VR) content, augmented reality (AR) content, or extended reality (XR) content via the display (Tavakoli [0023] the graphics content 102 may correspond to an augmented reality (AR) application or a virtual reality (VR) application).
Regarding Claim 12, Tavakoli in view of Schmidt and Young teaches the apparatus of claim 1, and further teaches wherein to output the set of images or videos via the display based on the first set of foveation levels or the second set of foveation levels, the at least one processor is configured to:
transmit, to at least one sensor, an indication of the first set of foveation levels or the second set of foveation levels (Tavakoli [0062] the eye position analyzer 510 may be configured to receive the eye position information 504 from the camera 550 (e.g., a sensor device) that tracks eye position of a user);
receive, from a multiplexer (MUX), a configuration for the first set of foveation levels or the second set of foveation levels (Tavakoli FIG. 5: 104: Fovea Estimation Engine); and
apply the configuration to an image signal processor (ISP) (Tavakoli [0023] FIG. 1 depicts an illustrative example of a device 100. The device 100 may be included in a graphics processing unit (GPU) that is configured to perform a graphics processing operation using graphics content 102).
Regarding Claim 13, Tavakoli in view of Schmidt and Young teaches the apparatus of claim 12, and further teaches wherein the at least one processor is further configured to: transmit the indication to a display processing unit (DPU) for outputting the set of images or videos with the first set of foveation levels or the second set of foveation levels via the display (Young [0056] The display device 304 may be in the form of a cathode ray tube (CRT), flat panel screen, touch screen, projector, retinal imager, or other device that displays text, numerals, graphical symbols, or other visual objects).
Regarding Claims 14-16 and 19, Tavakoli in view of Schmidt and Young teaches a method of image processing, comprising (Tavakoli Abst: An apparatus is configured to render graphics content to reduce latency of the graphics content; [0008] a method of operation of a device includes generating an indication of an area of interest of graphics content at the device based on scene information related to the graphics content; [0010] a computer-readable medium stores instructions executable by a processor to cause the processor to generate an indication of an area of interest of graphics content based on scene information related to the graphics content):
The metes and bounds of the claim substantially correspond to the claimed limitations set forth in claims 1-3 and 12; thus they are rejected on similar grounds and rationale as their corresponding limitations.
Regarding Claim 20, Tavakoli in view of Schmidt and Young teaches a computer-readable medium storing computer executable code, the code when executed by at least one processor causes the at least one processor to (Tavakoli Abst: An apparatus is configured to render graphics content to reduce latency of the graphics content; [0008] a method of operation of a device includes generating an indication of an area of interest of graphics content at the device based on scene information related to the graphics content; [0010] a computer-readable medium stores instructions executable by a processor to cause the processor to generate an indication of an area of interest of graphics content based on scene information related to the graphics content):
The metes and bounds of the claim substantially correspond to the claimed limitations set forth in claim 1; thus they are rejected on similar grounds and rationale as their corresponding limitations.
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tavakoli et al. (US 20200066024 A1), referred herein as Tavakoli in view of Schmidt et al. (US 20180227630 A1), referred herein as Schmidt, Young et al. (US 20170287446 A1), referred herein as Young (from IDS) and Lawrence (US 20190043154 A1), referred herein as Lawrence.
Regarding Claim 8, Tavakoli in view of Schmidt and Young teaches the apparatus of claim 1, but does not teach the claimed limitations herein.
However, Lawrence teaches wherein the second threshold is higher than the first threshold (Lawrence [0038] if the current level of concentration is greater than the prior level of concentration by at least a first threshold value, the parameters of the graphics system may be adjusted to increase one or more of the bandwidth, resolution, and frame rate. If the current level of concentration is lower than the prior level of concentration by at least a second threshold value, the parameters may be adjusted to reduce one or more of the bandwidth, resolution, and frame rate).
Lawrence discloses graphics systems relate to concentration based adaptive graphics quality, which is analogous to the present patent application.
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tavakoli to incorporate the teachings of Lawrence, and apply the multiple concentration thresholds into level of detail thresholds for the rendering engine for foveated area of interest of Tavakoli.
Doing so would be able to adjust the quality of the graphics processing based on the provided concentration information.
Claim(s) 10 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tavakoli et al. (US 20200066024 A1), referred herein as Tavakoli in view of Schmidt et al. (US 20180227630 A1), referred herein as Schmidt, Young et al. (US 20170287446 A1), referred herein as Young (from IDS) and Dunki-Jacobs (US 20090060381 A1), referred herein as Dunki.
Regarding Claim 10, Tavakoli in view of Schmidt and Young teaches the apparatus of claim 1, but does not teach the claimed limitations herein.
However, Lawrence teaches teaches wherein the at least one processor is further configured to: configure or modify the predetermined threshold, the first threshold, or the second threshold based on at least one of a lighting condition, a head movement, an eye movement, or a focus state (Dunki [0071] Each data point in the low resolution version represents multiple pixels of image data or a region within the image data. The region component 902 detects one or more candidate regions for gamma correction using the low resolution version of the image data and one or more predetermined thresholds… users define or modify threshold values used to automatically select regions for gamma correction).
Dunki discloses localized gamma correction that can be used to map image data to a display means, which is analogous to the present patent application.
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tavakoli to incorporate the teachings of Dunki, and apply modification of threshold values of intensity information into level of detail thresholds for the rendering engine for foveated area of interest of Tavakoli.
Doing so would be able to increase dynamic range and mitigating artifacts in imaging systems, such as scanned beam imagers.
Regarding Claim 18, Tavakoli in view of Schmidt and Young the method of claim 14. The metes and bounds of the claim substantially correspond to the claimed limitations set forth in claim 10; thus they are rejected on similar grounds and rationale as their corresponding limitations.
Response to Arguments
Applicant's arguments filed on 02 March 2026 with respect to the 103 rejection have been fully considered but they are not persuasive.
On page 9, Applicant's Remarks, with respect to claims 1 and 14, the applicant argues “Tavakoli does not appear to describe select[ing] or generat[ing] a weight map… Rather, Tavakoli merely discusses evaluating scene-related metrics to support foveated rendering decisions." The examiner respectfully disagrees with this argument. In Tavakoli, the evaluation metric stores ranking values of a frame at different regions based on techniques for ranking bins, triangles, drawcalls and/or gaze position (see Fig. 2-5), thus teaches the limitation of “select or generate a weight map for a periphery region of a display”. During the rendering process in Tavakoli,, the binning and ranking process may be performed using a different number of areas of the graphics content. Further, different areas may be rendered using different amounts of foveating. (see Fig. 6, par [0070]). Higher ranking score (see FIG. 6.202) represents higher level of detail, and will rendering in a higher resolution of the region (see par. [0032]). Therefore, Tavakoli in view of Schmidt and Young disclosed a fovea estimation engine that performs an equivalent function and achieves a same result of “calculate, based on the weight map, a weighted sum of at least one of scene statistics or saliency to obtain a value indicating a detail level specified by the periphery region of the display”, as claimed.
On page 10 of Applicant’s Remarks, the Applicant argues the corresponding dependent claims are not taught by the prior art, insomuch as they depend from claims that are not taught by the prior art. Examiner respectfully disagrees with these arguments, for the reasons discussed above
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Samantha (Yuehan) Wang whose telephone number is (571)270-5011. The examiner can normally be reached Monday-Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571)272-7440. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Samantha (YUEHAN) WANG/
Primary Examiner
Art Unit 2617
Prosecution Insights