DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed June 18, 2026 have been fully considered and are persuasive.
Regarding the rejection of claim 1 under 35 U.S.C. 102(a)(1) in view of Blackmon, Applicant argues that Blackmon does not disclose "wherein a display update period of the region-of-interest image on the display panel is shorter than a display update period of the background image on the display panel, the display update period being a period in which the timing controller updates a corresponding image on the display panel by outputting image data stored in the buffer to the display panel," as recited in amended independent claim 1. Blackmon discloses that the background image region and the foveated region of interest (ROI) can be rendered at different times, but the examiner agrees that Blackmon does not explicitly disclose updating the background image region and the foveated ROI on the display panel at different points in time. Accordingly, the prior art rejections of claim 1 and of dependent claims 2-10 and 13 are withdrawn and these claims are allowed. For similar reasons, the rejection of claim 19 is also withdrawn. Claim 19 uses slightly different language than claim 1 regarding this feature of the present invention, but the BRI for these limitations is that they mean the same thing.
Regarding the prior art rejection of independent claims 14 and 20, Applicant argues that Blackmon does not disclose the limitation "wherein a point in time when the background image is updated on the display panel is different from a point in time when the region-of-interest image is updated on the display panel," as recited in independent claim 14, and similarly recited in independent claim 20. The examiner agrees that Blackmon does not explicitly disclose this limitation because although Blackmon discloses rendering these regions at different points in time and then displaying them, Blackmon does not explicitly disclose that the different regions are displayed at different points in time on the display panel. Although the timing of rendering is typically coordinated with the timing of displaying images on the display panel, rendering image data at different instants in time does not necessarily mean that rendered images are displayed at different instants in time on the display panel. For this reason, the rejections of claims 14 and 20 are withdrawn. However, a new ground of rejection is set forth below.
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation (BRI) using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The BRI of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification.
In the following, some of the terms in the claims have been given BRIs in light of the specification. These BRIs are used for purposes of searching for prior art and examining the claims, but cannot be incorporated into the claims. Should Applicant believe that different interpretations are appropriate, Applicant should point to the portions of the specification that clearly support a different interpretation.
Position information: para. [0035], information based on a positional difference between the region-of-interest (ROI) image of the current frame and the region-of-interest image of the next frame;
Buffer: not defined in the present disclosure, and therefore the BRI is the plain meaning to one of ordinary skill in the art. The plain meaning is a temporary storage area in a memory device.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 14 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Publ. Appl. No. 2017/0169602 A1 to Blackmon et al. (hereinafter referred to as “Blackmon”) in view of U.S. Publ. Appl. No. 2023/0032431 A1 to Yoshida (hereinafter referred to as “Yoshida”).
Regarding claim 14, Blackmon discloses a display device comprising:
a processor (GPU 802, Fig. 8, paras. [0084]-[0096]) configured to generate a region-of-interest image of a current frame based on a background image of the current frame and a line-of-sight region of a user (gaze tracking logic 216 and ray tracing logic 220 in Fig. 2; also gaze tracking logic 214 and ray tracing surface identification logic 804 in Fig. 8; para. [0010], the gaze tracking logic is used “to determine a gaze position for an image to be rendered, wherein a region of interest of the image is based on the determined gaze position, e.g. the region of interest may surround the determined gaze position such that it represents a foveal region”; paras. [0011], [0073] and [0091], the ray tracing logic is used to render the identified regions of interest (ROIs) whereas the rasterization logic is used to render the background regions); and
a timing controller (Fig. 8, the combination of the control logic 822 and the combining logic 816 comprise timing controller logic) configured to output the background image (referred to in para. [0091] of Blackmon as “the rasterization region of the image”) and the region-of-interest image (referred to in para. [0091] of Blackmon as “the ray traced image values”) to a display panel (the control logic 822, Fig. 8, para. [0091], causes the rasterization image data corresponding to the background region and the ray traced image data corresponding to the ROI image data to be combined in the image buffer 814 by the combining logic 816 and subsequently output to the display panel); and
a buffer (image buffer 814) storing at least one of the background image and the region-of-interest image (para. [0091], the image buffer 814 stores the background rasterization image data and the ROI ray traced image data),
wherein the timing controller is configured to:
receive the background image, the region-of-interest image, and position information of the region-of-interest image in the background image from the processor through an interface (para. [0091], the combining logic 816 of the timing controller receives the background image from the rasterization logic 804 and receives the ROI image and the corresponding ray tracing position information from the ray tracing logic 806; the interface corresponds to the connections indicated by the arrows in Fig. 8 pointing to the components of the combining logic 816),
write the background image to the buffer (para. [0091], the blending logic 812 of the combining logic 814 writes the rasterization background image to the image buffer 814: “[i]n the blending region, the blending logic 812 can blend the image values determined by ray tracing with the rasterised image values which were stored in the image buffer 814.”),
based on the position information, write the region-of-interest image to a region of a background image corresponding to the position information (para. [0091], based on the position information determined via ray tracing, the blending logic 812 writes the ROI image to the image buffer 814), and
update at least one of the background image and the region-of-interest image to the buffer (para. [0091], the blending logic updates at least the ROI image: “[i]f there is a part of the region of interest which is not included in the blending region, then the image values determined by ray tracing for this part of the region of interest can be written into the image buffer 814, which may, or may not, overwrite image values previously stored in the image buffer 814 for this part of the region of interest.”); and
wherein a point in time when the background image is updated on the display panel is different from a point in time when the region-of-interest image is updated on the display panel (as indicated above in the rejection of claim 1, para. [0065] of Blackmon explicitly discloses that the background image region can be updated at different points in time than the higher-detail image region, i.e., the foveated ROI: “[t]he rendering techniques do not need to be synchronized. For example, a low detail rendering of the entire image can be performed using either ray tracing or rasterization, and then subsequent rendering operations can augment regions of the image with higher levels of detail. The high detail rendering may occur multiple times and across multiple displayed frames continuing to augment the image before the low detail image is required to be re-rendered.” As indicated above in the Response to Arguments section, while Blackmon discloses rendering the background image and the ROI image at different points in time, Blackmon does not explicitly disclose displaying them at different points in time).
Yoshida, in the same field of endeavor, discloses, in para. [0026] with respect to Figs. 9A-9C (duplicated below for convenience), displaying the higher resolution foveated ROI image data and the lower resolution background image data at different points in time (Para. [0026], Figs. 9A-9C, duplicated below: “[at one update timing of the display 100, the image is updated in one of the region 102 and the region 103. In the region 103, the displayed image is updated with the resolution lower than the resolution of the display 100. FIG. 9C is a view showing, with respect to time, images that the user can actually recognize visually when the display 100 is updated as shown in FIG. 9B. In the comparative example, the image including the regions having different resolutions is always displayed. On the other hand, in this embodiment, there are a timing at which the image having the low resolution is displayed, and a timing at which the image (foveated rendering image) including regions having different resolutions are displayed as in the comparative example. In this manner, in this embodiment, a frame including the image having the lower resolution than the display 100 and a frame in which the foveated rendering image is displayed are repeated alternately or in a predetermined order.”).
PNG
media_image1.png
200
400
media_image1.png
Greyscale
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the algorithms performed by the GPU 802 of Blackmon to update the background image and the foveated ROI image at different times as taught by Yoshida. One of ordinary skill in the art would have been motivated to make the modification to reduce the amount of data that has to be transmitted to the display, thereby improving bandwidth resources, as discussed in the Background of Yoshida. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (configuring the timing controller to cause the background images and the ROI images to be displayed at different points in time, e.g., alternately).
Regarding claim 20, to the extent that claim 20 recites the same limitations that are recited in claim 14, the rejection of claim 14 applies mutatis mutandis to claim 20. Blackmon discloses that the display device can be part of a wearable electronic device in which the processor generates the ROI based on a background image and a line of sight of a user tracked by using at least one sensor (para. [0098]: “]i]n one example, gaze tracking logic could shine low-power infrared focussed [sic] lights at the eye and have single (unfocussed) light sensors inside a head-mounted-display cavity that could infer the eye position based on the reflected light intensity at each sensor.”).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the algorithms performed by the GPU 802 of Blackmon to only update the image data stored in the image buffer 814 of the GPU 802 when the GPU 802 determines by comparing the current frame to the next frame that there has been a change in the position of the ROI and only update the image data in the image buffer 814 corresponding to the change as taught by Radhakrishan. One of ordinary skill in the art would have been motivated to make the modification to reduce the amount of image data in the image buffer 814 that needs to be updated/overwritten thereby reducing computational overhead and increasing the speed at which image data to be displayed can be processed. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying the software executed by the GPU 802 of Blackmon to determine which regions need to be updated and updating only those regions of the image stored in buffer 814).
Claims 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Blackmon in view of Yoshida and further in view of U.S. Publ. Appl. No. 2017/0032764 A1 to Radhakrishnan (hereinafter referred to as “Radhakrishnan”).
Regarding claim 15, Blackmon discloses that the reception controller (para. [0091], the combination of the controller logic 822, the combining logic 814 and the blending logic 812 comprise the reception controller) is configured to update the region-of-interest image in the buffer by overwriting the region-of-interest image of the next frame to the buffer based on position information of the region-of-interest image in the background image (para. [0091], in the combining logic 814, the blending logic 812 blends the rasterization image data and the ray traced image data together and stores the blended image data in the image buffer 814; para. [0091], the blending logic 812 can update and overwrite the blended image data based on the position information of the ROI determined via ray tracing: “[i]n the blending region, the blending logic 812 can blend the image values determined by ray tracing with the rasterised image values which were stored in the image buffer 814. If there is a part of the region of interest which is not included in the blending region, then the image values determined by ray tracing for this part of the region of interest can be written into the image buffer 814, which may, or may not, overwrite image values previously stored in the image buffer 814 for this part of the region of interest.”).
However, Blackmon does not explicitly disclose that the reception controller does this by comparing the region-of-interest image of the current frame with the region-of-interest image of the next frame.
Radhakrishan, in the same field of endeavor, teaches this limitation (paras. [0038]-[0039], processor 12 compares the ROI image of the current frame with the ROI image of the next frame to determine whether the position of one or more layers of the ROI have changed: “[p]rocessor 12 may determine that a position of one or more layers changed from a first image frame to a second image frame. In this example, the first image frame is an image frame that is displayed, and the second image frame is an image frame whose image content has been generated but has not yet been displayed”; when there is a difference between the region-of-interest image of the current frame and the region-of-interest image of the next frame, the processor transmits the region-of-interest image of the next frame to the timing controller (para. [0039], if the processor 12 determines that there is a difference between one or more layers of the ROIs of the current and next frame, the processor 12 transmits the ROI image of the next frame to the display processor 18, which comprises a timing controller: “[i]n response to determining that the position of the one or more layers changed, processor 12 may determine a region that needs to be updated in the second image frame (e.g., dirty region, an ROI, or the area to be updated) based on a construct, such as a rectangle, that encompasses the one or more layers in the first image frame whose position changed and one or more constructs that encompass respective layers of the set of one or more layers in the second image frame (e.g., all dirty rectangles). Processor 12 may then cause display processor 18 to compose the layers in the determined region (i.e., computed ROI)”).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the algorithms performed by the GPU 802 of Blackmon to only overwrite the image data stored in the image buffer 814 of the GPU 802 when the GPU 802 determines by comparing the current frame to the next frame that there has been a change in the position of the ROI and only update the image data in the image buffer 814 corresponding to the change as taught by Radhakrishan. One of ordinary skill in the art would have been motivated to make the modification to reduce the amount of image data in the image buffer 814 that needs to be updated/overwritten thereby reducing computational overhead and increasing the speed at which image data to be displayed can be processed. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying the software executed by the GPU 802 of Blackmon).
Regarding claim 16, Blackmon does not explicitly disclose that the reception controller updates the background image of the current frame by overwriting the background image of the next frame to a region of the buffer, excluding a region corresponding to the region-of-interest image of the current frame from the background image of the current frame, based on the position information.
Radhakrishan teaches this limitation because the processor 12 of Radhakrishan only transmits the image data of the next frame corresponding to the changed region based on the positional information such that only that image data is updated/overwritten by the display processor 18, which means that the ROI image data of the current frame is not updated/overwritten because it does not need to be updated since it has not changed (paras. [0020] and [0038]-[0039]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the algorithms performed by the GPU 802 of Blackmon to only overwrite the image data stored in the image buffer 814 of the GPU 802 that has changed while excluding the other image data from overwriting when the GPU 802 determines by comparing the current frame to the next frame that there has been a change in the position and/or content of the background image and only update the image data in the image buffer 814 corresponding to the changes as taught by Radhakrishan. One of ordinary skill in the art would have been motivated to make the modification to reduce the amount of image data in the image buffer 814 that needs to be updated/overwritten thereby reducing computational overhead and increasing the speed at which image data to be displayed can be processed. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying the software executed by the GPU 802 of Blackmon).
Regarding claim 17, Blackmon does not explicitly disclose these limitations. Radhakrishan discloses that when there is a difference in position between the region-of-interest image of the current frame stored in the buffer and a region-of-interest image of the next frame generated to update the region-of-interest image of the current frame, the processor transmits, to the timing controller, one of the background image of the current frame and the background image of the next frame generated to update the background image of the current frame, and the position information generated by comparing the region-of-interest image of the current frame with the region-of-interest images of the next frame and performs the updating (para. [0039], if the processor 12 determines that there is a difference between one or more layers of the ROIs of the current and next frame, the processor 12 transmits the ROI image of the next frame to the display processor 18, which comprises: “[i]n response to determining that the position of the one or more layers changed, processor 12 may determine a region that needs to be updated in the second image frame (e.g., dirty region, an ROI, or the area to be updated) based on a construct, such as a rectangle, that encompasses the one or more layers in the first image frame whose position changed and one or more constructs that encompass respective layers of the set of one or more layers in the second image frame (e.g., all dirty rectangles). Processor 12 may then cause display processor 18 to compose the layers in the determined region (i.e., computed ROI)”. When this happens, the display processor 18 transmits to its display timing control circuitry the unchanged background region image data along with the positional information, para. [0039]: “[f]or the other regions (e.g., those other than determined dirty region), display processor 18 may utilize the image content of the previous frame (e.g., the first frame). In some examples, processor 12 may cause display processor 18 to display the composed second image frame on a display device (e.g., display 19)”).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the algorithms performed by the GPU 802 of Blackmon to transmit, to the timing controller logic 822/816 for display, the background image data of the current frame along with the corresponding positional information of the changed ROI calculated by the ray tracing logic 806 since the background image data of the current frame outside of the ROI has not changed as taught by Radhakrishan. One of ordinary skill in the art would have been motivated to make the modification to reduce the amount of image data in the image buffer 814 that needs to be updated/overwritten thereby reducing computational overhead and increasing the speed at which image data to be displayed can be processed. In addition, the modification would reduce the number of rendering computations that need to be performed by the rasterization logic 804 that renders the background images, which would reduce the amount of power consumed by the GPU 802 and potentially increase the speed at which it can update displayed images. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying the software executed by the GPU 802 of Blackmon).
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Blackmon in view of U.S. Publ. Appl. No. 2019/0122642 A1 to Morein (hereinafter referred to as “Morein”).
Regarding claim 18, Blackmon discloses that the blending logic 812 mixes the ROI image created by the ray tracing logic with the rasterization background image created by the rasterization logic, but Blackmon does not explicitly teach that the timing controller comprises a scaler configured to receive the mixed image created by the blending logic 812 and stored in the image buffer 814 and to scale the mixed image to be displayed on the display panel.
Morein, in the same field of endeavor, discloses upscaling the background portion (referred to in Morein as the field of view (FOV) image) of the mixed image, at the time of displaying the image (para. [0037] and [0056]) in order to match the pixel resolution of the FOV image with that of the physical display (para. [0072]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the algorithms performed by the GPU 802 of Blackmon based on the teachings of Morein to upscale the FOV image portion of the mixed image stored in the image buffer 814 before displaying it on the display panel. One of ordinary skill in the art would have been motivated to make the modification to ensure that the resolution of the pixels representing the background portions of the mixed image match the resolution of the display panel so that the image is properly displayed. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (e.g., modifying the software executed by the GPU 802 of Blackmon to perform upscaling on the image data read out of the image buffer 814).
Allowable Subject Matter
Claims 1-10 and 13 are allowed.
The following is an examiner’s statement of reasons for allowance.
None of the prior art teaches or suggests the combination of limitations recited in claim 1 including “wherein a display update period of the region-of-interest image on the display panel is shorter than a display update period of the background image on the display panel, the display update period being a period in which the timing controller updates a corresponding image on the display panel by outputting image data stored in the buffer to the display panel.”
As indicated above in the Response to Arguments section of this Office Action, prior art including Blackmon teaches rendering ROI images and background images at during different time periods, but none of the prior art teaches that the display update period of the ROI image on the display monitor is shorted than the display update period of the background image on the display monitor. For this reason, claim 1 is allowable.
Claims 2-10 and 13 are allowable due to their direct or indirect dependencies from claim 1.
Claim 19 recites a similar limitation that is interpreted as meaning the same thing as the corresponding limitation in claim 1 discussed above. For this reason, claim 19 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL J SANTOS whose telephone number is (571)272-2867. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Bella can be reached at (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL J. SANTOS/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights