DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see page 7-8, filed 9/11/2025 with respect to 35 USC §103 rejection of claims 1-15 have been fully considered but were not persuasive. Applicant argues previously cited references does not disclose claim limitation of determine an effective resolution for each point on the display, based at least on the hardware parameters of the display and the optical path information. Applicant points to paragraph [0028] of Jarvenpaa and argues paragraph [0028] is silent to display resolution based on display hardware parameters and optical path information. Applicant argues [0028] discloses a foveated region and a peripheral region displayed at a low quality. Applicant argues Jarvenpaa relies on gaze-dependent foveation zones without considering intrinsic display characteristics and adjusts the image quality depending on where a user's gaze direction without any reference to resolution for each point on the display, any hardware parameters of the display, and any optical path information. Applicant argues Hua fails to cure the deficiencies of Jarvenpaa. In response, examiner points to page 7 of instant application specifications to define hardware parameters as a physical attribute or characteristic associated with the display of the display apparatus. Specifications also discloses hardware parameters of the display comprise at least one of: a display resolution, an optical axis of the display, a size of the display, a pixel size of pixels of the display, a pixel layout of the pixels of the display, a pose of the display within the display apparatus. Specifications also discloses optical path as that defines a path along which the light propagates from the display towards the user's eyes. Based on these definition, examiner points to paragraph [0071] of Jarvenpaa [0071] to disclose quality of the content in the foveation and peripheral regions may relate to, not least, one or more of: differing: resolution, bit rate, colour depth, compression level and depth of field. Jarvenpaa’s quality of foveation region related to resolution, bit rate, colour depth, compression level and depth of field teaches argued hardware parameters. In addition paragraph [0029] discloses foveation region FR (i.e. such that the user's gaze position/fixation point/point of focus is aligned with the foveation point FP) perceives the foveation region via the centre of the user's eye's retina. Jarvenpaa’s gaze position teaches argued optical path. The 35 USC §103 rejection of claims 1-15 has been affirmed.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jarvenpaa et al (US 20210174768 A1) in view of Hua et al (US 20240013752 A1).
Regarding claim 1, Jarvenpaa discloses a system comprising at least one server ([0063] device may be at least one of: a portable device, a handheld device, a wearable device, a wireless communications device, a user equipment device, a client device, a server device a mobile device, a hand held portable electronic device, a display device, a near eye display device, a virtual reality display device, and an augmented reality display device.) configured to:
receive, from a display apparatus, hardware parameters of a display of the display apparatus ([0034] gaze dependent foveated rendering) and optical path information pertaining to an optical path between the display and a user's eye ([0029] user looking at the foveation region FR (i.e. such that the user's gaze position/fixation point/point of focus is aligned with the foveation point FP) perceives the foveation region via the centre of the user's eye's retina, the fovea/fovea centralis);
determine an effective resolution for each point on the display, based at least on the hardware parameters of the display and the optical path information ([0028] A first region/zone of the image, referred to hereinafter as “foveation region” FR, is provided within which the image content, referred to hereinafter as “foveal content” FC, is displayed at a high quality. A second region/zone of the image outside of the first region, i.e. corresponding to the remaining portion of the image, referred to hereinafter as “peripheral region” PR, is provided within which the image content, referred to hereinafter as “peripheral content” PC, is displayed at a low quality);
process an image according to the at least one image processing setting ([0032] In foveated imaging, the quality of differing regions of the image displayed may be adjusted to match the ability of the user's eyes to perceive the image quality at differing regions, e.g. high image quality in a foveation region perceived by the user's sharp central/foveal vision, and low image quality in a peripheral region perceived by a user's peripheral vision.); and
send the image to the display apparatus ([0071] mage content inside the foveation region 701, foveal content 701a, is rendered/displayed at a different image quality to that of the image content, peripheral content 702a, rendered/displayed in the remaining peripheral region 702 outside of and surrounding the foveation region).
Hua discloses optical path information pertaining to an optical path between the display and a user's eye ([0082] eye is gazing naturally straight forward), the OXY plane is perpendicular to Z-axis, and the Y axis is pointing upward. In an HMD system, the Z-axis typically also coincides with the optical axis of the eyepiece)
determine a foveation setting to be employed for image processing ([0090] optical layout of a static foveated HMD design 400 based on the scheme shown in FIG. 11A. [0005] algorithmic approach in which foveation techniques are applied primarily to spatially variant image processing and video encoding and variable levels of detail graphics rendering to achieve real-time video communication and save data processing resources)
determine at least one image processing setting to be employed for image processing, based on the effective resolution for each point on the display and the foveation setting ([0062] where the pixel position P on the display plane along the given direction {right arrow over (r)} is specified by its field angle, θ, which is defined as the angular deviation of the pixel position from the Z-axis of the OXYZ reference system in the unit of degrees and θ=0 corresponds to the direction matching with the Z-axis of the OXYZ reference coordinate system)
Jarvenpaa and Hua are combinable because they are from the same field of invention.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify foveated imaging of Jarvenpaa to include optical path information pertaining to an optical path between the display and a user's eye , determine a foveation setting to be employed for image processing and determine at least one image processing setting to be employed for image processing, based on the effective resolution for each point on the display and the foveation setting as described by Hua.
The motivation for doing so would have been for offering elegant degradation of image resolution with the increase of eccentricity and eye movements (Hua, [0008]).
Therefore, it would have been obvious to combine Jarvenpaa and Hua to obtain the invention as specified in claim 1.
Regarding claim 2, Jarvenpaa discloses wherein the at least one server is further configured to:
receive, from the display apparatus, information indicative of a gaze direction of the user's eye ([0029] user's gaze position/fixation point/point of focus is aligned with the foveation point FP)); and
determine a gaze rotation as an angular separation between an optical axis of the display and the gaze direction of the user's eye ([1056] a gaze property is measured, such as a visual angle with reference to the display device (which may be related to visual angles of the displayed information).),
wherein the effective resolution for each point on the display is determined based also on the gaze rotation ([0157] where the content is audio/visual content, the audio output may also be adjusted).
Regarding claim 3, Jarvenpaa discloses wherein the at least one server utilises at least one foveation map to determine the effective resolution for each point on the display based on the gaze rotation ([0037] determined gaze position 703, for example, such that a foveation point (i.e. the centre/centroid of foveation region) is dynamically adjusted so as to correspond to the continually updated determined gaze position).
Regarding claim 4, Jarvenpaa discloses wherein the hardware parameters of the display comprise at least one of: a display resolution, an optical axis of the display, a size of the display, a pixel size of pixels of the display, a pixel layout of the pixels of the display, a pose of the display within the display apparatus ([0026] quality or detail of an image (e.g. not least for example its resolution, colour depth, compression level) varies over the image.).
Regarding claim 5, Jarvenpaa is silent to wherein the optical path information comprises at least one of: a length of the optical path, a relative arrangement of the optical path with respect to an optical axis of the display, information pertaining to optical elements arranged along the optical path.
Hua discloses wherein the optical path information comprises at least one of: a length of the optical path, a relative arrangement of the optical path with respect to an optical axis of the display, information pertaining to optical elements arranged along the optical path ([0086] where the origin O is located at the center of the entrance pupil 7 of the eye 5, Z axis is along the corresponding LoS when the eye gaze direction is parallel to the direction of head pose (in other words, no eye movements are engaged in both horizontal and vertical directions and the eye is gazing naturally straight forward), the OXY plane is perpendicular to Z-axis, and the Y axis is pointing upward. In an HMD system, the Z-axis typically also coincides with the optical axis of the eyepiece)
Jarvenpaa and Hua are combinable because they are from the same field of invention.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify foveated imaging of Jarvenpaa to include wherein the optical path information comprises at least one of: a length of the optical path, a relative arrangement of the optical path with respect to an optical axis of the display, information pertaining to optical elements arranged along the optical path as described by Hua.
The motivation for doing so would have been for offering elegant degradation of image resolution with the increase of eccentricity and eye movements (Hua, [0008]).
Therefore, it would have been obvious to combine Jarvenpaa and Hua to obtain the invention as specified in claim 5.
Regarding claim 6, Jarvenpaa discloses wherein when the foveation setting is indicative of foveation being enabled, the at least one image processing setting is one of:
(i) a spatially-variable resolution is to be employed for rendering the image;
(ii) a spatially-variable resolution is to be employed for rendering the image, and spatially-uniform downsampling is to be employed for downsampling the image for transport;
(iii) a spatially-variable resolution is to be employed for rendering the image, and spatially-variable downsampling is to be employed for downsampling the image for transport ([0032] foveated imaging, the quality of differing regions of the image displayed may be adjusted to match the ability of the user's eyes to perceive the image quality at differing regions, e.g. high image quality in a foveation region perceived by the user's sharp central/foveal vision, and low image quality in a peripheral region perceived by a user's peripheral vision.);
(iv) a spatially-invariant resolution is to be employed for rendering the image, and spatially-variable downsampling is to be employed for downsampling the image for transport
Regarding claim 7, Jarvenpaa discloses wherein when the image is processed according to the at least one image processing setting, the image comprises at least a first region having a first resolution and a second region having a second resolution, the first resolution being greater than the second resolution ([0031] reducing the quality of the image in a region of an image that is perceived by a viewer's peripheral vision (i.e. reducing the quality of the peripheral content of the peripheral region)), and
wherein the at least one server is further configured to utilize framebuffer bandwidth by performing ([0031] Foveated imaging may enable a reduction in computing resources (not least data processing, bandwidth and storage of image data)) at least one of:
decrease the first resolution to a third resolution and increase the second resolution to a fourth resolution, the third resolution being greater than the fourth resolution;
increase a size of an area of the display to which a first portion of at least one framebuffer is mapped, wherein the first region is stored in the first portion of the at least one framebuffer ([0034] the areas of the image where high quality foveal content is rendered and where low quality peripheral content is rendered dynamically vary in dependence upon the determined gaze position.).
Regarding claim 8, Jarvenpaa discloses wherein when the foveation setting is indicative of foveation being disabled, the at least one image processing setting is that a spatially-invariant resolution is to be employed for rendering the image, said resolution being equal to a maximum effective resolution amongst effective resolutions for all points on the display ([0110] recovery action responsive to a determination of a sub-optimal operational condition, namely ceasing/stopping foveated rendering and performing “normal” rendering instead wherein the entirety of the image is rendered in high quality or an intermediate quality level).
Regarding claim 9, Jarvenpaa discloses a method ([0063] device may be at least one of: a portable device, a handheld device, a wearable device, a wireless communications device, a user equipment device, a client device, a server device a mobile device, a hand held portable electronic device, a display device, a near eye display device, a virtual reality display device, and an augmented reality display device.) comprising:
receiving, from a display apparatus, hardware parameters of a display of the display apparatus ([0034] gaze dependent foveated rendering) and optical path information pertaining to an optical path between the display and a user's eye ([0029] user looking at the foveation region FR (i.e. such that the user's gaze position/fixation point/point of focus is aligned with the foveation point FP) perceives the foveation region via the centre of the user's eye's retina, the fovea/fovea centralis);
determining an effective resolution for each point on the display, based at least on the hardware parameters of the display and the optical path information ([0028] A first region/zone of the image, referred to hereinafter as “foveation region” FR, is provided within which the image content, referred to hereinafter as “foveal content” FC, is displayed at a high quality. A second region/zone of the image outside of the first region, i.e. corresponding to the remaining portion of the image, referred to hereinafter as “peripheral region” PR, is provided within which the image content, referred to hereinafter as “peripheral content” PC, is displayed at a low quality);
determining a foveation setting to be employed for image processing;
determining at least one image processing setting to be employed for image processing, based on the effective resolution for each point on the display and the foveation setting;
processing an image according to the at least one image processing setting ([0032] In foveated imaging, the quality of differing regions of the image displayed may be adjusted to match the ability of the user's eyes to perceive the image quality at differing regions, e.g. high image quality in a foveation region perceived by the user's sharp central/foveal vision, and low image quality in a peripheral region perceived by a user's peripheral vision.); and
sending the image to the display apparatus ([0071] mage content inside the foveation region 701, foveal content 701a, is rendered/displayed at a different image quality to that of the image content, peripheral content 702a, rendered/displayed in the remaining peripheral region 702 outside of and surrounding the foveation region).
Hua discloses optical path information pertaining to an optical path between the display and a user's eye ([0082] eye is gazing naturally straight forward), the OXY plane is perpendicular to Z-axis, and the Y axis is pointing upward. In an HMD system, the Z-axis typically also coincides with the optical axis of the eyepiece)
determining a foveation setting to be employed for image processing ([0090] optical layout of a static foveated HMD design 400 based on the scheme shown in FIG. 11A. [0005] algorithmic approach in which foveation techniques are applied primarily to spatially variant image processing and video encoding and variable levels of detail graphics rendering to achieve real-time video communication and save data processing resources)
determining at least one image processing setting to be employed for image processing, based on the effective resolution for each point on the display and the foveation setting ([0062] where the pixel position P on the display plane along the given direction {right arrow over (r)} is specified by its field angle, θ, which is defined as the angular deviation of the pixel position from the Z-axis of the OXYZ reference system in the unit of degrees and θ=0 corresponds to the direction matching with the Z-axis of the OXYZ reference coordinate system)
Jarvenpaa and Hua are combinable because they are from the same field of invention.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify foveated imaging of Jarvenpaa to include optical path information pertaining to an optical path between the display and a user's eye , determining a foveation setting to be employed for image processing and determining at least one image processing setting to be employed for image processing, based on the effective resolution for each point on the display and the foveation setting as described by Hua.
The motivation for doing so would have been for offering elegant degradation of image resolution with the increase of eccentricity and eye movements (Hua, [0008]).
Therefore, it would have been obvious to combine Jarvenpaa and Hua to obtain the invention as specified in claim 9.
Regarding claim 10, Jarvenpaa discloses receiving, from the display apparatus, information indicative of a gaze direction of the user's eye ([0029] user's gaze position/fixation point/point of focus is aligned with the foveation point FP)); and
determining a gaze rotation as an angular separation between an optical axis of the display and the gaze direction of the user's eye ([1056] a gaze property is measured, such as a visual angle with reference to the display device (which may be related to visual angles of the displayed information).),
wherein the effective resolution for each point on the display is determined based also on the gaze rotation ([0157] where the content is audio/visual content, the audio output may also be adjusted).
Regarding claim 11, Jarvenpaa discloses wherein the hardware parameters of the display comprise at least one of: a display resolution, an optical axis of the display, a size of the display, a pixel size of pixels of the display, a pixel layout of the pixels of the display, a pose of the display within the display apparatus ([0026] quality or detail of an image (e.g. not least for example its resolution, colour depth, compression level) varies over the image.).
Regarding claim 12, Jarvenpaa is silent to wherein the optical path information comprises at least one of: a length of the optical path, a relative arrangement of the optical path with respect to an optical axis of the display, information pertaining to optical elements arranged along the optical path.
Hua discloses wherein the optical path information comprises at least one of: a length of the optical path, a relative arrangement of the optical path with respect to an optical axis of the display, information pertaining to optical elements arranged along the optical path ([0086] where the origin O is located at the center of the entrance pupil 7 of the eye 5, Z axis is along the corresponding LoS when the eye gaze direction is parallel to the direction of head pose (in other words, no eye movements are engaged in both horizontal and vertical directions and the eye is gazing naturally straight forward), the OXY plane is perpendicular to Z-axis, and the Y axis is pointing upward. In an HMD system, the Z-axis typically also coincides with the optical axis of the eyepiece)
Jarvenpaa and Hua are combinable because they are from the same field of invention.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify foveated imaging of Jarvenpaa to include wherein the optical path information comprises at least one of: a length of the optical path, a relative arrangement of the optical path with respect to an optical axis of the display, information pertaining to optical elements arranged along the optical path as described by Hua.
The motivation for doing so would have been for offering elegant degradation of image resolution with the increase of eccentricity and eye movements (Hua, [0008]).
Therefore, it would have been obvious to combine Jarvenpaa and Hua to obtain the invention as specified in claim 12
Regarding claim 13, Jarvenpaa discloses wherein when the foveation setting is indicative of foveation being enabled, the at least one image processing setting is one of:
(i) a spatially-variable resolution is to be employed for rendering the image;
(ii) a spatially-variable resolution is to be employed for rendering the image, and spatially-uniform downsampling is to be employed for downsampling the image for transport;
(iii) a spatially-variable resolution is to be employed for rendering the image, and spatially-variable downsampling is to be employed for downsampling the image for transport ([0032] foveated imaging, the quality of differing regions of the image displayed may be adjusted to match the ability of the user's eyes to perceive the image quality at differing regions, e.g. high image quality in a foveation region perceived by the user's sharp central/foveal vision, and low image quality in a peripheral region perceived by a user's peripheral vision.);
(iv) a spatially-invariant resolution is to be employed for rendering the image, and spatially-variable downsampling is to be employed for downsampling the image for transport.
Regarding claim 14, Jarvenpaa discloses wherein when the image is processed according to the at least one image processing setting, the image comprises at least a first region having a first resolution and a second region having a second resolution, the first resolution being greater than the second resolution ([0031] reducing the quality of the image in a region of an image that is perceived by a viewer's peripheral vision (i.e. reducing the quality of the peripheral content of the peripheral region)),,
the method further comprises utilizing framebuffer bandwidth by performing ([0031] Foveated imaging may enable a reduction in computing resources (not least data processing, bandwidth and storage of image data)) at least one of:
decreasing the first resolution to a third resolution and increasing the second resolution to a fourth resolution, the third resolution being greater than the fourth resolution;
increasing a size of an area of the display to which a first portion of at least one framebuffer is mapped, wherein the first region is stored in the first portion of the at least one framebuffer ([0034] the areas of the image where high quality foveal content is rendered and where low quality peripheral content is rendered dynamically vary in dependence upon the determined gaze position.).
Regarding claim 15, Jarvenpaa discloses wherein when the foveation setting is indicative of foveation being disabled, the at least one image processing setting is that a spatially-invariant resolution is to be employed for rendering the image, said resolution being equal to a maximum effective resolution amongst effective resolutions for all points on the display ([0110] recovery action responsive to a determination of a sub-optimal operational condition, namely ceasing/stopping foveated rendering and performing “normal” rendering instead wherein the entirety of the image is rendered in high quality or an intermediate quality level).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHIVANG I PATEL whose telephone number is (571)272-8964. The examiner can normally be reached on M-F 9-5am.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached on 571-272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHIVANG I PATEL/ Primary Examiner, Art Unit 2615