Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 7, 8, 14 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication 2017/0213388 A1 (hereinafter Margolis) in view of U.S. Patent Application Publication 2021/004985 A1 (hereinafter Melkote) in view of U.S. Patent 10,785,471 B1 (hereinafter Hunt).
Regarding claim 1, the limitations “An apparatus for graphics processing, the apparatus comprising: processing circuitry having a pipeline at least partially implemented by processing resources, the pipeline as facilitated by the processing circuitry to: render input data to generate a first frame” are taught by Margolis (Margolis, e.g. abstract, paragraphs 20-104 describes a system for performing rendering for an HMD using a combination of conventional frame rendering and frame reprojection in order to generate display frames/fields at a higher rate than the conventional frame rendering pipeline renders frames, e.g. paragraphs 20-25. Margolis, e.g. figure 3, paragraphs 47-54, teaches that the system relies on a processing pipeline, modules 302-310, including rendering module 302 generating pre-rendered images that are displayed on the HMD, i.e. the claimed generated first frame based on rendering input data. Margolis, e.g. paragraphs 31, 46, indicates that the modules 302-310 may be part of the HMD, and rely on processing resources, or with respect to claim 14, execute stored computer readable instructions.)
The limitations “perform asynchronous space-warp (ASW) on the first frame on a block basis … based on content and focusing related information … associated with the first frame to generate a second frame” are taught by Margolis (Margolis, e.g. paragraphs 20-25, 47-54, teaches that the late stage reprojection (LSR) module generates one or more adjusted images from the pre-rendered image, where both the pre-rendered and adjusted images are displayed using the HMD, i.e. the LSR process generates the claimed second frame from the first frame, and both images are output on the HMD. Further, Margolis, e.g. paragraphs 60, 69, 82-84, teaches that the LSR process operates by determining motion vectors for each block of one or more pixels, and displacing/projecting each block using it’s motion vector to generate the adjusted image(s), i.e. the LSR process corresponds to the claimed asynchronous space warp performed on the first frame on a block basis. Finally, Margolis, e.g. paragraphs 25, 92-95, teaches that the LSR process is performed based on the content being focused on by the user, i.e. regions containing focus areas/objects are forward projected for every frame generated by the LSR process, whereas other regions may be forward projected at a lower rate as in paragraph 25.)
The limitations “perform asynchronous space-warp (ASW) on the first frame on a block basis with granularities for areas of the first frame based on content and focusing related information, wherein the granularities are determined prior to performing the ASW associated with the first frame to generate a second frame” are partially taught by Margolis (Margolis, e.g. paragraph 47, teaches that the LSR/ASW pipeline of figure 3 includes a pre-processing component, module 302, used to pre-render images, i.e. rendering the first frame on which the LSR/ASW process is performed, and Margolis, e.g. paragraphs 92-95, teaches that the LSR/ASW process may be selectively applied to plural areas of interest determined based on the content of the first image and the area(s) where the user is focused. While Margolis’ pre-processing component determines the content and focusing related information for the first frame, Margolis does not explicitly teach that the LSR/ASW process is performed with different granularities for the different areas of focus.) However, this limitation is taught by Melkote (Melkote, e.g. abstract, paragraphs 23-98, discloses a split-rendering virtual reality system including a host device performing rendering of images and a client/display device performing asynchronous time and space warping to correct for a user’s changing head position and/or scene motion from the last fully rendered frame, e.g. paragraphs 23-29. Further, Melkote, e.g. paragraphs 29, 66, 72-85, 90-98, further teaches that the ROI may be determined based on the content of the rendered image and/or the foveal region, where, e.g. paragraphs 29, 73, 76-78, 80-82, the motion vector grid size/granularity and motion vector filter kernel size/granularity are determined for the ROI based on said content/foveal region information within the ROI, i.e. as claimed granularities for areas of the first/rendered frame are determined prior to performing the ASW using the determined granularities. Finally, Melkote, e.g. paragraphs 83-85, teaches that the ASW process is performed by performing extrapolations on the vertices in the ROI using the determined size/granularity.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Margolis’ late stage reprojection system to use Melkote’s per-ROI motion vector grid/filter size/granularity determination technique in order to select an appropriate size/granularity for each ROI based on the content of the rendered image and the location where the user is focusing, thereby reducing apparent artifacts from the reprojection/warping process as taught by Melkote. In Margolis’ modified system, each area of interest, e.g. Margolis, paragraphs 92-95, corresponds to one of Melkote’s ROI, and as taught by Melkote with respect to the game engine/render side component, e.g. paragraphs 74-80, Margolis’ rendering module 302 would additionally determine the size/granularities for each area of interest based on the determined content and focusing information for the frame, and provide said size/granularities to the LSR module 308 for controlling the size/granularity of the motion vector filter kernel used for performing the reprojection/warping for each respective area of interest/ROI. Margolis, e.g. paragraph 69, teaches that the blocks used for reprojection can be any size, i.e. one or more pixels, and although Margolis does not indicate a basis for selecting a block size for reprojection, in the modified system Melkote’s per-ROI size/granularity for the motion vector grid/filter kernel corresponds to a selection of block size, i.e. a fine grid size corresponds to a smaller number of pixels per block whereas a coarser grid size corresponds to a larger number of pixels per block.
The limitation “output to a display device the first frame without an ASW adjustment and the second frame with the ASW adjustment” is implicitly taught by Margolis (As noted above, Margolis, e.g. paragraphs 20-25, 47-54, teaches that the late stage reprojection (LSR) module generates one or more adjusted images from the pre-rendered image, where both the pre-rendered and adjusted images are displayed using the HMD, i.e. the LSR process generates the claimed second frame from the first frame, and both images are output on the HMD. While Margolis does not explicitly state that the pre-rendered images, i.e. the first frame without ASW adjustments, are displayed along with the LSR adjusted images, i.e. the second frame with the ASW adjustment, are both displayed on the same display in sequence, i.e. the pre-rendered image being displayed followed by the one or more adjusted images from the pre-rendered image being displayed, followed by the next pre-rendered image, etc., one of ordinary skill in the art would have understood the implicit teaching that Margolis’ system operated this way based on Margolis’ description, e.g. paragraph 47, explicitly states that the pre-rendered images are “referred to as pre-rendered imaged because they are rendered prior to be[ing] displayed” (emphasis added), and the temporally appropriate display sequence is to display the pre-rendered image prior to the LSR adjusted images generated based thereon. While Margolis does teach the claim limitation, in the interest of compact prosecution Hunt is cited for explicitly teaching what one of ordinary skill in the art would have understood Margolis’ disclosure to implicitly teach, i.e. that frame rate upsampling is performed by displaying the lower frame rate/rendered frames in sequence with the higher frame rate/adjusted frames in order to achieve the increased effective display frame rate.) However this limitation is taught by Hunt (Hunt, e.g. abstract, cols 1-13, describes an HMD system performing frame rate upsampling on content rendered by a console using warping operations. Hunt, e.g. col 3, lines 37-63, col 7, lines 4-44, teaches that the synthetic frames generated by the upsampling module may be generated using any warping technique, including asynchronous space warping based on motion vectors calculated between blocks of current and previous frames, analogous to Margolis’ LSR adjusted images. Further, Hunt, e.g. col 3, lines 6-28, col 6, line 14 - col 7, line 3, describes how the upsampling module determines the number of synthetic frames to generate based on the console rendering/low frame rate and the target/high frame rate by generating one or more synthetic frames to be displayed in sequence with the original/rendered frame used to generate the synthetic frames. In Hunt’s example of upsampling a 100 fps console rendering/low frame rate to a target/high frame rate of 500 fps, frames must be displayed every 2ms using console rendered frames produced every 10ms, requiring 4 additional synthetic frames to be generated/displayed subsequent to the console rendered frame from which the synthetic frames are generated, using the formula of (10ms/2ms - 1), i.e. 10/2 = 5 frames are displayed for the 10ms period using 1 console rendered frame and (5-1) = 4 synthetic frames. That is, as noted above, Hunt explicitly teaches what one of ordinary skill in the art would have understood Margolis’ disclosure to implicitly teach, i.e. that frame rate upsampling is performed by displaying the lower frame rate/rendered frames in sequence with the higher frame rate/adjusted frames in order to achieve the increased effective display frame rate.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement Margolis’ late stage reprojection system, using Melkote’s per-ROI motion vector grid/filter size/granularity determination technique, by displaying the pre-rendered frames and the LSR adjusted frames in sequence because Margolis teaches that the pre-rendered frames are displayed and Hunt explicitly teaches what one of ordinary skill in the art would have understood Margolis’ disclosure to implicitly teach. It is noted that this does not necessarily constitute any actual modification to Margolis’ system, i.e. as noted above, one of ordinary skill in the art would have understood the implicit teaching that Margolis’ system operated this way based on Margolis’ description, e.g. paragraph 47. However, in the interest of compact prosecution, even if Applicant were able to persuasively demonstrate that one of ordinary skill in the art would not have interpreted Margolis’ description as teaching that both the pre-rendered and LSR adjusted frames are displayed, Hunt explicitly teaches that frame rate upsampling in HMD systems using ASW/LSR can be performed by displaying the rendered/low frame rate images in sequence with the generated/ASW adjusted images to achieve the higher frame rate, such that one of ordinary skill in the art would also have found it obvious to implement Margolis’ system to display the rendered/low frame rate images in sequence with the generated/ASW adjusted images to achieve the higher frame rate in view of Hunt’s explicit teaching to use the claimed display sequencing.
Regarding claim 7, the limitations “wherein the processing circuitry comprises graphics processing circuitry and wherein the processing resources comprise one or more of graphics processing resources associated with the graphics processing circuitry, programmable hardware, or fixed function hardware, wherein the apparatus includes a virtual reality (VR) device, wherein the pipeline comprises a VR pipeline” are taught by Margolis (Margolis, e.g. paragraphs 31, 35, teaches that the HMD device includes processors executing computer readable instructions, i.e. the claimed graphics processing circuitry/programmable hardware, and is used for displaying augmented/mixed reality images, i.e. Margolis’ device is a virtual reality device, and the pipeline is a virtual reality pipeline.)
Regarding claims 8 and 14, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 1 above.
Claims 2, 5, 6, 9, 12, 13, 15, 18, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication 2017/0213388 A1 (hereinafter Margolis) in view of U.S. Patent Application Publication 2021/004985 A1 (hereinafter Melkote) in view of U.S. Patent 10,785,471 B1 (hereinafter Hunt) as applied to claims 1, 8, and 14 above, and further in view of “Perceptual Model for Adaptive Local Shading and Refresh Rate” by Ashkay Jindal, et al. (hereinafter Jindal).
Regarding claim 2, the limitations “perform the ASW on areas of the first frame with different granularities determined from the content and focusing related information wherein the pipeline comprises an ASW pipeline to perform the ASW on the first frame, the ASW pipeline comprising: a pre-processing component to: determine the content and focusing related information; determine granularities for areas of the first frame based on the content and focusing related information” are taught by Margolis in view of Melkote (As discussed in the claim 1 rejection above, in Margolis’ modified system, each area of interest, e.g. Margolis, paragraphs 92-95, corresponds to one of Melkote’s ROI, and as taught by Melkote with respect to the game engine/render side component, e.g. paragraphs 74-80, Margolis’ rendering module 302 would additionally determine the size/granularities for each area of interest based on the determined content and focusing information for the frame, and provide said size/granularities to the LSR module 308 for controlling the size/granularity of the motion vector filter kernel used for performing the reprojection/warping for each respective area of interest/ROI, corresponding to the claimed pre-processing component determining granularities for areas based on determined content and focusing related information.)
The limitation “generate a screen-space image based on the granularities” is not explicitly taught by Margolis in view of Melkote (As discussed above, in Margolis’ modified system, each area of interest, e.g. Margolis, paragraphs 92-95, corresponds to one of Melkote’s ROI, and as taught by Melkote with respect to the game engine/render side component, e.g. paragraphs 74-80, Margolis’ rendering module 302 would additionally determine the size/granularities for each area of interest based on the determined content and focusing information for the frame, and provide said size/granularities to the LSR module 308 for controlling the size/granularity of the motion vector filter kernel used for performing the reprojection/warping for each respective area of interest/ROI. While this corresponds to the claimed pre-processing component determining granularities for areas based on determined content and focusing related information, Melkote does not explicitly teach the format for providing the size/granularities from the game engine/render side 10 to the display side 16.) However, this limitation is taught by Jindal (Jindal, abstract, sections 1, 4-7, discloses a variable rate shading (VRS) system using a perceptual model to adaptively control shading rates for regions of a rendered image according to the content of the rendered scene and a model of eye motion blur. Jindal, e.g. sections 1, 2.3, 5, 5.5, figures 1 middle, 10, 13, teaches that conventional VRS systems use a VRS state map which is a screen space image representing the shading rate of each tile, whereas the perceptual VRS system uses input auxiliary buffers rendered at the resolution of the VRS state map, and uses the CaMoJAB model described in section 4 to select a shading rate for each tile of the VRS map used to control the shading rates of the rendered image. Jindal, e.g. sections 4, 5.3, 5.4, teaches that the CaMoJAB model uses determined scene content information, i.e. motion vectors, texture IDs, mipmap levels, luminance, and focus information, i.e. conservative assumptions of eye movement for each tile, to determine the shading rate for each tile in order to meet a rendering budget. That is, Jindal teaches performing rendering by determining granularities, i.e. shading rates, for areas of the rendered frame based on determined content and focusing related information for each respective area, where a screen-space image based on the granularities is generated, i.e. the VRS state map. It is noted that Jindal’s shading rates/granularities are analogous to Melkote’s size/granularities for each area of interest based on the determined content and focusing information for the frame, i.e. both are used to quantify a rate of sample points, shading points for Jindal and motion vector vertices for Melkote, for a given area of the rendered image, and both are determined based on the scene content and user focusing information, such that one of ordinary skill in the art would have recognized that if the rendered image generated by Margolis’ module 302, analogous to Melkote’s game engine/render side 10, were rendered using Jindal’s perceptual VRS system, the VRS state map generated for controlling the shading rates of the rendered frame could be used to select the size/granularities for each area of interest/ROI as taught by Melkote, thereby avoiding a second analysis of the scene content/focusing information.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Margolis’ late stage reprojection system, using Melkote’s per-ROI motion vector grid/filter size/granularity determination technique, displaying the pre-rendered frames and the LSR adjusted frames in sequence as implicitly taught by Margolis and explicitly taught by Hunt, to use Jindal’s perceptual VRS system for rendering the images in Margolis’ rendering module 302 in order to maximize rendering quality for a given budget, e.g. Jindal, section 1, and further, as noted above, to use Jindal’s VRS state maps to select the size/granularities for each area of interest/ROI to avoid performing a second analysis of the scene content/focusing information.
Regarding claim 5, the limitations “wherein the ASW pipeline is further to: split the first frame into segmentations based on the granularities defined in the screen space image, wherein the ASW pipeline comprises a motion estimation component to: perform motion search on the segmentations; get segmentations in the first frame based on the screen-space image; get blocks in one or more segmentations; and perform a per-block motion search on the first frame” are taught by Margolis in view of Melkote and Jindal (Margolis, e.g. paragraphs 69, 82-85, teaches that the motion vector module 304 of the LSR module 308 performs a motion search for every block of the rendered image, where the motion vector estimation may be based on block matching, i.e. as claimed each valid block of the first/rendered image is used to perform a per-block motion search. Further, as discussed in the claim 2 rejection above, in Margolis’ modified system, using Melkote’s per-ROI motion vector grid/filter size/granularity determination technique, and using Jindal’s perceptual VRS system for rendering images, Jindal’s VRS state maps would be used to select the size/granularities for each area of interest/ROI to avoid performing a second analysis of the scene content/focusing information, such that each block of each area of interest/ROI corresponds to one of the granularities defined in the VRS state map, analogous to Jindal, figure 13. Finally, the areas of interest/ROIs determined from the VRS state maps correspond to the claimed segmentations based on granularities defined in the screen-space image, e.g. as noted by Margolis, paragraph 94, the separate areas of interest/ROIs correspond to a segmentation of the rendered image, i.e. as in Jindal, figure 13, VRS tiles having the highest 4x4 shading rate correspond to a first segment having the highest granularity, followed by the 2x4 and 4x2 shading rate segments, etc. It is noted that Applicant’s disclosure, e.g. paragraphs 281, 284, indicates that the motion search on the segmentations is performed by performing the per-block motion search on the blocks of each segmentation, and does not require otherwise evaluating and/or using the segmentations determined from the screen-space image, such that the scope of claim 5 is met by Margolis’ modified system, having Melkote’s sizes/granularities for each block determined using Jindal’s VRS state map, performing the per-block motion search for every block in the rendered image, i.e. performing the per-block motion search for all blocks is performing the motion search for all segmentations comprising said blocks, where blocks are associated with different segmentations based on the granularities as defined in the screen-space image.)
Regarding claim 6, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 5 above, except that claim 6 requires performing segmentation/per-block extrapolation for the blocks of the segmentations, rather the segmentation/per-block motion search for the blocks of the segmentation, which is also taught by Margolis’ modified system, e.g. Margolis, paragraphs 24, 61, 70-74, 86-88, teaches that the determined motion vector for each block is used to project the block to generate the display field for a target time after the current rendered frame, i.e. the claimed per-block extrapolation for the blocks of each segmentation. It is noted that Melkote, e.g. paragraph 84, analogously teaches that the warping based on the motion vectors corresponds to an extrapolation.
Regarding claims 9 and 15, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 2 above.
Regarding claims 12, 13, 18, and 19, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claims 5 and 6 above.
Claims 3, 10, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication 2017/0213388 A1 (hereinafter Margolis) in view of U.S. Patent Application Publication 2021/004985 A1 (hereinafter Melkote) in view of U.S. Patent 10,785,471 B1 (hereinafter Hunt) in view of “Perceptual Model for Adaptive Local Shading and Refresh Rate” by Ashkay Jindal, et al. (hereinafter Jindal) as applied to claims 2, 9, and 15 above, and further in view of “Luminance-Contrast-Aware Foveated Rendering” by Okan Tarhan Tursun, et al. (hereinafter Tursun).
Regarding claim 3, the limitations “wherein the content and focusing related information comprises a foveated area associated with the first frame, and wherein the pre-processing component is further to: determine the foveated area based on head pose/motions information collected from a head-mounted display (HMD) communicatively coupled with the apparatus; and assigning a finest granularity to the foveated area” are partially taught by Margolis in view of Melkote and Jindal (Margolis, e.g. paragraphs 25, 45, 47, 92-94, teaches that the system includes an HMD, determining the pose/motions of the HMD, and determining the areas of interest based on where the user’s gaze is focused, but does not address foveation, per se. Melkote, e.g. paragraphs 28, 41, 66, 77, similarly indicates the use of HMD/eye tracking for determining ROIs, and indicates that the ROI may be used for foveated rendering, but does not discuss the foveated area with respect to the determining the size/granularities for each area of interest, i.e. the claimed assigning a finest granularity to the foveated area. As discussed in the claim 2 rejection above, in Margolis’ modified system, using Melkote’s per-ROI motion vector grid/filter size/granularity determination technique, and using Jindal’s perceptual VRS system for rendering images, Jindal’s VRS state maps would be used to select the size/granularities for each area of interest/ROI to avoid performing a second analysis of the scene content/focusing information. Jindal, e.g. sections 5.4, 6, teaches that the perceptual VRS system does not rely on eye tracking or model foveation, but teaches that if gaze location were known, the allocation of the rendering budget would be improved, and foveated rendering could be performed to further increase quality, referring to the Tursun reference for details of modeling foveation using gaze location in a VRS based rendering system. That is, Margolis’ modified system of the claim 2 rejection uses Jindal’s unmodified perceptual VRS system for rendering and VRS state map for selecting the size/granularities for each area of interest/ROI, and therefore does not use the tracked HMD/eye gaze location for determining a foveated area and assigning a finest granularity to the foveated areas.) However, this limitation is taught by Jindal in view of Tursun (As noted above, Jindal, e.g. section 6, suggests combining the perceptual VRS system with Tursun’s foveated rendering when gaze location is available. Tursun, e.g. abstract, sections 1, 3-7, describes a system for luminance-contrast-aware foveated rendering which, analogous to Jindal’s perceptual VRS system, relies on a perceptual model to evaluate scene content and focusing information for each patch of an image to be rendered in order to select a VRS shading rate for the patch in a VRS state map, e.g. sections 4, 6.2, figure 3. Tursun, e.g. section 4.1, subsection Contrast sensitivity and retinal eccentricity, section 5, indicates that the eccentricity measured relative to a gaze position is used to control the foveation region and resolution reduction, where, e.g. section 7.2, in the implementation using an HMD the gaze position is defined using the eye tracking results, i.e. Tursun’s foveated VRS rendering system uses the gaze position determined from the HMD pose/motions to determine a foveated rendering region having higher shading/resolution rates selected closer to the gaze point and lower shading/resolution rates selected further from the gaze point, corresponding to the claimed assignation of finest granularity to the foveated area, e.g. as shown in the figure 3 example, where the gaze location is defined at the center of the image, the highest rates/granularities are assigned to the patches nearest the center.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Margolis’ late stage reprojection system, using Melkote’s per-ROI motion vector grid/filter size/granularity determination technique, displaying the pre-rendered frames and the LSR adjusted frames in sequence as implicitly taught by Margolis and explicitly taught by Hunt, using Jindal’s perceptual VRS system for rendering the images in Margolis’ rendering module, using Jindal’s VRS state maps to select the size/granularities for each area of interest/ROI, to modify Jindal’s perceptual VRS system to use the gaze position determined by Margolis’ HMD to include Tursun’s foveated VRS rendering technique because Jindal teaches that if the gaze position were known, Jindal’s perceptual VRS system could be improved by performing foveated rendering as taught by Tursun, both to improve the allocation of the rendering budget and the rendering quality. In the modified system, Margolis’ rendering module renders the first image using Jindal’s perceptual VRS system combined with Tursun’s foveated rendering, i.e. using the determined gaze point to evaluate eccentricity for each VRS tile/patch as in Tursun, section 4.1, and adjust the shading rates as in section 5, i.e. as claimed the pre-processing component determines the foveated area based on head pose/motions collected from the HMD, and assigns the finest granularity to the foveated area, analogous to Tursun, figure 3. Further, as discussed in the claim 2 modification, Jindal’s VRS state map for selecting the size/granularities for each area of interest/ROI, such that in the modified system using Jindal’s perceptual VRS system combined with Tursun’s foveated rendering, the areas of interest/ROIs which are located within the fovea, i.e. closest to the gaze point, will be assigned the highest shading rate, and by extension, the finest motion vector grid size/granularity.
Regarding claims 10 and 16, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 3 above.
Claims 4, 11, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication 2017/0213388 A1 (hereinafter Margolis) in view of U.S. Patent Application Publication 2021/004985 A1 (hereinafter Melkote) in view of U.S. Patent 10,785,471 B1 (hereinafter Hunt) in view of “Perceptual Model for Adaptive Local Shading and Refresh Rate” by Ashkay Jindal, et al. (hereinafter Jindal) in view of “Luminance-Contrast-Aware Foveated Rendering” by Okan Tarhan Tursun, et al. (hereinafter Tursun) as applied to claims 3, 10, and 16 above, and further in view of U.S. Patent Application Publication 2019/0005714 A1 (hereinafter Fuller).
Regarding claim 4, the limitations “wherein the content and focusing related information comprises information related to degree of detail associated with the first frame, and the pre-processing component is further to: determine the information related to degree of details of areas of the first frame based on sampler/shader output from rendering of the input data; and assign finer granularity to areas with more details” are taught by Margolis in view of Jindal (It is noted that Applicant’s disclosure, e.g. paragraph 249, gives the example of determining detail information by collecting LOD or Mipmap information from the shader to determine the claimed degree of detail of areas. Jindal, e.g. section 5, 5.1, 5.3 teaches that the input comprises a plurality of auxiliary buffers rendered at the VRS state map resolution, including texture IDs and mipmap levels, where the quality metric is dependent on said texture IDs and mipmap levels, i.e. as claimed, the pre-processing component performing Jindal’s perceptual VRS rendering determines degree of detail information for the first frame based on shader/sampler output from rendering the input data, where higher resolution mipmap levels require higher shading rates such that the ratio of pixel-to-texel area is close to 1:1, e.g. section 4.1, subsection Variable rate shading, i.e. finer shading rates/granularities are assigned to areas with higher resolution mipmap levels.)
The limitations “wherein the content and focusing related information comprises edge areas associated with the first frame; determine the edge areas based on edge information from an edge detector; and increase fineness of granularity to be assigned to the edge areas” are partially taught by Margolis in view of Jindal (Margolis, e.g. paragraph 82, teaches that edge detection may be performed for feature matching between frames in order to determine motion vectors. Further, Jindal, e.g. sections 4, 4.1, subsection Contrast sensitivity and energy pooling, describing the CaMoJAB model, and in particular the contrast sensitivity function based on the luminance, indicates that the function is based on spatial frequencies, i.e. as one of ordinary skill in the art would understand, the human visual system has increased sensitivity to distortions of image regions having higher intensity spatial frequency changes, i.e. highly textured regions including those containing edges, in comparison to distortions of image regions having lower intensity spatial frequency changes, i.e. regions which are relatively uniformly colored, such that one of ordinary skill in the art would understand that distortions caused by lower shading rates would be more noticeable for regions containing edges than mostly uniform regions. While Jindal, e.g. section 5.1, teaches using the previously rendered frame to approximate the luminance for a VRS tile, Jindal does not explicitly teach using the results of edge detection for a VRS tile to assign increased quality/shading rates to a VRS tile, i.e. the claimed increased fineness of granularity for areas determined to comprise edges using the edge detector.) However, this limitation is taught by Fuller (Fuller, e.g. abstract, paragraphs 18-67, describes a VRS rendering system which uses temporal reprojection to determine shading rates. Fuller, e.g. 18, 19, 50-60, teaches that the shading rate values (SRP) specified in the coarse SRP map, corresponding to Jindal’s VRS state map, may be increased or decreased depending on whether the corresponding area has high frequency details, and in particular, paragraph 57, applying an edge detection algorithm to identify areas having edges in order to increase the shading rate for that area, i.e. as claimed, increasing fineness of granularity assigned to areas which are determined to have edges based on an edge detector.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Margolis’ late stage reprojection system, using Melkote’s per-ROI motion vector grid/filter size/granularity determination technique, displaying the pre-rendered frames and the LSR adjusted frames in sequence as implicitly taught by Margolis and explicitly taught by Hunt, using Jindal’s perceptual VRS system for rendering the images in Margolis’ rendering module, using Jindal’s VRS state maps to select the size/granularities for each area of interest/ROI, with Jindal’s perceptual VRS system modified to use the gaze position determined by Margolis’ HMD to include Tursun’s foveated VRS rendering technique, to include Fuller’s VRS shading rate parameter adjustment based on performing edge detection on the corresponding areas of the previously rendered frame. As noted above, Jindal teaches that the perceptual VRS system already relies on rendering results from the previous frame, i.e. the luminance values, such that modifying the system to include Fuller’s edge detection VRS rate parameter adjustment would merely require additionally performing Fuller’s edge detection, and using the results to increase or decrease the VRS shading rate value for each area, where as noted above, areas comprising detected edges would have increased adjusted shading rates, corresponding to the claimed increasing fineness of granularity assigned to areas determined to have edges using the edge detector.
Regarding claims 14 and 17, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 4 above.
Response to Arguments
Applicant's arguments filed 5/7/26 have been fully considered but they are not persuasive.
Applicant’s amendment to the independent claims includes limitations previously addressed in view of Melkote in the claim 2 rejection. However, Applicant’s remarks are limited to arguing that the amended independent claims are not taught by Margolis and Hunt, without suggesting any reason why the amended independent claims would not be obvious in further view of Melkote as previously mapped in the claim 2 rejection. Therefore, Applicant’s remarks do not suggest any reason why the amended independent claims are not obvious in view of Margolis, Melkote, and Hunt, and cannot be considered persuasive.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT BADER whose telephone number is (571)270-3335. The examiner can normally be reached 11-7 m-f.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at 571-272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROBERT BADER/Primary Examiner, Art Unit 2611
Prosecution Insights