DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by O. Iqbal, V. I. T. Muro, S. Katoch, A. Spanias and S. Jayasuriya, "Adaptive Subsampling for ROI-Based Visual Tracking: Algorithms and FPGA Implementation," in IEEE Access, vol. 10, pp. 90507-90522, 2022, doi: 10.1109/ACCESS.2022.3200755. (hereinafter "Iqbal").
Regarding claim 1, Iqbal teaches a method comprising: rendering a frame comprising a plurality of regions (figs. 3, 10, 12), each region having subsampling characteristics (Section III B – the various object detection methods are used to determine subsample masks and do so by producing information for the region of interest, which is analogous to subsampling characteristics); and post-processing the frame based on information regarding the plurality of regions and the subsampling characteristics of each region (fig. 11, pg. 5 left column paragraph 8 – “This enables performance analysis of the ATOM and DiMP methods in the context of adaptive subsampling. Further, we also study what bearing the Kalman filter has on the performance”, section V B – “Our algorithm performance takes into consideration the delay of capturing an image, performing preprocessing, detection, postprocessing and updating the Kalman filter on every keyframe.”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2, 3, 8, 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Iqbal in view of Kilgariff (US 20150206511 A1).
Regarding claim 2, Iqbal teaches the method of claim 1. Iqbal fails to explicitly teach rendering the frame at a discrete graphics processing unit (dGPU).
However, Kilgariff teaches rendering the frame at a discrete graphics processing unit (dGPU) (paragraph [0004]). Kilgraff describes utilizing a dGPU to render images. Kilgraff is considered analogous to the claimed invention as it is in the same field of image processing. Therefore it would be obvious to one of ordinary skill in the art to combine the teachings of Iqbal with Kilgraff and specify the use of a dGPU to render frames in order to improve graphics processing capabilities.
Regarding claim 3, Iqbal in view of Kilgariff teaches the method of claim 2. Iqbal further teaches wherein post-processing the frame comprises post-processing the frame at an accelerated processing unit (APU) (Section V A “Hardware implementation” – “The image capture, preprocessing, digital ROI, Kalman filter update and prediction steps, and any other postprocessing required by each specific tracker is done in the processing system of the FPGA board.” The FPGA is a type of APU) based on a frame rate of a video stream comprising the frame (pg. 9 left column paragraph 2 – “R represents frame rate (fixed at 30 fps)”, pg. 5 right column paragraph 4 under “Datasets” – “The frames rates of all videos in our test datasets are 30 FPS”; All processing is fixed at the frame rate of the video stream, this also applies to post processing)
Regarding claim 8, Iqbal teaches A processing system, comprising: render a frame comprising a plurality of regions (figs. 3, 10, 12), each region having subsampling characteristics (Section III B – the various object detection methods are used to determine subsample masks and do so by producing information for the region of interest, which is analogous to subsampling characteristics); and an accelerated processing unit (APU) configured to post-process the frame (Section V A “Hardware implementation” – “The image capture, preprocessing, digital ROI, Kalman filter update and prediction steps, and any other postprocessing required by each specific tracker is done in the processing system of the FPGA board.” The FGPA is a type of APU) based on information regarding the plurality of regions and the subsampling characteristics of each region (fig. 11, pg. 5 left column paragraph 8 – “This enables performance analysis of the ATOM and DiMP methods in the context of adaptive subsampling. Further, we also study what bearing the Kalman filter has on the performance”, section V B – “Our algorithm performance takes into consideration the delay of capturing an image, performing preprocessing, detection, postprocessing and updating the Kalman filter on every keyframe.”).
Iqbal fails to explicitly teach a discrete graphics processing unit (dGPU) configured to render a frame.
However, Kilgariff teaches rendering the frame at a discrete graphics processing unit (dGPU) (paragraph [0004]). Kilgraff describes utilizing a dGPU to render images. Kilgraff is considered analogous to the claimed invention as it is in the same field of image processing. Therefore it would be obvious to one of ordinary skill in the art to combine the teachings of Iqbal with Kilgraff and specify the use of a dGPU to render frames in order to improve graphics processing capabilities.
Regarding 15, Iqbal teaches A processing system, comprising: render a frame comprising a plurality of regions (figs. 3, 10, 12) based on received information regarding the plurality of regions and subsampling characteristics of each region (Section III B – the various object detection methods are used to determine subsample masks and do so by producing information for the region of interest, which is analogous to subsampling characteristics), the plurality of regions comprising a first region having first subsampling characteristics (fig. 1, 2) and a second region having second subsampling characteristics different from the first subsampling characteristics (fig. 3 – regions have different IoU values); and post-process the frame based on the received information regarding the plurality of regions and subsampling characteristics of each region (fig. 11, pg. 5 left column paragraph 8 – “This enables performance analysis of the ATOM and DiMP methods in the context of adaptive subsampling. Further, we also study what bearing the Kalman filter has on the performance”, section V B – “Our algorithm performance takes into consideration the delay of capturing an image, performing preprocessing, detection, postprocessing and updating the Kalman filter on every keyframe.”).
Iqbal fails to explicitly teach a discrete graphics processing unit (dGPU).
However, Kilgraff teaches a discrete graphics processing unit (dGPU) (paragraph [0004]). Kilgraff describes utilizing a dGPU to render images. Kilgraff is considered analogous to the claimed invention as it is in the same field of image processing. Therefore it would be obvious to one of ordinary skill in the art to combine the teachings of Iqbal with Kilgraff and specify the use of a dGPU to render frames in order to improve graphics processing capabilities.
Claim(s) 4, 10, 11, 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Iqbal in view of Kilgraff and in further view of Tikhostoup (US 20230094384 A1).
Regarding claim 4, Iqbal in view of Kilgariff teaches The method of claim 3. Iqbal further teaches profiling an impact of performing post-processing of the frame (fig. 11, section IV Metrics) on the frame rate of the video stream (pg. 9 left column paragraph 2 – “R represents frame rate (fixed at 30 fps)”, pg. 5 right column paragraph 4 under “Datasets” – “The frames rates of all videos in our test datasets are 30 FPS”; All processing is fixed at the frame rate of the video stream, this also applies to post processing) and post-processing the frame at the APU (Section V A “Hardware implementation” – “The image capture, preprocessing, digital ROI, Kalman filter update and prediction steps, and any other postprocessing required by each specific tracker is done in the processing system of the FPGA board.” The FPGA is a type of APU) based further on the profiling (pg. 5 left column paragraph 6 – right column paragraph 1, “we modify the algorithms and allow the output ROI of one image frame D(I(x; y; t)) = bt dictate the ROI sensor mask of the next incoming frame”, “This enables performance analysis of the ATOM and DiMP methods in the context of adaptive subsampling”, “In this variation of the algorithm, the ATOM and DiMP are relegated to the sole purpose of feeding external measurements to the Kalman filter after certain intervals such that the Kalman filter is able to update and correct its trajectory”; Iqbal uses various methods of analysis and post-processing based on this analysis, repeatedly, for the frames in the video)
Iqbal fails to teach performing post-processing of the frame at the dGPU and post-processing the frame at the APU.
However, Tikhostoup teaches performing post-processing of the frame at the dGPU and post-processing the frame at the APU (paragraph [0134]). Tikhostoup is considered analogous to the claimed invention as it is in the same field of resource management. Therefore it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the teachings of Tikhostoup with Iqbal and Kilgariff to improve resource management.
Regarding claim 10 Iqbal in view of Kilgraff teaches the processing system of claim 9. Iqbal further teaches profiling circuitry configured to profile an impact of performing post-processing of the frame (figs. 3-7, 11, Section IV Metrics) on the frame rate of the video stream (pg. 9 left column paragraph 2 – “R represents frame rate (fixed at 30 fps)”, pg. 5 right column paragraph 4 under “Datasets” – “The frames rates of all videos in our test datasets are 30 FPS”; All processing is fixed at the frame rate of the video stream, this also applies to post processing).
Iqbal fails to teach performing post-processing of the frame at the dGPU
However, Tikhostoup teaches performing post-processing of the frame at the dGPU (paragraph [0134]). Tikhostoup is considered analogous to the claimed invention as it is in the same field of resource management. Therefore it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the teachings of Tikhostoup with Iqbal and Kilgariff to improve resource management.
Regarding claim 11, Iqbal in view of Kilgraff and in further view of Tikhostoup teach The processing system of claim 10. Iqbal further teaches wherein the profiling circuitry is further configured to task the APU with post-processing the frame (Section V A “Hardware implementation” – “The image capture, preprocessing, digital ROI, Kalman filter update and prediction steps, and any other postprocessing required by each specific tracker is done in the processing system of the FPGA board.” The FGPA is a type of APU) and the impact exceeding a threshold (Section IV Metrics).
Iqbal fails to explicitly teach post-processing the frame in response to the impact exceeding a threshold. However, it would have been obvious to one of ordinary skill in the art to utilize the described IoU threshold in the process of tracking performance (described in the “Metrics” section) with the updating of ROI predictions based on performance analysis because the IoU threshold is representative of performance.
Regarding claim 16, Iqbal in view of Kilgraff teaches The processing system of claim 15. Iqbal further teaches profiling circuitry configured to profile an impact of performing post-processing of the frame (figs. 3-7, 11, Section IV Metrics) on the frame rate of the video stream (pg. 9 left column paragraph 2 – “R represents frame rate (fixed at 30 fps)”, pg. 5 right column paragraph 4 under “Datasets” – “The frames rates of all videos in our test datasets are 30 FPS”; All processing is fixed at the frame rate of the video stream, this also applies to post processing).
Iqbal fails to teach performing post-processing of the frame at the dGPU
However, Tikhostoup teaches performing post-processing of the frame at the dGPU (paragraph [0134]). Tikhostoup is considered analogous to the claimed invention as it is in the same field of resource management. Therefore it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the teachings of Tikhostoup with Iqbal and Kilgariff to improve resource management.
Regarding 17, Iqbal in view of Kilgraff and Tikhostoup teaches the processing system of claim 16. Iqbal further teaches an accelerated processing unit (APU) configured to post-process the frame (Section V A “Hardware implementation” – “The image capture, preprocessing, digital ROI, Kalman filter update and prediction steps, and any other postprocessing required by each specific tracker is done in the processing system of the FPGA board.” The FPGA is a type of APU) based on the plurality of regions and the subsampling characteristics of each region (fig. 11, pg. 5 left column paragraph 8 – “This enables performance analysis of the ATOM and DiMP methods in the context of adaptive subsampling. Further, we also study what bearing the Kalman filter has on the performance”, section V B – “Our algorithm performance takes into consideration the delay of capturing an image, performing preprocessing, detection, postprocessing and updating the Kalman filter on every keyframe.”), and the impact exceeding a threshold (Section IV Metrics).
Iqbal fails to explicitly teach post-processing the frame in response to the impact exceeding a threshold. However, it would have been obvious to one of ordinary skill in the art to utilize the described IoU threshold in the process of tracking performance (described in the “Metrics” section) with the updating of ROI predictions based on performance analysis because the IoU threshold is representative of performance.
Claim(s) 5, 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Iqbal in view of Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. 2016. Towards foveated rendering for gaze-tracked virtual reality. ACM Trans. Graph. 35, 6, Article 179 (November 2016), 12 pages. https://doi.org/10.1145/2980179.2980246 (hereinafter "Patney").
Regarding claim 5, Iqbal teaches the method of claim 1. Iqbal fails to teach wherein the plurality of regions is further based on tracking of a gaze of a viewer of the frame.
Iqbal fails to teach wherein the plurality of regions is further based on tracking of a gaze of a viewer of the frame.
However, Patney teaches wherein the plurality of regions is further based on tracking of a gaze of a viewer of the frame (abstract, pg. 5 paragraph 4 of procedure, section 3.1). Patney is considered analogous to the claimed invention as it is in the same field of foveated rendering. Therefore it would have been obvious to one of ordinary skill in the art to combine the object detection based subsampling methods of Iqbal with the foveated rendering techniques of Patney, which includes tracking a gaze of a viewer, in order to improve performance, and provide a perceptually improved experience.
Regarding claim 6, Iqbal teaches the method of claim 1.
Iqbal fails to teach wherein the subsampling characteristics comprise at least one of a degree and a direction of subsampling applied to each region.
However, Patney teaches wherein the subsampling characteristics comprise at least one of a degree and a direction of subsampling applied to each region (fig. 9, section 3.1.1 – “The size of the subsampling filter increases linearly with retinal eccentricity.”). The motivation would have been the same as that of claim 5.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Iqbal in view of Besenthal, Simon & Maisch, Sebastian & Ropinski, Timo. (2019). Multi-Resolution Rendering for Computationally Expensive Lighting Effects. 10.48550/arXiv.1906.04576. (hereinafter "Besenthal").
Regarding claim 7, Iqbal teaches the method of claim 1. Iqbal fails to teach wherein post-processing comprises blending borders between regions of the frame having different subsampling characteristics.
However, Besenthal teaches blending borders between regions of the frame (section 3.3 & 4.2) having different subsampling characteristics (figs. 2 & 3 and their descriptions – the different resolutions of sub-images are analogous to different subsampling characteristics). Besenthal is considered analogous to the claimed invention as it is in the same field of rendering techniques. Therefore it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the teachings of Besenthal with Iqbal in view of Kilgraff in order to improve rendering speed.
Claim(s) 12, 13, 18, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Iqbal in view of Kilgraff and in further view of Patney.
Regarding claim 12, Iqbal in view of Kilgraff teaches The processing system of claim 8. Iqbal further teaches wherein the APU is further configured to assign the plurality of regions of the frame and subsampling characteristics of each region (section V paragraph 1, section III A “Video Subsampling and ROI prediction”, Section III B – the various object detection methods are used to determine subsample masks and do so by producing information for the region of interest, which is analogous to subsampling characteristics).
Iqbal fails to teach based on tracking of a gaze of a viewer of the frame.
However, Patney teaches wherein the plurality of regions is further based on tracking of a gaze of a viewer of the frame (abstract, pg. 5 paragraph 4 of procedure, section 3.1). Patney is considered analogous to the claimed invention as it is in the same field of foveated rendering. Therefore it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine Iqbal in view of Kilgraff to specify implementation of foveated rendering.
Regarding claim 13, Iqbal in view of Kilgraff teaches The processing system of claim 8. Iqbal fails to teach wherein the subsampling characteristics comprise at least one of a degree and a direction of subsampling applied to each region.
However, Patney teaches wherein the subsampling characteristics comprise at least one of a degree and a direction of subsampling applied to each region (fig. 9, section 3.1.1 – “The size of the subsampling filter increases linearly with retinal eccentricity.”). The motivation to combine Iqbal in view of Kilgraff with Patney would have been the same as that of claim 12.
Regarding claim 18, Iqbal in view of Kilgraff teaches The processing system of claim 17. Iqbal further teaches wherein the APU is further configured to assign the plurality of regions of the frame and subsampling characteristics of each region (section V paragraph 1, section III A “Video Subsampling and ROI prediction”, Section III B – the various object detection methods are used to determine subsample masks and do so by producing information for the region of interest, which is analogous to subsampling characteristics).
Iqbal in view of Kilgraff fails to teach so based on tracking of a gaze of a viewer of the frame.
However, Patney teaches wherein the plurality of regions is further based on tracking of a gaze of a viewer of the frame (abstract, pg. 5 paragraph 4 of procedure, section 3.1). Patney is considered analogous to the claimed invention as it is in the same field of foveated rendering. Therefore it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine Iqbal in view of Kilgraff to specify implementation of foveated rendering.
Regarding claim 19, Iqbal in view of Kilgraff teaches The processing system of claim 15.
Iqbal in view of Kilgraff fails to teach wherein the subsampling characteristics comprise at least one of a degree and a direction of subsampling applied to each region.
However, Patney teaches wherein the subsampling characteristics comprise at least one of a degree and a direction of subsampling applied to each region (fig. 9, section 3.1.1 – “The size of the subsampling filter increases linearly with retinal eccentricity.”). The motivation to combine would have been the same as that of claim 18.
Claim(s) 9, 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Iqbal in view of Kilgraff and in further view of Besenthal.
Regarding claim 9, Iqbal in view of Kilgraff teaches the processing system of claim 8.
wherein the system is further configured to render the frame response to a frame rate of a video stream comprising the frame exceeding a frame rate threshold (“The computationally inexpensive Kalman filter is used as a forecaster, and frames are skipped whenever the tracker is slower than the world frame rate.” Skipping rendering when the frame rate is below a threshold renders only frames that are at or above the world frame rate, this is analogous to rendering a frame in response to a frame rate exceeding a threshold).
Iqbal fails to explicitly teach wherein the dGPU is further configured to render the frame with a plurality of regions having different subsampling characteristics.
However, Kilgraff teaches teach wherein the dGPU is further configured to render the frame (paragraph [0004]).
Iqbal in view of Kilgraff fails to explicitly teach render the frame with a plurality of regions having different subsampling characteristics.
However, Besenthal teaches render the frame with a plurality of regions having different subsampling characteristics (figs. 2 & 3 and their descriptions – the different resolutions of sub-images are analogous to different subsampling characteristics). (figs. 2 & 3 and their descriptions – the different resolutions of sub-images are analogous to different subsampling characteristics)
Regarding claim 14, Iqbal in view of Kilgraff teaches The processing system of claim 8, wherein the APU is configured to post-process the frame (Iqbal Section V A “Hardware implementation” – “The image capture, preprocessing, digital ROI, Kalman filter update and prediction steps, and any other postprocessing required by each specific tracker is done in the processing system of the FPGA board.” The FGPA is a type of APU). Iqbal in view of Kilgraff fails to teach blending borders between regions of the frame having different subsampling characteristics.
However, Besenthal teaches blending borders between regions of the frame (section 3.3 & 4.2) having different subsampling characteristics (figs. 2 & 3 and their descriptions – the different resolutions of sub-images are analogous to different subsampling characteristics). Besenthal is considered analogous to the claimed invention as it is in the same field of rendering techniques. Therefore it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the teachings of Besenthal with Iqbal in view of Kilgraff in order to improve rendering speed.
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Iqbal in view of Kilgraff and in further view of Besenthal and Tikhostoup.
Regarding claim 20, Iqbal in view of Kilgraff teaches The processing system of claim 15. Iqbal fails to teach wherein the dGPU is configured to post-process the frame by blending borders between regions of the frame having different subsampling characteristics.
However, Besenthal teaches blending borders between regions of the frame (section 3.3 & 4.2) having different subsampling characteristics (figs. 2 & 3 and their descriptions – the different resolutions of sub-images are analogous to different subsampling characteristics). Besenthal is considered analogous to the claimed invention as it is in the same field of rendering techniques. Therefore it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the teachings of Besenthal with Iqbal in view of Kilgraff in order to improve rendering speed.
Iqbal in view of Kilgraff and Besenthal fails to teach wherein the dGPU is configured to post-process the frame.
However, Tikhostoup teaches wherein the dGPU is configured to post-process the frame (paragraph [0134]). Tikhostoup is considered analogous to the claimed invention as it is in the same field of resource management. Therefore it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the teachings of Tikhostoup with Iqbal and Kilgariff to improve resource management.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Okan Tarhan Tursun, Elena Arabadzhiyska-Koleva, Marek Wernikowski, Radosław Mantiuk, Hans-Peter Seidel, Karol Myszkowski, and Piotr Didyk. 2019. Luminance-contrast-aware foveated rendering. ACM Trans. Graph. 38, 4, Article 98 (August 2019), 14 pages. https://doi.org/10.1145/3306346.3322985
Malkin, E., Deza, A. and Poggio, T., 2020. CUDA-optimized real-time rendering of a foveated visual system. arXiv preprint arXiv:2012.08655.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Aidan W McCoy whose telephone number is (571)272-5935. The examiner can normally be reached 8:00 AM-5:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571)272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AIDAN W MCCOY/Examiner, Art Unit 2611
/TAMMY PAIGE GODDARD/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights