DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the Amendment filed on 10/08/2025.
Claims 1-20 are pending. Claims 4, 11, 19 have been amended.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1, 2, 8, 9, 15, 16, 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Park et al. (US 20180300838 A1, hereinafter Park) in view of Croxford et al. (US 20200410740 A1, hereinafter Croxford).
Regarding Claim 1, Park teaches a graphics processor (Park, Paragraph [0002], “Examples discussed herein relate to an integrated circuit that comprises a graphics processing unit GPU <read on graphics processor> to periodically render frame images in a frame buffer. These images are intended for output to a display device.”), a memory interface (Park, Paragraph [0017], “System 100 includes communication interface 120 , general processing system 130 , storage system 140 , display 160 , and graphics processing unit GPU 180 .” and “Storage system 140... may comprise a disk, tape, integrated circuit, RAM, ROM, EEPROM, flash memory, network storage, server, or other memory function <read on memory interface>”), a processing cluster coupled with the memory interface, the processing cluster including a plurality of processing resources coupled via a data interconnect (Park, Paragraph [0001], “Integrated circuits, and systems-on-a-chip SoC may include multiple independent processing units a.k.a., cores that read and execute instructions. These multi-core processing chips typically cooperate to implement multiprocessing.” and “GPU 180 is a type of specialized processor designed for the rapid creation of images in a memory called a frame buffer.” <read on processing cluster including a plurality of processing resources coupled via a data interconnect>); first circuitry to process input data via a processing resource of the plurality of processing resources, the first circuitry to (Park, Paragraph [0020], “GPU 180 is a type of specialized processor designed for the rapid creation of images in a memory called a frame buffer.” and “GPU 180 may be operated at a frequency that is determined by frequency manager 181 and/or real-time DCVS 182 . Frequency manager 181 and/or real-time DCVS 182 may select a per-frame operating frequency for GPU 180 based on timing information received from frame timing 185 .” <read on first circuitry to process input data via a processing resource of the plurality of processing resources>): process workloads submitted to a command queue of a graphics engine to render frame data for a frame (Park, Paragraph [0037], “GPU 180 periodically e.g., every frame renders frame images in a frame buffer <read on render frame data for a frame> intended for output to display 160”); track progress for submitted workloads for the frame (Park, Paragraph [0033], “For each frame, while the current frame is being rendered by GPU 180 , frequency manager 181 estimates the time remaining to complete the rendering of the current frame. The time remaining to complete the rendering of the current frame may be based on the previous rendering times of one or more immediately preceding frames.” <read on track progress for submitted workloads for the frame>); determine if the frame will meet a target display update deadline (Park, Paragraph [0028], “In an embodiment, when a command to start rendering each frame is received by GPU 180, frequency manager 181 compares the estimated workload (i.e., estimated rendering time) and the remaining time before the next rendering deadline (e.g., Vsync)” and “If the predicted remaining rendering time is larger than the remaining time to the next rendering deadline, frequency manager 181 calculates and applies a new frequency level e.g., higher frequency than the current level in order to avoid missing the rendering deadline.” and “If the predicted remaining rendering time is smaller than the remaining time to next rendering deadline, frequency manager 181 calculates and applies a new frequency level lower frequency than the current level to reduce power consumption without missing the rendering deadline.”; [0032], “The target time to complete the rendering of the current frame may be based on an indicator of the remaining time until a vertical synchronization signal e.g., Vsync occurs.” <read on determine if the frame will meet a target display update deadline>); continue to execute the workloads to render the frame data in response to a determination that the frame will meet the target display update deadline (Park, Paragraph [0028], “If the predicted remaining rendering time is smaller than the remaining time to next rendering deadline, frequency manager 181 calculates and applies a new frequency level lower frequency than the current level to reduce power consumption without missing the rendering deadline”). [[ request neural frame generation for the frame in response to a determination that the frame will not meet the target display update deadline]]; display a rendered or generated frame at the target display update deadline (Park, Paragraph [0031], “GPU 180 is to periodically render frame images in a frame buffer intended for output to display device 160 .” and “In an example, frame deadline times correspond to the timing of a vertical synchronization signal, V sync , being sent by GPU 180 to display 160 ”)
But Park does not explicitly disclose request neural frame generation for the frame in response to a determination that the frame will not meet the target display update deadline.
However, Croxford teaches request neural frame generation for the frame in response to a determination that the frame will not meet the target display update deadline (Croxford, Paragraph [0063], “the technology described herein can reduce distortions and artefacts, while providing significant savings in terms of memory bandwidth, processing resources and power, etc., when performing so-called spacewarp processing.”; [0149], “In an embodiment, the neural network is executed by a neural network processing unit (processor) (NPU) of the graphics processing system.” [0151], “Thus, in an embodiment the method comprises: (a graphics processing unit) generating the rendered frame; and a neural network processing unit of the graphics processing system generating the output (e.g. extrapolated) frame <read on neural frame generation>.” [0179], “, the output (e.g. transformed and/or extrapolated) frame is in an embodiment a frame generated for display <read on generated frame>”; [0063], “the technology described herein can reduce distortions and artefacts, while providing significant savings in terms of memory bandwidth, processing resources and power, etc., when performing so-called spacewarp processing.”)
Park and Croxford are analogous since both concern graphics processing units rendering frames for display under timing/deadline constraints, including VR/AR/MR use. Park provided a way of adjusting GPU frequency based on an estimate of frame rendering time and remaining time to a Vsync-based deadline so frames complete by the target time. Croxford provided a way of generating extrapolated frames using a neural network processing unit so that additional frames are produced while a next frame is being rendered, reducing artefacts and resource cost. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Croxford’s neural-network-based extrapolated frame generation into Park’s frame-deadline-controlled GPU, such that when Park’s system determines a frame risks missing the target display update deadline, it requests neural frame generation so a rendered or generated frame is still available at the deadline, improving smoothness and avoiding missed frames while saving processing resources.
Regarding Claim 2, the combination of Park and Croxford teaches the invention in Claim 1.
The combination further teaches second circuitry to perform neural frame generation for the frame (Croxford, Paragraph [0151], "Thus, in an embodiment the method comprises: (a graphics processing unit) generating the rendered frame; and a neural network processing unit of the graphics processing system generating the output (e.g. extrapolated) frame <read on second circuitry to perform neural frame generation for the frame>."), the second circuitry including a compute engine associated with the processing resource (Croxford, Paragraph [0149] , "In an embodiment, the neural network is executed by a neural network processing unit (processor) (NPU) <read on compute engine> of the graphics processing system <read on associated with the processing resource>.").
Croxford and Park are analogous since both relate to graphics processing systems that render image frames for output to a display, including VR/AR/MR use cases, and both are directed to solving problems of maintaining smooth frame output and reducing artefacts under resource and timing constraints. Park provided a way of managing per-frame GPU rendering deadlines using frequency scaling and Vsync-based timing estimation. Croxford provided a way of supplementing GPU rendering with a neural network processing unit (NPU) that serves as a compute engine within the graphics processing system to perform neural frame generation, producing extrapolated frames to fill in frames that would otherwise be missing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the NPU-based second circuitry taught by Croxford into the modified GPU system of Park, such that the combined system includes second circuitry comprising a compute engine associated with the processing resource to perform neural frame generation for the frame when the GPU risks missing its deadline. The motivation is to reduce distortions and artefacts while providing significant savings in terms of memory bandwidth, processing resources, and power.
Regarding Claim 8, it recites limitations similar in scope to the limitations of Claim 1 but as a method and the combination of Park and Croxford teaches all the limitations as of Claim 1. Therefore is rejected under the same rationale.
Regarding Claim 9, it recites limitations similar in scope to the limitations of Claim 2 and therefore is rejected under the same rationale.
Regarding Claim 15, it recites limitations similar in scope to the limitations of claim 1, but in a graphics processing system. As shown in the rejection, the combination of Park and Croxford disclose the limitations of claims 1. Additionally, Park discloses an system that maps to Fig. 1 and Paragraph [0090]-[0091] (Park, Fig. 1, Element 100 System, Element 140 Storage System, Element 180 GPU; Paragraph [0018], “System 100 includes communication interface 120, general processing system 130, storage system 140, display 160, and graphics processing unit (GPU) 180. General processing system 130 is operatively coupled to storage system 140. Storage system 140 stores software 150 and data 170. Software 150 may include operating system (OS) 151 and dynamic clock and voltage scaling software 152. General processing system 130 is operatively coupled to communication interface 120 and GPU 180”). Thus, Claim 15 is met by Park according to the mapping presented in the rejection of claims 1, given the graphics processor components corresponds to the graphics processing system.
Regarding Claim 16, the combination of Park and Croxford teaches the invention in Claim 15.
The combination further teaches second circuitry to track progress for submitted workloads for the frame (Park, Paragraph [0024], "frequency manager 181 uses information from frame timing 185 to monitor intra-frame timing and activity to control the operating frequency of GPU 180 for each frame"), determine if the frame will meet the target display update deadline (Park, Paragraph [0028], "frequency manager 181 compares the estimated workload i.e., estimated rendering time and the remaining time before the next rendering deadline e.g., V sync. This comparison detects early and/or late draw calls"; "If the predicted remaining rendering time is larger than the remaining time to the next rendering deadline, frequency manager 181 calculates and applies a new frequency level").
Park does not explicitly disclose but Croxford teaches in response to a determination that the frame will not meet the target display update deadline, signal the first circuitry and initiate a request for neural frame generation for the frame (Croxford, Paragraph [0149], "In an embodiment, the neural network is executed by a neural network processing unit (processor) (NPU) of the graphics processing system" <read on initiate a request for neural frame generation for the frame>; [0301], " As discussed above, in embodiments, the “spacewarp” extrapolation (interpolation) process is performed using a neural network executed by the neural network processing unit (NPU) 3 of the graphics processing system " <read on initiate a request for neural frame generation for the frame>; [0152], " the graphics processing system in an embodiment comprises a neural network processing unit (processor) (NPU) comprising the neural network circuit (and a graphics processing unit (processor) (GPU) comprising the rendering circuit) ").
Croxford and Park are analogous since both relate to GPU-based graphics processing systems that render frames for output to a display under strict timing constraints, including VR/AR/MR use cases, and both address the need to manage per-frame GPU workloads within a deadline. Park provided a way of tracking per-frame rendering progress and comparing estimated remaining rendering time to the remaining time until the Vsync deadline. Croxford provided a way of implementing second circuitry that determines at the end of each allocated time slice whether all submitted commands have completed, signals a halt to the first circuitry when unfinished commands remain (i.e., the frame will miss the deadline), and initiates an NPU-based neural frame generation request as an alternative. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Croxford's time-slice progress tracking and neural-fallback signaling into Park's frame-deadline GPU, such that second circuitry tracks progress for submitted workloads, determines whether the frame will meet the target display update deadline, and in response to a determination that it will not, signals the first circuitry and initiates a request for neural frame generation, ensuring a displayable frame is produced at the deadline. The motivation is to guarantee each application frequent and predictable access to the GPU and to avoid missed frame deadlines by switching to neural frame generation when the GPU workload will not complete in time.
Regarding Claim 17, the combination of Park and Croxford teaches the invention in
Claim 16.
The combination further teaches third circuitry to perform neural frame generation for
the frame (Croxford, Paragraph [0301], "the "spacewarp" extrapolation
(interpolation) process is performed using a neural network executed by the
neural network processing unit (NPU) 3 of the graphics processing system <read
on third circuitry to perform neural frame generation for the frame>"; [0142]-[0143], "the graphics processing system ... comprises ... a rendering circuit configured to generate a rendered frame; and [0144] a neural network circuit configured to generate an output frame from a rendered frame generated by the rendering circuit by using a neural network ... to extrapolate motion of one or more objects in the rendered frame <read on third circuitry to perform neural frame generation for the frame>."), the third circuitry including a compute engine associated with the processing resource (Croxford, Paragraph [0149], "In an embodiment, the neural network is executed by a neural network processing unit (processor) (NPU) of the graphics processing system <read on compute engine associated with the processing resource>."; Paragraph [0151], "the method comprises: (a graphics processing unit) generating the rendered frame; and a neural network processing unit of the graphics processing system generating the output (e.g. extrapolated) frame <read on compute engine associated with the processing resource>.").
Croxford and Park are analogous since both relate to graphics processing systems that render image frames for output to a display, including VR/AR/MR use cases, and both
are directed to solving problems of maintaining acceptable frame output under resource
and timing constraints. Park provided a way of managing per-frame GPU rendering
deadlines by using on-chip circuitry (frequency manager and frame timing) to estimate
remaining rendering time and adjust GPU operating frequency so frames complete
before a Vsync-based deadline. Croxford provided a way of supplementing GPU rendering with a neural network processing unit (NPU) that serves as a compute engine within the graphics processing system to perform neural frame generation (spacewarp/extrapolated frames) from GPU-rendered frames. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the NPU-based third circuitry taught by Croxford into the modified GPU system of Park, such that the combined system includes third circuitry comprising a compute engine (NPU) associated with the GPU processing resource to perform neural frame generation for the frame. The motivation is to offload neural-network-based frame generation from the GPU to a dedicated NPU in order to reduce artefacts and improve processing efficiency while more fully utilizing available processing resources in the graphics processing system.
Claim 3-7, 10-14, 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Park et al. (US 20180300838 A1, hereinafter Park) in view of Croxford et al. (US 20200410740 A1, hereinafter Croxford) as applied to Claim 1, 8, 15 above respectively and further in view of Reda et al. (US 20190297326 A1, hereinafter Reda).
Regarding Claim 3, the combination of Park and Croxford teaches the invention in Claim 1.
The combination does not explicitly disclose but Reda teaches the processing resource including a matrix accelerator to execute matrix multiply operations on behalf of the compute engine (Reda, Paragraph [0079]-[0080], "the cores 550 include 64 single-precision (32-bit) floating point cores, 64 integer cores, 32 double-precision (64-bit) floating point cores, and 8 tensor cores. Tensor cores [are] configured to perform matrix operations <read on matrix accelerator to execute matrix multiply operations> on behalf of the compute engine."; "the tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and inferencing”; [0081], “Tensor Cores operate on 16-bit floating point input data with 32-bit floating point accumulation. The 16-bit floating point multiply requires 64 operations and results in a full precision product that is then accumulated using 32-bit floating point addition with the other intermediate products for a 4×4×4 matrix multiply <read on matrix accelerator to execute matrix multiply operations on behalf of the compute engine>.").
Reda and Park are analogous since both relate to parallel processing units (PPUs/GPUs) with the same SM/tensor core architecture used for executing both graphics rendering pipelines and deep learning neural network operations, and both are directed to efficient neural network inference on GPU hardware. Park provided a way of managing frame rendering deadlines using the GPU's processing resources. Reda provided a way of executing video prediction neural networks on the same PPU by leveraging tensor cores within the SMs as dedicated matrix accelerators to execute the matrix multiply operations required by the neural network on behalf of its compute pipeline. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Reda's tensor core matrix accelerator teaching into the processing resource of the Park-Croxford combined system, such that the processing resource includes a matrix accelerator (tensor cores) to execute matrix multiply operations on behalf of the compute engine of the second circuitry, improving throughput and efficiency of neural frame generation. The motivation is to accelerate deep learning matrix arithmetic for neural network inferencing using dedicated matrix-multiply hardware.
Regarding Claim 4, the combination of Park, Croxford, and Reda teaches the invention in Claim 3.
The combination further teaches the second circuitry configured to perform operations associated with a temporally aware machine learning model via the compute engine to perform the neural frame generation for the frame (Reda, Paragraph [0039], "the video prediction system 200 includes first neural network 210, a second neural network 220, and a spatially-displaced convolution (SDC) module 230 <read on temporally aware machine learning model>. The video prediction system 200 receives a sequence of video frames 202 I1−t(x,y) as an input and generates a next predicted video frame 232 It+1(x,y) in the sequence <read on perform operations associated with a temporally aware machine learning model via the compute engine to perform the neural frame generation for the frame>."), the temporally aware machine learning model trained to estimate optical flow at a target timestamp (Reda, Paragraph [0040], "the first neural network 210 receives the sequence of video frames 202 and generates a plurality of optical flows 212 corresponding to pairs of video frames in the sequence <read on temporally aware machine learning model trained to estimate optical flow>."; Paragraph [0125], "the input 802 for each frame is provided in 3 dimensions: x-coordinate and y-coordinate in pixel space and a t-coordinate to identify different frames in the sequence <read on at a target timestamp>.") wherein the target timestamp indicates a time at which a generated frame is to be displayed relative to an input frame (Reda Paragraph [0039], "generates a next predicted video frame 232 It+1(x, y) in the sequence, where t is equal to the number of video frames in the sequence"; Paragraph [0041]: "the first neural network 210 is configured to receive five video frames [It(x, y), It−1(x, y), It−2(x, y), It−3(x, y), It−4(x, y)] in the sequence of video frames 202.<read on input frames>. The first neural network 210 then generates four optical flows [Ft(x, y), Ft−1(x, y), Ft−2(x, y), Ft−3(x, y)] <read on optical flow between the plurality of input frames used by the temporally aware model to estimate the flow at the target timestamp t+1 relative to the input frame at t>"];
Reda and Park are analogous since both relate to GPU/PPU-based processing systems that execute neural networks to generate video frames for display, and both address the problem of generating high-quality video frames efficiently using deep learning on parallel processing hardware. Park provided a way of managing frame rendering deadlines and GPU workloads using Vsync-based timing. Reda provided a way of implementing a temporally aware machine learning model on a PPU-based compute engine that uses a 3D convolutional neural network to estimate optical flow at a target timestamp, conditioned on both spatial (x, y) and temporal (t) dimensions of the input frames, for performing neural frame generation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the second circuitry of the Park-Croxford combined system using Reda's temporally aware machine learning model via the compute engine, such that the model is trained to estimate optical flow at a target timestamp in order to perform neural frame generation for the frame. The motivation is to accurately capture optical flow of objects at a specific target time defined relative to the last rendered input frame to enable high-quality predicted frame generation for display at the corresponding display interval
Regarding Claim 5, the combination of Park, Croxford, and Reda teaches the invention in Claim 4.
The combination further teaches the temporally aware machine learning model trained to estimate the optical flow at the target timestamp based on a plurality of input frames (Reda, Paragraph [0041], "The first neural network 210 is configured to receive five video frames It(x,y), It−1(x,y), It−2(x,y), It−3(x,y), It−4(x,y) in the sequence of video frames 202 <read on a plurality of input frames>."), render timestamps associated with the plurality of input frames (Reda, Paragraph [0125], "the input 802 for each frame is provided in 3 dimensions: x-coordinate and y-coordinate in pixel space and a t-coordinate to identify different frames in the sequence <read on render timestamps associated with the plurality of input frames>."), and optical flow between the plurality of input frames (Reda, Paragraph [0041], "The first neural network 210 then generates four optical flows Ft(x,y), Ft−1(x,y), Ft−2(x,y), Ft−3(x,y) <read on optical flow between the plurality of input frames>."; Paragraph , "Each optical flow Fj(x,y) maps each pixel p having pixel coordinates (x,y) in video frame Ij to a motion vector (u,v) that represents a displacement for that pixel to a corresponding pixel p̊ having pixel coordinates (x+u, y+v) in video frame Ij+1 <read on optical flow between the plurality of input frames>.").
Reda and Park are analogous as discussed above for Claim 4. Park provided a way of tracking per-frame render timing history across prior frames to estimate remaining rendering time for the current frame. Reda provided a way of training the temporally aware machine learning model on a plurality of prior video frames together with their temporal indices (render timestamps) and the pairwise optical flows between those frames, such that the model learns to estimate optical flow at the target timestamp from those multi-frame temporal inputs. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to train the temporally aware machine learning model of the Park-Croxford-Reda combined system using a plurality of input frames together with their render timestamps and the optical flows between them, as taught by Reda, such that the model can accurately estimate optical flow at the target timestamp from temporally sequenced frame and flow inputs. The motivation is to improve the accuracy of optical flow estimation and predicted frame generation by conditioning on multiple prior frames, their time indices, and inter-frame optical flows.
Regarding Claim 6, the combination of Park, Croxford, and Reda teaches the invention in Claim 5
The combination further teaches the second circuitry to: estimate optical flow at the target timestamp (Reda, Paragraph [0039], "the video prediction system 200 receives a sequence of video frames 202 . . . as an input and generates a next predicted video frame 232 It+1(x,y) in the sequence, where t is equal to the number of video frames in the sequence <read on estimate optical flow at the target timestamp>."; [0040], "The first neural network 210 receives the sequence of video frames 202 and generates a plurality of optical flows 212 corresponding to pairs of video frames in the sequence <read on estimate optical flow at the target timestamp>.") and warp a previously rendered frame based on the optical flow estimated at the target timestamp (Reda, Paragraph [0044], "The SDC module 230 generates each pixel in the predicted video frame 232 by sampling a spatially-displaced patch of pixels from the previous video frame It(x,y) 224, defined by the predicted displacement vector for the pixel <read on warp a previously rendered frame based on the optical flow estimated at the target timestamp>."; [0115], "the displacement vector 632 corresponds to a prediction near the edge of objects proximate the occluded pixel in the previous video frame, estimated from the pure backwards optical flow as well as the sequence of image frames <read on warp a previously rendered frame based on the optical flow estimated at the target timestamp>.").
Reda and Park are analogous as discussed above for Claim 4. Park provided a way of rendering successive frames under Vsync-based deadline control and tracking per-frame timing. Reda provided a way of estimating optical flow at the target prediction timestamp and then warping the previously rendered frame according to the estimated displacement vectors (optical flow) at that timestamp, using a spatially-displaced convolution module (SDC) to sample pixels from the previous frame shifted according to the predicted displacement, thereby generating the predicted next frame. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the second circuitry of the Park-Croxford combined system to estimate optical flow at the target display timestamp and warp the previously rendered frame based on that estimated optical flow, as taught by Reda's SDC-based video prediction pipeline, so that neural frame generation produces motion-consistent frames at the target display deadline. The motivation is to generate high-quality predicted video frames by sampling from a previously rendered frame using displacement vectors estimated from optical flow at the target time, avoiding artifacts from occlusions and dis-occlusions.
Regarding Claim 7, the combination of Park, Croxford, and Reda teaches the invention in Claim 6.
The combination further teaches that the optical flow estimated at the target timestamp is an extrapolated optical flow (Reda, [0003], "Video prediction is the task of inferring future frames of video in a video sequence based on past frames of video in the video sequence <read on extrapolated optical flow>."; [0006], "the deep learning neural network model estimates a displacement vector and a convolution kernel for each pixel of the predicted video frame <read on extrapolated optical flow>, . . . the previous video frame is a last video frame in the sequence of video frames that will immediately precede the predicted video frame in the updated sequence of video frames <read on extrapolated optical flow estimated at the target timestamp for a future frame>.") and the previously rendered frame is warped to extrapolate a generated frame (Reda, Paragraph [0039], "the video prediction system 200 receives a sequence of video frames 202 I1−t(x,y) as an input and generates a next predicted video frame 232 It+1(x,y) in the sequence, where t is equal to the number of video frames in the sequence <read on the previously rendered frame is warped to extrapolate a generated frame>."; [0044], "The SDC module 230 generates each pixel in the predicted video frame 232 by sampling a spatially-displaced patch of pixels from the previous video frame It(x,y) 224, defined by the predicted displacement vector for the pixel <read on the previously rendered frame is warped to extrapolate a generated frame>.").
Reda and Park are analogous as discussed above for Claim 4. Park provided a way of detecting when a frame will miss the next Vsync deadline, meaning the system needs to produce a frame at a future display time. Reda provided a way of extrapolating a future video frame by estimating optical flow at the target future timestamp based on past frames and flows, then warping the last previously rendered frame according to those extrapolated displacement vectors to generate the predicted future frame. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to configure the second circuitry of the Park-Croxford combined system to use Reda's extrapolated optical flow approach, such that the optical flow estimated at the target (future) display timestamp is an extrapolated optical flow and the previously rendered frame is warped according to that extrapolated optical flow to generate an extrapolated future frame at the upcoming Vsync deadline. The motivation is to generate a predicted future video frame by extrapolating motion beyond the last observed frame, enabling the display of a smooth, forward-looking generated frame when the GPU rendering pipeline will not complete in time.
Regarding Claim 10, it recites limitations similar in scope to the limitations of Claim 3 and therefore is rejected under the same rationale.
Regarding Claim 11, it recites limitations similar in scope to the limitations of Claim 4 and therefore is rejected under the same rationale.
Regarding Claim 12, it recites limitations similar in scope to the limitations of Claim 5 and therefore is rejected under the same rationale.
Regarding Claim 13, it recites limitations similar in scope to the limitations of Claim 6 and therefore is rejected under the same rationale.
Regarding Claim 14, it recites limitations similar in scope to the limitations of Claim 7 and therefore is rejected under the same rationale.
Regarding Claim 18, it recites limitations similar in scope to the limitations of Claim 3 and therefore is rejected under the same rationale.
Regarding Claim 19, it recites limitations similar in scope to the limitations of Claim 4 and therefore is rejected under the same rationale.
Regarding Claim 20, it recites limitations similar in scope to the limitations of Claim 5 and therefore is rejected under the same rationale.
Response to Arguments
Applicant's arguments with respect to claims 1-20, filed on 07/14/2026, with respect to rejection under 35 U.S.C. § 103 have been fully considered but are not persuasive.
Regarding Claim 1, 8, applicant asserts that (a) Park only responds to a predicted late frame by increasing GPU frequency and does not request neural frame generation, (b) Croxford's disclosure of neural frame generation is not tied to a determination that an in-progress rendered frame will not meet a target display update deadline, and (c) the proposed combination changes the operating principle of Park.
In response to the argument, as described in the rejection of Claim 1 above, the disputed limitation is taught by the combination of Park and Croxford, not by either reference in isolation. The rejection expressly acknowledges that "Park does not explicitly disclose request neural frame generation for the frame in response to a determination that the frame will not meet the target display update deadline." That is precisely why Croxford is combined with Park. Park in Paragraph [0028] and [0033] provides the "determination that the frame will not meet the target display update deadline" limitation by comparing the predicted remaining rendering time with the remaining time before the next Vsync deadline and, when the predicted remaining rendering time is larger than the remaining time to the deadline, recognizing that the rendering deadline is at risk of being missed. Croxford in turn provides the "request neural frame generation" limitation. In particular, Croxford in Paragraph [0016]-[0019] expressly teaches that "timewarp" processing (which Croxford at Paragraphs [0138] and [0147]-[0149] explicitly implements using a neural network executed by an NPU) "can be performed at a faster rate, such as 90 or 120 frames per second, than the graphics processing unit (GPU) 4 may be able to render frames at, such as 30 frames per second," and is used to "provide frames for display that have been updated based on a sensed head orientation (pose) at a faster rate than would otherwise be possible without the use of 'timewarp' processing," precisely to avoid "motion sickness caused by a low refresh rate and 'judder' artefacts." Croxford further teaches at Paragraph [0221] and FIG. 3 that "timewarp" processing 31 is performed "at successive intervals whilst waiting for a new application frame to be rendered," i.e., is performed when the GPU cannot itself supply a new rendered frame at the display refresh interval. Croxford Paragraph [0063] additionally states the motivation for using neural-network-based frame generation: to "reduce distortions and artefacts, while providing significant savings in terms of memory bandwidth, processing resources and power, etc., when performing so-called spacewarp processing." Croxford therefore expressly teaches generating a frame via neural network processing when the GPU cannot deliver a rendered frame at the display refresh interval, which is the very "in response to a determination that the frame will not meet the target display update deadline" condition detected by Park in Paragraph [0028].
Applicant's argument that the combination "changes the operating principle" of Park is misplaced. Park's operating principle is to ensure a displayable frame is available at each Vsync deadline; Park itself is not limited to a single response mechanism, as Park in Paragraph [0028] describes both increasing frequency (when a miss is predicted) and decreasing frequency (when a miss is not predicted) responses to the deadline comparison. Adding Croxford's neural-network-based extrapolated frame generation as a further response when the deadline is predicted to be missed does not remove Park's frequency-scaling response; it supplements Park's deadline-management framework with an additional, known technique for producing a displayable frame when GPU rendering will not complete in time. Under KSR, the combination of Park's deadline determination with Croxford's known neural-network-based extrapolated frame generation, each performing its known function, yields the predictable result of requesting neural frame generation when the frame will not meet the display deadline, and one of ordinary skill in the art would have been motivated to make that combination for the reasons expressly stated in Croxford Paragraph [0063] (reducing artefacts and saving processing resources) and Croxford Paragraphs [0016]-[0017] (providing frames at the display refresh rate when the GPU cannot). Applicant's characterization that the rejection uses the claims as a roadmap is therefore incorrect; the rationale is drawn from the express teachings of the cited references themselves. Hence the combination of Park and Croxford fully teaches the limitations of Claim 1. Therefore applicant's remarks cannot be considered persuasive.
Regarding Claim 15, applicant asserts that Park's response to the deadline determination is to raise GPU frequency and that Croxford's disclosure of an extrapolated output frame is not tied to a determination that workload execution for an in-progress frame will miss the target display update deadline. This argument is not persuasive for the same reasons discussed above with respect to Claim 1. In particular, Croxford at Paragraphs [0016]-[0019], [0138], [0147]-[0149], and [0221] expressly teaches performing neural-network-based timewarp/spacewarp processing to provide output frames for display at the display refresh interval when the GPU cannot render a new application frame within that interval, and Croxford Paragraph [0179] states that the "output (e.g. transformed and/or extrapolated) frame is in an embodiment a frame generated for display." When combined with Park's deadline determination in Paragraph [0028], the combination teaches "display a frame created via neural frame generation" "in response to a determination that workload execution for the frame will not meet the target display update deadline," for the same reasons articulated in the rejection of Claim 1.
Regarding Claim 16, applicant argues that Croxford Paragraphs [0149], [0152], and [0301] "identify an NPU and its neural processing" but do not disclose the claimed second circuitry that signals the first circuitry and initiates a neural frame generation request in response to the missed-deadline determination. This argument is not persuasive. Croxford at Paragraph [0152] expressly teaches a graphics processing system that "comprises a neural network processing unit (processor) (NPU) comprising the neural network circuit (and a graphics processing unit (processor) (GPU) comprising the rendering circuit)," and Croxford at Paragraph [0151] teaches that the "method comprises: (a graphics processing unit) generating the rendered frame; and a neural network processing unit of the graphics processing system generating the output (e.g. extrapolated) frame." These paragraphs establish that Croxford's graphics processing system includes a separate NPU (the claimed "second circuitry") that generates output frames via a neural network, distinct from the GPU (the claimed "first circuitry") that generates rendered frames. Croxford at Paragraphs [0016]-[0019] and [0221] further establishes that this NPU-based frame generation occurs at successive display intervals "whilst waiting for a new application frame to be rendered," i.e., is initiated when the GPU cannot supply the frame within the display refresh interval. Under the combination with Park, which supplies the tracking-progress and deadline-determination logic (Park Paragraphs [0024], [0028], and [0033]), the "second circuitry" of Croxford (the NPU) is invoked in response to the missed-deadline determination made by Park's frequency manager. Because the NPU necessarily receives an instruction to begin generating the output frame and the GPU is not to continue rendering that frame, the combined system inherently "signal[s] the first circuitry and initiate[s] a request for neural frame generation for the frame" as claimed. The motivation for the combination is stated in Croxford Paragraph [0063] and in the rejection of Claim 16 above (to guarantee a displayable frame is produced at each deadline and to reduce artefacts while saving processing resources).
In regard to Claims 2-7, 9-14, 17-20, they directly/indirectly depends on independent Claim 1, 8, 15 respectively. Applicant does not argue anything other than the independent claim 1, 8, 15. The limitations in those claims in conjunction with combination previously established as explained.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20150277686 A1 Systems and Methods for the Real-Time Modification of Videos and Images Within a Social Network Format
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YUJANG TSWEI whose telephone number is (571)272-6669. The examiner can normally be reached 8:30am-5:30pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YuJang Tswei/Primary Examiner, Art Unit 2614
Prosecution Insights