DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-20 are pending under this Office action.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 8/11/2025 has been entered.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4, 7-8, 11, 14-15 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Lobete, etc. (US 20210400170 A1) in view of Saulters (US 20140092109 A1), further in view of Schluessler, etc. (US 20220122566 A1), Masuda, etc. (US 20120147013 A1) and Campbell (US 20160042488 A1).
Regarding claim. 1, Lobete teaches that a processing system (See Lobete: Figs. 6-7 and 13, and [0083], “Referring to FIG. 6, there is shown a device 60 according to the first aspect of the disclosure. The device 60 comprises a display 61 which displays content at a display refresh rate. The display 61 provides a vertical synchronisation signal, VSYNC, to a compositor 62. In response to receiving the synchronisation signal, the compositor 62 triggers access to frames of content from the buffers 64 and 66 so that the frames of content may be displayed on the display at each display refresh rate. A first of the buffers 64 stores the most recent frames of content rendered by a game engine 65. A second of the buffers 66 stores the most recent frame of content rendered by other processes 67 of the device 60. The device 60 further comprises an interpolator 63 for interpolating between the rendered frames”) comprising:
an accelerator unit (AU) configured (See Lobete: Figs. 6-7 and 13, and [0123], “A method (or some operations of the method) according to various embodiments of the disclosure may be performed by at least one processor (e.g., a compositor 62, an interpolator 63), or by an electronic device (e.g., a device 60). An electronic device (e.g., a device 60) according to various embodiments may be include at least one processor (e.g., a compositor 62, an interpolator 63) or a memory (e.g., a buffer 64). An electronic device (e.g., a device 60) according to various embodiments may be further include a display (e.g., a display 61)”. Note that the processor may be mapped to the accelerator, but a secondary art will be searched to address the term “accelerator”) to:
generate an interpolated frame based on a first rendered frame and a second rendered frame (See Lobete: Figs. 6-7 and 13, and [0086], “The interpolator 63 generates an interpolated rendered frame for display by performing an interpolation operation using the obtained two rendered frames. The interpolation operation takes into account the difference between the timestamps of each of the two rendered frames and the timestamp of the synchronisation signal, when the timestamps of the rendered frames delayed by the predetermined delay value. For example, if one of the frames is closer to the delayed timestamp of the synchronisation signal, then this frame may have a grater influence in the interpolated rendered frame”);
determine an interpolated frame timing (See Lobete: Figs. 6-7 and 13, and [0129], “According to various embodiments of the disclosure, a device for processing rendered frames for display at a display refresh rate, the device may comprise, a buffer arranged to store a plurality of rendered frames rendered at a frame rendering rate and a time stamp for each of the rendered frames, a compositor arranged to obtain a timestamp of a synchronisation signal for synchronising the display of frames with the display refresh rate, and in response to obtaining the timestamp of the synchronisation signal, trigger access to the buffer to obtain two rendered frames having timestamps closest to the timestamp of the synchronisation signal, and an interpolator arranged to generate an interpolated rendered frame for display by performing an interpolation operation using the obtained two rendered frames. The interpolation operation may take into account the difference between the timestamps of each of the two rendered frames and the timestamp of the synchronisation signal”; and [0133], “According to various embodiments, the compositor may be arranged to trigger access to the buffer to obtain two rendered frames having timestamps closest to the timestamp of the synchronisation signal, when the timestamps of the rendered frames are delayed by a predetermined delay value. The interpolation operation may take into account the difference between the timestamps of each of the two rendered frames and the timestamp of the synchronisation signal, when the timestamps of the rendered frames are delayed by the predetermined delay value”. Note that the timestamps of the rendered frames are the rendering metrics/timing information used to determine the precise interpolated -frame timing/position relative to display sync) based on one or more rendering metrics indicating timing information associated with the first rendered frame and the second rendered frame; and
provide (See Lobete: Figs. 8-9 and 14, and [0089], “The top half plot 73 of FIG. 8 shows the rendering of frames of content 1, 2, 3, 4, 5, 6, 7. The y-axis indicates the motion within the content, and the x-axis 70 indicates the time. The time at which the frames 1, 2, 3, 4, 5, 6, 7 are rendered are indicated by the dashes 77a, 77b, 77c, 77d, 77e, 77f on the time axis 70. The time taken to render individual frames is variable which means that there is an uneven time spacing between the rendered frames. The solid lines in the plots 73, 75 of FIG. 8 indicate the actual motion of the content, whereas the dashed lines indicate the motion of the content that would be apparent to the user if the frames were displayed”; [0090], “FIG. 8 shows that the rendered frames of content are each delayed by a predetermined delay value. In this example, the delay is achieved by adjusting the timestamps of the rendered frames such that they are shifted into the future. In other words, the predetermined delay value is added to each of the timestamps. The delayed timestamps are indicated by the dashes 78a, 78b, 78c, 78d in FIG. 8. Only the delay of frames 1, 2, 3, 4 and 5 are specifically shown in FIG. 8”; and [0091], “The bottom half plot 75 shows the display refresh rate indicated by the dashes 79a, 79b, 79c, 79d, 79e, 79f, 79g, 79h, 79i on the time axis 70. At each display refresh, an interpolated rendered frame 1.5, 2.5, 3.2, 3.7, 4.2, and 4.8. The interpolated frame 1.5 is an interpolation of rendered frames 1 and 2. The time of the display refresh (i.e. the timestamp of the synchronisation signal) is approximately equidistant between the rendered frames 1 and 2 so both of rendered frames 1 and 2 have an equal influence on the interpolated rendered frame 1.5. The interpolated frame 2.5 is an interpolation of rendered frames 2 and 3. The time of the display refresh is approximately equidistant between the rendered frames 2 and 3 so both of rendered frames 2 and 3 have an equal influence on the interpolated rendered frame 2.5. The interpolated frame 3.2 is an interpolation of rendered frames 3 and 4. The rendered frame 3 is closer to the time of the display refresh, and so has a greater influence on the interpolated rendered frame 3.2 than the rendered frame 4. The interpolated frame 4.2 is an interpolation of rendered frames 4 and 5. The rendered frame 4 is closer to the time of the display refresh, and so has a greater influence on the interpolated rendered frame than the rendered frame 5. The interpolated frame 4.8 is an interpolation of rendered frames 4 and 5. The rendered frame 5 is closer to the time of the display refresh, and so has a greater influence on the interpolated rendered frame than the rendered frame”) the interpolated frame to a display based on the interpolated frame timing.
However, Lobete fails to explicitly disclose that an accelerator unit (AU) configured to; based on one or more rendering metrics indicating timing information associated with the first rendered frame and the second rendered frame; and the interpolated frame to a display based on the interpolated frame timing.
However, Saulters teaches that an accelerator unit (AU) configured to (See Saulters: Fig. 1, and [0034], “The central processing unit 110 is configured to process the data and instructions. The memory 120 and the storage device 130 are configured to store the data and instructions, such as computer-readable program codes, data structure and program module; and, the memory 120 and the storage device 130 may be volatile or non-volatile, removable or non-removable computer-readable medium. The input device 140 is configured to input the data and instructions. The graphic processing unit 150 is configured to process the data and instructions from the central processing unit 110 and associated with the processing of graphics, images or videos, such as for rendering a frame. The communication bus 160 is configured for the communication of data or instructions”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Lobete to have an accelerator unit (AU) configured to as taught by Saulters in order to save the processing time and the power consumption for rendering the lighting information of the second frame (See Saulters: Fig. 2, and [0006], “In order to save the processing time and power consumption, in the prior art, an application program, such as a game program, is programmed to have the information of the previous frame reused while rendering frames. For example, for rendering a first frame and a second frame for display, only the lighting information (may be other information) of the first frame is rendered, and the lighting information of the second frame repetitively employs the lighting information of the first frame, such that the processing time and power consumption for rendering the lighting information of the second frame may be saved”). Lobete teaches a method and system that may process the video frames and generate interpolated frames based on the frame rendering metrics and the display refresh rate; while Saulters teaches a system and method that may perform an interpolation to generate the second frame according to the rendered first frame and the rendered third frame using the conventional high-performance hardware GPU (accelerator) for real-time frame generation and interpolation. Therefore, it is obvious for one of ordinary skill in the art to modify Lobete by Saulters to use the GPU (accelerator) for the frame generation and interpolation. The motivation to modify Lobete by Saulters is “Use of known technique to improve similar devices (methods, or products) in the same way”.
However, Lobete, modified by Saulters, fails to explicitly disclose that based on one or more rendering metrics indicating timing information associated with the first rendered frame and the second rendered frame; and the interpolated frame to a display based on the interpolated frame timing.
However, Schluessler teaches that based on one or more rendering metrics indicating timing information associated with the first rendered frame and the second rendered frame (See Schluessler: Fig. 5, and [0065], “Illustrated processing block 72 provides for determining measured timing data in response to a presentation request (e.g., present call) from an application, wherein the measured timing data is associated with one or more previous frames and the presentation request is associated with one or more subsequent frames. Block 74 determines scheduling times for the subsequent frame(s) based on the measured timing data. The scheduling times may include, for example, a simulation time, a rendering time, a driver (e.g., UMD, KMD) submission time, a hardware (e.g., GPU) submission time, a display time, etc., or any combination thereof. In an embodiment, block 76 controls a pacing of the subsequent frame(s) on a display in accordance with the scheduling times”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Lobete to have based on one or more rendering metrics indicating timing information associated with the first rendered frame and the second rendered frame as taught by Schluessler in order to improve experiences in three-dimensional (3D) applications in an effective manner, and allows a graphics processing unit (GPU) to perform frame pacing to improve experience of a user in the 3D application in an efficient manner (See Schluessler: Figs. 9A-B, and [0080], “This mode can be used in isolation, or in conjunction with other modes. In isolation, the benefits of this mode may be no better than existing implementations of frame rate limiting. When used in conjunction with one or more other modes, the method 90 provides an improved user experience over existing methods. A user specifies a manual frame rate limit at block 90h, where illustrated block 90a initializes the system and the application calls present at block 90b. The frame pacer updates records of measured timing data since the last present at block 90c and calculates scheduling times of one or more subsequent frames at block 90d. Additionally, the frame pacer may inject a return delay at block 90e if the delay is needed to force the application to start/simulate the next frame at the scheduled time. In an embodiment, the application, CPU and/or GPU perform frame rendering work at block 90f, where the frame pacer schedules the frame for display using the previously calculated display time at block 90g”). Lobete teaches a method and system that may process the video frames and generate interpolated frames based on the frame rendering metrics and the display refresh rate; while Schluessler teaches a system and method that may use the measured timing data to determine display and scheduling timing for later frames. Therefore, it is obvious for one of ordinary skill in the art to modify Lobete by Schluessler to determine an interpolated frame timing based on the rendering metrics. The motivation to modify Lobete by Schluessler is “Use of known technique to improve similar devices (methods, or products) in the same way”.
However, Lobete, modified by Saulters and Schluessler, fails to explicitly disclose that the interpolated frame to a display based on the interpolated frame timing.
However, Masuda teaches that the interpolated frame to a display (See Masuda: Fig. 14, and [0009], In FIG. 14, a timer 601 is used for acquiring time information upon completion of the rendering process for rendering one frame. A CPU 602 uses keyframe information stored in a keyframe information storage unit 603, to generate an interpolation frame corresponding to the time acquired from the timer 601, render the interpolation frame on a graphic memory 604, and display the interpolation frame on a display screen 605. In this manner, a keyframe is interpolated to achieve smooth animation display, even when realizing a long distance or a short distance using the same number of keyframes”; Figs. 7A-B, and [0065], “Suppose here that the rendering start time (1 second) that is retained by the keyframe information 22 having the rendering start time following the current rendering start time starts 0.57 seconds after the current rendering start time. In this case, because the rendering time period estimated by the rendering time period computing unit 16 is 0.59 seconds, the time period remaining until the rendering start time included in the keyframe information 22 exceeds 0.57 seconds. Therefore, in the case of FIG. 7B, the rendering determination unit 15 notifies the animation controller 13 of that the interpolation component cannot be rendered. As a result, the interpolation component 52 is not rendered, and the key component 42 based on the keyframe information 22 is subsequently rendered by the display controller 18”) based on the interpolated frame timing.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Lobete to have the interpolated frame to a display based on the interpolated frame timing as taught by Masuda in order to compute the drawing time easily from ratio of an update drawing pixel count and a unit drawing pixel count (See Masuda: Fig. 1, and [0109], “According to this configuration, the rendering time period can be calculated easily from a ratio between the number of update rendering pixels required to display the interpolation screen component on the display unit, and the unit number of rendering pixels, and from the unit rendering time period”). Lobete teaches a method and system that may process the video frames and generate interpolated frames based on the frame rendering metrics and the display refresh rate; while Masuda teaches a system and method that may tightly couple rendering-performance metrics directly to the interpolation component itself, i.e. estimating the time required to generate or display the interpolated content based on stored rendering performance information, then using that estimate to decide whether and when to render/display the interpolated frame. Therefore, it is obvious for one of ordinary skill in the art to modify Lobete by Masuda to generate or display the interpolated content based on stored rendering performance information. The motivation to modify Lobete by Masuda is “Use of known technique to improve similar devices (methods, or products) in the same way”.
However, Lobete, modified by Saulters, Schluessler and Masuda, fails to explicitly disclose that the interpolated frame to a display based on the interpolated frame timing.
However, Campbell teaches that the interpolated frame to a display based on the interpolated frame timing (See Campbell: Fig. 1, and [0024], “Described herein is a method and system for frame pacing. In general, an estimate is made as to how long it takes to render a frame. This may be done by measuring how long it takes for a graphics processing unit (GPU) to render the frame. An average over several recent frames is used to smooth out differences in workload from frame to frame and render speed of the GPUs. A heartbeat is created that controls the progress of the GPUs and smooths out their presents. The determined appropriate amount of time is waited in the driver, (for example, the kernel mode driver (KMD)), so that the frames are evenly spaced. Frame pacing essentially postpones the flipping of a frame in one GPU that may come too early with respect to another GPU”; Figs. 4-5, and [0034], “Referring now to FIGS. 4 and 5, an estimate is made as to how long it takes to render a frame (505). This may be done by measuring how long it takes for the GPUs 410 and 415 to render the frame. For example, timestamp queries may be used to measure how long it takes for the GPUs to render the frames. An average over several recent frames is used to smooth out differences in workload from frame to frame and render speeds of the GPUs (510). A heartbeat is created that controls the progress of the GPUs and smooths out their presents (515), where a heartbeat is a pulse or steady ticking of when frames should be presented. The determined appropriate amount of time is waited in the kernel mode driver (KMD)) so that the frames are evenly spaced (520)”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Lobete to have the interpolated frame to a display based on the interpolated frame timing as taught by Campbell in order to enable the delay to effectively provide a minimum amount of time after which a GPU can present the frame (See Campbell: Figs. 6-7, and [0035], “As described and shown herein below, the Delay effectively provides a minimum amount of time after which a GPU can present. That is, if the rendering process is complete prior to the running of the Delay, then the GPU presents after the running of the Delay. A signal is sent by the dummy engine to GPU 0 (615). This is shown as “S” in FIG. 7. GPU 0 waits the requisite delay time (620). This is shown as “W” in FIG. 7. GPU 0 presents after the requisite delay time (625). This is shown by the “P” in FIG. 7. The sequence is then repeated for GPU 1. In particular, a render command is sent to GPU 1 (630). The UMD submits a delay request to a dummy engine in the KMD (635). A signal is sent by the dummy engine to GPU 1 (640). GPU 1 waits the requisite delay time (645). GPU 1 presents after the requisite delay time (650) is over and the rendering process is complete. That is, the present can be no earlier than the delay period and only if the rendering process is also complete. The sequence is then repeated for GPU 0 and GPU 1”). Lobete teaches a method and system that may process the video frames and generate interpolated frames based on the frame rendering metrics and the display refresh rate; while Campbell teaches a system and method that may measure the GPU rendering time and use this measured time to control display and spacing delays so the frames are evenly spaced. Therefore, it is obvious for one of ordinary skill in the art to modify Lobete by Campbell to control the display spacing based on the measured GPU rendering time. The motivation to modify Lobete by Campbell is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 4, Lobete, Saulters, Schluessler, Masuda and Campbell teach all the features with respect to claim 1 as outlined above. Further, Campbell teaches that the processing system of claim 1, wherein the one or more rendering metrics include a rendering time of the first rendered frame and a rendering time of the second rendered frame (See Campbell: Figs. 4-5, and [0034], “Referring now to FIGS. 4 and 5, an estimate is made as to how long it takes to render a frame (505). This may be done by measuring how long it takes for the GPUs 410 and 415 to render the frame. For example, timestamp queries may be used to measure how long it takes for the GPUs to render the frames. An average over several recent frames is used to smooth out differences in workload from frame to frame and render speeds of the GPUs (510). A heartbeat is created that controls the progress of the GPUs and smooths out their presents (515), where a heartbeat is a pulse or steady ticking of when frames should be presented. The determined appropriate amount of time is waited in the kernel mode driver (KMD)) so that the frames are evenly spaced (520). For example, a dummy schedulable engine is created in the KMD. A user mode driver (UMD) submits dummy command buffers to this dummy engine to request a delay, (which may be 90-95% of expected frame time). The KMD reports the command buffer as complete when the requested delay has passed (525). In the event that the rendering process takes longer than the delay, then the present will be done as soon as the rendering process is complete. In effect, the delay is a minimum wait time for a GPU to present a frame. The UMD submits signals of a Microsoft® (MS) synchronization object to the dummy engine. The UMD waits on this synchronization object on the regular 3D engines”).
Regarding claim 7, Lobete, Saulters, Schluessler, Masuda and Campbell teach all the features with respect to claim 1 as outlined above. Further, Lobete teaches that the processing system of claim 1, wherein the AU is configured to:
determine a delay in presenting the interpolated frame based on the one or more rendering metrics (See Lobete: Figs. 1-3, and [0051], “In an example online processing operation, the compositor may be arranged to trigger access to the buffer to obtain two rendered frames having timestamps closest to the timestamp of the synchronisation signal, when the timestamps of the rendered frames are delayed by a predetermined delay value. The interpolation operation may take into account the difference between the timestamps of each of the two rendered frames and the timestamp of the synchronisation signal, when the timestamps of the rendered frames are delayed by the predetermined delay value. Here, “when delayed by the predetermined delay value” does not mean that the timestamps of the rendered frames must be delayed by the predetermined delay value. The timestamps of the rendered frames may be unchanged and instead the predetermined delay value may be subtracted from the timestamp of the synchronisation signal so as to achieve an equivalent effect”; [0055], ‘The predetermined delay value may be between 3 and 160 milliseconds. The predetermined delay value may be between 8 and 160 milliseconds. The predetermined delay value may be between 16 and 160 milliseconds. The predetermined delay value may be between 3 and 160 milliseconds. The predetermined delay value may be between 3 and 120 milliseconds. The predetermined delay value may be between 3 and 80 milliseconds. The predetermined delay value may be between 30 and 50 milliseconds, and may be 40 milliseconds’; and [0053], “The predetermined delay value may be determined according to the frame rendering rate. In other words, the predetermined delay value may be determined according to the time taken to render one or more frames. The frame rendering rate may be an average frame rendering rate”); and
determine the interpolated frame timing based on the delay (See Lobete: Figs. 1-3, and [0051], “In an example online processing operation, the compositor may be arranged to trigger access to the buffer to obtain two rendered frames having timestamps closest to the timestamp of the synchronisation signal, when the timestamps of the rendered frames are delayed by a predetermined delay value. The interpolation operation may take into account the difference between the timestamps of each of the two rendered frames and the timestamp of the synchronisation signal, when the timestamps of the rendered frames are delayed by the predetermined delay value. Here, “when delayed by the predetermined delay value” does not mean that the timestamps of the rendered frames must be delayed by the predetermined delay value. The timestamps of the rendered frames may be unchanged and instead the predetermined delay value may be subtracted from the timestamp of the synchronisation signal so as to achieve an equivalent effect”).
Regarding claim 8, Lobete, Saulters, Schluessler, Masuda and Campbell teach all the features with respect to claim 1 as outlined above. Further, Lobete, Saulters, Schluessler, Masuda and Campbell teach that a method (See Lobete: Figs. 6-7 and 13, and [0083], “Referring to FIG. 6, there is shown a device 60 according to the first aspect of the disclosure. The device 60 comprises a display 61 which displays content at a display refresh rate. The display 61 provides a vertical synchronisation signal, VSYNC, to a compositor 62. In response to receiving the synchronisation signal, the compositor 62 triggers access to frames of content from the buffers 64 and 66 so that the frames of content may be displayed on the display at each display refresh rate. A first of the buffers 64 stores the most recent frames of content rendered by a game engine 65. A second of the buffers 66 stores the most recent frame of content rendered by other processes 67 of the device 60. The device 60 further comprises an interpolator 63 for interpolating between the rendered frames”), comprising:
generating an interpolated frame based on a first rendered frame and a second rendered frame ; (See Figs. 6-7 and 13, and [0086], “The interpolator 63 generates an interpolated rendered frame for display by performing an interpolation operation using the obtained two rendered frames. The interpolation operation takes into account the difference between the timestamps of each of the two rendered frames and the timestamp of the synchronisation signal, when the timestamps of the rendered frames delayed by the predetermined delay value. For example, if one of the frames is closer to the delayed timestamp of the synchronisation signal, then this frame may have a grater influence in the interpolated rendered frame”)
determining an interpolated frame timing (See Figs. 6-7 and 13, and [0129], “According to various embodiments of the disclosure, a device for processing rendered frames for display at a display refresh rate, the device may comprise, a buffer arranged to store a plurality of rendered frames rendered at a frame rendering rate and a time stamp for each of the rendered frames, a compositor arranged to obtain a timestamp of a synchronisation signal for synchronising the display of frames with the display refresh rate, and in response to obtaining the timestamp of the synchronisation signal, trigger access to the buffer to obtain two rendered frames having timestamps closest to the timestamp of the synchronisation signal, and an interpolator arranged to generate an interpolated rendered frame for display by performing an interpolation operation using the obtained two rendered frames. The interpolation operation may take into account the difference between the timestamps of each of the two rendered frames and the timestamp of the synchronisation signal”; and [0133], “According to various embodiments, the compositor may be arranged to trigger access to the buffer to obtain two rendered frames having timestamps closest to the timestamp of the synchronisation signal, when the timestamps of the rendered frames are delayed by a predetermined delay value. The interpolation operation may take into account the difference between the timestamps of each of the two rendered frames and the timestamp of the synchronisation signal, when the timestamps of the rendered frames are delayed by the predetermined delay value”. Note that the timestamps of the rendered frames are the rendering metrics/timing information used to determine the precise interpolated -frame timing/position relative to display sync) based on one or more rendering metrics indicating timing information associated with the first rendered frame and the second rendered frame (See Schluessler: Fig. 5, and [0065], “Illustrated processing block 72 provides for determining measured timing data in response to a presentation request (e.g., present call) from an application, wherein the measured timing data is associated with one or more previous frames and the presentation request is associated with one or more subsequent frames. Block 74 determines scheduling times for the subsequent frame(s) based on the measured timing data. The scheduling times may include, for example, a simulation time, a rendering time, a driver (e.g., UMD, KMD) submission time, a hardware (e.g., GPU) submission time, a display time, etc., or any combination thereof. In an embodiment, block 76 controls a pacing of the subsequent frame(s) on a display in accordance with the scheduling times”); and
providing (See Lobete: Figs. 8-9 and 14, and [0089], “The top half plot 73 of FIG. 8 shows the rendering of frames of content 1, 2, 3, 4, 5, 6, 7. The y-axis indicates the motion within the content, and the x-axis 70 indicates the time. The time at which the frames 1, 2, 3, 4, 5, 6, 7 are rendered are indicated by the dashes 77a, 77b, 77c, 77d, 77e, 77f on the time axis 70. The time taken to render individual frames is variable which means that there is an uneven time spacing between the rendered frames. The solid lines in the plots 73, 75 of FIG. 8 indicate the actual motion of the content, whereas the dashed lines indicate the motion of the content that would be apparent to the user if the frames were displayed”; [0090], “FIG. 8 shows that the rendered frames of content are each delayed by a predetermined delay value. In this example, the delay is achieved by adjusting the timestamps of the rendered frames such that they are shifted into the future. In other words, the predetermined delay value is added to each of the timestamps. The delayed timestamps are indicated by the dashes 78a, 78b, 78c, 78d in FIG. 8. Only the delay of frames 1, 2, 3, 4 and 5 are specifically shown in FIG. 8”; and [0091], “The bottom half plot 75 shows the display refresh rate indicated by the dashes 79a, 79b, 79c, 79d, 79e, 79f, 79g, 79h, 79i on the time axis 70. At each display refresh, an interpolated rendered frame 1.5, 2.5, 3.2, 3.7, 4.2, and 4.8. The interpolated frame 1.5 is an interpolation of rendered frames 1 and 2. The time of the display refresh (i.e. the timestamp of the synchronisation signal) is approximately equidistant between the rendered frames 1 and 2 so both of rendered frames 1 and 2 have an equal influence on the interpolated rendered frame 1.5. The interpolated frame 2.5 is an interpolation of rendered frames 2 and 3. The time of the display refresh is approximately equidistant between the rendered frames 2 and 3 so both of rendered frames 2 and 3 have an equal influence on the interpolated rendered frame 2.5. The interpolated frame 3.2 is an interpolation of rendered frames 3 and 4. The rendered frame 3 is closer to the time of the display refresh, and so has a greater influence on the interpolated rendered frame 3.2 than the rendered frame 4. The interpolated frame 4.2 is an interpolation of rendered frames 4 and 5. The rendered frame 4 is closer to the time of the display refresh, and so has a greater influence on the interpolated rendered frame than the rendered frame 5. The interpolated frame 4.8 is an interpolation of rendered frames 4 and 5. The rendered frame 5 is closer to the time of the display refresh, and so has a greater influence on the interpolated rendered frame than the rendered frame”) the interpolated frame to a display (See Masuda: Fig. 14, and [0009], In FIG. 14, a timer 601 is used for acquiring time information upon completion of the rendering process for rendering one frame. A CPU 602 uses keyframe information stored in a keyframe information storage unit 603, to generate an interpolation frame corresponding to the time acquired from the timer 601, render the interpolation frame on a graphic memory 604, and display the interpolation frame on a display screen 605. In this manner, a keyframe is interpolated to achieve smooth animation display, even when realizing a long distance or a short distance using the same number of keyframes”; Figs. 7A-B, and [0065], “Suppose here that the rendering start time (1 second) that is retained by the keyframe information 22 having the rendering start time following the current rendering start time starts 0.57 seconds after the current rendering start time. In this case, because the rendering time period estimated by the rendering time period computing unit 16 is 0.59 seconds, the time period remaining until the rendering start time included in the keyframe information 22 exceeds 0.57 seconds. Therefore, in the case of FIG. 7B, the rendering determination unit 15 notifies the animation controller 13 of that the interpolation component cannot be rendered. As a result, the interpolation component 52 is not rendered, and the key component 42 based on the keyframe information 22 is subsequently rendered by the display controller 18”) based on the interpolated frame timing (See Campbell: Fig. 1, and [0024], “Described herein is a method and system for frame pacing. In general, an estimate is made as to how long it takes to render a frame. This may be done by measuring how long it takes for a graphics processing unit (GPU) to render the frame. An average over several recent frames is used to smooth out differences in workload from frame to frame and render speed of the GPUs. A heartbeat is created that controls the progress of the GPUs and smooths out their presents. The determined appropriate amount of time is waited in the driver, (for example, the kernel mode driver (KMD)), so that the frames are evenly spaced. Frame pacing essentially postpones the flipping of a frame in one GPU that may come too early with respect to another GPU”; Figs. 4-5, and [0034], “Referring now to FIGS. 4 and 5, an estimate is made as to how long it takes to render a frame (505). This may be done by measuring how long it takes for the GPUs 410 and 415 to render the frame. For example, timestamp queries may be used to measure how long it takes for the GPUs to render the frames. An average over several recent frames is used to smooth out differences in workload from frame to frame and render speeds of the GPUs (510). A heartbeat is created that controls the progress of the GPUs and smooths out their presents (515), where a heartbeat is a pulse or steady ticking of when frames should be presented. The determined appropriate amount of time is waited in the kernel mode driver (KMD)) so that the frames are evenly spaced (520)”).
Regarding claim 11, Lobete, Saulters, Schluessler, Masuda and Campbell teach all the features with respect to claim 8 as outlined above. Further, Campbell teaches that the method of claim 8, wherein the one or more rendering metrics include a rendering time of the first rendered frame and a rendering time of the second rendered frame (See Campbell: Figs. 4-5, and [0034], “Referring now to FIGS. 4 and 5, an estimate is made as to how long it takes to render a frame (505). This may be done by measuring how long it takes for the GPUs 410 and 415 to render the frame. For example, timestamp queries may be used to measure how long it takes for the GPUs to render the frames. An average over several recent frames is used to smooth out differences in workload from frame to frame and render speeds of the GPUs (510). A heartbeat is created that controls the progress of the GPUs and smooths out their presents (515), where a heartbeat is a pulse or steady ticking of when frames should be presented. The determined appropriate amount of time is waited in the kernel mode driver (KMD)) so that the frames are evenly spaced (520). For example, a dummy schedulable engine is created in the KMD. A user mode driver (UMD) submits dummy command buffers to this dummy engine to request a delay, (which may be 90-95% of expected frame time). The KMD reports the command buffer as complete when the requested delay has passed (525). In the event that the rendering process takes longer than the delay, then the present will be done as soon as the rendering process is complete. In effect, the delay is a minimum wait time for a GPU to present a frame. The UMD submits signals of a Microsoft® (MS) synchronization object to the dummy engine. The UMD waits on this synchronization object on the regular 3D engines”).
Regarding claim 14, Lobete, Saulters, Schluessler, Masuda and Campbell teach all the features with respect to claim 8 as outlined above. Further, Lobete teaches that the method of claim 8, further comprising:
determining a delay in presenting the interpolated frame based on the one or more rendering metrics(See Lobete: Figs. 1-3, and [0051], “In an example online processing operation, the compositor may be arranged to trigger access to the buffer to obtain two rendered frames having timestamps closest to the timestamp of the synchronisation signal, when the timestamps of the rendered frames are delayed by a predetermined delay value. The interpolation operation may take into account the difference between the timestamps of each of the two rendered frames and the timestamp of the synchronisation signal, when the timestamps of the rendered frames are delayed by the predetermined delay value. Here, “when delayed by the predetermined delay value” does not mean that the timestamps of the rendered frames must be delayed by the predetermined delay value. The timestamps of the rendered frames may be unchanged and instead the predetermined delay value may be subtracted from the timestamp of the synchronisation signal so as to achieve an equivalent effect”; [0055], ‘The predetermined delay value may be between 3 and 160 milliseconds. The predetermined delay value may be between 8 and 160 milliseconds. The predetermined delay value may be between 16 and 160 milliseconds. The predetermined delay value may be between 3 and 160 milliseconds. The predetermined delay value may be between 3 and 120 milliseconds. The predetermined delay value may be between 3 and 80 milliseconds. The predetermined delay value may be between 30 and 50 milliseconds, and may be 40 milliseconds’; and [0053], “The predetermined delay value may be determined according to the frame rendering rate. In other words, the predetermined delay value may be determined according to the time taken to render one or more frames. The frame rendering rate may be an average frame rendering rate”); and
determining the interpolated frame timing based on the delay (See Lobete: Figs. 1-3, and [0051], “In an example online processing operation, the compositor may be arranged to trigger access to the buffer to obtain two rendered frames having timestamps closest to the timestamp of the synchronisation signal, when the timestamps of the rendered frames are delayed by a predetermined delay value. The interpolation operation may take into account the difference between the timestamps of each of the two rendered frames and the timestamp of the synchronisation signal, when the timestamps of the rendered frames are delayed by the predetermined delay value. Here, “when delayed by the predetermined delay value” does not mean that the timestamps of the rendered frames must be delayed by the predetermined delay value. The timestamps of the rendered frames may be unchanged and instead the predetermined delay value may be subtracted from the timestamp of the synchronisation signal so as to achieve an equivalent effect”).
Regarding claim 15, Lobete, Saulters, Schluessler, Masuda and Campbell teach all the features with respect to claim 1 as outlined above. Further, Lobete, Saulters, Schluessler, Masuda and Campbell teach that an accelerator unit (AU) (See Saulters: Fig. 1, and [0034], “The central processing unit 110 is configured to process the data and instructions. The memory 120 and the storage device 130 are configured to store the data and instructions, such as computer-readable program codes, data structure and program module; and, the memory 120 and the storage device 130 may be volatile or non-volatile, removable or non-removable computer-readable medium. The input device 140 is configured to input the data and instructions. The graphic processing unit 150 is configured to process the data and instructions from the central processing unit 110 and associated with the processing of graphics, images or videos, such as for rendering a frame. The communication bus 160 is configured for the communication of data or instructions”) comprising:
one or more processor cores (See Lobete: Figs. 6-7 and 13, and [0123], “A method (or some operations of the method) according to various embodiments of the disclosure may be performed by at least one processor (e.g., a compositor 62, an interpolator 63), or by an electronic device (e.g., a device 60). An electronic device (e.g., a device 60) according to various embodiments may be include at least one processor (e.g., a compositor 62, an interpolator 63) or a memory (e.g., a buffer 64). An electronic device (e.g., a device 60) according to various embodiments may be further include a display (e.g., a display 61)”. Note that the processor may be mapped to the accelerator, but a secondary art will be searched to address the term “accelerator”) configured to:
determine an interpolated frame timing (See Figs. 6-7 and 13, and [0129], “According to various embodiments of the disclosure, a device for processing rendered frames for display at a display refresh rate, the device may comprise, a buffer arranged to store a plurality of rendered frames rendered at a frame rendering rate and a time stamp for each of the rendered frames, a compositor arranged to obtain a timestamp of a synchronisation signal for synchronising the display of frames with the display refresh rate, and in response to obtaining the timestamp of the synchronisation signal, trigger access to the buffer to obtain two rendered frames having timestamps closest to the timestamp of the synchronisation signal, and an interpolator arranged to generate an interpolated rendered frame for display by performing an interpolation operation using the obtained two rendered frames. The interpolation operation may take into account the difference between the timestamps of each of the two rendered frames and the timestamp of the synchronisation signal”; and [0133], “According to various embodiments, the compositor may be arranged to trigger access to the buffer to obtain two rendered frames having timestamps closest to the timestamp of the synchronisation signal, when the timestamps of the rendered frames are delayed by a predetermined delay value. The interpolation operation may take into account the difference between the timestamps of each of the two rendered frames and the timestamp of the synchronisation signal, when the timestamps of the rendered frames are delayed by the predetermined delay value”. Note that the timestamps of the rendered frames are the rendering metrics/timing information used to determine the precise interpolated -frame timing/position relative to display sync) based on one or more rendering metrics indicating timing information associated with an interpolated frame (See Schluessler: Fig. 5, and [0065], “Illustrated processing block 72 provides for determining measured timing data in response to a presentation request (e.g., present call) from an application, wherein the measured timing data is associated with one or more previous frames and the presentation request is associated with one or more subsequent frames. Block 74 determines scheduling times for the subsequent frame(s) based on the measured timing data. The scheduling times may include, for example, a simulation time, a rendering time, a driver (e.g., UMD, KMD) submission time, a hardware (e.g., GPU) submission time, a display time, etc., or any combination thereof. In an embodiment, block 76 controls a pacing of the subsequent frame(s) on a display in accordance with the scheduling times”); and
provide (See Lobete: Figs. 8-9 and 14, and [0089], “The top half plot 73 of FIG. 8 shows the rendering of frames of content 1, 2, 3, 4, 5, 6, 7. The y-axis indicates the motion within the content, and the x-axis 70 indicates the time. The time at which the frames 1, 2, 3, 4, 5, 6, 7 are rendered are indicated by the dashes 77a, 77b, 77c, 77d, 77e, 77f on the time axis 70. The time taken to render individual frames is variable which means that there is an uneven time spacing between the rendered frames. The solid lines in the plots 73, 75 of FIG. 8 indicate the actual motion of the content, whereas the dashed lines indicate the motion of the content that would be apparent to the user if the frames were displayed”; [0090], “FIG. 8 shows that the rendered frames of content are each delayed by a predetermined delay value. In this example, the delay is achieved by adjusting the timestamps of the rendered frames such that they are shifted into the future. In other words, the predetermined delay value is added to each of the timestamps. The delayed timestamps are indicated by the dashes 78a, 78b, 78c, 78d in FIG. 8. Only the delay of frames 1, 2, 3, 4 and 5 are specifically shown in FIG. 8”; and [0091], “The bottom half plot 75 shows the display refresh rate indicated by the dashes 79a, 79b, 79c, 79d, 79e, 79f, 79g, 79h, 79i on the time axis 70. At each display refresh, an interpolated rendered frame 1.5, 2.5, 3.2, 3.7, 4.2, and 4.8. The interpolated frame 1.5 is an interpolation of rendered frames 1 and 2. The time of the display refresh (i.e. the timestamp of the synchronisation signal) is approximately equidistant between the rendered frames 1 and 2 so both of rendered frames 1 and 2 have an equal influence on the interpolated rendered frame 1.5. The interpolated frame 2.5 is an interpolation of rendered frames 2 and 3. The time of the display refresh is approximately equidistant between the rendered frames 2 and 3 so both of rendered frames 2 and 3 have an equal influence on the interpolated rendered frame 2.5. The interpolated frame 3.2 is an interpolation of rendered frames 3 and 4. The rendered frame 3 is closer to the time of the display refresh, and so has a greater influence on the interpolated rendered frame 3.2 than the rendered frame 4. The interpolated frame 4.2 is an interpolation of rendered frames 4 and 5. The rendered frame 4 is closer to the time of the display refresh, and so has a greater influence on the interpolated rendered frame than the rendered frame 5. The interpolated frame 4.8 is an interpolation of rendered frames 4 and 5. The rendered frame 5 is closer to the time of the display refresh, and so has a greater influence on the interpolated rendered frame than the rendered frame”) the interpolated frame to a display (See Masuda: Fig. 14, and [0009], In FIG. 14, a timer 601 is used for acquiring time information upon completion of the rendering process for rendering one frame. A CPU 602 uses keyframe information stored in a keyframe information storage unit 603, to generate an interpolation frame corresponding to the time acquired from the timer 601, render the interpolation frame on a graphic memory 604, and display the interpolation frame on a display screen 605. In this manner, a keyframe is interpolated to achieve smooth animation display, even when realizing a long distance or a short distance using the same number of keyframes”; Figs. 7A-B, and [0065], “Suppose here that the rendering start time (1 second) that is retained by the keyframe information 22 having the rendering start time following the current rendering start time starts 0.57 seconds after the current rendering start time. In this case, because the rendering time period estimated by the rendering time period computing unit 16 is 0.59 seconds, the time period remaining until the rendering start time included in the keyframe information 22 exceeds 0.57 seconds. Therefore, in the case of FIG. 7B, the rendering determination unit 15 notifies the animation controller 13 of that the interpolation component cannot be rendered. As a result, the interpolation component 52 is not rendered, and the key component 42 based on the keyframe information 22 is subsequently rendered by the display controller 18”) based on the interpolated frame timing (See Campbell: Fig. 1, and [0024], “Described herein is a method and system for frame pacing. In general, an estimate is made as to how long it takes to render a frame. This may be done by measuring how long it takes for a graphics processing unit (GPU) to render the frame. An average over several recent frames is used to smooth out differences in workload from frame to frame and render speed of the GPUs. A heartbeat is created that controls the progress of the GPUs and smooths out their presents. The determined appropriate amount of time is waited in the driver, (for example, the kernel mode driver (KMD)), so that the frames are evenly spaced. Frame pacing essentially postpones the flipping of a frame in one GPU that may come too early with respect to another GPU”; Figs. 4-5, and [0034], “Referring now to FIGS. 4 and 5, an estimate is made as to how long it takes to render a frame (505). This may be done by measuring how long it takes for the GPUs 410 and 415 to render the frame. For example, timestamp queries may be used to measure how long it takes for the GPUs to render the frames. An average over several recent frames is used to smooth out differences in workload from frame to frame and render speeds of the GPUs (510). A heartbeat is created that controls the progress of the GPUs and smooths out their presents (515), where a heartbeat is a pulse or steady ticking of when frames should be presented. The determined appropriate amount of time is waited in the kernel mode driver (KMD)) so that the frames are evenly spaced (520)”).
Regarding claim 18, Lobete, Saulters, Schluessler, Masuda and Campbell teach all the features with respect to claim 15 as outlined above. Further, Campbell teaches that the AU of claim 15, wherein the one or more rendering metrics include a generation time of the interpolated frame (See Campbell: Figs. 4-5, and [0034], “Referring now to FIGS. 4 and 5, an estimate is made as to how long it takes to render a frame (505). This may be done by measuring how long it takes for the GPUs 410 and 415 to render the frame. For example, timestamp queries may be used to measure how long it takes for the GPUs to render the frames. An average over several recent frames is used to smooth out differences in workload from frame to frame and render speeds of the GPUs (510). A heartbeat is created that controls the progress of the GPUs and smooths out their presents (515), where a heartbeat is a pulse or steady ticking of when frames should be presented. The determined appropriate amount of time is waited in the kernel mode driver (KMD)) so that the frames are evenly spaced (520). For example, a dummy schedulable engine is created in the KMD. A user mode driver (UMD) submits dummy command buffers to this dummy engine to request a delay, (which may be 90-95% of expected frame time). The KMD reports the command buffer as complete when the requested delay has passed (525). In the event that the rendering process takes longer than the delay, then the present will be done as soon as the rendering process is complete. In effect, the delay is a minimum wait time for a GPU to present a frame. The UMD submits signals of a Microsoft® (MS) synchronization object to the dummy engine. The UMD waits on this synchronization object on the regular 3D engines”).
Claims 2, 9 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Lobete, etc. (US 20210400170 A1) in view of Saulters (US 20140092109 A1), further in view of Schluessler, etc. (US 20220122566 A1), Masuda, etc. (US 20120147013 A1), Campbell (US 20160042488 A1) and Ungureanu, etc. (US 20140168229 A1).
Regarding claim 2, Lobete, Saulters, Schluessler, Masuda and Campbell teach all the features with respect to claim 1 as outlined above. However, Lobete, modified by Saulters, Schluessler, Masuda and Campbell, fails to explicitly disclose that the processing system of claim 1, wherein the AU includes an asynchronous scheduling circuitry configured to schedule instructions such that the one or more rendering metrics are determined concurrently with rendering one or more frames.
However, Ungureanu teaches that the processing system of claim 1, wherein the AU includes an asynchronous scheduling circuitry configured to schedule instructions such that the one or more rendering metrics are determined concurrently with rendering one or more frames (See Ungureanu: Fig. 6, and [0005], “Embodiments described herein relate to improving throughput of a CPU and a GPU working in conjunction to render graphics. Time frames for executing CPU and GPU work units are synchronized with a refresh rate of a display. Pending CPU work is performed when a time frame starts (a vsync occurs). When a prior GPU work unit is still executing on the GPU, then a parallel mode is entered. In the parallel mode, some GPU work and some CPU work is performed concurrently. When the parallel mode is exited, for example when there is no CPU work to perform, the parallel mode may be exited”; [0024], “FIG. 6 shows a process for dynamically alternating between a non-parallel mode and a parallel mode. At step 180 a refresh or vsync signal occurs or is received. The application code blocked at step 182 receives a refresh signal, event, etc., indicating the start of a new processing frame. Before, during, or after executing the currently pending CPU work unit, the GPU is checked to determine if the GPU is still executing a prior GPU work unit. If the prior frame's GPU unit is still executing, then at step 188 the process checks whether it is already in parallel mode. If not, then the process enters parallel mode and proceeds to step 192. If the process is in parallel mode, then the process continues to step 182. If the GPU is idle or not executing a prior frame's GPU work, then at step 190 the process either re-enters or stays in the non-parallel mode. In an embodiment where step 186 is performed at the beginning of the new frame, step 192 of executing the current CPU work unit is executed on the CPU, which generates a new GPU work unit. At step 194 the timing of execution of the GPU work is determined according to whether the process is in parallel mode. If the process is in parallel mode, then at step 194 the newly generated GPU work unit is scheduled to execute on the GPU at the start of the next frame. Or, in another embodiment shown in FIG. 8, the GPU work unit can begin executing when the current GPU work unit finishes, although such execution may still end at a next frame. As will be explained below, if the generated GPU work completes in a later frame then lag may be added (as will be the case when GPU work units start at the beginning of a later or next frame)”; and [0025], “To summarize, while in parallel mode, GPU work units may execute in parallel with CPU work units while maintaining synchronization with the refresh signal. Parallel mode may be entered when pending or executing GPU work is detected. Parallel mode may be exited when no CPU work is pending (when the software that is handling the events is idle)”. Note that pending CPU work is scheduled to run when a time frame start on VSYNC, overlapping with ongoing GPU rendering, and this constitutes asynchronous scheduling of instructions that enables concurrent determination of rendering related metrics/timing information while frames continue to be rendered on the GPU which is mapped to the accelerator unit).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Lobete to have the processing system of claim 1, wherein the AU includes an asynchronous scheduling circuitry configured to schedule instructions such that the one or more rendering metrics are determined concurrently with rendering one or more frames as taught by Ungureanu in order to save the processing time and the power consumption for rendering the lighting information of the second frame(See Ungureanu: Fig. 8, and [0027], “While the checking for execution of prior GPU work can be performed at the start of a new frame, such checking can be performed at other stages of the new frame, for example, when CPU work in the current frame finishes, during execution of the current CPU work (e.g., by a timer), or as otherwise convenient for the implementation. In one embodiment, in parallel mode, the GPU work units start at the beginning of frames following the frames of their respective CPU work units (see FIG. 7). FIG. 8 shows behavior of another embodiment where GPU work units begin executing mid-frame, for example, as soon as GPU work units are ready for processing. This approach, which may require additional synchronization logic and can in some cases reduce latency, increase overall concurrency, or hasten the exit from parallel mode”). Lobete teaches a method and system that may process the video frames and generate interpolated frames based on the frame rendering metrics and the display refresh rate; while Ungureanu teaches a system and method that may measure the GPU rendering time and use this measured time to control display and spacing delays so the frames are evenly spaced. Therefore, it is obvious for one of ordinary skill in the art to modify Lobete by Ungureanu to control the display spacing based on the measured GPU rendering time. The motivation to modify Lobete by Ungureanu is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 9, Lobete, Saulters, Schluessler, Masuda and Campbell teach all the features with respect to claim 8 as outlined above. Further, Ungureanu teaches that the method of claim 8, further comprising:
scheduling, at an asynchronous scheduling circuitry, instructions such that the one or more rendering metrics are determined concurrently with rendering one or more frames (See Ungureanu: Fig. 6, and [0005], “Embodiments described herein relate to improving throughput of a CPU and a GPU working in conjunction to render graphics. Time frames for executing CPU and GPU work units are synchronized with a refresh rate of a display. Pending CPU work is performed when a time frame starts (a vsync occurs). When a prior GPU work unit is still executing on the GPU, then a parallel mode is entered. In the parallel mode, some GPU work and some CPU work is performed concurrently. When the parallel mode is exited, for example when there is no CPU work to perform, the parallel mode may be exited”; [0024], “FIG. 6 shows a process for dynamically alternating between a non-parallel mode and a parallel mode. At step 180 a refresh or vsync signal occurs or is received. The application code blocked at step 182 receives a refresh signal, event, etc., indicating the start of a new processing frame. Before, during, or after executing the currently pending CPU work unit, the GPU is checked to determine if the GPU is still executing a prior GPU work unit. If the prior frame's GPU unit is still executing, then at step 188 the process checks whether it is already in parallel mode. If not, then the process enters parallel mode and proceeds to step 192. If the process is in parallel mode, then the process continues to step 182. If the GPU is idle or not executing a prior frame's GPU work, then at step 190 the process either re-enters or stays in the non-parallel mode. In an embodiment where step 186 is performed at the beginning of the new frame, step 192 of executing the current CPU work unit is executed on the CPU, which generates a new GPU work unit. At step 194 the timing of execution of the GPU work is determined according to whether the process is in parallel mode. If the process is in parallel mode, then at step 194 the newly generated GPU work unit is scheduled to execute on the GPU at the start of the next frame. Or, in another embodiment shown in FIG. 8, the GPU work unit can begin executing when the current GPU work unit finishes, although such execution may still end at a next frame. As will be explained below, if the generated GPU work completes in a later frame then lag may be added (as will be the case when GPU work units start at the beginning of a later or next frame)”; and [0025], “To summarize, while in parallel mode, GPU work units may execute in parallel with CPU work units while maintaining synchronization with the refresh signal. Parallel mode may be entered when pending or executing GPU work is detected. Parallel mode may be exited when no CPU work is pending (when the software that is handling the events is idle)”. Note that pending CPU work is scheduled to run when a time frame start on VSYNC, overlapping with ongoing GPU rendering, and this constitutes asynchronous scheduling of instructions that enables concurrent determination of rendering related metrics/timing information while frames continue to be rendered on the GPU which is mapped to the accelerator unit).
Regarding claim 16, Lobete, Saulters, Schluessler, Masuda and Campbell teach all the features with respect to claim 15 as outlined above. Further, Ungureanu teaches that the AU of claim 15, further comprising:
an asynchronous scheduling circuitry configured to schedule instructions such that the one or more rendering metrics are determined concurrently with rendering one or more frames (See Ungureanu: Fig. 6, and [0005], “Embodiments described herein relate to improving throughput of a CPU and a GPU working in conjunction to render graphics. Time frames for executing CPU and GPU work units are synchronized with a refresh rate of a display. Pending CPU work is performed when a time frame starts (a vsync occurs). When a prior GPU work unit is still executing on the GPU, then a parallel mode is entered. In the parallel mode, some GPU work and some CPU work is performed concurrently. When the parallel mode is exited, for example when there is no CPU work to perform, the parallel mode may be exited”; [0024], “FIG. 6 shows a process for dynamically alternating between a non-parallel mode and a parallel mode. At step 180 a refresh or vsync signal occurs or is received. The application code blocked at step 182 receives a refresh signal, event, etc., indicating the start of a new processing frame. Before, during, or after executing the currently pending CPU work unit, the GPU is checked to determine if the GPU is still executing a prior GPU work unit. If the prior frame's GPU unit is still executing, then at step 188 the process checks whether it is already in parallel mode. If not, then the process enters parallel mode and proceeds to step 192. If the process is in parallel mode, then the process continues to step 182. If the GPU is idle or not executing a prior frame's GPU work, then at step 190 the process either re-enters or stays in the non-parallel mode. In an embodiment where step 186 is performed at the beginning of the new frame, step 192 of executing the current CPU work unit is executed on the CPU, which generates a new GPU work unit. At step 194 the timing of execution of the GPU work is determined according to whether the process is in parallel mode. If the process is in parallel mode, then at step 194 the newly generated GPU work unit is scheduled to execute on the GPU at the start of the next frame. Or, in another embodiment shown in FIG. 8, the GPU work unit can begin executing when the current GPU work unit finishes, although such execution may still end at a next frame. As will be explained below, if the generated GPU work completes in a later frame then lag may be added (as will be the case when GPU work units start at the beginning of a later or next frame)”; and [0025], “To summarize, while in parallel mode, GPU work units may execute in parallel with CPU work units while maintaining synchronization with the refresh signal. Parallel mode may be entered when pending or executing GPU work is detected. Parallel mode may be exited when no CPU work is pending (when the software that is handling the events is idle)”. Note that pending CPU work is scheduled to run when a time frame start on VSYNC, overlapping with ongoing GPU rendering, and this constitutes asynchronous scheduling of instructions that enables concurrent determination of rendering related metrics/timing information while frames continue to be rendered on the GPU which is mapped to the accelerator unit).
Claims 3, 10 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Lobete, etc. (US 20210400170 A1) in view of Saulters (US 20140092109 A1), further in view of Schluessler, etc. (US 20220122566 A1), Masuda, etc. (US 20120147013 A1), Campbell (US 20160042488 A1), Ungureanu, etc. (US 20140168229 A1) and Swedberg, etc. (US 20070057952 A1).
Regarding claim 3, Lobete, Saulters, Schluessler, Masuda and Campbell teach all the features with respect to claim 1 as outlined above. However, Lobete, modified by Saulters, Schluessler, Masuda, Campbell and Ungureanu, fails to explicitly disclose that the processing system of claim 1, wherein the AU is configured to: determine the interpolated frame timing based on a comparison of the one or more rendering metrics to a target framerate.
However, Swedberg teaches that the processing system of claim 1, wherein the AU is configured to: determine the interpolated frame timing based on a comparison of the one or more rendering metrics to a target framerate (See Swedberg: Figs. 1-5, and [0045], “As can be seen in FIGS. 4A and 4B, upon waking, step 402 determines whether the previous frame was late. If so, step 404 adjusts CRefresh down by one (or by some other suitable value) for this frame. CRefresh is used in two slightly different ways. First CRefresh indicates for how many monitor refreshes this particular frame should be displayed. Second, CRefresh is used as the denominator in the equation RComposition=RRefresh/CRefresh to compute the composition rate. CRefresh is adjusted down in step 404 to indicate that this particular frame must be shortened by the number of refreshes that the thread woke up late. If the resulting adjusted CRefresh is zero, as evaluated at step 406, it indicates that the thread woke up so late that the entire chance to process the frame was missed. At step 414 CRefresh is adjusted back to its value, that is, to its value before processing step 404 (corresponding to 2.a above) that if executed changed it, and returns to sleep. Otherwise, the process continues to step 408 to begin recording data”; [0046], “To this end, the process in the adaptive scheduler 210 records the start time TFrameBegin (step 408) and determines the display time for the new frame TFrame (step 410). Step 412 computes the thresholds as set forth in FIG. 4A and the above computations”; [0036], “Similar timing information can be used is a slightly different manner to determine if the rate can be increased. For example, as represented in FIG. 3D, TEarlyDeadline is defined as TFrameBegin plus TFrameBudget/2 (or some other suitable fraction). If TFrameEnd is less than TEarlyDeadline (FIG. 3D), the system can potentially complete two frames within the budgeted time. The adaptive scheduler 210 will consider increasing the rate of frame composition if below the early deadline, and if currently composing slower than the monitor refresh rate. Note that using the elapsed time for a frame to determine if it can run at a faster frame composition rate depends on the rendering time being related to system load. Further note that in an alternative system, the adaptive scheduler 210 may periodically or occasionally attempt to increase the frame composition rate, even if not regularly finishing by the early deadline (TEarlyDeadline)”; and [0040], “TFrameEnd and TBudgetDeadline are variables used to monitor performance, with performance adjusted via RComposition. More particularly, in one example, implementing an adaptive composition scheduler operates by recording where TFrameEnd occurred relative to TEarlyDeadline, TBudgetDeadline, and THardDeadline for every frame, (or at least some sampled amount of frames). Transient conditions on the system along with interaction with the system scheduler may vary these times significantly, whereby making decisions on a single measurement would cause oscillations in the frame composition rate. As a result, because consistency of the frame composition rate provides for smooth display, such frame composition rate adjustments are made less frequently by the adaptive scheduler 210, corresponding to some number of frames, refreshes and/or period of time. For example, the information may be maintained for some period of time, PSample, and the aggregate performance information measured over this time used to make any adjustments”. Note that the comparison of measured metrics against a refresh-derived target framerate directly determines the timing at which frames including the interpolayed frames are scheduled and provided to the display).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Lobete to have the processing system of claim 1, wherein the AU is configured to: determine the interpolated frame timing based on a comparison of the one or more rendering metrics to a target framerate as taught by Swedberg in order to maintain a smooth predictable frame rate effectively and avoid glitches resulting from discontinuities in frame composition rate (See Swedberg: Fig. 1, and [0029], “One aspect of the desktop window manager 204 is directed towards avoiding visible glitches that result from discontinuities in frame composition rate. To this end, the desktop window manager composes arbitrary content at a smooth, consistent frame rate, in general by adaptively scaling the frame rate as necessary based on the current system load with respect to the system's capabilities. In one implementation, the desktop window manager 204 benefits from having a sufficient scheduling resolution and precision to support a high frame rate; the composition loop rarely or never blocks, other than waiting for the next frame time”). Lobete teaches a method and system that may process the video frames and generate interpolated frames based on the frame rendering metrics and the display refresh rate; while Swedberg teaches a system and method that may compare of the rendering metrics to a target framerate to determine the interpolated frame timing in order to maintain smooth predictable framerate. Therefore, it is obvious for one of ordinary skill in the art to modify Lobete by Swedberg to determine the interpolated frame timing based on the comparison of the rendering metrics against a target framerate. The motivation to modify Lobete by Swedberg is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 10, Lobete, Saulters, Schluessler, Masuda and Campbell teach all the features with respect to claim 8 as outlined above. Further, Swedberg teaches that the method of claim 8, wherein determining the interpolated frame timing comprises:
determining the interpolated frame timing based on a comparison of the one or more rendering metrics to a refresh rate of the display (See Swedberg: Figs. 1-5, and [0045], “As can be seen in FIGS. 4A and 4B, upon waking, step 402 determines whether the previous frame was late. If so, step 404 adjusts CRefresh down by one (or by some other suitable value) for this frame. CRefresh is used in two slightly different ways. First CRefresh indicates for how many monitor refreshes this particular frame should be displayed. Second, CRefresh is used as the denominator in the equation RComposition=RRefresh/CRefresh to compute the composition rate. CRefresh is adjusted down in step 404 to indicate that this particular frame must be shortened by the number of refreshes that the thread woke up late. If the resulting adjusted CRefresh is zero, as evaluated at step 406, it indicates that the thread woke up so late that the entire chance to process the frame was missed. At step 414 CRefresh is adjusted back to its value, that is, to its value before processing step 404 (corresponding to 2.a above) that if executed changed it, and returns to sleep. Otherwise, the process continues to step 408 to begin recording data”; [0046], “To this end, the process in the adaptive scheduler 210 records the start time TFrameBegin (step 408) and determines the display time for the new frame TFrame (step 410). Step 412 computes the thresholds as set forth in FIG. 4A and the above computations”; [0036], “Similar timing information can be used is a slightly different manner to determine if the rate can be increased. For example, as represented in FIG. 3D, TEarlyDeadline is defined as TFrameBegin plus TFrameBudget/2 (or some other suitable fraction). If TFrameEnd is less than TEarlyDeadline (FIG. 3D), the system can potentially complete two frames within the budgeted time. The adaptive scheduler 210 will consider increasing the rate of frame composition if below the early deadline, and if currently composing slower than the monitor refresh rate. Note that using the elapsed time for a frame to determine if it can run at a faster frame composition rate depends on the rendering time being related to system load. Further note that in an alternative system, the adaptive scheduler 210 may periodically or occasionally attempt to increase the frame composition rate, even if not regularly finishing by the early deadline (TEarlyDeadline)”; and [0040], “TFrameEnd and TBudgetDeadline are variables used to monitor performance, with performance adjusted via RComposition. More particularly, in one example, implementing an adaptive composition scheduler operates by recording where TFrameEnd occurred relative to TEarlyDeadline, TBudgetDeadline, and THardDeadline for every frame, (or at least some sampled amount of frames). Transient conditions on the system along with interaction with the system scheduler may vary these times significantly, whereby making decisions on a single measurement would cause oscillations in the frame composition rate. As a result, because consistency of the frame composition rate provides for smooth display, such frame composition rate adjustments are made less frequently by the adaptive scheduler 210, corresponding to some number of frames, refreshes and/or period of time. For example, the information may be maintained for some period of time, PSample, and the aggregate performance information measured over this time used to make any adjustments”. Note that the comparison of measured metrics against a refresh-derived target framerate directly determines the timing at which frames including the interpolayed frames are scheduled and provided to the display).
Regarding claim 17, Lobete, Saulters, Schluessler, Masuda and Campbell teach all the features with respect to claim 15 as outlined above. Further, Swedberg teaches that the AU of claim 15, wherein the one or more processor cores are configured to:
determine the interpolated frame timing based on a comparison of the one or more rendering metrics to a target framerate (See Swedberg: Figs. 1-5, and [0045], “As can be seen in FIGS. 4A and 4B, upon waking, step 402 determines whether the previous frame was late. If so, step 404 adjusts CRefresh down by one (or by some other suitable value) for this frame. CRefresh is used in two slightly different ways. First CRefresh indicates for how many monitor refreshes this particular frame should be displayed. Second, CRefresh is used as the denominator in the equation RComposition=RRefresh/CRefresh to compute the composition rate. CRefresh is adjusted down in step 404 to indicate that this particular frame must be shortened by the number of refreshes that the thread woke up late. If the resulting adjusted CRefresh is zero, as evaluated at step 406, it indicates that the thread woke up so late that the entire chance to process the frame was missed. At step 414 CRefresh is adjusted back to its value, that is, to its value before processing step 404 (corresponding to 2.a above) that if executed changed it, and returns to sleep. Otherwise, the process continues to step 408 to begin recording data”; [0046], “To this end, the process in the adaptive scheduler 210 records the start time TFrameBegin (step 408) and determines the display time for the new frame TFrame (step 410). Step 412 computes the thresholds as set forth in FIG. 4A and the above computations”; [0036], “Similar timing information can be used is a slightly different manner to determine if the rate can be increased. For example, as represented in FIG. 3D, TEarlyDeadline is defined as TFrameBegin plus TFrameBudget/2 (or some other suitable fraction). If TFrameEnd is less than TEarlyDeadline (FIG. 3D), the system can potentially complete two frames within the budgeted time. The adaptive scheduler 210 will consider increasing the rate of frame composition if below the early deadline, and if currently composing slower than the monitor refresh rate. Note that using the elapsed time for a frame to determine if it can run at a faster frame composition rate depends on the rendering time being related to system load. Further note that in an alternative system, the adaptive scheduler 210 may periodically or occasionally attempt to increase the frame composition rate, even if not regularly finishing by the early deadline (TEarlyDeadline)”; and [0040], “TFrameEnd and TBudgetDeadline are variables used to monitor performance, with performance adjusted via RComposition. More particularly, in one example, implementing an adaptive composition scheduler operates by recording where TFrameEnd occurred relative to TEarlyDeadline, TBudgetDeadline, and THardDeadline for every frame, (or at least some sampled amount of frames). Transient conditions on the system along with interaction with the system scheduler may vary these times significantly, whereby making decisions on a single measurement would cause oscillations in the frame composition rate. As a result, because consistency of the frame composition rate provides for smooth display, such frame composition rate adjustments are made less frequently by the adaptive scheduler 210, corresponding to some number of frames, refreshes and/or period of time. For example, the information may be maintained for some period of time, PSample, and the aggregate performance information measured over this time used to make any adjustments”. Note that the comparison of measured metrics against a refresh-derived target framerate directly determines the timing at which frames including the interpolayed frames are scheduled and provided to the display).
Claims 5, 12 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Lobete, etc. (US 20210400170 A1) in view of Saulters (US 20140092109 A1), further in view of Schluessler, etc. (US 20220122566 A1), Masuda, etc. (US 20120147013 A1), Campbell (US 20160042488 A1), Ungureanu, etc. (US 20140168229 A1), Swedberg, etc. (US 20070057952 A1) and Schneider, etc. (US 20130063459 A1).
Regarding claim 5, Lobete, Saulters, Schluessler, Masuda and Campbell, Ungureanu and Swedberg teach all the features with respect to claim 1 as outlined above. However, Lobete, modified by Saulters, Schluessler, Masuda, Campbell, Ungureanu and Swedberg, fails to explicitly disclose that the processing system of claim 1, wherein the AU is configured to: render a user interface (UI) within the first rendered frame.
However, Schneider teaches that the processing system of claim 1, wherein the AU is configured to: render a user interface (UI) within the first rendered frame (See Schneider: Figs. 1-3, and [0035], “The UI thread 202 may also be configured to generate composition structures 210 from the user interface hierarchy 206. If generated, these composition structures 210 may describe animations to apply to user interface elements represented by the static GPU data structures 208. As such, the UI thread 202 may also be configured to pass any generated composition structures 210 as composition data 212 to the composition thread 216, and the composition thread 216 may be configured to maintain a composition hierarchy 218. However, the composition hierarchy 218 in computer architecture 200 can potentially maintain far less composition information than the composition hierarchy 114 of computer architecture 100. For example, the composition hierarchy 218 may maintain only composition information related to the animation applied to the static GPU data structures 220. The composition thread 216 can then be configured to use the animation information in the composition hierarchy 218 to modify portions of the static GPU data structures 220 so that (as opposed generating new structures), when rendered by the GPU 228, any corresponding user interface elements are animated from frame to frame”; [0044], “Although not shown, the method 300 may also include an act of the UI thread 202 generating composition structures 210 corresponding to animations of the user interface, and passing composition data 212 to the composition thread 216. Thus, the composition thread 216 can store a composition hierarchy 218 which may include the composition data 212, which describes one or more animations of the user interface. As such, prior to sending GPU data and commands 224 to the GPU 228, the composition thread 216 can modify the static GPU data structures 220 to bring about animations of corresponding user interface elements when rendered. For instance, the composition thread 216 can modify information in the GPU data structures 220 that would bring about a change in one or more of position, scaling, rotation, transparency, clipping, color, perspective, opacity, model transform, or orientation of the corresponding user interface element. Also, while the composition thread 216 can modify the static GPU data structures 220 in response to composition data 212 received from the UI thread 202, the composition thread 216 can also initiate animations itself, in response to user input 222”; and [0045], “While carrying out the method 300, the UI thread 202 may perform some unbounded tasks (e.g., user interface layout tasks, network tasks, data binding tasks, rasterization, tessellation, other complex calculations), while the composition thread 216 may perform only bounded tasks. As such, the UI thread 202 may operate according to a first schedule, which may be relatively less frequent when compared to a second schedule with which the composition thread 216 operates, and/or the UI thread 202 may, at times, take relatively long amounts of time to execute. As such, the composition thread 216 can ensure that the user interface is consistent presented at high frame rates, and that the user interface remains responsive to user input”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Lobete to have the processing system of claim 1, wherein the AU is configured to: render a user interface (UI) within the first rendered frame as taught by Schneider in order to enhance the ability to scale the user interface framework to computing devices with relatively limited resources (See Schneider: Fig. 1, and [0013], “Embodiments may include functionality for performing primitive composition in a user interface (UI) thread. By composing primitives, such as Graphics Processing Unit (GPU) data structures, in a UI thread, a user interface framework is able to reduce the workload of a separate composition thread responsible for processing animations, processing input, and rendering a frame. This, in turn, enhances the ability to scale the user interface framework to computing devices having relatively limited resources, such as mobile devices, tablet computers, embedded devices, etc.”). Lobete teaches a method and system that may process the video frames and generate interpolated frames based on the frame rendering metrics and the display refresh rate; while Schneider teaches a system and method that may generate video frames corresponding to a user interface and sends GPU data and commands derived from static GPU data structure to GPU (the accelerator unit) for each of a plurality of video frame and GPU renders the frames that constitutes the user interface (including UI elements that remains static or are animated across frames,. Therefore, it is obvious for one of ordinary skill in the art to modify Lobete by Schneider to use GPU to render a user interface within the rendered frame including the interpolation frame, so that the first rendered frame includes UI content, a predictable combination. The motivation to modify Lobete by Schneider is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 12, Lobete, Saulters, Schluessler, Masuda and Campbell teach all the features with respect to claim 8 as outlined above. Further, Schneider teaches that the method of claim 8, further comprising: rendering a user interface (UI) within the interpolated frame (See Schneider: Figs. 1-3, and [0035], “The UI thread 202 may also be configured to generate composition structures 210 from the user interface hierarchy 206. If generated, these composition structures 210 may describe animations to apply to user interface elements represented by the static GPU data structures 208. As such, the UI thread 202 may also be configured to pass any generated composition structures 210 as composition data 212 to the composition thread 216, and the composition thread 216 may be configured to maintain a composition hierarchy 218. However, the composition hierarchy 218 in computer architecture 200 can potentially maintain far less composition information than the composition hierarchy 114 of computer architecture 100. For example, the composition hierarchy 218 may maintain only composition information related to the animation applied to the static GPU data structures 220. The composition thread 216 can then be configured to use the animation information in the composition hierarchy 218 to modify portions of the static GPU data structures 220 so that (as opposed generating new structures), when rendered by the GPU 228, any corresponding user interface elements are animated from frame to frame”; [0044], “Although not shown, the method 300 may also include an act of the UI thread 202 generating composition structures 210 corresponding to animations of the user interface, and passing composition data 212 to the composition thread 216. Thus, the composition thread 216 can store a composition hierarchy 218 which may include the composition data 212, which describes one or more animations of the user interface. As such, prior to sending GPU data and commands 224 to the GPU 228, the composition thread 216 can modify the static GPU data structures 220 to bring about animations of corresponding user interface elements when rendered. For instance, the composition thread 216 can modify information in the GPU data structures 220 that would bring about a change in one or more of position, scaling, rotation, transparency, clipping, color, perspective, opacity, model transform, or orientation of the corresponding user interface element. Also, while the composition thread 216 can modify the static GPU data structures 220 in response to composition data 212 received from the UI thread 202, the composition thread 216 can also initiate animations itself, in response to user input 222”; and [0045], “While carrying out the method 300, the UI thread 202 may perform some unbounded tasks (e.g., user interface layout tasks, network tasks, data binding tasks, rasterization, tessellation, other complex calculations), while the composition thread 216 may perform only bounded tasks. As such, the UI thread 202 may operate according to a first schedule, which may be relatively less frequent when compared to a second schedule with which the composition thread 216 operates, and/or the UI thread 202 may, at times, take relatively long amounts of time to execute. As such, the composition thread 216 can ensure that the user interface is consistent presented at high frame rates, and that the user interface remains responsive to user input”).
Regarding claim 19, Lobete, Saulters, Schluessler, Masuda and Campbell teach all the features with respect to claim 15 as outlined above. Further, Schneider teaches that the AU of claim 15, wherein the one or more processor cores are configured to:
render a user interface (UI) within the interpolated frame (See Schneider: Figs. 1-3, and [0035], “The UI thread 202 may also be configured to generate composition structures 210 from the user interface hierarchy 206. If generated, these composition structures 210 may describe animations to apply to user interface elements represented by the static GPU data structures 208. As such, the UI thread 202 may also be configured to pass any generated composition structures 210 as composition data 212 to the composition thread 216, and the composition thread 216 may be configured to maintain a composition hierarchy 218. However, the composition hierarchy 218 in computer architecture 200 can potentially maintain far less composition information than the composition hierarchy 114 of computer architecture 100. For example, the composition hierarchy 218 may maintain only composition information related to the animation applied to the static GPU data structures 220. The composition thread 216 can then be configured to use the animation information in the composition hierarchy 218 to modify portions of the static GPU data structures 220 so that (as opposed generating new structures), when rendered by the GPU 228, any corresponding user interface elements are animated from frame to frame”; [0044], “Although not shown, the method 300 may also include an act of the UI thread 202 generating composition structures 210 corresponding to animations of the user interface, and passing composition data 212 to the composition thread 216. Thus, the composition thread 216 can store a composition hierarchy 218 which may include the composition data 212, which describes one or more animations of the user interface. As such, prior to sending GPU data and commands 224 to the GPU 228, the composition thread 216 can modify the static GPU data structures 220 to bring about animations of corresponding user interface elements when rendered. For instance, the composition thread 216 can modify information in the GPU data structures 220 that would bring about a change in one or more of position, scaling, rotation, transparency, clipping, color, perspective, opacity, model transform, or orientation of the corresponding user interface element. Also, while the composition thread 216 can modify the static GPU data structures 220 in response to composition data 212 received from the UI thread 202, the composition thread 216 can also initiate animations itself, in response to user input 222”; and [0045], “While carrying out the method 300, the UI thread 202 may perform some unbounded tasks (e.g., user interface layout tasks, network tasks, data binding tasks, rasterization, tessellation, other complex calculations), while the composition thread 216 may perform only bounded tasks. As such, the UI thread 202 may operate according to a first schedule, which may be relatively less frequent when compared to a second schedule with which the composition thread 216 operates, and/or the UI thread 202 may, at times, take relatively long amounts of time to execute. As such, the composition thread 216 can ensure that the user interface is consistent presented at high frame rates, and that the user interface remains responsive to user input”).
Claims 6, 13 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Lobete, etc. (US 20210400170 A1) in view of Saulters (US 20140092109 A1), further in view of Schluessler, etc. (US 20220122566 A1), Masuda, etc. (US 20120147013 A1), Campbell (US 20160042488 A1), Ungureanu, etc. (US 20140168229 A1), Swedberg, etc. (US 20070057952 A1), Schneider, etc. (US 20130063459 A1) and Amichai, etc. (US 20160350197 A1).
Regarding claim 6, Lobete, Saulters, Schluessler, Masuda and Campbell, Ungureanu and Swedberg and Schneider teach all the features with respect to claim 5 as outlined above. However, Lobete, modified by Saulters, Schluessler, Masuda, Campbell, Ungureanu, Swedberg and Schneider, fails to explicitly disclose that the processing system of claim 5, wherein the one or more rendering metrics include a UI rendering time of the first rendered frame.
However, Amichai teaches that the processing system of claim 5, wherein the one or more rendering metrics include a UI rendering time of the first rendered frame (See Amichai: figs. 1-4, and [0019], “A number of time durations may be defined based on FIG. 1 and the preceding description. Initial feedback duration (120) may be the difference between time 103 and 104. UI responsive duration (122) may be the difference between time 103 and 106. Finally, UI complete duration (124) may be the difference between time 103 and 107. It should be noted that some monitoring tools may track user level application performance by analyzing the UI complete time duration (124), which is the time until the UI screen is completely rendered or loaded. The present disclosure may focus on UI responsive duration (122). This UI responsive duration may be used as a Key Performance Indicator (KPI) that provides insight into user-level UI responsiveness. This UI responsive duration may also be used as an industry benchmark for comparing various application UI's, for example, because UI responsiveness can be measured and compared regardless of what background activities are running”; [0037], “Rendered view analyzer 310 may detect all the UI views that change when a user interacts with a UI control (as detected by interaction detector 308). For example, if a user clicks a “next” button on a UI screen, the entire UI screen or a portion of the UI screen (e.g., at least one UI view) may change, refresh or load. The user interaction with the next button may be detected by the interaction detector 308. Then, based on this detection, rendered view analyzer 310 may determine all the UI views that change because of the user interaction. Rendered view analyzer 310 may detect which UI views change by interacting with user interaction hooks 302. Rendered view analyzer 310 may determine, out all the UI views that change, refresh or load (and will ultimately be displayed on the next UI screen after the user interaction is detected), which ones are “actionable controls.” This determination of which ones are actionable controls may be based on information received by actionable controls accessor 306 (from the monitoring system). Thus, rendered view analyzer 310 may determine which actionable controls (e.g., commonly used UI controls) changed based on the user interaction (and will ultimately be displayed on the next UI screen), and these may be referred to as “change controls,” Furthermore, rendered view analyzer 310 may determine timing specifics (e.g., start load time, end load time, etc.) with regard to the changing of these actionable controls, for example, by interacting with user interaction hooks 302”; and [0041], “FIG. 4 is a flowchart of an example method 400 for measuring user interface responsiveness. Method 400 may be described below as being executed or performed by a computing device, for example, client 220 of FIG. 2. Other suitable systems and/or computing devices may be used as well. Method 400 may be implemented in the form of executable instructions stored on at least one machine-readable storage medium of the computing device and executed by at least one processor of the computing device. Alternatively or in addition, method 400 may be implemented in the form of electronic circuitry (e.g., hardware). In alternate embodiments of the present disclosure, one or more steps of method 400 may be executed substantially concurrently or in a different order than shown in FIG. 4. In alternate embodiments of the present disclosure, method 400 may include more or less steps than are shown in FIG. 4. In some embodiments, one or more of the steps of method 400 may, at certain times, be ongoing and/or may repeat”. Note that these measured UI-related time are rendering metrics associated with the rendering of the UI frame (the first rendered frame in the sequence of frames being processed, and the system determines these times in the context of rendering successive UI screens/ frames).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Lobete to have the processing system of claim 5, wherein the one or more rendering metrics include a UI rendering time of the first rendered frame as taught by Amichai in order to prevent a user from using the application (See Amichai: Fig. 1, and [0010], “As mentioned above, for an application (e.g., computer software), it is desirable for user interface (UI) screens to change, refresh or load in a timely manner, for example, such that a user is not prevented from using the application because a UI screen is loading. Therefore, it may be useful to measure how quickly UI screens change, refresh or load, for example, such that an individual or entity (e.g., the owner of the application) can track how its application is performing. Specifically, it may be useful to measure UI performance for applications running on real computing devices (i.e., clients) being used by real users in the real world. In order to track data regarding the execution of applications, the individual or entity may use a monitoring tool running on a monitoring system (e.g., a server). The monitoring tool may collect data from various clients running the application, and may aid the individual or entity in analyzing the data. Specifically, agents running on the various clients may collect information about the execution of the application and may send data to the monitoring tool”). Lobete teaches a method and system that may process the video frames and generate interpolated frames based on the frame rendering metrics and the display refresh rate; while Amichai teaches a system and method that may measure various times for the UI elements in response to the user interactions to determine who is using which UI elements in order to prevent interferences between used in using the UI elements. Therefore, it is obvious for one of ordinary skill in the art to modify Lobete by Amichai to measure UI element responsive time and rendering times and using these rendering metrics to determine the frame timing. The motivation to modify Lobete by Amichai is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 13, Lobete, Saulters, Schluessler, Masuda and Campbell teach all the features with respect to claim 8 as outlined above. Further, Lobete teaches that the method of claim 12, wherein the one or more rendering metrics includes a UI rendering time of the interpolated frame (See Amichai: figs. 1-4, and [0019], “A number of time durations may be defined based on FIG. 1 and the preceding description. Initial feedback duration (120) may be the difference between time 103 and 104. UI responsive duration (122) may be the difference between time 103 and 106. Finally, UI complete duration (124) may be the difference between time 103 and 107. It should be noted that some monitoring tools may track user level application performance by analyzing the UI complete time duration (124), which is the time until the UI screen is completely rendered or loaded. The present disclosure may focus on UI responsive duration (122). This UI responsive duration may be used as a Key Performance Indicator (KPI) that provides insight into user-level UI responsiveness. This UI responsive duration may also be used as an industry benchmark for comparing various application UI's, for example, because UI responsiveness can be measured and compared regardless of what background activities are running”; [0037], “Rendered view analyzer 310 may detect all the UI views that change when a user interacts with a UI control (as detected by interaction detector 308). For example, if a user clicks a “next” button on a UI screen, the entire UI screen or a portion of the UI screen (e.g., at least one UI view) may change, refresh or load. The user interaction with the next button may be detected by the interaction detector 308. Then, based on this detection, rendered view analyzer 310 may determine all the UI views that change because of the user interaction. Rendered view analyzer 310 may detect which UI views change by interacting with user interaction hooks 302. Rendered view analyzer 310 may determine, out all the UI views that change, refresh or load (and will ultimately be displayed on the next UI screen after the user interaction is detected), which ones are “actionable controls.” This determination of which ones are actionable controls may be based on information received by actionable controls accessor 306 (from the monitoring system). Thus, rendered view analyzer 310 may determine which actionable controls (e.g., commonly used UI controls) changed based on the user interaction (and will ultimately be displayed on the next UI screen), and these may be referred to as “change controls,” Furthermore, rendered view analyzer 310 may determine timing specifics (e.g., start load time, end load time, etc.) with regard to the changing of these actionable controls, for example, by interacting with user interaction hooks 302”; and [0041], “FIG. 4 is a flowchart of an example method 400 for measuring user interface responsiveness. Method 400 may be described below as being executed or performed by a computing device, for example, client 220 of FIG. 2. Other suitable systems and/or computing devices may be used as well. Method 400 may be implemented in the form of executable instructions stored on at least one machine-readable storage medium of the computing device and executed by at least one processor of the computing device. Alternatively or in addition, method 400 may be implemented in the form of electronic circuitry (e.g., hardware). In alternate embodiments of the present disclosure, one or more steps of method 400 may be executed substantially concurrently or in a different order than shown in FIG. 4. In alternate embodiments of the present disclosure, method 400 may include more or less steps than are shown in FIG. 4. In some embodiments, one or more of the steps of method 400 may, at certain times, be ongoing and/or may repeat”. Note that these measured UI-related time are rendering metrics associated with the rendering of the UI frame (the first rendered frame in the sequence of frames being processed, and the system determines these times in the context of rendering successive UI screens/ frames).
Regarding claim 20, Lobete, Saulters, Schluessler, Masuda and Campbell teach all the features with respect to claim 15 as outlined above. Further, Amichai teaches that the AU of claim 19, wherein the one or more rendering metrics include a UI rendering time of the interpolated frame (See Amichai: figs. 1-4, and [0019], “A number of time durations may be defined based on FIG. 1 and the preceding description. Initial feedback duration (120) may be the difference between time 103 and 104. UI responsive duration (122) may be the difference between time 103 and 106. Finally, UI complete duration (124) may be the difference between time 103 and 107. It should be noted that some monitoring tools may track user level application performance by analyzing the UI complete time duration (124), which is the time until the UI screen is completely rendered or loaded. The present disclosure may focus on UI responsive duration (122). This UI responsive duration may be used as a Key Performance Indicator (KPI) that provides insight into user-level UI responsiveness. This UI responsive duration may also be used as an industry benchmark for comparing various application UI's, for example, because UI responsiveness can be measured and compared regardless of what background activities are running”; [0037], “Rendered view analyzer 310 may detect all the UI views that change when a user interacts with a UI control (as detected by interaction detector 308). For example, if a user clicks a “next” button on a UI screen, the entire UI screen or a portion of the UI screen (e.g., at least one UI view) may change, refresh or load. The user interaction with the next button may be detected by the interaction detector 308. Then, based on this detection, rendered view analyzer 310 may determine all the UI views that change because of the user interaction. Rendered view analyzer 310 may detect which UI views change by interacting with user interaction hooks 302. Rendered view analyzer 310 may determine, out all the UI views that change, refresh or load (and will ultimately be displayed on the next UI screen after the user interaction is detected), which ones are “actionable controls.” This determination of which ones are actionable controls may be based on information received by actionable controls accessor 306 (from the monitoring system). Thus, rendered view analyzer 310 may determine which actionable controls (e.g., commonly used UI controls) changed based on the user interaction (and will ultimately be displayed on the next UI screen), and these may be referred to as “change controls,” Furthermore, rendered view analyzer 310 may determine timing specifics (e.g., start load time, end load time, etc.) with regard to the changing of these actionable controls, for example, by interacting with user interaction hooks 302”; and [0041], “FIG. 4 is a flowchart of an example method 400 for measuring user interface responsiveness. Method 400 may be described below as being executed or performed by a computing device, for example, client 220 of FIG. 2. Other suitable systems and/or computing devices may be used as well. Method 400 may be implemented in the form of executable instructions stored on at least one machine-readable storage medium of the computing device and executed by at least one processor of the computing device. Alternatively or in addition, method 400 may be implemented in the form of electronic circuitry (e.g., hardware). In alternate embodiments of the present disclosure, one or more steps of method 400 may be executed substantially concurrently or in a different order than shown in FIG. 4. In alternate embodiments of the present disclosure, method 400 may include more or less steps than are shown in FIG. 4. In some embodiments, one or more of the steps of method 400 may, at certain times, be ongoing and/or may repeat”. Note that these measured UI-related time are rendering metrics associated with the rendering of the UI frame (the first rendered frame in the sequence of frames being processed, and the system determines these times in the context of rendering successive UI screens/ frames).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/Primary Examiner, Art Unit 2618
Prosecution Insights