DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1-20 are currently pending in the present application, with claims 1 and 11 being independent. Claims 1-20 have been amended.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 04 September 2023 has been considered by the examiner.
Response to Arguments
Applicant’s arguments, see page 8, filed 24 February 2026, with respect to objection to the drawings, along with accompanying amendments received on the same date, have been fully considered and are persuasive. The objection to the drawings has been withdrawn.
Applicant’s arguments, see page 8, filed 24 February 2026, with respect to objection to the claims, along with accompanying amendments received on the same date, have been fully considered and are persuasive. The objection to the claims has been withdrawn.
Applicant’s arguments, see page 9, filed 24 February 2026, with respect to 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph rejection of claims 10 and 20, along with accompanying amendments received on the same date, have been fully considered and are not persuasive. The 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph rejection of claims 10 and 20 has been maintained.
With respect to claims 10 and 20, given the plain and ordinary meaning of the words in light of the corresponding disclosure, the scope of the claimed limitations remains unclear. For instance, as currently drafted, it remains unclear as to the relationship between the plurality of pointers, the memory access log entries, and the performance log entries. The claim calls for the plurality of performance log entries to include a plurality of points to the plurality of memory access log entries. Does each log entry include a pointer to a memory access log entry? Or is it that there is a plurality of pointers pointing to a (or multiple) log entries spread throughout the performance log entries? The examiner respectfully requests the applicant clarify the scope of the claimed limitation. For the purposes of further examination, the examiner is interpreting the claim along the lines of paragraph 47, which recites, in part, each performance log entry includes a time range, statistics, and one or more pointers to memory access logs. Accordingly, the 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph rejection of claims 10 and 20 has been maintained
Applicant’s arguments, see page 9-10, filed 24 February 2026, with respect to 35 U.S.C. 101 rejection of claims 1-7, 10-17, and 20, along with accompanying amendments received on the same date, have been fully considered and are partially persuasive. The 35 U.S.C. 101 rejection of claims 1-3, 5-7, 10-13, 15-17, and 20 has been maintained, while the 35 U.S.C. 101 rejection of claims 4 and 14 has been withdrawn.
Applicant argues that claims 1 and 11 cannot be considered a mental process, “tie the performance statistics to generation of a visual element, and recite that the visualization for these performance statistics are incorporated into the rendering of the visual element. These rendering and memory operations are explicitly computer operations that cannot be performed in the human mind”. The examiner respectfully disagrees.
As drafted, the claimed limitations are mental processes that could be done with a human using pen and paper. For instance, the limitation “obtaining performance statistics associated with memory accesses, the performance statistics being involved in determining a visual element of a frame”, is a process that, under its broadest reasonable interpretation, is data gathering, obtaining performance statistical values associated with memory accesses, the performance statistics being involved in determining a visual element of a frame. The second step merely generates a visualization for the statistics in the drawing/rendering of the visual element in the frame. The final step merely displays the visualization. The three limitations can be done by processes that could be performed using a pen and paper. Thus, the claim, as drafted, falls at least within the “Mental Processes” grouping of abstract ideas.
This judicial exception is not integrated into a practical application because the computer/processor (with the 112(f) interpretation applied thereto), is recited at a high-level of generality, such that it amounts to no more than mere instructions to apply the exception using a generic computer component. Claim 1 does not require a computer or machine. Accordingly, the additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Claims 1 and 11 are directed to an abstract idea.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above, with respect to integration of the abstract idea into a practical application, the additional elements of using a processor/computer/display to perform the claimed limitations amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Also, as noted, claim 1 does not require a generic computer component. Thus, claims 1 and 11 are not patent eligible and the 35 U.S.C. 101 rejection of claims 1-3, 5-7, 10-13, 15-17, and 20 has been maintained.
With respect to applicant’s argument, that rendering and memory operations are explicitly computer operations that cannot be performed in the human mind, the examiner notes the memory operations as currently drafted are recited in an obtaining/data gathering step, so it does not matter if the memory operations are explicitly computer operations. Even if they were, as recited, the memory operations are recited at a high-level of generality, such that it would amount to no more than mere instructions to apply the exception using a generic computer component. The rendering operations (wherein the visualization is incorporated into rendering the visual element of the frame) is also merely incorporated into drawing the element of the frame, which could be done by pen and paper. Even if it was an explicitly computer operation, as recited, the rendering operation is recited at such a high-level of generality it amounts to no more than mere instructions to generate the visualization with the performance statistic.
Applicant’s arguments, see pages 11-12, filed 24 February 2026, with respect to prior art rejection, have been fully considered and are not persuasive. Applicant argues that Ritts does not teach “obtaining performance statistics associated with memory accesses involved in determining a visual element of a frame, and then generating a visualization of the performance statistics that is incorporated into rendering the visual element”. Applicant further argues that Ritts teaches displaying state/performance information and that this state/performance information cannot be incorporated into rendering the visual element as recited in claims 1 and 11. The examiner respectfully disagrees.
Ritts teaches a graphics device may provide graphics commands, along with state and/or performance information to an application computing device executing a performance analysis application, see for instance, paragraph 26. The graphics device may be used, for example for graphics applications, video game applications, video playback applications, ..., or video streaming applications, see for instance, paragraph 28. Various forms of performance data may also be included within the state/performance information, such as metrics or hardware counter data from one or more control processor, vertex processor, graphics processing unit, and display processor, see for instance, paragraph 76. Performance metrics include pixels written, shader instructions issued, number of memory writes, etc, see for instance, paragraph 58. Different graphical identifiers (e.g., colors, shading/cross-hatching identifiers, other graphical identifiers) may be assigned to each of the graphics elements based upon the computed values of the performance metric for the graphics elements, and may process the graphics elements to render the graphics scene for the purpose of display, see for instance, paragraph 59. In such fashion...displaying rendered graphics elements during processing of the graphics elements and visually displaying such elements according to the graphical identifiers that are assigned to the graphics elements, see for instance, paragraph 59. The developer may tune various aspects of the application to determine if any changes may improve one or more performance characteristics of the executed application upon inspection of an updated visual representation of performance metrics for rendered elements on display device, see for instance, paragraph 61. A color yellow may be associated with a metric value of three; a color blue may be associated with a metric value of five; and a color red may be associated with a metric value of seven, and based upon these color designations, the performance analysis application may then assign the color yellow to the first graphics element, assign the color blue to the second graphics element, and assign the color red to the third graphics element, see for instance, paragraph 130. As described previously, application computing device may display a representation of a graphics image according to received graphics commands and state/performance information, see for instance, paragraph 77. Application computing device may also display a visual representation of state/performance information, see for instance, paragraph 77. The use of performance analysis application may allow an application developer to measure and visualize a large set of properties of processing of individual 2D or 3D elements by the graphics processing hardware of graphics processing unit, see for instance, paragraph 78. Each element of the image rendered within the display area may be assigned a graphical identifier based upon computed metric values of a specified performance metric for the draw commands that are executed to render the elements, see for instance, paragraph 150. Visualizing the metrics data may allow a game developer, for example, to take a quick survey of how a device is rendering a particular scene, see for instance, paragraph 177. The graphics driver is capable of providing state and performance information, along with the draw commands, the performance analysis application, such that the analysis application can recreate the draw commands and display graphical representations of the calculated metrics, on a per-element basis, see for instance, paragraphs 124 and 150 and fig. 12.
That is, Ritts teaches obtaining performance statistics associated with memory accesses involved in determining a visual element of a frame, and then generating a visualization of the performance statistics that is incorporated into rendering the visual element as currently claimed.
With respect to applicant’s argument that Ritts teaches displaying state/performance information and that this state/performance information cannot be incorporated into rendering the visual element as recited in claims 1 and 11, Ritts teaches performance metrics include pixels written, shader instructions issued, number of memory writes, etc, see for instance, paragraph 58. Different graphical identifiers (e.g., colors, shading/cross-hatching identifiers, other graphical identifiers) may be assigned to each of the graphics elements based upon the computed values of the performance metric for the graphics elements, and may process the graphics elements to render the graphics scene for the purpose of display, see for instance, paragraph 59. In such fashion...displaying rendered graphics elements during processing of the graphics elements and visually displaying such elements according to the graphical identifiers that are assigned to the graphics elements, see for instance, paragraph 59. As described previously, application computing device may display a representation of a graphics image according to received graphics commands and state/performance information, see for instance, paragraph 77. Application computing device may also display a visual representation of state/performance information, see for instance, paragraph 77. The use of performance analysis application may allow an application developer to measure and visualize a large set of properties of processing of individual 2D or 3D elements by the graphics processing hardware of graphics processing unit, see for instance, paragraph 78. Each element of the image rendered within the display area may be assigned a graphical identifier based upon computed metric values of a specified performance metric for the draw commands that are executed to render the elements, see for instance, paragraph 150. Visualizing the metrics data may allow a game developer, for example, to take a quick survey of how a device is rendering a particular scene, see for instance, paragraph 177. A color yellow may be associated with a metric value of three; a color blue may be associated with a metric value of five; and a color red may be associated with a metric value of seven, and based upon these color designations, the performance analysis application may then assign the color yellow to the first graphics element, assign the color blue to the second graphics element, and assign the color red to the third graphics element, see for instance, paragraph 130. The graphics driver is capable of providing state and performance information, along with the draw commands, the performance analysis application, such that the analysis application can recreate the draw commands and display graphical representations of the calculated metrics, on a per-element basis, see for instance, paragraphs 124 and 150 and fig. 12. That is, as currently claimed, the performance information is incorporated into rendering the visual element in Ritts.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: a memory visualization generator configured to...in claim 11.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Claims 1 and 11 recite “the performance statistics being involved in determining a visual element of a frame; generating a visualization for the performance statistics, the visualization is incorporated into rendering the visual element of the frame”. The originally filed disclosure does not appear to set forth or describe the performance statistics being involved in determining a visual element of the frame. Applicant points to paragraph 57 for support, which describes overlaying or annotating a frame. However, paragraph 57 (as well of the remainder of the disclosure) does not appear to teach the performance statistics being involved in determining a visual element of a frame; generating a visualization for the performance statistics, the visualization is incorporated into rendering the visual element of the frame. Accordingly, the examiner respectfully requests the applicant clarify as to where support can be found for amended subject matter.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
With respect to claims 1 and 11, given the plain and ordinary meaning of the words themselves when interpreted on their own or in light of the corresponding disclosure, the scope of the claimed limitations is unclear. For instance, it is not immediately clear as to how and to what extent the statistics are involved in determining a visual element of the frame and how the visualization is incorporated into rendering the visual element of the frame. The specification provides examples for the scene annotator in paragraph 57 that generates color or color modifications. However, changing a color of a pixel would be different than the currently recited subject matter. In addition, the disclosure, appears to associate “a visual element” with the overlay portion of the description. Visual elements with respect to the overlay generator are primarily discussed in paragraphs 72-75. Paragraph 72 sets forth that the overlay generator generates one or more visual elements for each time period within a time range, each visual element illustrates aspects of one or more statistics of the stored statistics. Paragraph 73 sets forth “where the statistics include performance log statistics 328, the overlay generator 504 obtains the performance statistics 328 from the performance log entries 322 that correspond to the times. In examples where the statistics include memory access log statistics 334, the overlay generator 504 obtains the memory access log statistics 334 that correspond to the time point for the visual element, and generates the visual element for those memory access log statistics 334”. However, the disclosure does not appear to clarify the scope of the claimed limitation and it remains unclear as to how and to what extent the statistics are involved in determining a visual element of the frame and how the visualization is incorporated into rendering the visual element of the frame. The examiner respectfully requests the applicant clarify the scope of the claimed limitation.
Claims depending thereon do not cure the noted deficiency and are accordingly also rejected using substantially similar rationale as to that set forth for the claims from which they depend.
With respect to claims 4 and 14, given the plain and ordinary meaning of the words themselves when interpreted on their own or in light of the corresponding disclosure, the scope of the claimed limitations is unclear. For instance, it is not immediately clear as to what is meant by the time point being obtained by scrubbing backwards or forwards through time. What is being “scrubbed” and how is it being scrubbed through time to obtain the time point? It appears from paragraphs 60 and 61 that a time point can be obtained by scrubbing a current time point backwards or forwards in time through memory tracing. For instance, paragraph 60 sets forth “the memory visualization generator is capable of "scrubbing" this current point in time backwards or forwards in time, adjusting the visualizations generated accordingly. The "current" point in time determines the time of the memory accesses for which the visualization generator generates memory visualizations. Thus, scrubbing the current point in time backwards or forwards follows the memory traces in time and adjusts the memory visualizations that are displayed. For example, scrubbing backwards in time follows the dependencies of the memory traces backwards and scrubbing forwards in time follows the dependencies of the memory traces forwards, adjusting the visualizations accordingly. In some examples, this "scrubbing" occurs automatically (e.g., at the direction of the memory visualization generator itself or at the direction of any other element) or manually (e.g., in response to a user input)”. Paragraph 61 adds “By scrubbing forwards and backwards, it is possible to visualize different points in time within the process of rendering the frame”. However, the scope of the claimed limitation remains unclear when interpreted in light of the corresponding disclosure as it is still unclear what is being “scrubbed” and how is it being scrubbed through time to obtain the time point. The examiner respectfully requests the applicant clarify the scope of the claimed invention.
With respect to claims 10 and 20, given the plain and ordinary meaning of the words themselves when interpreted on their own or in light of the corresponding disclosure, the scope of the claimed limitations is unclear. For instance, it is not immediately clear as to the relationship between the plurality of pointers, the memory access log entries, and the performance log entries. The claim calls for the plurality of performance log entries to include a plurality of points to the plurality of memory access log entries. Does each log entry include a pointer to a memory access log entry? Or is it that there is a plurality of pointers pointing to a (or multiple) log entries spread throughout the performance log entries? The examiner respectfully requests the applicant clarify the scope of the claimed limitation. For the purposes of further examination, the examiner is interpreting the claim along the lines of paragraph 47, which recites, in part, each performance log entry includes a time range, statistics, and one or more pointers to memory access logs.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-3, 5-7, 10-13, 15-17, and 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim(s) recite(s) obtaining statistics for operation of a device, the statistics including either or both of performance statistics and memory access statistics; generating a plurality of visualizations of the statistics in one of an overlay mode or a scene annotation mode; and displaying the plurality of visualization. Claim 11 recites similar subject matter and is being interpreted under 112(f).
As drafted, the claimed limitations are mental processes that could be done with a human using pen and paper. For instance, the limitation “obtaining performance statistics associated with memory accesses, the performance statistics being involved in determining a visual element of a frame”, is a process that, under its broadest reasonable interpretation, is data gathering, obtaining performance statistical values associated with memory accesses, the performance statistics being involved in determining a visual element of a frame. The second step merely generates a visualization for the statistics in the drawing/rendering of the visual element in the frame. The final step merely displays the visualization. The three limitations can be done by processes that could be performed using a pen and paper. Thus, the claim, as drafted, falls at least within the “Mental Processes” grouping of abstract ideas.
This judicial exception is not integrated into a practical application because the computer/processor (with the 112(f) interpretation applied thereto), is recited at a high-level of generality, such that it amounts to no more than mere instructions to apply the exception using a generic computer component. Claim 1 does not require a computer or machine. Accordingly, the additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Claims 1 and 11 are directed to an abstract idea.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above, with respect to integration of the abstract idea into a practical application, the additional elements of using a processor/computer/display to perform the claimed limitations amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Also, as noted, claim 1 does not require a generic computer component. Thus, claims 1 and 11 are not patent eligible.
Claims 2-3, 5-7, 10, 12, 13, 15-17, and 20 recite additional limitations that are processes that could be performed by a human using pen and paper. For instance, claim 2 further requires in the data gathering/obtaining step, the memory accesses include accesses for a sequence of dependent memory operations for generating the visual element of the frame. Accordingly, the considerations laid out for claims 1 and 11 apply to the noted dependent claims. Therefore, as currently drafted, the noted dependent claims are not patent eligible.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-8 and 11-18 is/are rejected under 35 U.S.C. 102(a)(1)/102(a)(2) as being anticipated by Ritts et al. (US PG Publication 2011/0018884).
Regarding claim 1, Ritts teaches a method comprising: obtaining performance statistics associated with memory accesses (A graphics device may provide graphics commands, along with state and/or performance information to an application computing device executing a performance analysis application, see for instance, paragraph 26. The graphics device may be used, for example for graphics applications, video game applications, video playback applications, ..., or video streaming applications, see for instance, paragraph 28. Various forms of performance data may also be included within the state/performance information, such as metrics or hardware counter data from one or more control processor, vertex processor, graphics processing unit, and display processor, see for instance, paragraph 76. Performance metrics include pixels written, shader instructions issued, number of memory writes, etc, see for instance, paragraph 58), the performance statistics being involved in determining a visual element of a frame (see for instance, paragraphs 58-61 and 76. Performance analysis application may assign graphical identifiers (e.g., colors, shading/cross-hatching identifiers, other graphical identifiers) to each of the graphics elements based upon the computed values of the performance metric for the graphics elements, and may process the graphics elements to render the graphics scene for the purpose of display, see for instance, paragraph 59. In such fashion...displaying rendered graphics elements during processing of the graphics elements and visually displaying such elements according to the graphical identifiers that are assigned to the graphics elements, see for instance, paragraph 59. The developer may tune various aspects of the application to determine if any changes may improve one or more performance characteristics of the executed application upon inspection of an updated visual representation of performance metrics for rendered elements on display device, see for instance, paragraph 61. A color yellow may be associated with a metric value of three; a color blue may be associated with a metric value of five; and a color red may be associated with a metric value of seven, and based upon these color designations, the performance analysis application may then assign the color yellow to the first graphics element, assign the color blue to the second graphics element, and assign the color red to the third graphics element, see for instance, paragraph 130);
generating a visualization for the performance statistics, wherein the visualization is incorporated into rendering the visual element of the frame (As described previously, application computing device may display a representation of a graphics image according to received graphics commands and state/performance information, see for instance, paragraph 77. Application computing device may also display a visual representation of state/performance information, see for instance, paragraph 77. The use of performance analysis application may allow an application developer to measure and visualize a large set of properties of processing of individual 2D or 3D elements by the graphics processing hardware of graphics processing unit, see for instance, paragraph 78. Each element of the image rendered within the display area may be assigned a graphical identifier based upon computed metric values of a specified performance metric for the draw commands that are executed to render the elements, see for instance, paragraph 150. Visualizing the metrics data may allow a game developer, for example, to take a quick survey of how a device is rendering a particular scene, see for instance, paragraph 177. The graphics driver is capable of providing state and performance information, along with the draw commands, the performance analysis application, such that the analysis application can recreate the draw commands and display graphical representations of the calculated metrics, on a per-element basis, see for instance, paragraphs 124 and 150 and fig. 12. Performance analysis application may assign graphical identifiers (e.g., colors, shading/cross-hatching identifiers, other graphical identifiers) to each of the graphics elements based upon the computed values of the performance metric for the graphics elements, and may process the graphics elements to render the graphics scene for the purpose of display, see for instance, paragraph 59. A color yellow may be associated with a metric value of three; a color blue may be associated with a metric value of five; and a color red may be associated with a metric value of seven, and based upon these color designations, the performance analysis application may then assign the color yellow to the first graphics element, assign the color blue to the second graphics element, and assign the color red to the third graphics element, see for instance, paragraph 130); and
displaying the visualization (As described previously, application computing device may display a representation of a graphics image according to received graphics commands and state/performance information, see for instance, paragraph 77. Application computing device may also display a visual representation of state/performance information, see for instance, paragraph 77. The use of performance analysis application may allow an application developer to measure and visualize a large set of properties of processing of individual 2D or 3D elements by the graphics processing hardware of graphics processing unit, see for instance, paragraph 78. Each element of the image rendered within the display area may be assigned a graphical identifier based upon computed metric values of a specified performance metric for the draw commands that are executed to render the elements, see for instance, paragraph 150. Visualizing the metrics data may allow a game developer, for example, to take a quick survey of how a device is rendering a particular scene, see for instance, paragraph 177. The graphics driver is capable of providing state and performance information, along with the draw commands, the performance analysis application, such that the analysis application can recreate the draw commands and display graphical representations of the calculated metrics, on a per-element basis, see for instance, paragraphs 124 and 150 and fig. 12. The developer may select the statistics button to view statistical information, which may comprise one or more graphs, tables, or other forms of statistical information, associated with the display of graphics image 210, see for instance, paragraph 162).
Regarding claim 2, Ritts teaches the method of claim 1 and further teaches wherein the memory accesses include accesses for a sequence of dependent memory operations for generating the visual element of the frame (see for instance, paragraphs 104 and 127 and fig. 12. Graphics commands may include on or more application instructions, see for instance, paragraph 41. Graphics processing system may execute graphics commands to display an image, or a scene of rendered graphics elements, on display device, see for instance, paragraph 42. When a draw command included within application instructions is executed by graphics processing unit, one or more of graphics drivers may obtain the values of one or more counters of graphics processing unit that encompasses the work performed by graphics processing unit in order to execute the draw command, see for instance, paragraph 56. For example, the graphics elements may be draw commands (e.g., commands included within application instructions 21) to render a graphics scene, see for instance, paragraph 57. Graphics image data may be processed in this processing pipeline during execution of graphics commands, which may be part of application instructions, see for instance, paragraph 73. In some cases, one or more of control processor, vertex processor, graphics processing unit, and display processor may have performance issues, or serve as potential bottlenecks within the processing pipeline, during the execution of graphics commands, see for instance, paragraph 73. As described previously, application computing device may display a representation of a graphics image according to received graphics commands and state/performance information, see paragraph 77. Application computing device may also display a visual representation of state/performance information, see for instance, paragraph 77).
Regarding claim 3, Ritts teaches the method of claim 2 and further teaches wherein the visualization corresponds to a time point that is associated with a memory operation of the sequence of dependent memory operations (Graphics commands may include on or more application instructions, see for instance, paragraph 41. Graphics processing system may execute graphics commands to display an image, or a scene of rendered graphics elements, on display device, see for instance, paragraph 42. When a draw command included within application instructions is executed by graphics processing unit, one or more of graphics drivers may obtain the values of one or more counters of graphics processing unit that encompasses the work performed by graphics processing unit in order to execute the draw command, see for instance, paragraph 56. For example, the graphics elements may be draw commands (e.g., commands included within application instructions 21) to render a graphics scene, see for instance, paragraph 57. Graphics image data may be processed in this processing pipeline during execution of graphics commands, which may be part of application instructions, see for instance, paragraph 73. In some cases, one or more of control processor, vertex processor, graphics processing unit, and display processor may have performance issues, or serve as potential bottlenecks within the processing pipeline, during the execution of graphics commands, see for instance, paragraph 73. As described previously, application computing device may display a representation of a graphics image according to received graphics commands and state/performance information, see paragraph 77. Application computing device may also display a visual representation of state/performance information, see for instance, paragraph 77. Each element of the image rendered within the display area may be assigned a graphical identifier based upon computed metric values of a specified performance metric for the draw commands that are executed to render the elements, see for instance, paragraph 150. Visualizing the metrics data may allow a game developer, for example, to take a quick survey of how a device is rendering a particular scene, see for instance, paragraph 177. The graphics driver is capable of providing state and performance information, along with the draw commands, the performance analysis application, such that the analysis application can recreate the draw commands and display graphical representations of the calculated metrics, on a per-element basis, see for instance, paragraphs 124 and 150 and fig. 12).
Regarding claim 4, Ritts teaches the method of claim 3, wherein the time point is obtained by scrubbing backwards or forwards though time (“One or more of the counters may be enabled for data collection. In some cases, the graphics device may enable or disable the counters, while in some instances, the performance analysis application may enable or disable the counters based upon one or more selections of a user. For example, if an application developer is interested in viewing performance metrics information for one or more specified metrics at any point in time, the performance analysis application may provide a request to enable data collection from those counters of the graphics processing unit that are used in determining or calculating the specified, desired metrics”, see paragraph 119).
Regarding claim 5, Ritts teaches the method of claim 1 and further teaches wherein generating the visualization comprises performing a two-pass technique in which the performance statistics are recorded in a first pass and the visualization is generated in a second pass based on the performance statistics recorded in the first pass (see for instance, paragraphs 104 and 127 and fig. 12. The performance analysis application may then check to see if it has received a full frame of animation from the graphics driver and then for each received graphics element and associated GPU counter data, the performance analysis application computes values for one or more specified performance metrics using the received GPU counter values, see for instance, paragraph 127. Each element of the image rendered within the display area may be assigned a graphical identifier based upon computed metric values of a specified performance metric for the draw commands that are executed to render the elements, see for instance, paragraph 150. The graphics driver is capable of providing state and performance information, along with the draw commands, the performance analysis application, such that the analysis application can recreate the draw commands and display graphical representations of the calculated metrics, on a per-element basis, see for instance, paragraphs 124 and 150 and fig. 12).
Regarding claim 6, Ritts teaches the method of claim 5 and further teaches wherein generating the visualization includes using a modified shader program in the second pass, wherein the modified shader program is a modified version of a shader program used to generate the frame in the first pass (see for instance, paragraphs 142-145).
Regarding claim 7, Ritts teaches the method of claim 6 and further teaches wherein the shader program comprises a pixel shader program or a compute kernel (see for instance, paragraphs 55, 57, and 69. Outgoing pixel fragments can obtain their values directly from performance counter data, such as performance counter data provided by graphics processing unit, during the implementation or execution of performance analysis application, see for instance, paragraph 69. A fragment shader engine could be given access to one or more performance counters by exposing special registers to the shader code, see for instance, paragraph 69. Pixel fragment values may be based upon, or derived from, the data in the special registers, see for instance, paragraph 69. Graphical identifiers that are assigned to the pixel fragments may be determined by using blending to combine multiple ones of the pixel fragments based upon values of one or more counters, see for instance, paragraph 69).
Regarding claim 8, Ritts teaches the method of claim 7 and further teaches wherein the modified shader program is configured to generate a modification to a pixel value generated by the modified shader program in the first pass, wherein the modification is for one of color, depth, luminance, or transparency (see for instance, paragraphs 55, 57, 69, 104, 109, and 127 and fig. 12. Outgoing pixel fragments can obtain their values directly from performance counter data, such as performance counter data provided by graphics processing unit, during the implementation or execution of performance analysis application, see for instance, paragraph 69. A fragment shader engine could be given access to one or more performance counters by exposing special registers to the shader code, see for instance, paragraph 69. Pixel fragment values may be based upon, or derived from, the data in the special registers, see for instance, paragraph 69. Graphical identifiers that are assigned to the pixel fragments may be determined by using blending to combine multiple ones of the pixel fragments based upon values of one or more counters, see for instance, paragraph 69. The performance analysis application may then check to see if it has received a full frame of animation from the graphics driver and then for each received graphics element and associated GPU counter data, the performance analysis application computes values for one or more specified performance metrics using the received GPU counter values, see for instance, paragraph 127. Each element of the image rendered within the display area may be assigned a graphical identifier based upon computed metric values of a specified performance metric for the draw commands that are executed to render the elements, see for instance, paragraph 150. The graphics driver is capable of providing state and performance information, along with the draw commands, the performance analysis application, such that the analysis application can recreate the draw commands and display graphical representations of the calculated metrics, on a per-element basis, see for instance, paragraphs 124 and 150 and fig. 12. The developer can request to modify one or more of the graphics commands, see for instance, paragraph 80).
Regarding claim 11, claim 11 recites substantially similar claim elements as to that set forth in claim 1 and those elements are rejected using substantially similar rationale as to that which was set forth in claim 1. In addition, Ritts teaches a device comprising: a processing device; and a memory visualization generator to perform operations (see for instance, paragraphs 155-157 and 180-185).
Regarding claim 12, claim 12 is the device claim of the method claim 2 and is accordingly rejected using substantially similar rationale as to that set forth in claim 2.
Regarding claim 13, claim 13 is the device claim of the method claim 3 and is accordingly rejected using substantially similar rationale as to that set forth in claim 3.
Regarding claim 14, claim 14 is the device claim of the method claim 4 and is accordingly rejected using substantially similar rationale as to that set forth in claim 4.
Regarding claim 15, claim 15 is the device claim of the method claim 5 and is accordingly rejected using substantially similar rationale as to that set forth in claim 5.
Regarding claim 16, claim 16 is the device claim of the method claim 6 and is accordingly rejected using substantially similar rationale as to that set forth in claim 6.
Regarding claim 17, claim 17 is the device claim of the method claim 7 and is accordingly rejected using substantially similar rationale as to that set forth in claim 7.
Regarding claim 18, claim 18 is the device claim of the method claim 8 and is accordingly rejected using substantially similar rationale as to that set forth in claim 8.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 9 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ritts et al. (US PG Publication 2011/0018884) in view of Levit-Gurevich (US PG Publication 2023/0109752).
Regarding claim 9, Ritts teaches the method of claim 7, but does not appear to explicitly state a kernel and thus does not explicitly teach wherein the modified shader program comprises a modified compute kernel configured to generate a modification to a generated output value in the first pass, wherein the modification is for one of color, depth, luminance, or transparency.
In the same art of performance metrics, Levit-Gurevich teaches that GPUs execute instructions commonly referred to as kernels, compute kernels, and/or shaders, see for instance, paragraph 42. The term kernel is used for general purpose computational tasks and the term shader is typically used when a kernel is used for graphics-related tasks, see for instance, paragraph 42. A GPU programmer develops kernels/shaders in a high-level programming language such as, for example, HLSL, OpenCL, etc, and then compiles the code into a binary version of the kernel, which is then executed by the GPU, see for instance, paragraph 42. GPU profiling may be used to identify improvements in GPU operation...some examples disclosed herein improve operation of the GPU by measuring operating parameters of the GPU based on analysis of the GLITs and determining whether to adjust operation of the GPU based on the measured operating parameters (e.g., operational statistics, performance statistics, etc), see for instance, paragraph 44. The GLIT engine can determine operating parameters associated with the GPU, which may be used to determine improvement(s) to the operation of the GPU, see for instance, paragraph 89.
It would have been obvious to one of ordinary skill in the art having the teachings of Ritts and Levit-Gurevich in front of them before the effective filing date of the claimed invention to incorporate measuring operating parameters and kernels as taught by Levit-Gurevich into Ritts performance metrics visualization system, as having a computing kernel and adjusting operation of the GPU based on the measured operating parameters, such as described by Levit-Gurevich was well known at the time of the effective filing date invention and would have yielded predictable results in combination with Ritts.
The modification of Ritts with Levit-Gurevich would have explicitly allowed the shader program to comprise a modified compute kernel configured to generate a modification to a generated output value in the first pass, wherein the modification is for one of color, depth, luminance, or transparency.
The motivation for combining Ritts with Levit-Gurevich would have been to improve the user experience, enhance functionality and use known graphics methods, see for instance, Levit-Gurevich, paragraphs 44 and 48.
Regarding claim 19, claim 19 is the device claim of the method claim 9 and is accordingly rejected using substantially similar rationale as to that set forth in claim 9.
Claim(s) 10 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ritts et al. (US PG Publication 2011/0018884) in view of Tsai et al. (US PG Publication 2016/0011955).
Regarding claim 10, Ritts teaches the method of claim 1, but does not appear to specifically state that the performance metrics are stored in a log file and thus does not explicitly teach wherein: the performance statistics are stored in a plurality of performance log entries; memory access statistics for the memory accesses are stored in a plurality of memory access log entries; and the plurality of performance log entries include a plurality of pointers to the plurality of memory access log entries.
In the same art of graphical performance metrics, Tsai teaches low level graphics API calls and parameters for a frame are captured and saved or logged to a file as an API call trace or graphics diagnostic trace, see for instance, paragraph 12. The trace allows the developer to inspect API state, view pixel history, pipeline objects, render target content, etc, see for instance, paragraph 12. Timing queries can be inserted between each draw call or between groups of draw calls in the logged API trace, see for instance, paragraph 13. Embodiments of the invention allow the developer to do additional in-depth offline graphics frame analysis using the logged API calls, see for instance, paragraph 13.
It would have been obvious to one of ordinary skill in the art having the teachings of Ritts and Tsai in front of them before the effective filing date of the claimed invention to incorporate storing information in log files as taught by Tsai into Ritts performance metrics visualization system, as storing parameters in a log file, such as described by Tsai was well known at the time of the effective filing date invention and would have yielded predictable results in combination with Ritts.
The modification of Ritts with Tsai would have explicitly allowed the performance statistics to be stored in a plurality of performance log entries; memory access statistics for the memory accesses are stored in a plurality of memory access log entries; and the plurality of performance log entries include a plurality of pointers to the plurality of memory access log entries.
The motivation for combining Ritts with Tsai would have been to improve the user experience, enhance functionality, use known logging techniques, and improve execution time, see for instance, Tsai, paragraph 5.
Regarding claim 20, claim 20 is the device claim of the method claim 10 and is accordingly rejected using substantially similar rationale as to that set forth in claim 10.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US PG Publication 2022/0309056 to Chivukula et al teaches having multiple log files and using a schema to correlate the information, see for instance, paragraphs 43, 36, and 65
US PG Publication 2019/0308099 to Lalonde et al teaches performance criteria 572 and usage statistics 574 that are stored in association with the sequences of compiled shaders that are optimized, wherein each of the performance criteria 572 corresponds to one or more of total power consumption, a code size, an average latency, total cache usage, and total memory usage of a sequence of compiled shaders during the course of executing a plurality of online gaming sessions, and the usage statistics 574 characterize execution of each of a plurality of online gaming sessions, see for instance, paragraph 112.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL J COBB whose telephone number is (571)270-3875. The examiner can normally be reached Monday - Friday, 11am - 7pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at 571-272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL J COBB/Primary Examiner, Art Unit 2615
Prosecution Insights