DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-6, 10, 11 and 14-20 are rejected under 35 U.S.C. 103 as being unpatentable over Wang (CN112150631A; all citations to English language machine translation with this office action).
Regarding claim 1:
Wang teaches: an image processing method (page 3, rendering method) comprising:
rendering, by a rendering processor (page 4, processor such as a GPU), a first image frame of content using a first rendering model (pages 3-5, the method can proceed frame by frame (p. 5, first partial paragraph), which teaches a first frame using the best rendering parameters (i.e. a rendering model) for that frame);
receiving input data relating to a plurality of properties of the rendering processor (page 6, receiving drawing parameters, rendering parameters, and data related to GPU energy consumption),
wherein the properties of the rendering processor comprise: one or more static properties of the rendering processor (see e.g., pages 6-7, non-limiting examples of static properties: drawing pipeline related data, GPU model, and/or number of GPU stream processors); and one or more dynamic properties of the rendering processor indicative of a current load on the rendering processor (see e.g., page 7, non-limiting examples of dynamic properties, any one of: (1) working frequency corresponding to GPU energy consumption; (2) working voltage corresponding to GPU energy consumption; (3) working current corresponding to GPU energy consumption; and/or (4) The rendering core occupancy rate corresponding to GPU energy consumption);
inputting the input data to a machine learning model trained to select a rendering model, amongst a plurality of rendering models, for use in rendering an image frame, in dependence on the properties of a processor for rendering the image frame (page 9, input into a trained neural network; pages 9-13, description of the neural network; page 14-15: output or obtain the best drawing parameter(-s) to be selected for rendering. See also page 17, Step 6 and/or page 3, summary paragraph at the top));
selecting, by the trained machine learning model, a second rendering model in dependence on the input data (see mapping directly above, pages 14, 15, 17, and page 5, frame by frame analysis, so this would be for the second frame); and
rendering, by the rendering processor, at least part of a second image frame of the content using the second rendering model in place of the first rendering model (page 5, frame by frame, the second frame can have different rendering model than the first).
It would have been obvious for one of ordinary skill in the art to have further modified the applied reference(-s), in view of same, to have obtained the above, and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A). Wang teaches all of the features of Applicant’s claim 1, as mapped above, and also teaches frame by frame analysis or optimization of rendering parameters and energy consumption/usage.
The prior art included each element recited in claim 1, although not necessarily in a single embodiment, with the only difference being between the claimed element and the prior art being the lack of actual combination of certain elements in a single prior art embodiment, as mapped and described above.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 2:
Wang teaches: the method of claim 1, wherein the machine learning model is trained to select the rendering model to maximise quality of the image frame (page 3, best rendering parameters), based at least in part on the properties of the processor for rendering the image frame (e.g. pages 6-7).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied reference(-s) in view of same to have obtained the above, motivated to control rendering in a manner that achieves both hardware and image optimizations.
Regarding claim 3:
Wang teaches: the method of claim 1, wherein each rendering model comprises a shading model (page 6, last partial paragraph to page 7).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied reference(-s) in view of same to have obtained the above, motivated to make use of known rendering stages for optimization of same.
Regarding claim 4:
It would have been obvious for one of ordinary skill in the art to have further modified the applied reference(-s), in view of same, to have obtained: the method of claim 3, wherein rendering the second image frame comprises: generating a mesh of objects in the second image frame (Wang, page 6, input mesh data); and
applying the second rendering model to the mesh to shade the mesh (pages 6-7, vertex shaders, geometric shaders, to shade mesh; second rendering model mapped in claim 1),
wherein generating the mesh of objects in the second image frame is initiated before, or simultaneously, with one of the steps of receiving the input data, inputting the input data to the machine learning model, or selecting the second rendering model (pages 6-7, data related to the drawing pipeline includes data related to shading mesh. Modifying the applied reference(-s), in view of same, such that the input generated mesh is generated before or simultaneously with receiving input data, as taught by Wang, would have been obvious and predictable to one of ordinary skill in the art),
and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A).
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 5:
Wang teaches: the method of claim 1, wherein the static properties of a processor comprise one or more selected from the list consisting of: architecture; factory clock speed; one or more supported rendering models; number of cores; one or more thermal properties; and one or more memory properties of the processor (pages 6-7).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied reference(-s) in view of same to have obtained the above, motivated to optimize rendering in view of architecture and rendering environments.
Regarding claim 6:
Wang teaches: the method of claim 1, wherein the dynamic properties of a processor comprise one or more selected from the list consisting of: computational resource usage; memory usage; temperature; fan speed; and power consumption (page 7, GPU hardware information, GPU working status and GPU energy consumption examples).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied reference(-s) in view of same to have obtained the above, motivated to optimize rendering in view of architecture and rendering environments.
Regarding claim 10:
Wang teaches: the method of claim 1, wherein: the machine learning model is trained to select the rendering model in dependence on a currently used rendering model (page 18, when drawing parameters are different, gradually transition); and wherein the input data further comprises an identifier of the first rendering model (page 18, input error data of a first rendering model can be used in an example of 10 drawing frames as an interval).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied reference(-s) in view of same to have obtained the above, motivated to optimize rendering in view of architecture and rendering environments.
Regarding claim 11:
Wang teaches: the method of claim 1, wherein: the machine learning model is trained to select the rendering model in dependence on one or more characteristics of the image frame to be rendered (p. 6-7);
the input data further comprises the one or more characteristics of the second image frame (p. 6-7),
wherein the one or more characteristics of the second image frame comprise one or more selected from the list consisting of: resolution; level of detail; and number of light sources (page 8).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied reference(-s) in view of same to have obtained the above, motivated to optimize rendering in view of architecture and rendering environments.
Regarding claim 14: see also claim 1.
Wang teaches: an image processing system (page 19, device) comprising: a rendering processor… a communication processor…an input processor (page 19, microprocessor, CPU, DSP, page 4, GPU).
The functions of all the processors of the system of claim 14 correspond to those of the method of claim 1; the same rationale for rejection applies.
Regarding claim 15: see claim 2.
These claims are similar; the same rationale for rejection applies.
Regarding claim 16: see claim 3.
These claims are similar; the same rationale for rejection applies.
Regarding claim 17: see claim 4.
These claims are similar; the same rationale for rejection applies.
Regarding claim 18: see claim 5.
These claims are similar; the same rationale for rejection applies.
Regarding claim 19: see claim 6.
These claims are similar; the same rationale for rejection applies.
Regarding claim 20: see also claim 1.
Wang teaches: a non-transitory computer-readable medium storing computer executable instructions, which when executed by a processor, causes a computer system to perform an image processing method (page 19, and see method mapped in claim 1) comprising:
The method of claim 20 corresponds to the method of claim 1; the same rationale for rejection applies.
Claim(s) 7 is rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of De Haan (U.S. Patent App. Pub. No. 2006/0209947).
Regarding claim 7:
It would have been obvious for one of ordinary skill in the art to have combined and modified the applied reference(-s), in view of same, to have obtained: the method of claim 1, wherein inputting the input data and selecting the second rendering model are performed in dependence upon determining that one or more dynamic properties of the rendering processor have changed relative to their previous values by at least a predetermined threshold; and
wherein, upon determining that one or more dynamic properties of the rendering processor have changed relative to their previous values by less than the predetermined threshold, the rendering comprises rendering, by the rendering processor, the second image frame using the first rendering model, and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A).
De Haan teaches that it is known to use thresholds to determine whether there is a static (non-changing) or dynamic (a change has occurred) situation (see para. 24). Applying this concept to the dynamic properties of the rendering processor, as taught by Wang, to know whether to use the first (previous) rendering model, or that a new rendering model is needed, would have been obvious and predictable to one of ordinary skill, with additional motivation to minimize processor usage (i.e. no need to recalculate a new model if there is static change from frame to frame in terms of dynamic properties of the processor).
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Claim(s) 8 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Zhang, Y., Wang, R., Huo, Y., Hua, W., & Bao, H. (2021). Powernet: Learning-based real-time power-budget rendering. IEEE Transactions on Visualization and Computer Graphics, 28(10), 3486-3498 (“Zhang”, cited in IDS).
Regarding claim 8:
It would have been obvious for one of ordinary skill in the art to have combined and modified the applied reference(-s), in view of same, to have obtained: the method of claim 1, wherein the machine learning model is trained with training data comprising: the properties of a plurality of different processors at a plurality of different loads and using a plurality of different rendering models to render one or more image frames (Wang, pages 10-12) (Zhang, Section 5.3); and
data relating to quality of the image frames rendered by the processors at the respective loads and using the respective rendering models (Wang, pages 10-12) (Zhang, Sections 6.1, 6.2),
and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A).
The prior art included each element recited in claim 8, although not necessarily in a single embodiment, with the only difference being between the claimed element and the prior art being the lack of actual combination of certain elements in a single prior art embodiment, as mapped and described above.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 9:
Wang teaches: the method of claim 8, wherein the data relating to the quality of the image frames comprises a ranking, based on image quality, of the rendering models for each given processor at each given load (page 7, image corresponding to worst quality rendering for a certain rendering technology teaches Applicant’s claimed ranking of data. See also pages 8 and 11).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied reference(-s) in view of same to have obtained the above, motivated to optimize rendering in view of architecture and rendering environments.
Claim(s) 12 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Holmes (U.S. Patent App. Pub. No. 2020/0090396 A1).
Regarding claim 12:
It would have been obvious for one of ordinary skill in the art to have combined and modified the applied reference(-s), in view of same, to have obtained: the method of claim 1, further comprising: receiving gaze data indicative of a gaze location of a user for the second image frame,
wherein rendering the at least part of the second image frame comprises: rendering a first part of the second image frame, corresponding to the gaze location of the user, using the second rendering model; and
rendering a second part of the second image frame using a third rendering model of the plurality of rendering models, wherein the third rendering model has a lower associated resource usage than the second rendering model, and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A).
Holmes teaches that foveated rendering is known, which varies rendering based on gaze of a user. See paras. 49, 65, 66. A first part, corresponding to gaze location, will be rendered at a higher level of detail and/or higher resolution (para. 26, 49), and other regions (i.e. a second part) rendered at a lesser level of detail/resolution. Id. Modifying the applied references, such to apply foveated rendering to the system/methods of Wang, teaches the above gaze tracking, first and second parts, and separate rendering models, as claimed.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 13:
It would have been obvious for one of ordinary skill in the art to have further modified the applied reference(-s), in view of same, to have obtained: the method of claim 12, further comprising selecting the third rendering model, wherein selecting the third rendering model comprises: inputting the input data to a second machine learning model trained to select a rendering model, amongst a plurality of rendering models, for use in rendering an image frame to minimise resource usage in dependence on the properties of a processor for rendering the image frame; and
selecting, by the second trained machine learning model, the third rendering model based on the input data, and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A).
Wang teaches that the energy consumption budget threshold can be user determined and set in a number of different ways (see pages. 14-16). This teaches Applicant’s claimed second model, which places more emphasis on energy consumption over image quality, for example.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Sarah Lhymn whose telephone number is (571)270-0632. The examiner can normally be reached M-F, 9:00 AM to 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Sarah Lhymn
Primary Examiner
Art Unit 2613
/Sarah Lhymn/Primary Examiner, Art Unit 2613
Prosecution Insights