DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 7 January 2026 have been fully considered but they are not persuasive.
As to the independent claims, applicant argues that the operations in Nellutla are not generic application-level image-processing operations parameterized by hints from an application, but instead are rendering operations associated with fixed bins in a frame. Applicant further argues that the "information" associated with each bin in Nellutla is load information, which is not an estimated tile dilation size or an estimated data footprint associated with a first operation. Examiner notes that it is the Nama reference, not the Nellutla reference, that is cited for the limitation of the plan for execution being based on an estimated tile dilation size and the estimated data footprint associated with the first operation, wherein the estimated tile dilation size and the estimated data footprint are estimated based on information received from an application that utilized the image. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Applicant argues that even if one were to interpret Nama as involving notions of data size and padding that loosely resemble footprint and dilation, Nama does not describe computing, for a selected first operation among one or more operations that perform on an image, an estimated tile dilation size and an estimated data footprint associated with that specific operation, much less doing so based on information received from an application that utilized the image. Applicant states that the tiling parameters in Nama are derived from internal network-layer parameters rather than application-supplied hints describing arbitrary image operation. Examiner notes that the cited portion of Nama describes the configuration information that is cited to correspond to the claimed values as generated based on the compilation of the application (col. 6, lines 36-45). In other words, based on the information received from the application in Nama, a compiled application and associated configuration information, including the data size and padding information, is computed.
Applicant argues that in applicant’s disclosure, the estimated tile dilation size is a compact per operation metric that characterizes how far an operation reads relative to the location identified by a thread identifier, and the estimated data footprint is a per operation metric that characterizes the total data read and written when executing that
operation. Applicant states that these values are derived from information such as data types, estimated accessed data sizes, data dependencies, compression ratios, and indications of one-to-one correspondence or dilation patterns, which are provided via hints from the application. In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the specificity of elements being argued is recited as part of the overall disclosure, but not recited in the rejected claims. Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Applicant argues that Nellutla divides a frame into bins and schedules rendering work for those bins, but there is no teaching that an application-level sequence of operations is recorded once and then replayed for each tile in accordance with a separate auto-tiling plan derived from per operation estimates. Examiner notes that with respect to the “replaying” concept, claim 1 recites “executing the first operation by replaying instructions to be executed on the image to be divided into a plurality of tiles according to the auto-tiling plan”. In Nellutla, execution takes place in a number of passes each including load, rendering, and store instructions (section 0053), appearing to fit the claimed replaying of instructions.
As to claims 3 and 21, applicant argues that in the present application,
these hints are explicit metadata provided from the application through an API
extension, and they are used to estimate the total data footprint for the operation so
that a number of tiles can be chosen to keep the footprint within cache, while Nama's
configuration information is generated from the network itself and its layer
parameters. Examiner notes that the configuration information in Nama is determined in association with the compiled application, as discussed above and also in the rejection. As to claims 6 and 22, applicant argues that hints support the computation of a compact scalar tile dilation size value which characterizes the reach of the operation relative to thread position. Examiner notes that hints being explicit metadata and supporting the computation of a compact scalar tile dilation size value which characterizes the reach of the operation relative to thread position may be described in the overall disclosure, but claims 3, 6, 21, and 22 do not require the hints to have these specific characteristics.
As to claim 4, applicant argues that Nellutla does not compute a per operation data footprint metric from application hints and then select the number of tiles so that the data resulting from execution of that operation fits into local cache, instead using internal load estimates to select power modes and bin ordering for fixed bins. Examiner notes that claim 4 does not require a metric from application hints, instead requiring a plan specifying a number of tiles selected such that resulting data from an operation executed according to the plan fits into a local cache. Nellutla discloses, at section 0052, that the rendering target is broken into specifically sized tiles/bins, which would correspond to a certain number of tiles to make up the whole image. The numbers must be pre-planned (reading on part of an “auto-tiling plan”) such that each bin can be assigned a corresponding portion of a stream.
As to claim 9, applicant argues that the present application describes a specific auto-tiling extension in the API, through which chains of operations are recorded once in a command list and tagged for auto-tiling, while Nellutla's instruction streams are not described as an auto-tiling command list in an API extension that contains operations flagged for auto-tiling based on application hints. Examiner notes that claim 9 does not require operations flagged for auto-tiling based on application hints, instead requiring storing an auto-tiling command list in an application programming interface, the auto-tiling command list comprising operations to be selected for auto-tiling execution; and selecting the first operation for auto-tiling execution from the auto-tiling command list. In Nellutla, commands are stored and executed via API (see, for example, section 0046) and operations are automatically sorted and selected based on what can be grouped or an amount of processing necessary. Operations are then performed based on how the list of operations has been sorted.
As to claims 7 and 17, applicant argues that Bolz does not describe computing and storing, for a selected operation, a tile dilation size as a single scalar maximum distance value expressed relative to a location identified by a thread identifier, as recited in the claims. Examiner notes that the claims do not appear to require a single scalar value. In any case, the offset values in section 0094 of the reference are computed as distance values from an origin of a compute tile identified in a thread. In the case of a 4x4 compute tile and 6x6 read region, offsets are -1 through 4, corresponding to a maximum dilation distance/size of 1 in each direction compared to the offsets of 0-3 within the compute tile.
Applicant further states that Bolz does not describe storing such a per operation tile dilation size together with other per operation information and using that value, along with a per operation data footprint derived from application hints, to generate an auto-tiling plan and to replay operation instructions across tiles according to that plan. Examiner notes that these limitations are taught by Nellutla and Nama as explained in above arguments and in the rejection to claim 1.
As to claim 8, applicant argues that Bui does not disclose storing, for image-processing operations that are to be auto-tiled, information received from an application that utilized the image where that information consists of per operation hints describing data types, accessed data sizes, dependencies, compression ratios, or dilation patterns, nor does it disclose using that stored information to estimate tile dilation size and data footprint and to generate an auto-tiling plan for replay. Applicant states that even if Bui is read as showing that metadata may be stored together with other data, it does not supply the specific type of information or the auto-tiling context recited in claim 8. Examiner notes that the specific type of data claimed is taught in the Nellutla and Nama references. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
As to claim 10, applicant argues Minkin does not describe storing an application level sequence of image operations together with application-supplied hints, does not teach using those hints to estimate per operation tile dilation size and data footprint, and does not disclose generating an auto-tiling plan and replaying the recorded operations multiple times across tiles according to that plan. Examiner notes that these limitations are taught by Nellutla and Nama as explained in above arguments and in the rejection to claim 1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 3, 4, 6, 9, 11, 13, 14, 16, and 18-24 are rejected under 35 U.S.C. 103 as being unpatentable over Nellutla (U.S. Publication 2019/0221009) in view of Nama (U.S. Patent 11,195,080).
As to claim 1, Nellutla discloses a method of auto-tiled workload processing comprising:
selecting for execution a first operation among one or more operations to perform on an image; generating an auto-tiling plan for execution of the first operation; and executing the first operation by replaying instructions to be executed on the image to be divided into a plurality of tiles according to an auto-tiling plan (p. 2, section 0029; p. 5, section 0052-p. 6, section 0053; p. 6, sections 0056-0057; operations are performed/executed in a group and selected in an order based on how the image has been tiled/binned; the auto-tiling is performed based on the size of a cache; load, rendering, and store instructions are replayed on successive rendering passes).
Nellutla does not disclose, but Nama discloses wherein the plan for execution is based on an estimated tile dilation size and the estimated data footprint associated with the first operation (col. 36, line 46-col. 37, line 24; col. 38, lines 4-51; for an operation, the total size of the data, which reads on a footprint of the data is used to determine a tiling configuration plan including number of tiles and tile size; further, the determined/estimated padding size for the tile in processing, which reads on the size of tile expansion/dilation, can also be used to determine tiling configuration), wherein the estimated tile dilation size and the estimated data footprint are estimated based on information received from an application that utilized the image (col. 6, lines 36-45; col. 7, lines 6-33; col. 20, line 46-col. 21, line 20; configuration files, including configuration of size of overlapping padded tiles, are generated from a compiled application that indicates dimensions of the image tensor data as metadata, reading on information that is used to determine padding/dilation size and image data size/footprint). The motivation for this is that convolution can reduce dimensionality and different processes can affect the input/output size in different ways, and would need to be compensated in particular ways- for example, a non-tiled image would need to be compensated differently from an image that is padded/dilated then tiled (col. 14, lines 8-54). It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to modify Nellutla to have the plan for execution based on an estimated tile dilation size and the estimated data footprint associated with the first operation, wherein the estimated tile dilation size and the estimated data footprint are estimated based on information received from an application that utilized the image in order to compensate for reduction in dimensionality by different processes as taught by Nama.
As to claim 3, Nellutla does not disclose, but Nama discloses wherein the information received from the application comprises a hint, and wherein the estimated data footprint is estimated based on the hint (col. 6, lines 36-45; col. 7, lines 6-33; col. 20, line 46-col. 21, line 20; configuration files are generated from a compiled application that indicates dimensions of the image tensor data as metadata, reading on a hint used to determine image data size/footprint). Motivation for the combination is given in the rejection to claim 1.
As to claim 4, Nellutla discloses wherein the auto-tiling plan specifies a selected number of tiles, and the selected number of tiles is selected such that data, resulting from executing the first operation according to the auto-tiling plan, fits into a local cache (p. 5, section 0052; the size of processed and rasterized tiles/bins, which would include tiles/bins that have at least had a rasterization operation applied, is determined based on the size of the cache).
As to claim 6, Nellutla does not disclose, but Nama discloses wherein the information received from the application comprises a hint, and wherein the estimated tile dilation size is estimated based on the hint (col. 6, lines 36-45; col. 7, lines 6-33; col. 18, line 31-col. 19, line 19; col. 20, line 46-col. 21, line 20; configuration files, including configuration of size of overlapping padded tiles, are generated from a compiled application that indicates dimensions of the image tensor data as metadata, reading on a hint that is used to determine padding/dilation size). Motivation for the combination is given in the rejection to claim 1.
As to claim 9, Nellutla discloses a method further comprising storing an auto-tiling command list in an application programming interface, the auto-tiling command list comprising operations to be selected for auto-tiling execution (p. 4-5, section 0046; p. 5, section 0048; p. 8, section 0074; the API stores the commands); and selecting the first operation for auto-tiling execution from the auto-tiling command list (p. 2, section 0029; p. 5, section 0052-p. 6, section 0053; p. 6, sections 0056-0057; operations are performed in a group and selected in an order based on how the image has been tiled/binned).
As to claim 11, see the rejection to claim 1. Further, Nellutla discloses a processing device (p. 4, section 0043- p. 5, section 0048) used for auto-tiled workload processing comprising: memory comprising cache memory (p. 2, section 0029); and a processor configured to perform the method (p. 4, section 0043- p. 5, section 0048).
As to claim 13, see the rejection to claim 3.
As to claim 14, see the rejection to claim 4.
As to claim 16, see the rejection to claim 6.
As to claim 18, Nellutla discloses a device further comprising a display device, wherein the image is provided for display on the display device (p. 4, section 0041; p. 7, section 0063; p. 8, section 0078; p. 9, sections 0084-0085).
As to claim 19, see the rejection to claim 9.
As to claim 20, see the rejection to claim 1. Further, Nellutla discloses a non-transitory computer-readable storage medium having instructions thereon for causing a computer to execute a method of auto-tiled workload processing (p. 1, section 0008-p. 2, section 0019; p. 4, sections 0043-0044; p. 10, section 0094-0096; p. 12, sections 0124-0125).
As to claim 21, Nellutla does not disclose, but Nama discloses wherein the hint includes at least one of data types, estimates for accessed data sizes, data dependencies or estimated compression ratios (col. 6, lines 36-45; col. 7, lines 6-33; col. 20, line 46-col. 21, line 20; the image tensor dimensions would correspond to an amount of data in pixels that is accessed for that input). Motivation for the combination is given in the rejection to claim 1.
As to claim 22, Nellutla does not disclose, but Nama discloses wherein the hint includes an indication whether a 1: 1 correspondence exists between a portion of an input image and a portion of an output image (col. 14, lines 8-14; an indication that that input and output are the same size, i.e. 1:1 correspondence between pixels, is a hint used to allow the padding/size to increase to compensate for convolution operations). Motivation for the combination is given in the rejection to claim 1.
As to claim 23, see the rejection to claim 21.
As to claim 24, see the rejection to claim 22.
Claims 7 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Nellutla in view of Nama and further in view of Bolz (U.S. Publication 2018/0165787).
As to claim 7, Nellutla does not disclose, but Bolz discloses wherein the estimated tile dilation size is expressed as maximum distance relative to a location indicated by a thread identifier (p. 8, section 0088-p. 9, section 0094; a thread operation requires caching of tiles outside the compute tile by some specified distance; for example, offsets are defined by the shader executing the thread that correspond to distances from the origin of the compute tile; for a 4x4 render tile requiring a 6x6 tile, the size would correspond to a maximum of 1 pixel outside the render tile in each direction). The motivation for this is to execute filtering operations that require adjacent pixels, such as blur operations. It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to modify Nellutla and Nama to have the estimated tile dilation size be expressed as maximum distance relative to a location indicated by a thread identifier in order to execute filtering operations that require adjacent pixels, such as blur operations as taught by Bolz.
As to claim 17, see the rejection to claim 7.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Nellutla in view of Nama and further in view of Bui (U.S. Publication 2020/0042286).
As to claim 8, Nellutla discloses storing, for the one or more operations, instructions to be executed on the image (p. 2, section 0029; p. 4, section 0043-0045; p. 5, sections 0047-0049; p. 5, section 0052; command/instruction streams are generated and compiled into a program for execution, stored in a system or other memory, for an image that is divided into a number of bins/tiles). Nellultla does not expressly disclose, but Bui discloses storing the information received from the application when the instructions for the one or more operations are stored (p. 10, sections 0092-0095; information including image information from a module/application associated with commands/instructions for an operation are stored together at the same time with the commands/instructions). The motivation for this is to train a model and reduce user effort for editing operations. It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to modify Nellutla and Nama to store information when the instructions for the operations are stored in order to train a model and reduce user effort for editing operations as taught by Bui.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Nellutla in view of Nama and further in view of Minkin (U.S. Publication 2022/0309336).
As to claim 10, Nellutla discloses executing operations according to the auto-tiling plan as discussed in the rejection to claim 1. Nellutla does not disclose, but Minkin discloses storing, a single time, the instructions for each of the one or more operations to be executed on the image; and executing one or more of the operations multiple times on the image (p. 38, sections 0392-0393; using SIMD or SIMT, a single instruction is stored and used to operate on the input data, which in this case is the image data, multiple times). The motivation for this is to support parallel execution of a large number of threads without providing multiple independent instruction units. It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to modify Nellutla and Nama to store, a single time, the instructions for each of the operations to be executed on the image and execute one or more of the operations multiple times on the image in order to support parallel execution of a large number of threads without providing multiple independent instruction units as taught by Minkin.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AARON M RICHER whose telephone number is (571)272-7790. The examiner can normally be reached 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571)272-7440. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AARON M RICHER/Primary Examiner, Art Unit 2617