DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant claims benefit of a prior-filed U.S. Provisional Application No. 62348802, filed on 06/10/2016, which is acknowledge by the examiner.
Drawings
The drawings were received on 06/09/2017. These drawings are acceptable.
Claim Interpretation
Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Examiner notes that the following terms as claimed are treated as data processing elements and include the following interpretations among those cited by the prior, as the specification nor the claims have limited the interpretation of noted terms to any specific embodiments or preferred interpretations:
Stack: a collection or set of data element or any grouping of data
See example in specification paragraphs 0026-0028: Figure 1 is a block diagram illustrating exemplary components of a system for reducing processor demand during convolution using data packing. As illustrated, input channels 102 contain multiple layers 1 to IC=Input Channel images also referred to as an image stack.… In one embodiment, a pre-processor component 108 processes the image stack contained in the input channels to generate packed input 110. The packed input 110 is contained in an input channel block to contain as much of the image stack of data as is needed to insure that the pipeline of data needed for convolution is continuous and to reduce the convolution processor 112 demands during convolution…
Kernel: any element capable of being executed or process by a computing element/system/unit/process
Block: is a data element that includes a sequential set of elements/items/data/space to form it of any N-dimensional space
Packing/packed: any data processing technique/activity/function or any process for configuring data for processes or as a result of processing. Examiner does not consider this a term of art and the applicant has not provided sufficient evidence on record (e.g. in the original disclaimer or filed remarks) that make clear what one of ordinary skill in the art (e.g. a computer scientist) would understand as a standard meaning/definition of term.
In the instant case the applicant has not made the claim language clear regarding which interpretation is being required. Examiner relies on the broadest reasonable interpretation (BRI) in light of specification paragraph, see MPEP 2111:
0008: In one embodiment data packing includes input data packing and output data packing. Input data packing includes pre-processing input data representing a digital image signal into an input channel block of contiguous memory. Output data packing includes convolving the input data representing the digital image signal into an output channel block of contiguous memory sized in accordance with an architecture of the convolution processor.
Additionally, the MPEP notes that “[A] patentee can act as his own lexicographer to specifically define terms of a claim contrary to their ordinary meaning," in such a situation the written description must clearly redefine a claim term "so as to put a reasonable competitor or one reasonably skilled in the art on notice that the patentee intended to so redefine that claim term."); Hormone Research Foundation Inc. v. Genentech Inc., 904 F.2d 1558, 15 USPQ2d 1039 (Fed. Cir. 1990). Accordingly, when there is more than one meaning for a term, it is incumbent upon applicant to make clear which meaning is being relied upon to claim the invention.” See MPEP 2173.05
Examiner recommends that applicant clarify claim limitation to make clear the intended scope as the MPEP allows for broadest reasonable interpretation, in light of the specification, see MPEP 2111.
Response to Arguments
Applicant's arguments filed 06/03/2025 have been fully considered and found unpersuasive.
Regarding the applicant remarks directed to the rejection of claims under 35 USC 112, upon further consideration of amendments the issues has been resolved; and the rejection made in the previous office action has been withdrawn.
Regarding the applicant remarks directed to the rejection of claims under 35 USC 103, the Applicant’s arguments with respect to claims 1-20 have been considered.
Applicant arguments are directed to the amended claim limitations that were not previously examined by the examiner. See current office action that addresses the claim amendments.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation "the output block size" in the claim limitation. There is insufficient antecedent basis for this limitation in the claim as it is unclear which recitation of an output block size the recitation refers back to in the limitations “determining an output block size based on the processor architecture; determining an input block size based on an output block size…”
Regarding independent claims 7 and 15, the claims recite similar limitations to does noted above and are thus rejected under the same rationale.
Regarding the dependent claims that depend on claims 1, 7, and 15 respectively, the claims fail to resolve the noted deficiencies above and are thus rejected under the noted rationale.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 7, and 15 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Yang et al. (US 20160342888, hereinafter ‘Yang’).
Regarding independent claim 1 limitation, Yang teaches a computer-implemented method of managing data for convolution processing, the method comprising: (in Abstract: Aspects of the present disclosure are directed to techniques that improve performance of CNN systems through the effect of improved memory efficiencies for CNNs operating on GPUs [a computer-implemented method of managing data for convolution processing, the method comprising]…)
determining a usable number of registers in a processor architecture of a device, the device comprising a memory, an input channel for receiving an input stack of input data, an output channel for receiving an output stack of output data, and a convolution kernel containing a stack of weights for convolving the stack of input data into the stack of output data; in 0037: In order to accelerate the CNN learning process, many-core architectures including GPUs have been employed in state-of-art CNN frameworks [determining a usable number of registers in a processor architecture of a device, the device comprising a memory] such as Caffe …, cuda-convnet … Of particular interest, it has been shown that the GPU-based implementation can achieve a performance improvement of 10 to 60 times over single instruction, multiple data (SIMD) central processing units (CPUs). Additionally, Nvidia has recently released a library—cuDNN—to accelerate a set of core CNN layers on GPUs. And in 0068-0069: We now discuss our experimental methodology before our characterizations and optimizations for deep learning applications. Since deep learning frameworks—i.e., Caffe, cuda-convnet and cuDNN have been commonly used and specifically optimized for GPGPUs, we describe memory efficiency on three major layers including the convolutional layer, the pooling layer and the softmax layer while employing these frameworks/library. Note that Caffe selects NCHW data layout [determining a usable number of registers in a processor architecture of a device, the device comprising a memory] and implements the convolutional layer on top of cuBLAS, and cuda-convnet, uses CHWN as its data layout [determining a usable number of registers in a processor architecture of a device, the device comprising a memory]…; And in [0070] … Table 1 shows the configurations for the benchmarking layers of the five networks. With reference to that table, from the left to right, for a convolutional layer it shows the number of images (Ni) [an input channel for receiving an input stack of input data,], the number of output feature channels (Co) [an output channel for receiving an output stack of output data,], the height/width of each image (H/W), width/height of each kernel filter (Fw IFh) [and a convolution kernel containing a stack of weights for convolving the stack of input data into the stack of output data], the number of input feature channels (C9), and the stride (S)…)
determining an output block size based on the processor architecture; determining an input block size based on an output block size and a convolution weight size; (in 0052-0053: In order to utilize the massive computation power provided by GPGPUs, CNN frameworks oftentimes process multiple images in a batch. In this way, the input of a convolutional layer includes multiple images as a 4D array. The algorithm [based on the processor architecture] in the prediction stage proceeds as follows in equation [1].
PNG
media_image1.png
140
578
media_image1.png
Greyscale
Where in is the input array, filter is the weights/parameters of the layer [determining an input block size based on an output block size and a convolution weight size] and out is the output array [determining an output block size based on the processor architecture]. As shown in Equation [1], all three arrays are 4D arrays. For the input array, in, the first dimension is the number of image, and following it are the number of input feature channels, the height and weight of each input image. For the synapse matrix, which is also a four-dimensional matrix, the first dimension is the number of output feature channels, the second dimension is the number of input feature channels, and the other are the size of the 2D convolution filter kernel. The output is also four-dimensional matrix. However as compared to the input, the second dimension is the number of the output channels instead of the number of the input channels.
Examiner notes the out array block as recited in equation 1 is computed by determining an output block size based on the processor architecture and in array block as recited in equation 1 is computed by determining an input block size based on an output block size and a convolution weight size; )
determining dimensions of neighboring pixels within the input stack to be used to determine each output value in the output stack; packing the input data from the input stack into a continuous block of register memory based at least in part on the dimensions of the neighboring pixels and the input block size; (in 0063-0065 First, as we have discovered, data layout [determining dimensions of neighboring pixels within the input stack to be used to determine each output value in the output stack] has a significant impact on GPU memory bandwidth. Using matrix-vector multiplication (MV) as example, we have two data layouts to store an input matrix. One layout is a column-major format and the other layout is a row-major format. As may be appreciated, each output pixel of MV results from a product of a row of the input matrix [determining dimensions of neighboring pixels within the input stack to be used to determine each output value in the output stack] and the vector, while a straightforward implementation of MV allows one thread to compute one output pixel. Therefore, if we are employing the row-major format, each thread needs to access a number of consecutive pixels in memory, and different threads in a warp will access different rows without satisfying coalesced memory requirement(s)... However—and as will now be readily appreciated by those skilled in the art—selecting a best data layout for an application is not a trivial problem—especially for the deep learning algorithm which uses four-dimensional arrays. As may be observed in Equation (1), four dimensions are used to identify the Number of images, the feature Channels, the Height, and the Width. Note that the nomenclature used in Equation (1) uses abbreviations for each dimension using its first capital letter. Therefore, the data layout in Equation (1) is NCHW for the arrays, in [packing the input data from the input stack into a continuous block of register memory based at least in part on the dimensions of the neighboring pixels and the input block size;] and out. For such a four-dimensional array, there are overall 24 (i.e., 4!) data layouts. If we consider that different layers can have different data layouts, the number of choices can easily reach thousands. Therefore, the selection of data layout is a challenge for developers. Note that existing frameworks choose to use a fixed data layout. For example, cuda-convnet uses CHWN, while Caffe and cuDNN use NCHW.)
packing the convolution kernel into another continuous block of register memory; convolving the input stack into the output stack using the stack of weights in the convolution kernel; and packing the output stack using the output block size. (in 0052-0053: In order to utilize the massive computation power provided by GPGPUs, CNN frameworks oftentimes process multiple images in a batch. In this way, the input of a convolutional layer includes multiple images as a 4D array. The algorithm [based on the processor architecture … using the output block size] in the prediction stage proceeds as follows in equation [1] [convolving the input stack into the output stack using the stack of weights in the convolution kernel;].
PNG
media_image1.png
140
578
media_image1.png
Greyscale
Where in is the input array, filter is the weights/parameters of the layer and out is the output array. As shown in Equation [1], all three arrays are 4D arrays. For the input array, in, the first dimension is the number of image, and following it are the number of input feature channels, the height and weight of each input image. For the synapse matrix, which is also a four-dimensional matrix, the first dimension is the number of output feature channels, the second dimension is the number of input feature channels, and the other are the size of the 2D convolution filter kernel [for packing filter recited in equation 1 as claimed packing the convolution kernel into another continuous block of register memory …; including claimed the stack of weights in the convolution kernel]. The output is also four-dimensional matrix [packing the output stack using the output block size]. However as compared to the input, the second dimension is the number of the output channels instead of the number of the input channels.
Examiner notes the filter array block as recited in equation 1 is computed by packing the convolution kernel into another continuous block of register memory; and equation 1 computing
And in 0065... However—and as will now be readily appreciated by those skilled in the art—selecting a best data layout for an application is not a trivial problem—especially for the deep learning algorithm which uses four-dimensional arrays. As may be observed in Equation (1), four dimensions are used to identify the Number of images, the feature Channels, the Height, and the Width. Note that the nomenclature used in Equation (1) uses abbreviations for each dimension using its first capital letter. Therefore, the data layout in Equation (1) is NCHW for the arrays, in [based on the processor architecture … using the output block size] and out. For such a four-dimensional array, there are overall 24 (i.e., 4!) data layouts. If we consider that different layers can have different data layouts, the number of choices can easily reach thousands. Therefore, the selection of data layout is a challenge for developers. Note that existing frameworks choose to use a fixed data layout. For example, cuda-convnet uses CHWN, while Caffe and cuDNN use NCHW [based on the processor architecture … using the output block size].)
Regarding independent claims 7 and 15, the limitations are similar to claim 1 limitations and are thus rejected under the same rationale.
Claims 1, 7, and 15 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Ovsiannikov et al. (US 9411726, hereinafter ‘Niko’).
Regarding independent claim 1 limitation, Niko teaches a computer-implemented method of managing data for convolution processing, the method comprising: (in 4:36-40: FIG. 1C is a schematic view of an operation according to an embodiment. Referring to FIGS. 1B and 1C, in an embodiment, the operation performed by a circuit 106 may be part of a convolution [a computer-implemented method of managing data for convolution processing, the method comprising]. In an embodiment, classification algorithms utilizing convolutional networks may use large amounts of calculations…; And in 14:14-54: FIG. 18 is a schematic view of an electronic system [a computer-implemented method of managing data for convolution processing, the method comprising] which may include a system according to an embodiment. The electronic system 1800 may be part of a wide variety of electronic devices including, but not limited to portable notebook computers, Ultra-Mobile PCs (UMPC), Tablet PCs, servers, workstations, mobile telecommunication devices, and so on. Moreover, the electronic system 1800 may be implemented in part using a system-on-chip architecture... The memory system 1812 may be configured to store codes for operating the processor 1814, data processed by the processor 1814, or externally input data. The memory system 1812 may include a controller and a memory. The memory system may include an interface to computer readable media. Such computer readable media may store instructions [a computer-implemented method of managing data for convolution processing, the method comprising] to perform the variety of operations describe above.)
determining a usable number of registers in a processor architecture of a device, the device comprising a memory, an input channel for receiving an input stack of input data, an output channel for receiving an output stack of output data, and a convolution kernel containing a stack of weights for convolving the stack of input data into the stack of output data; (in 5:1-26: In convolution, a Krow×Kcol kernel window scans maps, input vectors, images, or the like in the previous layer performing [an input channel for receiving an input stack of input data] a multiply-accumulate operation with the convolution kernel weights [a convolution kernel containing a stack of weights for convolving the stack of input data into the stack of output data]. Convolution results from multiple input maps are added together. A circuit 106 may be configured to perform such convolution to generate a pixel of a map in a current layer m [an input channel for receiving an input stack of input data] based on one or more maps of a previous layer, layer m−1 [an output channel for receiving an output stack of output data]. Another circuit 106 may be configured to perform a non-linear operation on the pixels of the newly generated map in layer m. Another circuit 106 may be configured to perform the pooling operation to reduce a resolution of a map in layer m. In particular, one or more circuits 106 [determining a usable number of registers in a processor architecture of a device, the device comprising a memory] may be configured to operate as a convolution multiply-accumulate arithmetic unit. The data stored in the memory 102 [determining a usable number of registers in a processor architecture of a device, the device comprising a memory] may be data representing pixels of an image or map. Here, in layer m−1, multiple maps 150 have multiple pixels…. The result of the operation may be output to the memory 102 and/or another circuit 106. That is, the map 156 or other maps may be stored in the memory 102 and/or may exist as a stream of data passing through the circuits 106...; And in 6:62-7:3: The memory 114 may be configured to store values [determining a usable number of registers in a processor architecture of a device, the device comprising a memory]. The memory may be a set of registers […a usable number of registers in a processor architecture of a device, the device comprising a memory], static memory, dynamic memory, or the like. Any circuit that may store values may be used as the memory 114. The multiplier 112 may be configured to multiply an input value 111 by a value 117 output from the memory 114. The product 113 may be stored in the accumulator 110, added to an existing value in the accumulator 110, or the like. The value of the accumulator 110 may be output as value 115…)
determining an output block size based on the processor architecture; determining an input block size based on an output block size and a convolution weight size; (As depicted in Fig. 1C:
PNG
media_image2.png
486
870
media_image2.png
Greyscale
And in 4:61-5:26: A convolutional network may be organized into multiple layers. Here, examples of layers m−1, m, and m+1 are illustrated; however, any number of layers may be used. Each layer may contain one or more “maps” or images, which are representations of information in the previous layer. To compute a map, each layer may perform 3 steps: a convolution, a non-linear transform, and pooling. In convolution, a Krow×Kcol kernel window scans maps, input vectors, images, or the like in the previous layer performing a multiply-accumulate operation with the convolution kernel weights. Convolution results from multiple input maps are added together. A circuit 106 may be configured to perform such convolution to generate a pixel of a map in a current layer m based on one or more maps of a previous layer, layer m−1. Another circuit 106 may be configured to perform a non-linear operation on the pixels of the newly generated map in layer m… Here, in layer m−1, multiple maps 150 have multiple pixels. The circuit 106 may be configured to multiply an input pixel stream from the memory 102 by cached weights, accumulate weighted values and output a result when each convolution completes. The output may be stored in a pixel 154 of a map 156 in another layer m. The result of the operation may be output to the memory 102 and/or another circuit 106. That is, the map 156 or other maps may be stored in the memory 102 and/or may exist as a stream of data passing through the circuits 106.)
determining dimensions of neighboring pixels within the input stack to be used to determine each output value in the output stack; (in Ass depicted in Fig. 1C And 6:2-8: … For example, the memory 102 may be configured to stream data, such as pixels of a row of an image [determining dimensions of neighboring pixels within the input stack to be used to determine each output value in the output stack], to various circuits. The circuits 106 may, in turn, be configured to output another stream of data/pixels. This stream may be an input to another circuit 106 or group of circuits that generate another stream….)
packing the input data from the input stack into a continuous block of register memory based at least in part on the dimensions of the neighboring pixels and the input block size; packing the convolution kernel into another continuous block of register memory; (As depicted in Fig. 1C:
PNG
media_image2.png
486
870
media_image2.png
Greyscale
And in 4:61-5:26: A convolutional network may be organized into multiple layers. Here, examples of layers m−1, m, and m+1 are illustrated; however, any number of layers may be used. Each layer may contain one or more “maps” or images, which are representations of information in the previous layer. To compute a map, each layer may perform 3 steps: a convolution, a non-linear transform, and pooling. In convolution, a Krow×Kcol kernel window scans maps, input vectors, images, or the like in the previous layer performing a multiply-accumulate operation with the convolution kernel weights. Convolution results from multiple input maps are added together. A circuit 106 may be configured to perform such convolution to generate a pixel of a map in a current layer m based on one or more maps of a previous layer, layer m−1. Another circuit 106 may be configured to perform a non-linear operation on the pixels of the newly generated map in layer m… Here, in layer m−1, multiple maps 150 have multiple pixels. The circuit 106 may be configured to multiply an input pixel stream from the memory 102 by cached weights [packing the convolution kernel into another continuous block of register memory], accumulate weighted values and output a result when each convolution completes. The output may be stored in a pixel 154 of a map 156 in another layer m [packing the input data from the input stack into a continuous block of register memory based at least in part on the dimensions of the neighboring pixels and the input block size]. The result of the operation may be output to the memory 102 and/or another circuit 106. That is, the map 156 or other maps may be stored in the memory 102 and/or may exist as a stream of data passing through the circuits 106. And in 7:4-44: In a particular example, this circuit 106a may be configured to perform a multiply and accumulate function as part of a convolution. The memory 114 may be configured to store weights for the convolution [packing the convolution kernel into another continuous block of register memory]. The appropriate weight may be output as value 117 from the memory when the corresponding value is present as the input value 111. The accumulator 110 may be configured to create a running sum of the products 113 for different value and weight combinations. Once all of the products of the associated convolution have been performed, the output value 115 may be generated representing a value of the result of the convolution… Although the memory 114 and the memory 116 are illustrated as separate, the memories 114 and 116 may be combined together. Thus, in an embodiment, a single memory may be configured to store weights for a convolution and the running accumulated values, or perform other storage functions that the memories 114 and 116 would separately perform.)
convolving the input stack into the output stack using the stack of weights in the convolution kernel; and packing the output stack using the output block size. (in As depicted in Fig. 5 And in 5:1-25: In convolution, a Krow×Kcol kernel window scans maps, input vectors, images, or the like in the previous layer performing a multiply-accumulate operation with the convolution kernel weights [convolving the input stack into the output stack using the stack of weights in the convolution kernel]. Convolution results from multiple input maps are added together [packing the output stack using the output block size]. A circuit 106 may be configured to perform such convolution to generate a pixel of a map in a current layer m [packing the output stack using the output block size] based on one or more maps of a previous layer, layer m−1… The output may be stored in a pixel 154 of a map 156 in another layer m. The result of the operation may be output to the memory 102 and/or another circuit 106. That is, the map 156 or other maps may be stored in the memory 102 and/or may exist as a stream of data passing through the circuits 106…)
Regarding independent claims 7 and 15, the limitations are similar to claim 1 limitations and are thus rejected under the same rationale.
Claims 1, 7, and 15 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Chung et al. (US 20160379109, hereinafter ‘Chung’).
Regarding independent claim 1 limitation, Chung teaches a computer-implemented method of managing data for convolution processing, the method comprising: (in Abstract: A hardware acceleration component is provided for implementing a convolutional neural network [a computer-implemented method of managing data for convolution processing, the method comprising]...; And in [0059] FIG. 1 shows an overview of a data processing system 102 that includes a software plane 104 and a hardware acceleration plane 106. Software plane 104 includes a collection of software-driven components [a computer-implemented method of managing data for convolution processing, the method comprising] (each denoted by the symbol “S” in FIG. 1), whereas hardware acceleration plane 106 includes a collection of hardware acceleration components (each denoted by the symbol “H” in FIG. 1.). [0060] For instance, a software-driven host component may correspond to a server computer that executes machine-readable instructions using one or more central processing units (CPUs). Each CPU, in turn, may execute the instructions on one or more hardware threads…)
determining a usable number of registers in a processor architecture of a device, the device comprising a memory, an input channel for receiving an input stack of input data, an output channel for receiving an output stack of output data, and a convolution kernel containing a stack of weights for convolving the stack of input data into the stack of output data; (in 0256-00259 As used herein, a “convolutional neural network” or (CNN) refers to a neural network that includes one or more convolutional layers [determining a usable number of registers in a processor architecture of a device, the device comprising a memory], and also may include pooling and/or non-linearization operations applied after some or all of the convolutional layers. FIG. 42 is a simplified diagram depicting a three-dimensional (3D) CNN 4202 that includes three 3D volumes (4204, 4206, 4208). Each 3D volume (4204, 4206, 4208) represents an input to a layer [an input channel for receiving an input stack of input data,], and is transformed into a new 3D volume feeding a subsequent layer [an output channel for receiving an output stack of output data]. In the example of FIG. 42, there are two convolutional layers 4210 and 4212. Volume 4204 (with 3 planes) is an input to convolutional layer 4210, which generates volume 4206 (with H planes), which in turn is an input to convolutional layer 4212, which generates volume 4208 (with K planes).In an implementation, volume 4204 includes image data in D=3 planes (e.g., r, g, b), with each plane including an J×J array of image data (e.g., pixel data). Persons of ordinary skill in the art will understand that more or fewer than three planes may be used, that each plane need not include a square array, and that data other than image data may be used. A 3D input volume 4214 of dimensions L×L×D is convolved with H weight kernels [and a convolution kernel containing a stack of weights for convolving the stack of input data into the stack of output data] (e.g., weight kernel 4216) of dimension L×L×D and stride S. Each weight kernel (e.g., weight kernel 4216) is shifted in a sliding-window-like fashion (with a shift offset defined by stride S) across the input volume (e.g., volume 4204). During each shift, each weight in to the 3D weight kernel is multiplied and added with corresponding pair-wise input elements from the overlapping region of input volume 4214…)
determining an output block size based on the processor architecture; (in Abstract: A hardware acceleration component is provided for implementing a convolutional neural network. The hardware acceleration component includes an array of N rows and M columns of functional units, an array of N input data buffers configured to store input data, and an array of M weights data buffers configured to store weights data. Each of the N input data buffers is coupled to a corresponding one of the N rows of functional units. Each of the M weights data buffers is coupled to a corresponding one of the M columns of functional units. Each functional unit in a row is configured to receive a same set of input data. Each functional unit in a column is configured to receive a same set of weights data from the weights data buffer coupled to the row. Each of the functional units is configured to perform a convolution of the received input data and the received weights data, and the M columns of functional units are configured to provide M planes of output data [determining an output block size based on the processor architecture;].)
determining an input block size based on an output block size and a convolution weight size; (in 0257-0262: FIG. 42 is a simplified diagram depicting a three-dimensional (3D) CNN 4202 that includes three 3D volumes (4204, 4206, 4208). Each 3D volume (4204, 4206, 4208) represents an input to a layer, and is transformed into a new 3D volume feeding a subsequent layer. In the example of FIG. 42, there are two convolutional layers 4210 and 4212. Volume 4204 (with 3 planes) is an input to convolutional layer 4210, which generates volume 4206 (with H planes), which in turn is an input to convolutional layer 4212, which generates volume 4208 (with K planes). In an implementation, volume 4204 includes image data in D=3 planes (e.g., r, g, b) with each plane including an J×J array of image data (e.g., pixel data). Persons of ordinary skill in the art will understand that more or fewer than three planes may be used, that each plane need not include a square array, and that data other than image data may be used. A 3D input volume 4214 of dimensions L×L×D is convolved with H weight kernels (e.g., weight kernel 4216) of dimension L×L×D and stride S [determining an output block size based on the processor architecture]. Each weight kernel (e.g., weight kernel 4216) is shifted in a sliding-window-like fashion (with a shift offset defined by stride S) across the input volume (e.g., volume 4204). During each shift, each weight in to the 3D weight kernel is multiplied and added with corresponding pair-wise input elements from the overlapping region of input volume 4214. For example, FIGS. 43A-43D show an example convolution of a first volume 4302 with a first set of kernel weights 4304 to generate a first plane 4308x of a second volume 4308 (shown in FIG. 43D), and also shows convolution of first volume 4302 with a second set of kernel weights 4306 to generate a second plane 4308y of a second volume 4308. First volume 4302 includes 3 planes (4302r, 4302g, 4302b) with each plane including a 9×9 array of image data. Each of first set of kernel weights 4304 and second set of kernel weights [and a convolution weight size] 4308 has a dimension 3×3×3. Second volume 4308 includes H planes (4308x, 4308y, . . . , 4308q), with each plane including a 4×4 array of data. As illustrated in FIG. 43A, data value x0 of first plane 4308x of second volume 4308 is determined by multiplying every weight in first set of kernel weights 4304 with every pair-wise input element from the overlapping region of a first input volume (4302r.sub.0, 4302g.sub.0, 4302b.sub.0) [determining an input block size based on an output block size and a convolution weight size], as follows: First set of kernel weights 4304 then slides S=2 horizontal data values of first volume 4302. As illustrated in FIG. 43B, data value x1 of first plane 4308x of second volume 4308 is determined by multiplying every weight in first set of kernel weights 4304 with every pair-wise input element from the overlapping region of a second input volume (4302r.sub.1, 4302g.sub.1, 4302b.sub.1), as follows: …. )
determining dimensions of neighboring pixels within the input stack to be used to determine each output value in the output stack; (in 0257-0262: FIG. 42 is a simplified diagram depicting a three-dimensional (3D) CNN 4202 that includes three 3D volumes (4204, 4206, 4208). Each 3D volume (4204, 4206, 4208) represents an input to a layer, and is transformed into a new 3D volume feeding a subsequent layer. In the example of FIG. 42, there are two convolutional layers 4210 and 4212. Volume 4204 (with 3 planes) is an input to convolutional layer 4210, which generates volume 4206 (with H planes) [determining dimensions of neighboring pixels within the input stack to be used to determine each output value in the output stack;], which in turn is an input to convolutional layer 4212, which generates volume 4208 (with K planes). In an implementation, volume 4204 includes image data in D=3 planes (e.g., r, g, b), with each plane including an J×J array of image data (e.g., pixel data) [determining dimensions of neighboring pixels within the input stack to be used to determine each output value in the output stack;]. Persons of ordinary skill in the art will understand that more or fewer than three planes may be used, that each plane need not include a square array, and that data other than image data may be used. A 3D input volume 4214 of dimensions L×L×D is convolved with H weight kernels (e.g., weight kernel 4216) of dimension L×L×D and stride S. Each weight kernel (e.g., weight kernel 4216) is shifted in a sliding-window-like fashion (with a shift offset defined by stride S) across the input volume (e.g., volume 4204) [determining dimensions of neighboring pixels within the input stack to be used to determine each output value in the output stack]. During each shift, each weight in to the 3D weight kernel is multiplied and added with corresponding pair-wise input elements from the overlapping region of input volume 4214…, And in [0264] This process continue until all data values of second plane 4308y of second volume 4308 are complete, and also continues for each of the H weight volumes to generate the H planes in of second volume 4308. Referring again to FIG. 42, volume 4206 [to determine each output value in the output stack] then becomes an input layer to convolutional layer 4212, which includes K weight volumes to generate the K planes of volume 4208. )
packing the input data from the input stack into a continuous block of register memory based at least in part on the dimensions of the neighboring pixels and the input block size; packing the convolution kernel into another continuous block of register memory; ( As depicted in Fig. 42 and Fig. 43A-D in 0257-0262: FIG. 42 is a simplified diagram depicting a three-dimensional (3D) CNN 4202 that includes three 3D volumes (4204, 4206, 4208). Each 3D volume (4204, 4206, 4208) represents an input to a layer, and is transformed into a new 3D volume feeding a subsequent layer. In the example of FIG. 42, there are two convolutional layers 4210 and 4212. Volume 4204 (with 3 planes) is an input to convolutional layer 4210, which generates volume 4206 (with H planes), which in turn is an input to convolutional layer 4212, which generates volume 4208 (with K planes). In an implementation, volume 4204 includes image data in D=3 planes (e.g., r, g, b), with each plane including an J×J array of image data (e.g., pixel data) [packing the input data from the input stack into a continuous block of register memory based at least in part on the dimensions of the neighboring pixels and the input block size]. Persons of ordinary skill in the art will understand that more or fewer than three planes may be used, that each plane need not include a square array, and that data other than image data may be used. A 3D input volume 4214 of dimensions L×L×D is convolved with H weight kernels (e.g., weight kernel 4216) of dimension L×L×D [packing the convolution kernel into another continuous block of register memory] and stride S. Each weight kernel (e.g., weight kernel 4216) is shifted in a sliding-window-like fashion (with a shift offset defined by stride S) across the input volume (e.g., volume 4204). During each shift, each weight in to the 3D weight kernel is multiplied and added with corresponding pair-wise input elements from the overlapping region of input volume 4214. For example, FIGS. 43A-43D show an example convolution of a first volume 4302 with a first set of kernel weights 4304 to generate a first plane 4308x of a second volume 4308 (shown in FIG. 43D), and also shows convolution of first volume 4302 [packing the input data from the input stack into a continuous block of register memory based at least in part on the dimensions of the neighboring pixels and the input block size] with a second set of kernel weights 4306 [ packing the convolution kernel into another continuous block of register memory] to generate a second plane 4308y of a second volume 4308. First volume 4302 includes 3 planes (4302r, 4302g, 4302b) with each plane including a 9×9 array of image data… )
convolving the input stack into the output stack using the stack of weights in the convolution kernel; and packing the output stack using the output block size. (in 0257-0262: FIG. 42 is a simplified diagram depicting a three-dimensional (3D) CNN 4202 that includes three 3D volumes (4204, 4206, 4208). Each 3D volume (4204, 4206, 4208) represents an input to a layer, and is transformed into a new 3D volume feeding a subsequent layer. In the example of FIG. 42, there are two convolutional layers 4210 and 4212. Volume 4204 (with 3 planes) is an input to convolutional layer 4210, which generates volume 4206 (with H planes), which in turn is an input to convolutional layer 4212, which generates volume 4208 (with K planes). In an implementation, volume 4204 includes image data in D=3 planes (e.g., r, g, b), with each plane including an J×J array of image data (e.g., pixel data)... During each shift, each weight in to the 3D weight kernel is multiplied and added with corresponding pair-wise input elements from the overlapping region of input volume 4214. For example, FIGS. 43A-43D show an example convolution of a first volume 4302 with a first set of kernel weights 4304 to generate a first plane 4308x of a second volume 4308 (shown in FIG. 43D), and also shows convolution of first volume 4302 with a second set of kernel weights 4306 to generate a second plane 4308y of a second volume 4308 [convolving the input stack into the output stack using the stack of weights in the convolution kernel]. First volume 4302 includes 3 planes (4302r, 4302g, 4302b) with each plane including a 9×9 array of image data…: First set of kernel weights 4304 then slides S=2 horizontal data values of first volume 4302. As illustrated in FIG. 43B, data value x1 of first plane 4308x of second volume 4308 is determined by multiplying every weight in first set of kernel weights 4304 with every pair-wise input element from the overlapping region of a second input volume (4302r.sub.1, 4302g.sub.1, 4302b.sub.1), as follows: … This process continues, with first set of kernel weights 4304 sliding S=2 horizontal values of first volume 4302 each iteration until the first row of data values (x0, x1, x3, x3) of first plane 4308x is complete [convolving the input stack into the output stack using the stack of weights in the convolution kernel]. First set of kernel weights 4304 then slides down S=2 rows and back to the leftmost column of first volume 4302 to calculate the second row of data values (x4, x5, x6, x7) of first plane 4308x. This process continues until all four rows of data values of first plane 4308x are complete. And in 0266: FIG. 44 shows an implementation of a convolutional layer on an acceleration component 4402, which may be physically implemented as an FPGA device. Acceleration component 4402 includes an input buffer array 4404, a kernel weights buffer array 4406, a functional unit array 4408, and an output buffer array 4410. Acceleration component 4402 performs a convolution of an input volume of D planes of input data and M sets of kernel weights data Weights.sub.0, Weights.sub.1, . . . , Weights.sub.M-1, to generate an output volume of M planes of output data [packing the output stack using the output block size].)
Regarding independent claims 7 and 15, the limitations are similar to claim 1 limitations and are thus rejected under the same rationale.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 7, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Martineau et al. (US 20170076179, hereinafter ‘Mart’) in further view of Francini et al. (US 11755913, hereinafter ‘Fran’).
Regarding independent claim 1 limitation, Mart teaches a computer-implemented method of managing data for convolution processing, the method comprising: (in [0028] FIG. 1 illustrates an example computing architecture 10 for implementing an object recognition system 100, in one or more embodiments... As described in detail later herein, one or more applications may execute/operate on the one or more processor devices 41 to create, initialize, and iteratively train the object recognition system 100. The object recognition system 100 comprises one or more convolutional neural networks (CNNs) [a computer-implemented method of managing data for convolution processing, the method comprising] 105 And in [0079] The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments… It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions [a computer-implemented method of managing data for convolution processing, the method comprising].)
determining a usable number of registers in a processor architecture of a device, the device comprising a memory, an input channel for receiving an input stack of input data, an output channel for receiving an output stack of output data, and a convolution kernel containing a stack of weights for convolving the stack of input data into the stack of output data; (in 0033-0034 Returning to FIG. 2A, in one embodiment, the storage devices 42 [determining a usable number of registers in a processor architecture of a device, the device comprising a memory] of the initialization and training system 40 maintain one or more databases... In one embodiment, the corresponding window size is indicative of a size of a window representing a region of an input the layer 106 processes (e.g., if the input is an image, the window may represent an image tile of the image, such as an image tile having dimensions 5 x 5 along the x and y dimensions) [an input channel for receiving an input stack of input data]. In one embodiment, the corresponding stride interval is indicative of a distance the window is slid along the x and y dimensions before the region of the input the window represents is computed... And in 0050 -0051: Weights 220 in the first convolutional layer 110 comprises 2D patterns of a homogenous feature. After applying the weights 220 [a convolution kernel containing a stack of weights for convolving the stack of input data into the stack of output data] to the input 51, output is produced [an output channel for receiving an output stack of output data] and fed to a subsequent convolutional layer 110 as input [… for receiving an output stack of output data]. In one embodiment, the subsequent convolutional layer 110 utilizes a tall thin 3D filter 240 of size t.sub.w×t.sub.h×t.sub.d, wherein t.sub.w=1, t.sub.h=1, and t.sub.d=d. The tall thin 3D filter 240 implements a tall thin convolution over the output to create a 3D pattern 260 with a new semantic definition based on learned linear combinations of flat 2D filters. The output may also be fed to one or more other layers of the CNN 105.)
determining an output block size based on the processor architecture; (In [0004] Convolutional neural networks (CNNs) [the processor architecture] are deep learning neural networks that provide translational independence via a convolutions of a filter over an input. Assume an input occupying a three-dimensional (3D) space with three dimensions: x-dim