Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This action is responsive to the following communication: Amendment filed Sept. 17, 2026. This action is made non-final.
Claims 1-20 are pending in the case. Claims 1, 17 and 20 are independent claims.
Claim Objections
Claims 13-16 are objected to because of their dependency from rejected claim. Claims 13-16 would be allowable if written in independent form.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-12, 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over NURVITADHI et al. (hereinafter Nur) U.S. Patent Publication No. 2018/0189638 in view of Nama et al. (hereinafter Nama) U.S. Patent Publication No. 2022/0309325.
With respect to independent claim 1, Nur teaches a method of implementing in hardware a dynamic neural network for operation on an input tensor having a variable dimension (see e.g., Para [3]-[5][82]-[85] -]-“The RNN 100 includes an input value (I.sub.t) 102, a node 104 (sometimes referred to as a neuron, etc.) that is executed a number of times via the recurrent connection 105, and an output value (O.sub.t) 106. RNNs, within the node 104, typically perform a variety operations; commonly, these are matrix operations and/or vector operations (e.g., a dense matrix*vector, or vector-wise add, multiply, hyperbolic tangent, sigmoid, etc.).”), the method comprising:
receiving a representation of the dynamic neural network (see e.g., Para [80][113]- “Thus, a more mathematical representation of the GRU data flow can be generated using a variety of data structures such as a graph, which is referred to herein as a “flow graph.”” “The flow graph identifies a plurality of operations to be performed to implement the RNN algorithm and further identifies data dependencies between ones of the plurality of operations. The plurality of operations includes one or more matrix operations and one or more vector operations.”);
transforming the representation of the dynamic neural network into a static network adapted to operate on a fixed size input into a static network adapted to operate on a fixed size input (see e.g., Para [83][84][93]-[101][115] [116] – “an accelerator instance optimized for an ASIC for a particular market segment, programmable to support all RNN applications in this segment. “ “the framework can be configured to generate more efficient fixed-control units within the accelerator at the cost of programmability. “ “mapping the plurality of operations of the flow graph to an accelerator hardware template to yield the accelerator instance comprising register transfer language code that describes how one or more matrix processing units (MPUs) and one or more vector processing units (VPUs) are to be arranged to perform the RNN algorithm … the mapping is based upon hardware design constraints indicating amounts or capabilities of hardware elements that can be utilized in the accelerator instance.” – The representation of the dynamic neural network is transformed into templates which are static and is arranged to perform the RNN algorithm), the static network being adapted to perform operations on a fixed-size input (see e.g., Para [84][85][93]-[101]-“the framework can be configured to generate more efficient fixed-control units within the accelerator at the cost of programmability.”)which are equivalent to the operations performed by the dynamic neural network on its input tensor (see e.g., [83]-[85][93]-[101][115] [116] – same represented RNN algorithm can be performed); and
implementing a plurality of instances of the static network in hardware for operation on an input (see e.g., [82]-[85][93]-[101][115]), each instance of the static network being arranged to operate on a respective fixed size input of the sequence (see e.g., [83]-[85][93]-[101][115] [116]).
Nur does not expressly show input tensor split into a sequence of overlapping fixed size inputs along its variable dimension. However, Nama teaches the similar feature (see e.g. Fig. 4A para [60]-[66] – “tiling of an input tensor 402 into a plurality of tiles 404a, . . . , 404d and subsequent convolution of the tiles, where neighboring tiles in the input tensor 402 partially overlap. ”). Both Nur and Nama are directed to input tensor sequencing methods. Accordingly, it would have been obvious to the skilled artisan before the effective filing date of the claimed invention having Nur and Hofer in front of them to modify the system of Nur to include the above feature. The motivation to combine Nur and Nama comes from Nama. Nama discloses the motivation to provide overlapping inputs so that data context can be preserved (see e.g. Abstract para [60]-[66]). This motivation for combination also applies to the remaining claims which depend on this combination.
With respect to dependent claim 2, the modified Nur teaches the implementing includes defining a combination operation arranged to combine the output of each instance of the static network so as to provide an output of the dynamic neural network operated on the input tensor (see e.g. Nama Fig. 4A para [60]-[66] – “a convolution operation within a processing node or layer 406 of a neural network, in which a kernel is convolved with each tile 404, to generate a corresponding tile 424 of an output tensor 412. The lower portion of FIG. 4A illustrates how individual tile 404 is convolved with the kernel to generate a corresponding tile 424a (note that the lower portion of the figure shows the tiles in non-overlapping manner, for clearly depicting the tile-wise convolution operations). For example, tile 404a is convolved to generate a corresponding tile 424a, tile 404b is convolved to generate a corresponding tile 424b, and so on. The output tensor 412 is a combination of the tiles 424a, . . . , 424d. Although not illustrated, the tiles 424a, . . . , 424d can be further convolved or processed by another operation (e.g., max-pooling) within the neural network.”).
With respect to dependent claim 3, the modified Nur teaches the defining the combination operation comprises implementing the combination operation in hardware (see e.g. Nama Para [45] – “Examples of the data processor 110 include Graphics Processing Unit (GPU), Central Processing Unit (CPU), Field Programmable Gate Arrays (FPGAs), Coarse-Grained Reconfigurable Architectures (CGRAs), Application-Specific Integrated Circuits (ASICs), and Application Specific Instruction-set Processor (ASIP). In an example where the data processor 110 is a reconfigurable data processor, examples of the data processor 110 includes FPGAs, CGRAs, ASICs, and ASIP.”).
With respect to dependent claim 4, the modified Nur teaches each of the fixed size inputs of the sequence of overlapping fixed size inputs is the same size (see e.g. Nama Para [45] – “the input tensor 402 has a dimension of 34×34 pixels, and individual tiles 404 has a dimension of 18×18 pixels. Thus, in an embodiment, each tile within the input tensor 402 has the same dimension.”).
With respect to dependent claim 5, the modified Nur teaches the transforming comprises selecting the size of the overlap between the overlapping fixed size inputs of the sequence in dependence on the receptive field of the first layer of the static network (see e.g. Nama Fig. 4A Para [65]-[67] – “ To generate an output tile of a certain size, the corresponding input tile size is determined from the receptive field of the filter used for the convolution operation. For example, a tiling that is to be performed at a section output is initially determined. Then, using the information about the receptive field of each operation in the section, an algorithm (e.g., discussed with respect to FIG. 12A) works backwards through the section until it reaches the input. In other words, the tile size of the output is used to calculate the tile size of the input. During a convolution operation, dimensions of an input tile (e.g., input tile 404 of the input tensor 402) can be different from the dimensions of the corresponding output tile (e.g., output tile 424 of the output tensor 412).”) .
With respect to dependent claim 6, the modified Nur teaches the transforming comprises selecting the size of the fixed size input in dependence on the characteristics of the hardware (see e.g. Para [116] – “the mapping is based upon hardware design constraints indicating amounts or capabilities of hardware elements that can be utilized in the accelerator instance.” Also see Nama Fig. 4A Para [60] – “the number of tiles is based on a variety of factors, such as a size of the input tensor 402, a memory and/or processing capacity of the network processing the tensors “).
With respect to dependent claim 7, the modified Nur teaches the static network includes the same set of layers as the dynamic neural network, the layers of the dynamic neural network representing the operations performed by the dynamic neural network (see e.g. Para [80][81][113]-[115] – “mapping the plurality of operations of the flow graph to an accelerator hardware template to yield the accelerator instance comprising register transfer language code that describes how one or more matrix processing units (MPUs) and one or more vector processing units (VPUs) are to be arranged to perform the RNN algorithm”).
With respect to dependent claim 8, the modified Nur teaches the dynamic neural network is for operation on an input tensor having a plurality of variable dimensions and the fixed size input is fixed in size in respect of each of the variable dimensions, the plurality of instances of the static network being for operation on an input tensor split along each of variable dimensions into a sequence of overlapping fixed size inputs (see e.g. Nama Fig. 4A Para [62]-[67] – Both horizontal and vertical overlap is disclosed).
With respect to dependent claim 9, the modified Nur teaches the transforming comprises selecting the size of the fixed size input and/or the size of the overlap in respect of each of the variable dimensions independently of selecting the size of the fixed size input and/or the size of the overlap in respect of the other variable dimensions of the plurality of variable dimensions (see e.g. Nama Fig. 4A Para [65]-[67] – “In equations 1 and 2, W.sub.i and H.sub.i are a width and a height, respectively, of the input tile; K.sub.w and K.sub.h are a width and a height, respectively, of the convolution kernel used during the convolution operation; P.sub.w and P.sub.h are convolution padding used in horizontal and vertical directions, respectively of the convolution operation; and S.sub.W and S.sub.h are strides in horizontal and vertical directions, respectively, of the convolution operation.” Width/height and horizontal/vertical kernel size are separately defined).
With respect to dependent claim 10, the modified Nur teaches the transforming comprises, prior to forming the static network: determining whether padding of the layers of the dynamic neural network may be propagated into the input tensor whilst satisfying the receptive field of each layer of the dynamic neural network (see e.g. Nama Para [78] – “Due to tiling and the receptive fields of the convolutional operations in a section, the peripheral input tiles may contain pixels outside the boundary of the original input. These out of bounds pixels are zero-padded for every successive convolutional layer in the section. For any given convolution layer, a relatively small number of pixels can be outside the boundary of the original input, but this can increase and exacerbate as many successive convolutional layers are applied.”); and if the determination is positive, propagating the padding of the layers of the static network into the fixed size input to the static network (see e.g. Nama Para [65][84][95] – “the corresponding input tile size is determined from the receptive field of the filter used for the convolution operation. For example, a tiling that is to be performed at a section output is initially determined. Then, using the information about the receptive field of each operation in the section, an algorithm (e.g., discussed with respect to FIG. 12A) works backwards through the section until it reaches the input. In other words, the tile size of the output is used to calculate the tile size of the input.” Nama does not expressly show “the determination.” However, it would have been obvious to include this feature because padding feasibility must be determined before processing.)
With respect to dependent claim 11, the modified Nur teaches the determination is performed if the dynamic neural network does not introduce padding between layers of the dynamic network and is otherwise not performed (The examiner notes that it would have been obvious to include this feature based on the teachings of Nama. Nama expressly teaches eliminating intermediate paddling by moving it to an input boundary – see e.g. Claim 7, 39 – “ the first padding configuration applies a cumulative padding to pad the input into a padded input, wherein the second padding configuration applies no padding to leave the intermediate representation unpadded, wherein the third padding configuration applies no padding to leave the further intermediate representation unpadded.”).
With respect to dependent claim 12, the modified Nur teaches the implementing is performed such that the overlaps of each fixed size input to each of the instances of the static network are shared with the fixed size inputs to adjacent instances of the static network for operation on the sequence of overlapping fixed size inputs (see e.g. Nama Fig. 4A Para [63][71] –“Thus, pixels within the overlapping section 405 are common to both tiles 404a and 404b. Similarly, a 2×18 bottom section of the tile 404a overlaps with a 2×18 top section of the tile 404c, and a 2×2 right-bottom section of the tile 404a overlaps with a left-top section of the tile 404d. As illustrated, the central 2×2 overlap region 407 is common to all the four tiles 404a, . . . , 404d.”), but that inputs to layers of the instances of the static network subsequent to the first layer are not shared with the respective layers of adjacent instances of the static network (see e.g. Nama Fig. 4B and Para [76] –“Thus, individual 16×16 tiles 444 form the output tensor 446 of size 32×32. Thus, there is no overlap among the tiles 444 in the output tensor 446.”).
With respect to dependent claim 17, the modified Nur teaches a data processing system for implementing a dynamic neural network for operation on an input tensor having a variable dimension, the system comprising: a transformation unit configured to receive a representation of the dynamic neural network and transform the representation into a static network adapted to operate on a fixed size input, the static network being configured to perform operations on the fixed size input which are equivalent to the operations performed by the dynamic neural network on its input tensor (see e.g., Nur Para [84][113]-[115], Mama Para [60]-[64] and discussion above with respect to claim 1); a hardware accelerator for processing neural networks (see e.g., Nur Para [93]-[100] and [115]); and control logic configured to implement a plurality of instances of the static network at the hardware accelerator for operation on an input tensor split into a sequence of overlapping fixed size inputs along its variable dimension, each instance of the static network being arranged to operate on a respective fixed size input of the sequence (see e.g., Nur Para [84][113]-[115], Mama Para [60]-[64] and discussion above with respect to claim 1).
With respect to dependent claim 18, the modified Nur teaches the hardware accelerator and the control logic are adapted to perform feed-forward neural networks on input tensors of fixed size (see e.g., Mama Para [54]-[64] and discussion above with respect to claim 1).
With respect to dependent claim 19, the modified Nur teaches the hardware accelerator and the control logic are incapable of performing the received representation of the dynamic neural network (Nur does not expressly require that the hardware accelerator to be capable of the “performing.” Therefore, it would have been obvious to include the above feature).
Claim 20 is rejected for the similar reasons discussed above with respect to claim 1.
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain.” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)). Further, a reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill the art, including nonpreferred embodiments. Merck & Co. v. Biocraft Laboratories, 874 F.2d 804, 10 USPQ2d 1843 (Fed. Cir.), cert. denied, 493 U.S. 975 (1989). See also Upsher-Smith Labs. v. Pamlab, LLC, 412 F.3d 1319, 1323, 75 USPQ2d 1213, 1215 (Fed. Cir. 2005); Celeritas Technologies Ltd. v. Rockwell International Corp., 150 F.3d 1354, 1361, 47 USPQ2d 1516, 1522-23 (Fed. Cir. 1998).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PEIYONG WENG whose telephone number is (571)270-1660. The examiner can normally be reached on Mon.-Fri. 8 am to 5 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Matthew Ell, can be reached on (571) 270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://portal.uspto.gov/external/portal. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/PEI YONG WENG/Primary Examiner, Art Unit 2141
Prosecution Insights