Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Detailed Action
This action is in response to the claims filed 5/31/2024:
Claims 1 – 20 are pending.
Claims 1, 19, and 20 are independent.
Specification
The disclosure is objected to because of the following informalities:
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 101
101 Rejection
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 USC § 101 because the claimed invention is directed to non-statutory subject matter.
Regarding Claim 1: Claim 1 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 1 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: Claim 1 under its broadest reasonable interpretation is a series of mental processes. For example, but for the generic computer components language, the above limitations in the context of this claim encompass machine learning processing, including the following:
forming a graph that represents the flow of data through the plurality layers of the neural network, the graph comprising: a plurality of vertices, each vertex of the plurality of vertices being representative of an output channel of a layer of the plurality of layers of the neural network; and one or more edges, each edge of the one or more edges representing the potential flow of non-zero data between respective output channels represented by a respective pair of vertices (observation, evaluation, and judgement),
identifying, by traversing the graph, one or more redundant channels comprised by the plurality of layers of the neural network (observation, evaluation, and judgement)
Therefore, claim 1 recites an abstract idea which is a judicial exception.
Step 2A Prong Two Analysis: Claim 1 recites additional elements “A computer implemented method”. However, these additional features are computer components recited at a high-level of generality, such that they amount to no more than mere instructions to apply the judicial exception using a generic computer component. An additional element that merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea, does not integrate the judicial exception into a practical application (See MPEP 2106.05(f)). Claim 1 also recites additional elements “receiving a neural network comprising a plurality of layers” and “outputting a compressed neural network in which the identified one or more redundant channels are not present in the compressed neural network” which amounts to gathering and outputting data which is insignificant extra-solution activity (See MPEP 2106.05(g)). Therefore, claim 1 is directed to a judicial exception.
Step 2B Analysis: Claim 1 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the lack of integration of the abstract idea into a practical application, the additional elements recited in claim 1 amount to no more than mere instructions to apply the judicial exception using a generic computer component and insignificant extra-solution activity. The gathering and outputting data is considered well-understood, routine, and conventional in the art (See MPEP 2106.05(d)(II)(i)).
For the reasons above, claim 1 is rejected as being directed to non-patentable subject matter under §101. This rejection applies equally to independent claims 19 and 20, which recite a system and a computer program product, respectively, as well as to dependent claims 2-18.
Independent claim 19 recites additional instructions to apply the judicial exception using generic computer components “A processing system for compressing a neural network, the processing system comprising at least one processor configured to:”.
Independent claim 20 recites additional instructions to apply the judicial exception using generic computer components “A non-transitory computer readable storage medium having stored thereon computer readable instructions that, when executed at a computer system, cause the computer system to compress a neural network, by:”.
The additional limitations of the dependent claims are addressed briefly below:
Dependent claim 2 recites additional insignificant extra-solution activity of gathering and outputting data “a redundant channel is a channel, comprised by a layer of the plurality of layers of the neural network, that can be removed from the neural network without changing the output of the neural network”
Dependent claim 3 recites additional observation, evaluation, and judgement “traversing the graph in reverse topologically sorted order and/or topologically sorted order
Dependent claim 4 recites additional observation, evaluation, and judgement “a plurality of vertex subsets, each vertex subset of the plurality of vertex subsets being representative of a respective layer of the plurality of layers of the neural network, each vertex subset of the plurality of vertex subsets comprising one or more vertices, each vertex of the one or more vertices being representative of an output channel of the respective layer of the neural network; and the one or more edges, each edge of the one or more edges: connecting two vertices, said two vertices being comprised by different vertex subsets of the graph; and being representative of the potential flow of non-zero data between the respective channels of the respective layers of the neural network represented by those vertices”
Dependent claim 5 recites additional observation, evaluation, and judgement “a vertex subset of the plurality of vertex subsets is representative of a fully-connected layer of the plurality of layers of the neural network, each vertex of the one or more vertices comprised by that vertex subset being representative of a respective output channel of that fully-connected layer, and wherein forming the graph comprises: determining a matrix representative of a set of coefficients of the fully-connected layer, the matrix comprising one or more elements representative of non-zero coefficients and one or more elements representative of zero coefficients; for each of the one or more elements representative of a non-zero coefficient: identifying: an output channel of the fully-connected layer comprising that non-zero coefficient; an input channel of the fully-connected layer comprising that non-zero coefficient; and an output channel of a preceding layer of the plurality of layers of the neural network corresponding to the identified input channel of the fully-connected layer; and connecting, using an edge, a vertex in the vertex subset representative of the identified output channel of the fully-connected layer to a vertex in a different vertex subset of the plurality of vertex subsets representative of the identified output channel of the preceding layer”
Dependent claim 6 recites additional observation, evaluation, and judgement “a vertex subset of the plurality of vertex subsets is representative of a convolution layer of the plurality of layers of the neural network, each vertex of the one or more vertices comprised by that vertex subset being representative of a respective output channel of that convolution layer, and forming the graph comprises: determining a matrix representative of a set of coefficients of the convolution layer, the matrix comprising one or more elements representative of non-zero values and one or more elements representative of zero values; for each of the one or more elements representative of a non-zero value: identifying: an output channel of the convolution layer; an input channel of the convolution layer; and an output channel of a preceding layer of the plurality of layers of the neural network corresponding to the identified input channel of the convolution layer; and connecting, using an edge, a vertex in the vertex subset representative of the identified output channel of the convolution layer to a vertex in a different vertex subset of the plurality of vertex subsets representative of the identified output channel of the preceding layer”
Dependent claim 7 recites additional observation, evaluation, and judgement “the convolution layer comprises a set of coefficients arranged in one or more filters, each of the one or more filters arranged in one or more channels, each channel of each filter comprising a respective subset of the set of coefficients of the convolution layer, and wherein determining the matrix comprises: for each channel of each filter: determining whether that channel of that filter comprises a non-zero coefficient; and in response to determining that that channel of that filter comprises at least one non-zero coefficient, representing that channel of that filter with an element representative of a non-zero value in the matrix; or in response to determining that that channel of that filter comprises exclusively zero coefficients, representing that channel of that filter with an element representative of a zero value in the matrix.”
Dependent claim 8 recites additional observation, evaluation, and judgement “wherein a vertex subset of the plurality of vertex subsets further comprises a bias vertex representative of one or more biases of a layer of the plurality of layers of the neural network subsequent to the layer of the plurality of layers of the neural network that that vertex subset is representative of, said bias vertex being connected, by one or more edges, to one or more vertices of the vertex subset representative of that subsequent layer, each of said edges representing a non-zero bias of the one or more biases represented by the bias vertex being associated with a respective output channel of the one or more output channels represented by the vertex subset representative of that subsequent layer”
Dependent claim 9 recites additional observation, evaluation, and judgement “a vertex subset of the plurality of vertex subsets is representative of an add layer of the plurality of layers of the neural network, said vertex subset comprising a number of vertices equal to the number of channels in each of a plurality of activation data sets that that add layer is configured to sum, each of said plurality of activation data sets having the same number of channels, each vertex comprised by that vertex subset being representative of a respective summation operation performed between a set of respective channels of the plurality of activation data sets such that each vertex comprised by that vertex subset is representative of a respective output channel of the add layer; and each vertex comprised by that vertex subset being connected, by respective edges, to vertices in different vertex subsets, said vertices being representative of output channels of preceding layers of the plurality of layers of the neural network, said output channels corresponding to the channels of the set of respective channels of the plurality of activation data sets between which the summation operation represented by that vertex is performed”
Dependent claim 10 recites additional observation, evaluation, and judgement “a vertex subset of the plurality of vertex subsets is representative of a flatten layer of the plurality of layers of the neural network, said vertex subset comprising n groups of vertices, n being equal to the number of channels of data of an activation data set on which the flatten layer is configured to perform a flatten operation, each group of vertices comprising m vertices, m being equal to the number of values in each channel of data of said activation data set, each vertex comprised by said vertex subset being representative of a respective output channel of the flatten layer; and each vertex comprised by each group of vertices in that vertex subset being connected, by a respective edge, to a vertex in a different vertex subset, said vertex representative of an output channel of a preceding layer of the plurality of layers of the neural network, said output channel corresponding to the channel of the activation data set on which the part of the flatten operation represented by that group of vertices is performed”
Dependent claim 11 recites additional observation, evaluation, and judgement “the plurality of vertex subsets are arranged in a sequence representative of the sequence in which the plurality of layers of the neural network are arranged, and wherein identifying the one or more redundant channels comprises: assigning each of the incoming edges of the vertices of the vertex subset representative of the sequentially last layer of the plurality of layers of the neural network a first state; traversing the sequence of vertex subsets, from the vertex subset representative of the sequentially penultimate layer of the plurality of layers of the neural network, to the vertex subset representative of the sequentially first layer of the plurality of layers of the neural network, assessing each of the one or more vertices in each vertex subset to determine whether that vertex has at least one outgoing edge assigned the first state, and: if yes, assigning each of the incoming edges of that vertex the first state; and if not, not assigning each of the incoming edges of that vertex the first state; subsequently, traversing the sequence of vertex subsets, from the vertex subset representative of the sequentially first layer of the plurality of layers of the neural network, to the vertex subset representative of the sequentially penultimate layer of the plurality of layers of the neural network, assessing each of the one or more vertices in each vertex subset to determine whether that vertex has at least one incoming edge assigned the first state, and: if yes, assigning each of the outgoing edges of that vertex the first state; and if not, causing each of the outgoing edges of that vertex to not be assigned the first state; and subsequently, identifying one or more vertices that do not have any outgoing edges assigned the first state, said one or more identified vertices representing the one or more redundant channels comprised by the plurality of layers of the neural network”
Dependent claim 12 recites additional observation, evaluation, and judgement “the plurality of vertex subsets are arranged in a sequence representative of the sequence in which the plurality of layers of the neural network are arranged, and wherein identifying the one or more redundant channels comprises: assigning each of the incoming edges of the one or more vertices comprised by the vertex subset representative of the sequentially first layer of the plurality of layers of the neural network a first state; traversing the sequence of vertex subsets, from the vertex subset representative of the sequentially first layer of the plurality of layers of the neural network, to the vertex subset representative of the sequentially penultimate layer of the plurality of layers of the neural network, assessing each of the one or more vertices in each vertex subset to determine whether that vertex has at least one incoming edge assigned the first state, and: if yes, assigning each of the outgoing edges of that vertex the first state; and if not, not assigning each of the outgoing edges of that vertex the first state; subsequently, traversing the sequence of vertex subsets, from the vertex subset representative of the sequentially penultimate layer of the plurality of layers of the neural network, to the vertex subset representative of the sequentially first layer of the plurality of layers of the neural network, assessing each of the one or more vertices in each vertex subset to determine whether that vertex has at least one outgoing edge assigned the first state, and: if yes, assigning each of the incoming edges of that vertex with first state; and if not, causing each of the incoming edges of that vertex to not be assigned the first state; and subsequently, identifying one or more vertices that do not have any outgoing edges assigned with first state, said one or more identified vertices representing the one or more redundant channels comprised by the plurality of layers of the neural network”
Dependent claim 13 recites additional observation, evaluation, and judgement “the plurality of vertex subsets are arranged in a sequence representative of the sequence in which the plurality of layers of the neural network are arranged, and wherein identifying the one or more redundant channels comprises: assigning each of the incoming edges of the vertices of the vertex subset representative of the sequentially last layer of the plurality of layers of the neural network a first state; traversing the sequence of vertex subsets, from the vertex subset representative of the sequentially penultimate layer of the plurality of layers of the neural network, to the vertex subset representative of the sequentially first layer of the plurality of layers of the neural network, assessing each of the one or more vertices in each vertex subset to determine whether that vertex has at least one outgoing edge assigned the first state, and: if yes, assigning each of the incoming edges of that vertex the first state; and if not, not assigning each of the incoming edges of that vertex the first state; and subsequently, identifying one or more vertices that do not have any outgoing edges assigned the first state, said one or more identified vertices representing the one or more redundant channels comprised by the plurality of layers of the neural network”
Dependent claim 14 recites additional observation, evaluation, and judgement “the plurality of vertex subsets are arranged in a sequence representative of the sequence in which the plurality of layers of the neural network are arranged, and wherein identifying the one or more redundant channels comprises: assigning each of the incoming edges of the one or more vertices comprised by the vertex subset representative of the sequentially first layer of the plurality of layers of the neural network a first state; traversing the sequence of vertex subsets, from the vertex subset representative of the sequentially first layer of the plurality of layers of the neural network, to the vertex subset representative of the sequentially penultimate layer of the plurality of layers of the neural network, assessing each of the one or more vertices in each vertex subset to determine whether that vertex has at least one incoming edge assigned the first state, and: if yes, assigning each of the outgoing edges of that vertex the first state; and if not, not assigning each of the outgoing edges of that vertex the first state; and subsequently, identifying one or more vertices that do not have any outgoing edges assigned the first state, said one or more identified vertices representing the one or more redundant channels comprised by the plurality of layers of the neural network”
Dependent claim 15 recites additional observation, evaluation, and judgement “an edge can be an outgoing edge of a first vertex and/or an incoming edge of a second vertex; an incoming edge of a vertex is representative of the potential flow of non-zero data into the output channel represented by that vertex; and an outgoing edge of a vertex is representative of the potential flow of non-zero data from the output channel represented by that vertex”
Dependent claim 16 recites additional insignificant extra-solution activity of gathering and outputting data (See MPEP 2106.05(g)) “storing the compressed neural network for subsequent implementation” which is well-understood, routine, and conventional in the art (See MPEP 2106.05(d)(II)(iv)).
Dependent claim 16 recites additional insignificant extra-solution activity of gathering and outputting data (See MPEP 2106.05(g)) “outputting a computer readable description of the compressed neural network that, when implemented at a system for implementing a neural network, causes the compressed neural network to be executed” which is well-understood, routine, and conventional in the art (See MPEP 2106.05(d)(II)(i)).
Dependent claim 17 recites additional instructions to apply the judicial exception using generic computer components “configuring hardware logic to implement the compressed neural network, wherein the hardware logic comprises a neural network accelerator”
Therefore, when considering the elements separately and in combination, they do not add significantly more to the inventive concept. Accordingly, claims 1-20 are rejected under 35 U.S.C. § 101.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-4, 6-8, and 11-20 are rejected under U.S.C. §102(a)(1) as being anticipated by Schoonhoven (“LEAN: GRAPH-BASED PRUNING FOR CONVOLU TIONAL NEURAL NETWORKS BY EXTRACTING LONGEST CHAINS”, 2021) and the corresponding Github repository at https://github.com/schoonhovenrichard/LEAN_CNN_pruning/tree/master.
PNG
media_image1.png
282
1208
media_image1.png
Greyscale
FIG. 1 of Schoonhoven
Regarding claim 1, Schoonhoven teaches A computer implemented method of compressing a neural network, the method comprising:([p. 10] "In order to aid reproducibility the authors have published (Python) code for LEAN pruning. There are installation instructions that aim to help users install and run the code on their machines, as well as explanations and documentation for the code and scripts. The scripts are intended to be usable on relatively simple machines (with GPU and CUDA)")
receiving a neural network comprising a plurality of layers;([p. 3] "CNNs are composed of layers of operations which pass images from one layer to the next")
forming a graph that represents the flow of data through the plurality layers of the neural network, the graph comprising: a plurality of vertices, each vertex of the plurality of vertices being representative of an output channel of a layer of the plurality of layers of the neural network; and([p. 4] "Figure 1: (A) Example CNN architecture with the number of channels indicated above each layer. (B) Associated pruning graph. Every channel is a node, and every operator is an edge connecting input and output nodes. The edge weights are the corresponding operator norms." See FIG. 1(B) "Pruning Graph")
one or more edges, each edge of the one or more edges representing the potential flow of non-zero data between respective output channels represented by a respective pair of vertices;([p. 26] "edges = np.where(adjmat != 0) […] if val == 0: print(inidx, outidx, val) raise Exception("PAUSE")" Schoonhoven explicitly constructs graph edges where the adjacency matrix values are not zero and explicitly rejects values of zero. An edge by definition connects an ordered pair of vertices)
identifying, by traversing the graph, one or more redundant channels comprised by the plurality of layers of the neural network; and([p. 24] "topo_order = nx.topological_sort(G) […] for v in topo_order:" [p. 6] "After every LEAN pruning step is concluded, some post-processing is performed. In some cases there are channels which receive no input data at all, or which are equal to a homogeneous constant image for all input data. We therefore remove nodes without incoming edges as well as nodes where the succeeding batch normalization has running variance below some threshold (10−40 by default)" See also Algorithm 2 on p. 5. The method explicitly uses graph paths to decide what survives and critically p. 5 says after each LEAN pruning step the system performs post-processing because some channels no longer receive any input data. It therefore removes graph nodes having no incoming edges.)
outputting a compressed neural network in which the identified one or more redundant channels are not present in the compressed neural network.([p. 20] "The pruned networks are saved to the pruned_models folder after every pruning step.").
Regarding claim 2, Schoonhoven teaches The method of claim 1, wherein: a redundant channel is a channel, comprised by a layer of the plurality of layers of the neural network,(Schoonhoven [p. 3] "CNNs are composed of layers of operations which pass images from one layer to the next. Every operation, e.g., convolution, has an input x and output y. The input and output consist of one or more images, called channels")
that can be removed from the neural network without changing the output of the neural network.(Schoonhoven [p. 6] "We therefore remove nodes without incoming edges as well as nodes where the succeeding batch normalization has running variance below some threshold (10−40 by default). A low running variance can occur when the output of a convolution is always zero after applying the ReLU activation function, for instance." An identically zero output channel contributes zero through every downstream linear convolutional connection. Eliminating that zero channel together with its corresponding channel-to-channel operations therefore does not alter the values computed by the remaining network. Schoonhoven's redundancy propagation reinforces this: once an upstream channel is fully pruned, subsequent masks for connections using that channel are zeroed, and the associated bias cleanup is performed as part of the same redundancy procedure.).
Regarding claim 3, Schoonhoven teaches The method of claim 1, the method comprising traversing the graph in reverse topologically sorted order and/or topologically sorted order.(Schoonhoven [p. 24] "topo_order = nx.topological_sort(G) […] for v in topo_order:").
Regarding claim 4, Schoonhoven teaches The method of claim 1, the graph comprising: a plurality of vertex subsets, each vertex subset of the plurality of vertex subsets being representative of a respective layer of the plurality of layers of the neural network, each vertex subset of the plurality of vertex subsets comprising one or more vertices, each vertex of the one or more vertices being representative of an output channel of the respective layer of the neural network; and(Schoonhoven [p. 4] "Figure 1: (A) Example CNN architecture with the number of channels indicated above each layer. (B) Associated pruning graph. Every channel is a node, and every operator is an edge connecting input and output nodes. The edge weights are the corresponding operator norms." [p. 20] "To create an adjacency list, let every channel be a node index, and add to the list [inidx, outidx, normvalue] the edges between the channels." See FIG. 1(B) "Pruning Graph" which shows each layer as a vertex subset of the graph)
the one or more edges, each edge of the one or more edges: connecting two vertices, said two vertices being comprised by different vertex subsets of the graph; and(Schoonhoven [p. 4] "Figure 1: (A) Example CNN architecture with the number of channels indicated above each layer. (B) Associated pruning graph. Every channel is a node, and every operator is an edge connecting input and output nodes. The edge weights are the corresponding operator norms." [p. 20] "To create an adjacency list, let every channel be a node index, and add to the list [inidx, outidx, normvalue] the edges between the channels." See FIG. 1(B) "Pruning Graph" which shows each layer as a vertex subset of the graph)
being representative of the potential flow of non-zero data between the respective channels of the respective layers of the neural network represented by those vertices.(Schoonhoven [p. 26] "edges = np.where(adjmat != 0) […] if val == 0: print(inidx, outidx, val) raise Exception("PAUSE")" Schoonhoven explicitly constructs graph edges where the adjacency matrix values are not zero and explicitly rejects values of zero. An edge by definition connects an ordered pair of vertices).
Regarding claim 6, Schoonhoven teaches The method of claim 4, wherein a vertex subset of the plurality of vertex subsets is representative of a convolution layer of the plurality of layers of the neural network, each vertex of the one or more vertices comprised by that vertex subset being representative of a respective output channel of that convolution layer, and forming the graph comprises:(Schoonhoven [p. 4] "Figure 1: (A) Example CNN architecture with the number of channels indicated above each layer. (B) Associated pruning graph. Every channel is a node, and every operator is an edge connecting input and output nodes. The edge weights are the corresponding operator norms." [p. 20] "To create an adjacency list, let every channel be a node index, and add to the list [inidx, outidx, normvalue] the edges between the channels." See FIG. 1(B) "Pruning Graph" which shows each layer as a vertex subset of the graph)
determining a matrix representative of a set of coefficients of the convolution layer, the matrix comprising one or more elements representative of non-zero values and one or more elements representative of zero values;(Schoonhoven [p. 20] "A function, or code block, to define the CNN as a pruning graph. One can immediately define the adjacency list, or create an adjacency matrix and use the pre-supplied convert_matr_to_adjlist function to convert it. To create an adjacency list, let every channel be a node index, and add to the list [inidx, outidx, normvalue] the edges between the channels. Similarly, the adjacency matrix has at the appropriate positions (columns and rows are the in and out indices) the norms of the operators" [p. 6] "Split the image and filter into the coloured sections, with white entries representing zeroes")
for each of the one or more elements representative of a non-zero value: identifying: an output channel of the convolution layer; an input channel of the convolution layer; and(Schoonhoven [p. 3] "Every operation, e.g., convolution, has an input x and output y. The input and output consist of one or more images, called channels […] A convolution with stride s defines a map h : Rm×n → Rm s× n" [p. 6] "Split the image and filter into the coloured sections, with white entries representing zeroes" See also FIG. 1)
an output channel of a preceding layer of the plurality of layers of the neural network corresponding to the identified input channel of the convolution layer; and(Schoonhoven [p. 3] "Every operation, e.g., convolution, has an input x and output y. The input and output consist of one or more images, called channels […] A convolution with stride s defines a map h : Rm×n → Rm s× n" See also FIG. 1)
connecting, using an edge, a vertex in the vertex subset representative of the identified output channel of the convolution layer to a vertex in a different vertex subset of the plurality of vertex subsets representative of the identified output channel of the preceding layer.(Schoonhoven [p. 5 §4.1] "we say that every CNN operation has an input x and an output y, consisting of channels xi and yi. For each channel, we add a single node in the pruning graph. An edge connects two nodes corresponding to input channel xi and output channel yi if channel xi is used in the computation of channel yi").
Regarding claim 7, Schoonhoven teaches The method of claim 6, wherein the convolution layer comprises a set of coefficients arranged in one or more filters, (Schoonhoven [p. 1] "Neural networks consist of learnable parameters, including the scalar components of the convolutional filters" [p. 3] "in a convolutional operation with input channels x1,...,xN, an output channel yj is computed by convolving input images with learned filters […] Here hij is the filter related to the convolution operator that acts between channel xi and yj, and bj is an additive bias parameter. In a similar way, every CNN operation produces an output which consists of a number of channels")
each of the one or more filters arranged in one or more channels, each channel of each filter comprising a respective subset of the set of coefficients of the convolution layer, and wherein determining the matrix comprises:(Schoonhoven [p. 1] "Neural networks consist of learnable parameters, including the scalar components of the convolutional filters" [p. 3] "in a convolutional operation with input channels x1,...,xN, an output channel yj is computed by convolving input images with learned filters […] Here hij is the filter related to the convolution operator that acts between channel xi and yj, and bj is an additive bias parameter. In a similar way, every CNN operation produces an output which consists of a number of channels")
for each channel of each filter: determining whether that channel of that filter comprises a non-zero coefficient; and(Schoonhoven [p. 1] "convolution operators can only be removed once all scalar parameters of the filter kernel have been pruned" [p. 26] "edges = np.where(adjmat != 0) […] if val == 0: print(inidx, outidx, val) raise Exception("PAUSE")" Schoonhoven explicitly constructs graph edges where the adjacency matrix values are not zero and explicitly rejects values of zero. An edge by definition connects an ordered pair of vertices)
in response to determining that that channel of that filter comprises at least one non-zero coefficient, representing that channel of that filter with an element representative of a non-zero value in the matrix; or(Schoonhoven [p. 1] "convolution operators can only be removed once all scalar parameters of the filter kernel have been pruned" [p. 26] "edges = np.where(adjmat != 0) […] if val == 0: print(inidx, outidx, val) raise Exception("PAUSE")" Schoonhoven explicitly constructs graph edges where the adjacency matrix values are not zero and explicitly rejects values of zero. An edge by definition connects an ordered pair of vertices)
in response to determining that that channel of that filter comprises exclusively zero coefficients, representing that channel of that filter with an element representative of a zero value in the matrix.(Schoonhoven [p. 1] "convolution operators can only be removed once all scalar parameters of the filter kernel have been pruned" [p. 26] "edges = np.where(adjmat != 0) […] if val == 0: print(inidx, outidx, val) raise Exception("PAUSE")" Schoonhoven explicitly constructs graph edges where the adjacency matrix values are not zero and explicitly rejects values of zero. An edge by definition connects an ordered pair of vertices).
Regarding claim 8, Schoonhoven teaches The method of claim 4, wherein a vertex subset of the plurality of vertex subsets further comprises a bias vertex representative of one or more biases of a layer of the plurality of layers of the neural network subsequent to the layer of the plurality of layers of the neural network that that vertex subset is representative of, (Schoonhoven [p. 3] "yj = i=1 hij ∗ xi +bj […] hij is the filter related to the convolution operator that acts between channel xi and yj, and bj is an additive bias parameter" [p. 5] "For each channel, we add a single node in the pruning graph")
said bias vertex being connected, by one or more edges, to one or more vertices of the vertex subset representative of that subsequent layer, each of said edges representing a non-zero bias of the one or more biases represented by the bias vertex being associated with a respective output channel of the one or more output channels represented by the vertex subset representative of that subsequent layer.(Schoonhoven [p. 20] "A function to prune biases for convolutional layers. Example: prune_biases_MSD in pruning_algorithms.py . The function prune_biases_MSD shows how we can iterate over the pruning masks in the model, and check if the convolutional channel is fully pruned. If this is the case, we prune the bias of that channel" [p. 49] "bias_mask = torch.Tensor(np.ones(mod.bias.size())).to(model_device) [...] for mask in conv_masks:
if it == model.depth: # We do not prune the output biases [...] if mask.sum() == 0: bias_mask[it] = 0" Schoonhoven discloses within its graph-based representation and pruning process an identifiable graph element representing and pruning non-zero biases and their per-channel relationships to downstream output channel vertices. Schoonhoven's bias pruning is not incidental, it establishes direct non-zero value graph-pruning dependency).
Regarding claim 11, Schoonhoven teaches The method of claim 4, wherein the plurality of vertex subsets are arranged in a sequence representative of the sequence in which the plurality of layers of the neural network are arranged, and wherein identifying the one or more redundant channels comprises:(Schoonhoven [p. 3] "Every operation, e.g., convolution, has an input x and output y. The input and output consist of one or more images, called channels […] A convolution with stride s defines a map h : Rm×n → Rm s× n" [p. 5] "the pruning graph is a Directed Acyclic Graph (DAG)" [p. 6] "Split the image and filter into the coloured sections, with white entries representing zeroes" See also FIG. 1)
assigning each of the incoming edges of the vertices of the vertex subset representative of the sequentially last layer of the plurality of layers of the neural network a first state;(Schoonhoven [p. 4] "Figure 1: (A) Example CNN architecture with the number of channels indicated above each layer. (B) Associated pruning graph. Every channel is a node, and every operator is an edge connecting input and output nodes. The edge weights are the corresponding operator norms")
traversing the sequence of vertex subsets, from the vertex subset representative of the sequentially penultimate layer of the plurality of layers of the neural network, to the vertex subset representative of the sequentially first layer of the plurality of layers of the neural network, (Schoonhoven [p. 59] "BackwardPathIterator" [p. 57] "// Walk backward from largest node to find all edges in longest path" [p. 63] "// Implement an iterator to easily walk backwards over a directed path […] BackwardEdgeIterator […] self.node_idx = incoming_edge.from;" Schoonhoven explicitly performs the downstream result of the path analysis and walks upstream through predecessor connectivity)
assessing each of the one or more vertices in each vertex subset to determine whether that vertex has at least one outgoing edge assigned the first state, and: if yes, assigning each of the incoming edges of that vertex the first state; and if not, not assigning each of the incoming edges of that vertex the first state;(Schoonhoven [p. 58] "//// Returns true if edge was marked, else false (if edge was skippable). fn mark_single_edge(&mut self, edge_idx: usize) -> bool { if self.skippable[edge_idx] { return false; } // Mark edge self.marked[edge_idx] = true; self.num_marked += 1;" [p. 32] "if not pruned[edge_idx]" Schoonhoven explicitly maintains a boolean state for every edge and this returned boolean state has direct network meaning)
subsequently, traversing the sequence of vertex subsets, from the vertex subset representative of the sequentially first layer of the plurality of layers of the neural network, to the vertex subset representative of the sequentially penultimate layer of the plurality of layers of the neural network,(Schoonhoven [p. 66] "// Walk forward through path and // - mark edges along the path // - remove edges from hashsets // - mark nodes along path for invalidation […] // Recalculate downstream nodes" Schoonhoven explicitly reverses traversal direction to recalculate downstream nodes)
assessing each of the one or more vertices in each vertex subset to determine whether that vertex has at least one incoming edge assigned the first state, and: if yes, assigning each of the outgoing edges of that vertex the first state; and if not, causing each of the outgoing edges of that vertex to not be assigned the first state; and(Schoonhoven [p. 67] "// Recalculate node_distance
for &edge_idx in (&node_all_incoming_edge_idxs[dest_node_idx]).iter() […] for &edge_idx in (&node_all_outgoing_edge_idxs[dest_node_idx]).iter() […] for (_, i) in norm_idx_pairs.iter() {pb.set(num_marked as u64);if num_to_mark <= num_marked {
break;} marked[*i] = true; num_marked += 1;")
subsequently, identifying one or more vertices that do not have any outgoing edges assigned the first state, said one or more identified vertices representing the one or more redundant channels comprised by the plurality of layers of the neural network.(Schoonhoven [p. 32] "if prunedQ and default_mask[i, j].sum() == 0: # it already was pruned, ergo it has no edge continue code = codebook[code_iter] edge_idx = code[1] if not pruned[edge_idx]:").
Regarding claim 12, Schoonhoven teaches The method of claim 4, wherein the plurality of vertex subsets are arranged in a sequence representative of the sequence in which the plurality of layers of the neural network are arranged, (Schoonhoven [p. 3] "Every operation, e.g., convolution, has an input x and output y. The input and output consist of one or more images, called channels […] A convolution with stride s defines a map h : Rm×n → Rm s× n" [p. 6] "Split the image and filter into the coloured sections, with white entries representing zeroes" See also FIG. 1)
and wherein identifying the one or more redundant channels comprises: assigning each of the incoming edges of the one or more vertices comprised by the vertex subset representative of the sequentially first layer of the plurality of layers of the neural network a first state;(Schoonhoven [p. 58] "//// Returns true if edge was marked, else false (if edge was skippable). fn mark_single_edge(&mut self, edge_idx: usize) -> bool { if self.skippable[edge_idx] { return false; } // Mark edge self.marked[edge_idx] = true; self.num_marked += 1;" [p. 32] "if not pruned[edge_idx]" Schoonhoven explicitly maintains a boolean state for every edge and this returned boolean state has direct network meaning)
traversing the sequence of vertex subsets, from the vertex subset representative of the sequentially first layer of the plurality of layers of the neural network, to the vertex subset representative of the sequentially penultimate layer of the plurality of layers of the neural network, (Schoonhoven [p. 59] "BackwardPathIterator" [p. 57] "// Walk backward from largest node to find all edges in longest path" [p. 63] "// Implement an iterator to easily walk backwards over a directed path […] BackwardEdgeIterator […] self.node_idx = incoming_edge.from;" Schoonhoven explicitly performs the downstream result of the path analysis and walks upstream through predecessor connectivity)
assessing each of the one or more vertices in each vertex subset to determine whether that vertex has at least one incoming edge assigned the first state, and: if yes, assigning each of the outgoing edges of that vertex the first state; and if not, not assigning each of the outgoing edges of that vertex the first state;(Schoonhoven [p. 67] "// Recalculate node_distance for &edge_idx in (&node_all_incoming_edge_idxs[dest_node_idx]).iter() […] for &edge_idx in (&node_all_outgoing_edge_idxs[dest_node_idx]).iter() […] for (_, i) in norm_idx_pairs.iter() {pb.set(num_marked as u64); if num_to_mark <= num_marked {
break;} marked[*i] = true; num_marked += 1;")
subsequently, traversing the sequence of vertex subsets, from the vertex subset representative of the sequentially penultimate layer of the plurality of layers of the neural network, to the vertex subset representative of the sequentially first layer of the plurality of layers of the neural network, (Schoonhoven [p. 66] "// Walk forward through path and // - mark edges along the path // - remove edges from hashsets // - mark nodes along path for invalidation […] // Recalculate downstream nodes" Schoonhoven explicitly reverses traversal direction to recalculate downstream nodes)
assessing each of the one or more vertices in each vertex subset to determine whether that vertex has at least one outgoing edge assigned the first state, and: if yes, assigning each of the incoming edges of that vertex with first state; and if not, causing each of the incoming edges of that vertex to not be assigned the first state; and(Schoonhoven [p. 67] "// Recalculate node_distance
for &edge_idx in (&node_all_incoming_edge_idxs[dest_node_idx]).iter() […] for &edge_idx in (&node_all_outgoing_edge_idxs[dest_node_idx]).iter() […] for (_, i) in norm_idx_pairs.iter() { pb.set(num_marked as u64); if num_to_mark <= num_marked {
break;} marked[*i] = true; num_marked += 1;")
subsequently, identifying one or more vertices that do not have any outgoing edges assigned with first state, said one or more identified vertices representing the one or more redundant channels comprised by the plurality of layers of the neural network.(Schoonhoven [p. 32] "if prunedQ and default_mask[i, j].sum() == 0: # it already was pruned, ergo it has no edge continue code = codebook[code_iter] edge_idx = code[1] if not pruned[edge_idx]:").
Regarding claim 13, Schoonhoven teaches The method of claim 4, wherein the plurality of vertex subsets are arranged in a sequence representative of the sequence in which the plurality of layers of the neural network are arranged, and wherein identifying the one or more redundant channels comprises:(Schoonhoven [p. 3] "Every operation, e.g., convolution, has an input x and output y. The input and output consist of one or more images, called channels […] A convolution with stride s defines a map h : Rm×n → Rm s× n" [p. 6] "Split the image and filter into the coloured sections, with white entries representing zeroes" See also FIG. 1)
assigning each of the incoming edges of the vertices of the vertex subset representative of the sequentially last layer of the plurality of layers of the neural network a first state;(Schoonhoven [p. 58] "//// Returns true if edge was marked, else false (if edge was skippable). fn mark_single_edge(&mut self, edge_idx: usize) -> bool { if self.skippable[edge_idx] { return false; } // Mark edge self.marked[edge_idx] = true; self.num_marked += 1;" [p. 32] "if not pruned[edge_idx]" Schoonhoven explicitly maintains a boolean state for every edge and this returned boolean state has direct network meaning)
traversing the sequence of vertex subsets, from the vertex subset representative of the sequentially penultimate layer of the plurality of layers of the neural network, to the vertex subset representative of the sequentially first layer of the plurality of layers of the neural network, (Schoonhoven [p. 59] "BackwardPathIterator" [p. 57] "// Walk backward from largest node to find all edges in longest path" [p. 63] "// Implement an iterator to easily walk backwards over a directed path […] BackwardEdgeIterator […] self.node_idx = incoming_edge.from;" Schoonhoven explicitly performs the downstream result of the path analysis and walks upstream through predecessor connectivity)
assessing each of the one or more vertices in each vertex subset to determine whether that vertex has at least one outgoing edge assigned the first state, and: if yes, assigning each of the incoming edges of that vertex the first state; and if not, not assigning each of the incoming edges of that vertex the first state; and (Schoonhoven [p. 67] "// Recalculate node_distance for &edge_idx in (&node_all_incoming_edge_idxs[dest_node_idx]).iter() […] for &edge_idx in (&node_all_outgoing_edge_idxs[dest_node_idx]).iter() […] for (_, i) in norm_idx_pairs.iter() { pb.set(num_marked as u64); if num_to_mark <= num_marked {
break; } marked[*i] = true; num_marked += 1;")
subsequently, identifying one or more vertices that do not have any outgoing edges assigned the first state, said one or more identified vertices representing the one or more redundant channels comprised by the plurality of layers of the neural network.(Schoonhoven [p. 32] "if prunedQ and default_mask[i, j].sum() == 0: # it already was pruned, ergo it has no edge continue code = codebook[code_iter] edge_idx = code[1] if not pruned[edge_idx]:").
Regarding claim 14, Schoonhoven teaches The method of claim 4, wherein the plurality of vertex subsets are arranged in a sequence representative of the sequence in which the plurality of layers of the neural network are arranged, and wherein identifying the one or more redundant channels comprises:(Schoonhoven [p. 3] "Every operation, e.g., convolution, has an input x and output y. The input and output consist of one or more images, called channels […] A convolution with stride s defines a map h : Rm×n → Rm s× n" [p. 6] "Split the image and filter into the coloured sections, with white entries representing zeroes" See also FIG. 1)
assigning each of the incoming edges of the one or more vertices comprised by the vertex subset representative of the sequentially first layer of the plurality of layers of the neural network a first state;(Schoonhoven [p. 58] "//// Returns true if edge was marked, else false (if edge was skippable). fn mark_single_edge(&mut self, edge_idx: usize) -> bool { if self.skippable[edge_idx] { return false; } // Mark edge self.marked[edge_idx] = true; self.num_marked += 1;" [p. 32] "if not pruned[edge_idx]" Schoonhoven explicitly maintains a boolean state for every edge and this returned boolean state has direct network meaning)
traversing the sequence of vertex subsets, from the vertex subset representative of the sequentially first layer of the plurality of layers of the neural network, to the vertex subset representative of the sequentially penultimate layer of the plurality of layers of the neural network, (Schoonhoven [p. 59] "BackwardPathIterator" [p. 57] "// Walk backward from largest node to find all edges in longest path" [p. 63] "// Implement an iterator to easily walk backwards over a directed path […] BackwardEdgeIterator […] self.node_idx = incoming_edge.from;" Schoonhoven explicitly performs the downstream result of the path analysis and walks upstream through predecessor connectivity)
assessing each of the one or more vertices in each vertex subset to determine whether that vertex has at least one incoming edge assigned the first state, and: if yes, assigning each of the outgoing edges of that vertex the first state; and if not, not assigning each of the outgoing edges of that vertex the first state; and(Schoonhoven [p. 67] "// Recalculate node_distance for &edge_idx in (&node_all_incoming_edge_idxs[dest_node_idx]).iter() […] for &edge_idx in (&node_all_outgoing_edge_idxs[dest_node_idx]).iter() […] for (_, i) in norm_idx_pairs.iter() { pb.set(num_marked as u64); if num_to_mark <= num_marked {
break; } marked[*i] = true; num_marked += 1;")
subsequently, identifying one or more vertices that do not have any outgoing edges assigned the first state, said one or more identified vertices representing the one or more redundant channels comprised by the plurality of layers of the neural network. (Schoonhoven [p. 32] "if prunedQ and default_mask[i, j].sum() == 0: # it already was pruned, ergo it has no edge continue code = codebook[code_iter] edge_idx = code[1] if not pruned[edge_idx]:").
Regarding claim 15, Schoonhoven teaches The method of claim 11, wherein: an edge can be an outgoing edge of a first vertex and/or an incoming edge of a second vertex;(Schoonhoven [p. 3] "Every operation, e.g., convolution, has an input x and output y. The input and output consist of one or more images, called channels […] A convolution with stride s defines a map h : Rm×n → Rm s× n" [p. 6] "Split the image and filter into the coloured sections, with white entries representing zeroes" See also FIG. 1)
an incoming edge of a vertex is representative of the potential flow of non-zero data into the output channel represented by that vertex; and(Schoonhoven [p. 3] "Every operation, e.g., convolution, has an input x and output y. The input and output consist of one or more images, called channels […] A convolution with stride s defines a map h : Rm×n → Rm s× n" [p. 6] "Split the image and filter into the coloured sections, with white entries representing zeroes" See also FIG. 1)
an outgoing edge of a vertex is representative of the potential flow of non-zero data from the output channel represented by that vertex.(Schoonhoven [p. 3] "Every operation, e.g., convolution, has an input x and output y. The input and output consist of one or more images, called channels […] A convolution with stride s defines a map h : Rm×n → Rm s× n" [p. 6] "Split the image and filter into the coloured sections, with white entries representing zeroes" See also FIG. 1).
Regarding claim 16, Schoonhoven teaches The method of claim 1, further comprising storing the compressed neural network for subsequent implementation.(Schoonhoven [p. 20] "The pruned networks are saved to the pruned_models folder after every pruning step.").
Regarding claim 17, Schoonhoven teaches The method of claim 1, further comprising outputting a computer readable description of the compressed neural network that, when implemented at a system for implementing a neural network, causes the compressed neural network to be executed.(Schoonhoven [p. 7] "As U-Net architecture we use a fully-convolutional (FCN) U-Net4 network, i.e., a U-Net with 4
scaling operations. We used a U-Net4 architecture from the PyTorch-UNet repository" [p. 10] "In order to aid reproducibility the authors have published (Python) code for LEAN pruning. There are installation instructions that aim to help users install and run the code on their machines, as well as explanations and documentation for the code and scripts. The scripts are intended to be usable on relatively simple machines (with GPU and CUDA) [...] the U-Net4 and ResNet50 models and pruning methods are also supplied [...] This way, users can check the experimental results, and experiment themselves with the LEAN method").
Regarding claim 18, Schoonhoven teaches The method of claim 1, further comprising configuring hardware logic to implement the compressed neural network, wherein the hardware logic comprises a neural network accelerator.(Schoonhoven [p. 7] "As U-Net architecture we use a fully-convolutional (FCN) U-Net4 network, i.e., a U-Net with 4 scaling operations. We used a U-Net4 architecture from the PyTorch-UNet repository" [p. 10] "In order to aid reproducibility the authors have published (Python) code for LEAN pruning. There are installation instructions that aim to help users install and run the code on their machines, as well as explanations and documentation for the code and scripts. The scripts are intended to be usable on relatively simple machines (with GPU and CUDA) [...] the U-Net4 and ResNet50 models and pruning methods are also supplied [...] This way, users can check the experimental results, and experiment themselves with the LEAN method" pruning interpreted as a neural network accelerator. Executing said pruned neural network on GPU interpreted as hardware acceleration logic).
Regarding claim 19, claim 19 is directed towards a system for performing the method of claim 1. Therefore, the rejection applied to claim 1 also applies to claim 19. Claim 19 recites additional elements A processing system for compressing a neural network, the processing system comprising at least one processor configured to:(Schoonhoven [p. 10] "In order to aid reproducibility the authors have published (Python) code for LEAN pruning. There are installation instructions that aim to help users install and run the code on their machines, as well as explanations and documentation for the code and scripts. The scripts are intended to be usable on relatively simple machines (with GPU and CUDA)").
Regarding claim 20, claim 20 is directed towards computer readable storage for performing the method of claim 1. Therefore, the rejection applied to claim 1 also applies to claim 20. Claim 20 recites additional elements A non-transitory computer readable storage medium having stored thereon computer readable instructions that, when executed at a computer system, cause the computer system to compress a neural network, by: (Schoonhoven [p. 10] "In order to aid reproducibility the authors have published (Python) code for LEAN pruning. There are installation instructions that aim to help users install and run the code on their machines, as well as explanations and documentation for the code and scripts. The scripts are intended to be usable on relatively simple machines (with GPU and CUDA)").
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim 5 is rejected under U.S.C. §103 as being unpatentable over the combination of Schoonhoven and Samek (US20220114455A1).
Regarding claim 5, Schoonhoven teaches The method of claim 4.
However, Schoonhoven doesn't explicitly teach, wherein a vertex subset of the plurality of vertex subsets is representative of a fully-connected layer of the plurality of layers of the neural network, each vertex of the one or more vertices comprised by that vertex subset being representative of a respective output channel of that fully-connected layer, and wherein forming the graph comprises:
determining a matrix representative of a set of coefficients of the fully-connected layer, the matrix comprising one or more elements representative of non-zero coefficients and one or more elements representative of zero coefficients;
for each of the one or more elements representative of a non-zero coefficient: identifying: an output channel of the fully-connected layer comprising that non-zero coefficient;
an input channel of the fully-connected layer comprising that non-zero coefficient; and
an output channel of a preceding layer of the plurality of layers of the neural network corresponding to the identified input channel of the fully-connected layer; and
connecting, using an edge, a vertex in the vertex subset representative of the identified output channel of the fully-connected layer to a vertex in a different vertex subset of the plurality of vertex subsets representative of the identified output channel of the preceding layer.
Samek, in the same field of endeavor, teaches a vertex subset of the plurality of vertex subsets is representative of a fully-connected layer of the plurality of layers of the neural network, each vertex of the one or more vertices comprised by that vertex subset being representative of a respective output channel of that fully-connected layer, and wherein forming the graph comprises: ([¶0037] "FIG. 5 shows schematically the definition of a layered ML predictor such as a neural network by way of defining the interconnections between two layers of nodes of the ML predictor by way of a mapping function such as a weight matrix with additionally illustrating examples for structured pruning of individual model components, including weights or mapping paths—mapping routes from component i of the input to component j of the adjacent output —, in order to show that pruning of such weights or mapping paths in a structured manner is considered as structured pruning" [¶0081] "dense layer or fully-connected layers are illustrated in FIG. 5")
determining a matrix representative of a set of coefficients of the fully-connected layer, the matrix comprising one or more elements representative of non-zero coefficients and one or more elements representative of zero coefficients;([¶0100] "the non-pruned away portion of the ML predictor 10 or non-quantized-to-zero portion of the ML-predictor 10")
for each of the one or more elements representative of a non-zero coefficient: identifying: an output channel of the fully-connected layer comprising that non-zero coefficient;([¶0100] "the non-pruned away portion of the ML predictor 10 or non-quantized-to-zero portion of the ML-predictor 10" [¶0126] "The apparatus 2 then removes portions of the ML predictor 10 exclusively interconnected to one or more predetermined uninterested output nodes 18 of the ML predictor to obtain the actual ML predictor 10 and its representation 32 on the basis of which the aforementioned pruning and/or quantization starts" [¶0136] "the ML predictor 16 may be trained in such a manner that the one or more output nodes are indicative of the prediction" See also FIG. 5)
an input channel of the fully-connected layer comprising that non-zero coefficient; and([¶0100] "the non-pruned away portion of the ML predictor 10 or non-quantized-to-zero portion of the ML-predictor 10" [¶0126] "The apparatus 2 then removes portions of the ML predictor 10 exclusively interconnected to one or more predetermined uninterested output nodes 18 of the ML predictor to obtain the actual ML predictor 10 and its representation 32 on the basis of which the aforementioned pruning and/or quantization starts" [¶0136] "the ML predictor 16 may be trained in such a manner that the one or more output nodes are indicative of the prediction" See also FIG. 5)
an output channel of a preceding layer of the plurality of layers of the neural network corresponding to the identified input channel of the fully-connected layer; and([¶0100] "the non-pruned away portion of the ML predictor 10 or non-quantized-to-zero portion of the ML-predictor 10" [¶0126] "The apparatus 2 then removes portions of the ML predictor 10 exclusively interconnected to one or more predetermined uninterested output nodes 18 of the ML predictor to obtain the actual ML predictor 10 and its representation 32 on the basis of which the aforementioned pruning and/or quantization starts" [¶0136] "the ML predictor 16 may be trained in such a manner that the one or more output nodes are indicative of the prediction" See also FIG. 5)
connecting, using an edge, a vertex in the vertex subset representative of the identified output channel of the fully-connected layer to a vertex in a different vertex subset of the plurality of vertex subsets representative of the identified output channel of the preceding layer. ([¶0036] "FIG. 5 shows schematically the definition of a layered ML predictor such as a neural network by way of defining the interconnections between two layers of nodes of the ML predictor by way of a mapping function such as a weight matrix with additionally illustrating examples for structured pruning of individual model components, including weights or mapping paths—mapping routes from component i of the input to component j of the adjacent output —, in order to show that pruning of such weights or mapping paths in a structured manner is considered as structured pruning;").
Schoonhoven as well as Samek are directed towards neural network pruning. Therefore, Schoonhoven as well as Samek are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Schoonhoven with the teachings of Samek by pruning a fully-connected layer. Samek provides as additional motivation for combination ([¶0081] “Structured pruning is not limited to neural network type architectures. For instance, the matrix 50 shown may be a collection of interconnection functions mij. Further, although dense layer or fully-connected layers are illustrated in FIG. 5, convolutional layers may be subject to structured pruning as well”). This motivation for combination also applies to the remaining claims which depend on this combination.
Claims 9 and 10 are rejected under U.S.C. §103 as being unpatentable over the combination of Schoonhoven and Sui (US20190303762A1).
Regarding claim 9, Schoonhoven teaches The method of claim 4, wherein: a vertex subset of the plurality of vertex subsets is representative of an add layer of the plurality of layers of the neural network, said vertex subset comprising a number of vertices equal to the number of channels in each of a plurality of activation data sets that that add layer is configured to sum, (Schoonhoven [p. 5] "some CNNs contain operations that are meant to dis tribute features throughout the network, but are not implemented with learnable parameters, e.g., residual connections in ResNet (He et al., 2016). We include residual connections in the pruning graph with an edge weight of 1, but label them as unprunable to prevent the residual connections from being removed from the network")
each vertex comprised by that vertex subset being connected, by respective edges, to vertices in different vertex subsets, said vertices being representative of output channels of preceding layers of the plurality of layers of the neural network,(Schoonhoven [p. 3] "Every operation, e.g., convolution, has an input x and output y. The input and output consist of one or more images, called channels […] A convolution with stride s defines a map h : Rm×n → Rm s× n" See also FIG. 1).
However, Schoonhoven doesn't explicitly teach each of said plurality of activation data sets having the same number of channels, each vertex comprised by that vertex subset being representative of a respective summation operation performed between a set of respective channels of the plurality of activation data sets such that each vertex comprised by that vertex subset is representative of a respective output channel of the add layer; and
said output channels corresponding to the channels of the set of respective channels of the plurality of activation data sets between which the summation operation represented by that vertex is performed..
Sui, in the same field of endeavor, teaches each of said plurality of activation data sets having the same number of channels, each vertex comprised by that vertex subset being representative of a respective summation operation performed between a set of respective channels of the plurality of activation data sets such that each vertex comprised by that vertex subset is representative of a respective output channel of the add layer; and ([¶0009] "the optimization method may further comprise: a subsequent adjacent layer that further reads required data from other feature maps from the off-chip memory. The subsequent adjacent layer may be an element-wise add (ELTWISE) layer." element-wise addition requires same-shaped date)
said output channels corresponding to the channels of the set of respective channels of the plurality of activation data sets between which the summation operation represented by that vertex is performed.([¶0066] "a network computational graph having branches introduces an element-wise add (ELTWISE) layer which is used for adding and merging a plurality of convolution layers").
Schoonhoven as well as Sui are directed towards neural network pruning. Therefore, Schoonhoven as well as Sui are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Schoonhoven with the teachings of Sui by using elementwise add and flattening layers in a pruning graph. Sui provides as additional motivation for combination ([¶0085] “Operations of the FLATTEN layer and the CONCAT layer are operations for special data rearrangement and dimension transformation, can be pruned through specific rules of storing data and/or reading mode.”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 10, Schoonhoven teaches each vertex comprised by each group of vertices in that vertex subset being connected, by a respective edge, to a vertex in a different vertex subset, said vertex representative of an output channel of a preceding layer of the plurality of layers of the neural network, (Schoonhoven [p. 3] "Every operation, e.g., convolution, has an input x and output y. The input and output consist of one or more images, called channels […] A convolution with stride s defines a map h : Rm×n → Rm s× n" See also FIG. 1).
However, Schoonhoven doesn't explicitly teach The method of claim 4, wherein: a vertex subset of the plurality of vertex subsets is representative of a flatten layer of the plurality of layers of the neural network,
said vertex subset comprising n groups of vertices, n being equal to the number of channels of data of an activation data set on which the flatten layer is configured to perform a flatten operation, each group of vertices comprising m vertices, m being equal to the number of values in each channel of data of said activation data set, each vertex comprised by said vertex subset being representative of a respective output channel of the flatten layer; and
said output channel corresponding to the channel of the activation data set on which the part of the flatten operation represented by that group of vertices is performed.
Sui, in the same field of endeavor, teaches The method of claim 4, wherein: a vertex subset of the plurality of vertex subsets is representative of a flatten layer of the plurality of layers of the neural network, ([¶0084] "FIG. 8 shows that a FLATTEN layer is pruned according to an embodiment of the present invention. FIG. 9 shows that a CONCAT (concatenation) layer is pruned according to an embodiment of the present invention. Because a concatenation layer relates to merging of network branches, marking in the graph is needed" See FIG. 8)
said vertex subset comprising n groups of vertices, n being equal to the number of channels of data of an activation data set on which the flatten layer is configured to perform a flatten operation, each group of vertices comprising m vertices, m being equal to the number of values in each channel of data of said activation data set, each vertex comprised by said vertex subset being representative of a respective output channel of the flatten layer; and ([¶0085] "the operation of a FLATTEN layer is to carry out “flattening” for a feature map which has been processed by single convolution, namely, one-dimensional processing")
said output channel corresponding to the channel of the activation data set on which the part of the flatten operation represented by that group of vertices is performed.([¶0084] "FIG. 8 shows that a FLATTEN layer is pruned according to an embodiment of the present invention. FIG. 9 shows that a CONCAT (concatenation) layer is pruned according to an embodiment of the present invention. Because a concatenation layer relates to merging of network branches, marking in the graph is needed" See FIG. 8).
Schoonhoven as well as Sui are directed towards neural network pruning. Therefore, Schoonhoven as well as Sui are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Schoonhoven with the teachings of Sui by using elementwise add and flattening layers in a pruning graph. Sui provides as additional motivation for combination ([¶0085] “Operations of the FLATTEN layer and the CONCAT layer are operations for special data rearrangement and dimension transformation, can be pruned through specific rules of storing data and/or reading mode.”). This motivation for combination also applies to the remaining claims which depend on this combination.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Wang (“Convolutional Neural Network Pruning with Structural Redundancy Reduction”, 2021) is directed towards a pruning graph for neural network compression.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SIDNEY VINCENT BOSTWICK whose telephone number is (571)272-4720. The examiner can normally be reached M-F 7:30am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached on (571)270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SIDNEY VINCENT BOSTWICK/Examiner, Art Unit 2124
Prosecution Insights