Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Remarks
This Office Action is responsive to Applicants' Amendment filed on January 02, 2026, in which claims 1, 2, 7, 8, 11, and 19 have been amended. Claims 5 and 6 have been newly cancelled. No claims have been newly added.
Claims 1-4 and 7-20 are currently pending.
Response to Arguments
With regards to the objections to claims 2, 7, and 8 for minor informalities, Applicant has amended the claims to correct the previously noted informalities, and thus the objections are withdrawn.
With regards to the rejection of claim 6 under 35 U.S.C. 112(b), claim 6 has been cancelled, rendering the rejection moot.
With regards to the rejections of claims Claims 1, 3, 4, 10-15, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Smilkov and Carter “A Neural Network Playground”, in view of Liu et al. “DeepTracker: Visualizing the Training Process of Convolutional Neural Networks”, further in view of Nie et al. “Visualizing Deep Neural Networks for Text Analytics”, Examiner finds Applicant’s arguments that the claims as amended overcome the rejections are persuasive, however the arguments are moot in view of a new grounds of rejection, as presented below, as necessitated by Applicant’s amendments to the claims.
Prior Art
The following references are used for prior art claim rejections:
Smilkov and Carter “A Neural Network Playground”
Liu et al. “DeepTracker: Visualizing the Training Process of Convolutional Neural Networks”
Nie et al. “Visualizing Deep Neural Networks for Text Analytics”
Yuan and Xiao “Scaling-Based Weight Normalization for Deep Neural Networks”
Patro and Sahu “Normalization: A Preprocessing Stage”
Park et al. “SQuantizer: Simultaneous Learning for Both Sparse and Low-precision Neural Networks”
Rogozhnikov et al. “Neural networks in 3d with webGL”
Pushpoth et al. (U.S. Patent Application Publication No. 2021/0042973)
Vyas and Calyam “An Interactive Graphical Visualization Approach to CNNs and RNNs”
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 4, 10-15, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Smilkov and Carter “A Neural Network Playground”, hereinafter Smilkov, in view of Liu et al. “DeepTracker: Visualizing the Training Process of Convolutional Neural Networks”, hereinafter Liu, further in view of Nie et al. “Visualizing Deep Neural Networks for Text Analytics”, hereinafter Nie, further in view of Vyas and Calyam “An Interactive Graphical Visualization Approach to CNNs and RNNs”, hereinafter Vyas.
Regarding claim 1,
Smilkov teaches A method for updating a visual representation of a neural network, said method comprising: (Smilkov shows a visual representation of a neural network that reflects the changes caused by iterative training, which corresponds to updating the visual representation)
PNG
media_image1.png
686
1430
media_image1.png
Greyscale
after a selected number of iterations in which input data is fed into a neural network, obtaining network data describing the neural network, (Smilkov shows that the displayed network data is obtained after input data is fed into a neural network for 313 iterations)
wherein the network data includes state data describing a state of the neural network and structure data describing a structure of the neural network; (Smilkov shows that the network data includes state data such as weight values, such as the indicated weight of -2.1 for the second neuron in the second hidden layer, and structure data such as number of hidden layers)
generating a visual representation of the neural network, (Smilkov shows a visual representation of a neural network)
wherein the visual representation includes a set of nodes comprising one or more input nodes, one or more hidden layer nodes, [and one or more output nodes,] (Smilkov shows a column of one or more input nodes (features), one or more columns of one or more hidden layer nodes (hidden layers), Smilkov does not explicitly show output nodes)
and wherein the visual representation further includes edges connecting various ones of said nodes; (Smilkov shows edges connecting nodes between layers)
Liu teaches the following further limitations that Smilkov does not teach:
normalizing at least some of the network data, ((Liu Pg. 18) “In our system, two different methods (i.e., filter-based or iteration-based) are used to normalize weight changes at the filter-level”)
updating the visual representation using the normalized network data, (Liu Pg. 8, Fig. 2 shows that the visual representations of the validation view and the correlation view, which do not use normalized network data, are updated to incorporate the layer view, which does use normalized network data, to form the cube view)
PNG
media_image2.png
447
830
media_image2.png
Greyscale
wherein, as a result of updating the visual representation using the normalized network data, a display [of the nodes and/or of the edges] is modified in a manner to reflect a relative relationship that exists between the nodes and/or the edges, ((Liu Pg. 18) “For example, Figure 7(d) visualizes the filter changes in one of the CONV layer belonging to the second CONV module using filter-based normalization. Our experts found that the changes are drastic in stage s1 and become relatively small in the later stages because of the decrease in learning rate and the convergence of the model”, (Liu Pg. 3) “A CONV layer comprises numerous neurons that are connected to a local region in the previous layer’s output volume through weighted edges, many of which share the same weights through a parameter sharing scheme. The weights in each neuron compose a filter, the basic unit for detecting visual features in the input image”, modifying a visual representation using normalized data to reflect a relative relationship between filters in a layer corresponds to modifying the representation to reflect a relationship between nodes and edges, because a filter consists of the weights of each node’s edges, Smilkov but not Liu displays nodes and edges directly)
and wherein the relative relationship is based on the normalized network data; ((Liu Pg. 18) “For filter-based normalization, changes are grouped and normalized by filters, which aims to help experts see the change distribution over iterations for individual filters. Similarly, iteration-based normalization allows users to examine the distribution over filters for individual iterations”)
and displaying the updated visual representation. (Liu Pg. 19, Fig. 9 shows a complete visual representation updated with normalized data in the layer view)
PNG
media_image3.png
596
823
media_image3.png
Greyscale
At the time of filing, one of ordinary skill in the art would have motivation to combine Smilkov and Liu by taking the method for neural network visualization, including iterations, state data, structure data, input and hidden layer nodes, and edges, taught by Smilkov, and adding updating the model based on normalization in order to better show a relative relationship between neurons, taught by Liu, as Liu teaches: (Liu Pg. 18) “For filter-based normalization, changes are grouped and normalized by filters, which aims to help experts see the change distribution over iterations for individual filters”, that is, that the normalization aids in comprehension of important information. Such a combination would be obvious.
Nie teaches the following further limitations that neither Smilkov nor Liu explicitly teaches:
wherein the visual representation includes…and one or more output nodes, (Nie Pg. 4, Fig. 4 shows an output layer consisting of several output nodes)
PNG
media_image4.png
686
1182
media_image4.png
Greyscale
At the time of filing, one of ordinary skill in the art would have motivation to combine Smilkov, Liu, and Nie by taking the method for neural network visualization, including iterations, state data, structure data, input and hidden layer nodes, edges, and normalization of data, taught jointly by Smilkov and Liu, and adding display of output neurons, taught by Nie, as doing so provides more information to the user as to how an output class, corresponding to the highest valued output node, was chosen based on the flow of weight data throughout the network and to the given output node. Such a combination would be obvious.
Vyas teaches the following further limitations that neither Smilkov, nor Liu, nor Nie explicitly teaches:
wherein said normalizing includes normalizing, for each layer of the neural network and by a visualization component that is external to the neural network, the state data within that layer to a common scale ((Vyas Pg. 3) “The main interface of our CNN implementation displays the network architecture and layer information as shown in Fig. 1. The architecture visualizes the dataflow and the input and output shapes of the data at different layers. Users can use a slider to view layer information specific to the type of layer. For convolution layers, we display the filter weights and normalize the weight to display a grayscale image of the weights”, (Vyas Pg. 6) “Using a web-based, graphical approach, we visualized various aspects of the model including the overall model architecture”, a web-based graphical approach is a visualization component external to the visualized neural network, grayscale is a common scale) based on a range of weight values for that layer, (Vyas Pg. 4, Fig. 1 shows that the greyscale display image, which is for a particular convolutional layer, is normalized based on the range of weight values: the highest weight value of +0.62 is white in the greyscale image, the lowest weight value of -0.39 is black in the greyscale image, and the weights in between are shades of grey)
PNG
media_image5.png
442
1140
media_image5.png
Greyscale
such that each layer is normalized differently from at least one other layer in the neural network; (Vyas Pg. 4, Fig. 1 shows that each greyscale display image, produced by normalization of weights in a layer, is unique to the layer, the greyscale display images for other layers would be normalized based on those weights, which would in most cases be different)
At the time of filing, one of ordinary skill in the art would have motivation to combine Smilkov, Liu, Nie, and Vyas by taking the method for neural network visualization, including iterations, state data, structure data, input nodes, hidden layer nodes, output nodes, edges, and normalization of data, taught jointly by Smilkov, Liu, and Nie, and having the normalization be performed by an external visualization component and be applied to layer state data to render it in a common scale based on the range of weight values of the layer, with each layer being normalized differently from at least one other, taught by Vyas, as doing so provides a user with an intuitive scale for conceptualizing the operations of a layer, aiding user understanding of the neural network. Such a combination would be obvious.
Regarding claim 3,
Smilkov, Liu, Nie, and Vyas jointly teach The method of claim 1,
Smilkov further teaches:
wherein the selected number of iterations is at least two iterations. (Smilkov shows 313 iterations selected, which is at least two iterations)
At the time of filing, one of ordinary skill in the art would have motivation to combine the method jointly taught by Smilkov, Liu, Nie, and Vyas for the parent claim of claim 3, claim 1. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 4,
Smilkov, Liu, Nie, and Vyas jointly teach The method of claim 1,
Smilkov further teaches:
wherein the network data further includes weights for the set of nodes (Smilkov shows weights for each node reflected by the thickness of the output edges, exact numbers such as a weight of -2.1 are also available by hovering over the edge with a user cursor)
At the time of filing, one of ordinary skill in the art would have motivation to combine the method jointly taught by Smilkov, Liu, Nie, and Vyas for the parent claim of claim 4, claim 1. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 10,
Smilkov, Liu, Nie, and Vyas jointly teach The method of claim 1,
Liu further teaches:
wherein generating the visual representation is performed using a three-dimensional (3D) mesh (Liu Pg. 19, Fig. 9 shows a 3D visual representation)
At the time of filing, one of ordinary skill in the art would have motivation to combine the method jointly taught by Smilkov, Liu, Nie, and Vyas for the parent claim of claim 10, claim 1. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 11,
Claim 11 recites a computer system comprising a processor and a memory for performing the function of the method of claim 1.
Specifically, claim 11 recites A computer system that updates a visual representation of a neural network, said computer system comprising: at least one processor; and at least one hardware storage device that stores instructions that are executable by the at least one processor to cause the computer system to: [perform the method of claim 11]. Liu teaches: (Liu Pg. 7) “DeepTracker is a web-based application developed under the full-stack framework, MEAN.ts (i.e., MongoDB, Express, AngularJs, Node, and Typescript). The back-end part of our application is deployed in a server with 3.10GHz Intel Xeon E5-2687W CPU and 32GB memory.”
All other limitations in claim 11 are substantially the same as those in claim 1, therefore the same rationale for rejection applies.
Regarding claim 12,
Smilkov, Liu, Nie, and Vyas jointly teach The computer system of claim 11,
Nie further teaches:
wherein the state data includes weights and thresholds for the set of nodes ((Nie Pg. 5) “The system provides five basic functions: coloring to represent node and edge activation values,…and tooltips to expose detailed information about a node’s or edge’s weight, pre and post-activation, and bias values”, (Nie Pg. 2) “A common activation function is ReLU, which prunes negative parts of an input to zero and retains positive parts”, activations in which a threshold is set so only positive parts of an input are kept correspond to state data on thresholds)
At the time of filing, one of ordinary skill in the art would have motivation to combine the system jointly taught by Smilkov, Liu, Nie, and Vyas for the parent claim of claim 12, claim 11. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 13,
Smilkov, Liu, Nie, and Vyas jointly teach The computer system of claim 11,
Smilkov further teaches:
wherein the iterations are a part of a training phase for the neural network. (Smilkov shows that the training loss decreases as the iterations progress)
At the time of filing, one of ordinary skill in the art would have motivation to combine the system jointly taught by Smilkov, Liu, Nie, and Vyas for the parent claim of claim 13, claim 11. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 14,
Smilkov, Liu, Nie, and Vyas jointly teach The computer system of claim 11,
Smilkov further teaches:
wherein the iterations are a part of an evaluation phase for the neural network where the neural network is being tuned (Smilkov shows that when the simulation is paused, training stops and the neural network can be evaluated)
At the time of filing, one of ordinary skill in the art would have motivation to combine the system jointly taught by Smilkov, Liu, Nie, and Vyas for the parent claim of claim 14, claim 11. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 15,
Smilkov, Liu, Nie, and Vyas jointly teach The computer system of claim 11,
Liu further teaches:
wherein the selected number of iterations is based on a determination as to whether the neural network has reached a convergence state ((Liu Pg. 8) “we conducted our experiments with…ImageNet Dataset…ImageNet 2012 is also among the largest and most challenging publicly available datasets. The dataset includes 1,000 classes of images, with each class containing 1,300 training images and 50 validation images. Training such a model needs around 120 epoches (nearly 1.2 millions iterations when batch size is 128) to achieve convergence”)
At the time of filing, one of ordinary skill in the art would have motivation to combine the system jointly taught by Smilkov, Liu, Nie, and Vyas for the parent claim of claim 15, claim 11. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Regarding claim 17,
Smilkov, Liu, Nie, and Vyas jointly teach The computer system of claim 11,
Smilkov further teaches:
wherein modifying the display of the nodes and/or of the edges includes modifying a displayed thickness of the nodes and/or edges (Smilkov shows that the edges output by each node has a thickness reflecting the value of the weight)
At the time of filing, one of ordinary skill in the art would have motivation to combine the system jointly taught by Smilkov, Liu, Nie, and Vyas for the parent claim of claim 17, claim 11. No new embodiments are introduced, so the reason to combine is the same as for the parent claim.
Claims 2, 8, 16, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Smilkov, in view of Liu, further in view of Nie, further in view of Vyas, further in view of Yuan and Xiao “Scaling-Based Weight Normalization for Deep Neural Networks”, hereinafter Yuan, further in view of Patro and Sahu “Normalization: A Preprocessing Stage”, hereinafter Patro.
Regarding claim 2,
Smilkov, Liu, Nie, and Vyas jointly teach The method of claim 1, wherein said normalizing includes:
Vyas further teaches:
identifying weights for a subset of nodes that are included in a same layer of the neural network; ((Vyas Pg. 3) “The main interface of our CNN implementation displays the network architecture and layer information as shown in Fig. 1. The architecture visualizes the dataflow and the input and output shapes of the data at different layers. Users can use a slider to view layer information specific to the type of layer. For convolution layers, we display the filter weights and normalize the weight to display a grayscale image of the weights”, Vyas Pg. 4, Fig. 1 shows that the weights displayed are for a particular layer of the neural network)
computing a range for the weights; (Vyas Pg. 4, Fig. 1 shows that the greyscale display image, which is for a particular convolutional layer, is normalized based on the range of weight values: the highest weight value of +0.62 is white in the greyscale image, the lowest weight value of -0.39 is black in the greyscale image, and the weights in between are shades of grey)
Yuan teaches the following further limitation more explicitly than Vyas teaches and that neither Smilkov, nor Liu, nor Nie teaches:
and normalizing each weight ((Yuan Abstract) “we proposed the scaling-based weight normalization”) by dividing each weight … ((Yuan Pg. 6, Algorithm 1) “8: scale the weights and biases: wk (i, :) ← wk (i, :) /αk+1 (i)”)
At the time of filing, one of ordinary skill in the art would have motivation to combine Smilkov, Liu, Nie, Vyas, and Yuan by taking the method of claim 1, including identifying weights for nodes in a layer and computing a range for the weights, taught jointly by Smilkov, Liu, Nie, and Vyas, and adding normalization of the weights of neurons via division, taught by Yuan, as Yuan teaches: (Yuan Pg. 9) “The additional computation cost introduced by performing scaling-based normalization is negligible. Our proposed method can compatible with the commonly used optimization algorithms and collaborates well with the batch normalization. The extensive empirical results show that our proposed method improves the performance of various state-of-the-art network architectures over various datasets”, and moreover it is well known in the art for data normalization operations to encompass division. Such a combination would be obvious.
Patro teaches the following further limitation more explicitly than Vyas and that neither Smilkov, nor Liu, nor Nie, nor Yuan teaches:
and normalizing each [weight] by dividing each [weight] by the computed range. ((Patro Pg. 2) “Min-Max normalization is a simple technique where the technique can specifically fit the data in a pre-defined boundary with a pre-defined boundary. As per Min-Max normalization technique, [Equation]”, the min-max normalization of Patro shows normalization by dividing by a range, Patro does not teach normalization of weights)
PNG
media_image6.png
42
465
media_image6.png
Greyscale
At the time of filing, one of ordinary skill in the art would have motivation to combine Smilkov, Liu, Nie, Vyas, Yuan, and Patro by taking the method of claim 1, alongside identifying weights for nodes in a layer, computing a range for the weights, and normalization of neural network weights using division, taught jointly by Smilkov, Liu, Nie, Vyas, and Yuan, and adding normalization by dividing a data point by the computed range, taught by Patro, as such a technique, known by the term “min-max normalization”, is a well-known normalization technique within the art, and performs a similar function to the normalization technique used by Yuan, and substituting the normalization technique of Patro for the weight normalization technique of Yuan yields the predictable outcome of also normalizing the weights. Such a combination would be obvious.
Regarding claim 8,
Smilkov, Liu, Nie, and Vyas jointly teach The method of claim 1, wherein said normalizing includes:
Vyas further teaches:
identifying weights for a subset of nodes that are included in a same layer of the neural network; ((Vyas Pg. 3) “The main interface of our CNN implementation displays the network architecture and layer information as shown in Fig. 1. The architecture visualizes the dataflow and the input and output shapes of the data at different layers. Users can use a slider to view layer information specific to the type of layer. For convolution layers, we display the filter weights and normalize the weight to display a grayscale image of the weights”, Vyas Pg. 4, Fig. 1 shows that the weights displayed are for a particular layer of the neural network)
computing a range for the weights; (Vyas Pg. 4, Fig. 1 shows that the greyscale display image, which is for a particular convolutional layer, is normalized based on the range of weight values: the highest weight value of +0.62 is white in the greyscale image, the lowest weight value of -0.39 is black in the greyscale image, and the weights in between are shades of grey)
Smilkov further teaches:
wherein a raw value for at least one weight is negative. (Smilkov shows that the value for at least one weight is negative, specifically -2.1)
Yuan teaches the following further limitation more explicitly than Vyas teaches and that neither Smilkov, nor Liu, nor Nie teaches:
and normalizing each weight’s raw value ((Yuan Abstract) “we proposed the scaling-based weight normalization”) by dividing each weight’s raw value … ((Yuan Pg. 6, Algorithm 1) “8: scale the weights and biases: wk (i, :) ← wk (i, :) /αk+1 (i)”, the value of the weight that is divided in Yuan’s algorithm is unmodified and thus raw)
At the time of filing, one of ordinary skill in the art would have motivation to combine Smilkov, Liu, Nie, Vyas, and Yuan for the method of claim 8, for the same reasons to combine given in claim 2. Such a combination would be obvious.
Patro teaches the following further limitation more explicitly than Vyas and that neither Smilkov, nor Liu, nor Nie, nor Yuan teaches:
and normalizing each [weight] by dividing each [weight] by the computed range. ((Patro Pg. 2) “Min-Max normalization is a simple technique where the technique can specifically fit the data in a pre-defined boundary with a pre-defined boundary. As per Min-Max normalization technique, [Equation]”, the min-max normalization of Patro shows normalization by dividing by a range, Patro does not teach normalization of weights)
PNG
media_image6.png
42
465
media_image6.png
Greyscale
At the time of filing, one of ordinary skill in the art would have motivation to combine Smilkov, Liu, Nie, Vyas, Yuan, and Patro for the method of claim 8, for the same reasons to combine given in claim 2. Such a combination would be obvious.
Regarding claim 16,
Smilkov, Liu, Nie, and Vyas jointly teach The computer system of claim 11, wherein said normalizing includes:
Vyas further teaches:
identifying weights for a subset of nodes that are included in a same layer of the neural network; ((Vyas Pg. 3) “The main interface of our CNN implementation displays the network architecture and layer information as shown in Fig. 1. The architecture visualizes the dataflow and the input and output shapes of the data at different layers. Users can use a slider to view layer information specific to the type of layer. For convolution layers, we display the filter weights and normalize the weight to display a grayscale image of the weights”, Vyas Pg. 4, Fig. 1 shows that the weights displayed are for a particular layer of the neural network)
Yuan teaches the following further limitation more explicitly than Vyas teaches and that neither Smilkov, nor Liu, nor Nie teaches:
and normalizing each weight ((Yuan Abstract) “we proposed the scaling-based weight normalization”) by dividing each weight … ((Yuan Pg. 6, Algorithm 1) “8: scale the weights and biases: wk (i, :) ← wk (i, :) /αk+1 (i)”)
At the time of filing, one of ordinary skill in the art would have motivation to combine Smilkov, Liu, Nie, Vyas, and Yuan for the system of claim 16, for the same reasons to combine given in claim 2. Such a combination would be obvious.
Patro teaches the following further limitations more explicitly than Vyas and that neither Smilkov, nor Liu, nor Nie, nor Yuan teaches:
identifying a first [weight] having a highest value among said [weights]; ((Patro Pg. 2) “As per Min-Max normalization technique, [Equation]”, the largest (max) value in the dataset A is identified, Patro does not teach weights)
PNG
media_image6.png
42
465
media_image6.png
Greyscale
identifying a second [weight] having a lowest value among said [weights]; ((Patro Pg. 2) “As per Min-Max normalization technique, [Equation]”, the smallest (min) value in the dataset A is identified, Patro does not teach weights)
computing a range by subtracting the lowest value from the highest value; ((Patro Pg. 2) “As per Min-Max normalization technique, [Equation] Where, A’ contains Min-Max Normalized data one…If A is the range of original data”)
and normalizing each [weight] by dividing each [weight] by the computed range. ((Patro Pg. 2) “Min-Max normalization is a simple technique where the technique can specifically fit the data in a pre-defined boundary with a pre-defined boundary. As per Min-Max normalization technique, [Equation]”, the min-max normalization of Patro shows normalization by dividing by a range, Patro does not teach normalization of weights)
At the time of filing, one of ordinary skill in the art would have motivation to combine Smilkov, Liu, Nie, Vyas, Yuan, and Patro for the system of claim 16, for the same reasons to combine given in claim 2. Such a combination would be obvious.
Regarding claim 19,
Claim 19 recites a method for performing the function of the system of claim 16, including the limitations inherited from claim 16’s parent claim, claim 11. All other limitations in claim 19 are substantially the same as those in claims 11 and 16, therefore the same rationale for rejection applies.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Smilkov, in view of Liu, further in view of Nie, further in view of Vyas, further in view of Park et al. “SQuantizer: Simultaneous Learning for Both Sparse and Low-precision Neural Networks”, hereinafter Park.
Regarding claim 7,
Smilkov, Liu, Nie, and Vyas jointly teach The method of claim 1, wherein said normalizing includes:
Vyas further teaches:
identifying weights for a subset of nodes that are included in a same layer of the neural network; ((Vyas Pg. 3) “The main interface of our CNN implementation displays the network architecture and layer information as shown in Fig. 1. The architecture visualizes the dataflow and the input and output shapes of the data at different layers. Users can use a slider to view layer information specific to the type of layer. For convolution layers, we display the filter weights and normalize the weight to display a grayscale image of the weights”, Vyas Pg. 4, Fig. 1 shows that the weights displayed are for a particular layer of the neural network)
computing a range for the weights; (Vyas Pg. 4, Fig. 1 shows that the greyscale display image, which is for a particular convolutional layer, is normalized based on the range of weight values: the highest weight value of +0.62 is white in the greyscale image, the lowest weight value of -0.39 is black in the greyscale image, and the weights in between are shades of grey)
Park teaches the following further limitations that neither Smilkov, nor Liu, nor Nie, nor Vyas teaches:
computing an absolute value for each weight; ((Park Pg. 4) “Based on Equation 3 to Equation 6, we quantize a full-precision non-zero element of sparse weight sparseWnl (I, j) to k-bit wsq”, Park Pg. 4, Equation 4 shows that the absolute value of the sparse weight is taken)
PNG
media_image7.png
60
433
media_image7.png
Greyscale
and normalizing each weight's absolute value by dividing each weight's absolute value by the computed range. (Park Pg. 4, Equation 4 shows that the absolute value of the sparse weight is divided by a computed range)
At the time of filing, one of ordinary skill in the art would have motivation to combine Smilkov, Liu, Nie, Vyas, and Park by taking the method of claim 1, alongside normalization of neural network weights in a layer using a range, taught jointly by Smilkov, Liu, Nie, and Vyas, and adding use of the absolute value of the weights, taught by Park, as doing so makes the normalized weights simpler to understand due to only being non-negative, thus aiding in user comprehension of the visualization. Such a combination would be obvious.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Smilkov, in view of Liu, further in view of Nie, further in view of Vyas, further in view of Rogozhnikov et al. “Neural networks in 3d with webGL”, hereinafter Rogozhnikov.
Regarding claim 9,
Smilkov, Liu, Nie, and Vyas jointly teach The method of claim 1,
Rogozhnikov teaches the following further limitations that neither Smilkov, nor Liu, nor Nie, nor Vyas teaches:
wherein generating the visual representation is performed using a shader. ((Rogozhnikov) “This demonstration employs a variation of raymarching technique in computer graphics (also known as volume ray casting). It is highly recommended to have a good GPU-acceleration to enjoy the picture, because in raymarching everything is computed with shaders only (and neural network is calculated with shaders as well).”)
At the time of filing, one of ordinary skill in the art would have motivation to combine Smilkov, Liu, Nie, Vyas, and Rogozhnikov by taking the method of claim 1, taught jointly by Smilkov, Liu, Nie, and Vyas, and adding creation of the neural network visualization using a shader, taught by Rogozhnikov, as shaders are a common and well-known technique within the art for creating 3D visuals, and rendering the visualization in 3D can aid the user in the comprehension of the visualized information. Such a combination would be obvious.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Smilkov, in view of Liu, further in view of Nie, further in view of Vyas, further in view of Pushpoth et al. (U.S. Patent Application Publication No. 2021/0042973), hereinafter Pushpoth.
Regarding claim 18,
Smilkov, Liu, Nie, and Vyas jointly teach The computer system of claim 11,
Pushpoth teaches the following further limitations that neither Smilkov, nor Liu, nor Nie, nor Vyas teaches:
wherein modifying the display of the nodes includes modifying a border of at least one node ((Pushpoth [0050]) “FIG. 3 shows an embodiment of child node 210 as the selected node by generating the rectangle-shaped polygon for child node 210 with an increased border thickness and increased thickness of the line connecting child node 210 and first node 205. By increasing the thickness of the connecting line and the border around child node 210, the user is able to identify child node 210 as the selected child node”)
At the time of filing, one of ordinary skill in the art would have motivation to combine Smilkov, Liu, Nie, Vyas, and Pushpoth by taking the system of claim 11, taught jointly by Smilkov, Liu, Nie, and Vyas, and adding modifying the border of at least one displayed node, taught by Pushpoth, as doing so performs a similar function to the saturation of colors of nodes with higher activations to guide user attention to them within Nie, and substituting the node display technique of Nie for the node display technique of Pushpoth yields the predictable outcome of also assisting the user in identifying the most relevant nodes. Such a combination would be obvious.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Smilkov, in view of Liu, further in view of Nie, further in view of Vyas, further in view of Yuan, further in view of Patro, further in view of Pushpoth.
Regarding claim 20,
Smilkov, Liu, Nie, Vyas, Yuan, and Patro jointly teach The method of claim 19,
Pushpoth teaches the following further limitations that neither Smilkov, nor Liu, nor Nie, nor Vyas, nor Yuan, nor Patro explicitly teach:
wherein modifying the display of the nodes includes modifying a border of at least one node ((Pushpoth [0050]) “FIG. 3 shows an embodiment of child node 210 as the selected node by generating the rectangle-shaped polygon for child node 210 with an increased border thickness and increased thickness of the line connecting child node 210 and first node 205. By increasing the thickness of the connecting line and the border around child node 210, the user is able to identify child node 210 as the selected child node”)
At the time of filing, one of ordinary skill in the art would have motivation to combine Smilkov, Liu, Nie, Vyas, Yuan, Patro, and Pushpoth by taking the method of claim 19, taught jointly by Smilkov, Liu, Nie, Vyas, Yuan, and Patro, and adding modifying the border of at least one displayed node, taught by Pushpoth, as doing so performs a similar function to the saturation of colors of nodes with higher activations to guide user attention to them within Nie, and substituting the node display technique of Nie for the node display technique of Pushpoth yields the predictable outcome of also assisting the user in identifying the most relevant nodes. Such a combination would be obvious.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Healey et al. (U.S. Patent No. 9,934,462) discloses a neural network visualization technique, including edges and nodes of the neural network, using a quilt graph.
Xia et al. (U.S. Patent No. 12,198,046) discloses a neural network visualization technique, including layer parameters and loss function values based on layer parameters.
Smilkov et al. “Direct-Manipulation Visualization of Deep Networks” discloses details and motivations related to the cited prior art of Smilkov and Carter “A Neural Network Playground”.
Sohl-Dickstein et al. (U.S. Patent Application Publication No. 2022/0108149) discloses a neural network system with pre-normalized layers and regularization normalization layers.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VICTOR A NAULT whose telephone number is (703) 756-5745. The examiner can normally be reached M - F, 12 - 8.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached at (571) 270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/V.A.N./Examiner, Art Unit 2124
/Kevin W Figueroa/Primary Examiner, Art Unit 2124
Prosecution Insights