DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1, 19, and 21 were amended.
Claims 1-16, 19, and 21 are pending and examined herein.
Claims 1-16, 19, and 21 are rejected under 35 U.S.C. 103.
Response to Arguments
Applicant’s arguments, see page 10, filed 03/26/2026, with respect to the objection of claim 19 have been fully considered and are persuasive. The objection of claim 19 has been withdrawn.
Applicant’s arguments, see pages 1-4, filed 12/17/2025, with respect to the rejection(s) of claim(s) 1-16, 19, and 21 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Konishi (US 2021/0056420 A1), Tensorboard (“TensorBoard: Graph Visualization”, 2019), Szymanski (“Conceptual Capacity and Efffective Complexity of Neural Networks”, March 2021), and Cassimon (“Designing Resource-Constrained Neural Networks Using Neural Architecture Search Targeting Embedded Devices”, 2020).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-3, 5-7, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Konishi (US 2021/0056420 A1), Tensorboard (“TensorBoard: Graph Visualization”, 2019), Szymanski (“Conceptual Capacity and Efffective Complexity of Neural Networks”, March 2021), and Cassimon (“Designing Resource-Constrained Neural Networks Using Neural Architecture Search Targeting Embedded Devices”, 2020).
Regarding claim 1, Konishi teaches
A method of optimizing a neural network model, the method comprising: ([0005] states "In view of this, the present disclosure provides a neural network construction device that contributes to an improvement in the efficiency of obtaining an optimal neural improvement in the efficiency of obtaining an optimal neural network by narrowing candidate neural networks. Furthermore, the present disclosure provides a neural network construction method and a recording medium that are used by the neural network construction device.")
receiving first model information about a first neural network model, the first neural network model comprising layers; ([0056] states "The obtainer may further obtain learning data on the neural network, the determination unit may output data indicating a model generated by the generator and determined as meeting the second condition, the learning unit may perform, using the learning data, learning of the model indicated in the data output by the determination unit, and the outputter may output at least a part of the model that has already been learned." The data indicating a model generated by the generator is interpreted as the first model information. Fig. 1 shows that the model comprises layers.)
receiving device information about a first target device that is used to execute the first
neural network model; (The abstract states "A neural network construction device includes: an obtainer which obtains resource information related to a computational resource of an embedded device and performance constraints related to processing performance of the embedded device;". The resource information is interpreted as the device information.)
performing an analysis on execution of the first neural network model first target device, based on the first model information, the device information, and at least one of a plurality of suitability determination algorithms; and ([0157] states "Next, generator 13 calculates a duration of the inference process using the source code obtained by the conversion in Step S515 (S516)." As the calculation is of the inference process, this involves the model information. [0157] further states "Furthermore, generator 13 calculates a duration of a process with said number of execution cycles using information having an impact on the processing time such as the operating frequency, etc., of the arithmetic processing device included in the resource information obtained in Step S501." [0158] states "Next, determination unit 14 determines whether or not the duration calculated in Step S516 meets target processing time which is the second condition included in the condition information obtained in Step S501, in other words, the performance constraints (S517)." The determination is interpreted as performing the analysis, and the performance constraints are interpreted as the suitability determination algorithms.)
outputting a result of the analysis such that the first model information and the result of
the analysis are displayed on a screen, ([0088], in regards to FIG. 3, states "Output device 4 is, for example, a display such as a monitor, and displays text and graphics on a screen to prompt a user to input data or to present the progress or the result of a process performed by arithmetic processing device 3.")
Konishi does not appear to explicitly teach
wherein the outputting comprises:
displaying a graphical representation on a graphical user interface (GUI), the graphical representation showing layer boxes corresponding to the layers, a connection between the layer boxes corresponding to a connection between the layers in the neural network model, and a structure of the layers between an input and an output of the first neural network model;
displaying, on the GUI that displays the graphical representation, a menu including a plurality of selectable GUI elements respectively corresponding to the plurality of suitability determination algorithms, the plurality of selectable GUI elements including a first GUI element corresponding to a first algorithm for a performance score, a second GUI element corresponding to a complexity score, and a third algorithm for a memory footprint score, and a fourth GUI element corresponding to a total score based on the performance score, the complexity score, and the memory footprint score; and
receiving, via the GUI, a user input to select one of the plurality of GUI elements from the menu on the GUI, and based on a corresponding suitability determination algorithm of the selected GUI element, displaying the layer boxes of the graphical representation on the GUI in a manner such that a first layer box that corresponds to a layer having a higher score is displayed according to a first color, and a second layer box that corresponds to a layer having a lower score according to the corresponding suitability determination algorithm is displayed according to a second color different from the first color,
wherein, for any one of other GUI elements among the first through fourth GUI elements that is selected by the user input via the GUI, the GUI displays the layer boxes of the graphical representation in the same manner of displaying a layer box of a layer having a higher score based on a corresponding suitability determination algorithm of a selected one of other GUI elements according to the first color and displaying a layer box of a layer having a lower score based on the corresponding suitability determination algorithm of the selected one of other GUI elements according to the second color.
However, Tensorboard—directed to analogous art—teaches
wherein the outputting comprises: (Pages 1-5 shows output of the system.)
displaying a graphical representation on a graphical user interface (GUI), the graphical representation showing layer boxes corresponding to the layers, a connection between the layer boxes corresponding to a connection between the layers in the neural network model, and a structure of the layers between an input and an output of the first neural network model; (Page 1 shows the graph, with nodes labeled with operations such as conv, referring to a convolutional layer of a neural network. Therefore, these nodes are interpreted as layer boxes. The layer boxes are connected by connections. Page 2 states “Tensorflow graphs have two kinds of connections: data dependencies and control dependencies. Data dependencies show the flow of tensors between two ops and are shown as solid arrows, while control dependencies use dotted lines.” The data dependency connections are interpreted as the connection between the layers. Therefore, the connection corresponds to connections between the layers in the neural network model. As the connections show the flow of tensors, meaning a flow from an input to an output, the graph shows a structure of the layers between an input and an output of the first neural network model. )
displaying, on the GUI that displays the graphical representation, a menu including a plurality of selectable GUI elements respectively corresponding to the plurality of suitability determination algorithms, the plurality of selectable GUI elements including a first GUI element corresponding to a first algorithm for a performance score, a second GUI element corresponding to a …, and a third algorithm for a memory footprint score, and a fourth GUI element corresponding to a …; and (Page 5 shows the Color menu, interpreted as the menu, which includes four radio buttons, interpreted as the four selectable GUI elements. It includes a first algorithm for a performance score (compute time) and a third algorithm for a memory footprint (memory).)
receiving, via the GUI, a user input to select one of the plurality of GUI elements from the menu on the GUI, and based on a corresponding suitability determination algorithm of the selected GUI element, displaying the layer boxes of the graphical representation on the GUI in a manner such that a first layer box that corresponds to a layer having a higher score is displayed according to a first color, and a second layer box that corresponds to a layer having a lower score according to the corresponding suitability determination algorithm is displayed according to a second color different from the first color. (Page 5 shows that, when a color button is selected (a user input is received), layer boxes are colored closer to a first color, in this case white, when the layer box has a lower score and colored in another color, in this case red, when the score is higher.)
wherein, for any one of other GUI elements among the first through fourth GUI elements that is selected by the user input via the GUI, the GUI displays the layer boxes of the graphical representation in the same manner of displaying a layer box of a layer having a higher score based on a corresponding suitability determination algorithm of a selected one of other GUI elements according to the first color and displaying a layer box of a layer having a lower score based on the corresponding suitability determination algorithm of the selected one of other GUI elements according to the second color. (Page 5 states that “In the controls on the left hand side, you will be able to color the nodes by total memory or total compute time.” One of ordinary skill in the art would reason that, when selecting either memory or compute time, the layer boxes will be colored based on the score with higher scores closer to one color, while lower scores are closer to another color. Note that the other GUI elements in the menu do not necessarily have a score. However, it would be obvious to one of ordinary skill in the art to use the other suitability determination algorithms taught by Syzmanski and Marchisio, which do have scores, and color them in the same manner.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi and Tensorboard because as stated by Tensorboard on page 1, “TensorFlow computation graphs are powerful but complicated. The graph visualization can help you understand and debug them.”
The combination of Konishi and Tensorboard does not appear to explicitly teach
a second algorithm for a complexity score
a fourth algorithm for a total score based on [other scores] and the memory footprint score
However, Szymanski—directed to analogous art—teaches
a second algorithm for a complexity score (Algorithm 1 on page 8 shows the computation of conceptual capacity. Section 5 discusses how effective the conceptual capacity is as a measure of effective complexity. Section 5.1 states "It is quite evident that conceptual capacity rates the SPIRAL-trained neural network as the most complex, then XOR, and then LIN, inline with the expectations based on the visualisation of their respective function maps." As the conceptual capacity is correlated with the complexity of the neural network model, the algorithm is used to analyze both capacity and complexity.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi and Tensorboard with the complexity and capacity algorithm of Szymanski because, as stated by Szymanski, "Empirical evaluations show that this new measure is correlated with the complexity of the mapping function and thus the generalisation capabilities of the corresponding network. It captures the effective, as oppose to the theoretical, complexity of the network function."
The combination of Konishi, Tensorboard, and Szymanski does not appear to explicitly teach
a fourth algorithm for a total score based on [other scores] and the memory footprint score
However, Cassimon—directed to analogous art—teaches
a fourth algorithm for a total score based on [other scores] and the memory footprint score (Section 3.1, Equation 2 provides an equation for a hard memory constraint. The result of this equation is interpreted as the memory footprint score. The equation is based on the "the amount of available memory on the target device" (Section 3.1). Equation (1) provides a reward function for the neural network model. The equation includes hard constraints and soft constraints. According to Section 3.1, the memory calculation and the latency calculation are hard constraints, and the soft constraints include a possible compression calculation, a cache size calculation, and a network performance calculation. The result of the reward function is interpreted as the total score of the neural network model.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi, OpenVINO, and Szymanski with the memory footprint and combination score taught by Cassimon because as Cassimon states in reference to the memory footprint score, "We chose to take the parameter size into account, because this allows us to compare this to the available memory of real platforms, and check the feasibility of running our networks on embedded devices." Additionally, Cassimon states, in reference to the extra constraints in the reward function, "We will design these constraints to maximize the generated networks’ performance, while keeping resource requirements low."
Regarding claim 2, the rejection of claim 1 is incorporated herein. Further, Konishi teaches
wherein the first algorithm is used to determine a performance efficiency of the structure and the layers of the first neural network model associated with the first target device. ([0156] states "Next, generator 13 generates a source code for temporary use by converting a portion corresponding to the inference process of the neural network (S515)." [0157] states "Next, generator 13 calculates a duration of the inference process using the source code obtained by the conversion in Step S515 (S516)." As the inference process of the neural network is a result of the structure and layers of the neural network, calculating the duration of the inference process is calculating the performance efficiency of the structure and layers of the neural network. [0157] further states "Furthermore, generator 13 calculates a duration of a process with said number of execution cycles using information having an impact on the processing time such as the operating frequency, etc., of the arithmetic processing device included in the resource information obtained in Step S501." [0158] states "Next, determination unit 14 determines whether or not the duration calculated in Step S516 meets target processing time which is the second condition included in the condition information obtained in Step S501, in other words, the performance constraints (S517).")
Regarding claim 3, the rejection of claim 2 is incorporated herein. Further, Konishi teaches
wherein the performing the analysis includes: performing, based on [the first algorithm], a first analysis on the first neural network model based on the first algorithm. ([0158] states "Next, determination unit 14 determines whether or not the duration calculated in Step S516 meets target processing time which is the second condition included in the condition information obtained in Step S501, in other words, the performance constraints (S517)." Determining whether the performance constraints are met is interpreted as performing the analysis. The duration is interpreted as the first algorithm.)
Konishi does not appear to explicitly teach
a GUI element corresponding to the first algorithm being selected by the user input
However, Tensorboard—directed to analogous art—teaches
a GUI element corresponding to the first algorithm being selected by the user input (Page 5 states that “In the controls on the left hand side, you will be able to color the nodes by total memory or total compute time.”.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi and Tensorboard for the reasons given above in regards to claim 1.
Regarding claim 5, the rejection of claim of claim 1 is incorporated herein. The combination of Konishi and Tensorboard does not appear to explicitly teach
wherein the second algorithm is used to analyze a complexity and a capacity of a structure and the layers of the first neural network model.
However, Szymanski—directed to analogous art—teaches
wherein the second algorithm is used to analyze a complexity and a capacity of a structure and the layers of the first neural network model. (Algorithm 1 on page 8 shows the computation of conceptual capacity. Section 5 discusses how effective the conceptual capacity is as a measure of effective complexity. Section 5.1 states "It is quite evident that conceptual capacity rates the SPIRAL-trained neural network as the most complex, then XOR, and then LIN, inline with the expectations based on the visualisation of their respective function maps." As the conceptual capacity is correlated with the complexity of the neural network model, the algorithm is used to analyze both capacity and complexity.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi and Tensorboard with the algorithms taught by Syzmanski for the reasons given above in regards to claim 1.
Regarding claim 6, the rejection of claim 5 is incorporated herein. Further, Konishi does not appear to explicitly teach
performing, based on a GUI element corresponding to the second algorithm being selected by the user input, a second analysis on the first neural network model based on the second algorithm.
However, Tensorboard—directed to analogous art—teaches
a GUI element corresponding to the first algorithm being selected by the user input (Page 5 shows the Color menu, interpreted as the menu, which includes four radio buttons, interpreted as the four selectable GUI elements. It includes a first algorithm for a performance score (compute time) and a third algorithm for a memory footprint (memory). Page 5 states that “In the controls on the left hand side, you will be able to color the nodes by total memory or total compute time.”)
The combination of Konishi and Tensorboard does not appear to explicitly teach
performing, based on [algorithm], a second analysis on the first neural network model based on the second algorithm.
However, Szymanski—directed to analogous art—teaches
performing, based on the [algorithm], a second analysis on the first neural network model based on the second algorithm. (Section 6.1 states "We can evaluate the conceptual capacity of a model during training, to see what happens to the architecture as it is learning a task. Figure 8 shows such analysis of a VGG16 network at different stages of training on the CIFAR10 dataset (for details of the training see Appending A.7).")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi and Tensorboard with the algorithms taught by Syzmanski for the reasons given above in regards to claim 1.
Regarding claim 7, the rejection of claim 6 is incorporated herein. Further, Konishi does not appear to explicitly teach
obtaining fourth scores of the structure and the layers of the first neural network model by determining the complexity of the structure and the layers of the first neural network model;
obtaining fifth scores of the structure and the layers of the first neural network model by measuring the capacity of the structure and the layers of the first neural network model; and
obtaining complexity scores of the structure and the layers of the first neural network model based on the fourth scores and the fifth scores.
However, Szymanski—directed to analogous art—teaches
obtaining fourth scores of the structure and the layers of the first neural network model by determining the complexity of the structure and the layers of the first neural network model; (Section 3.2 states "Next, we introduce a similarity score or kernel function on df, which we will use to evaluate f's complexity." Therefore, the similarity score is interpreted as the fourth score.)
obtaining fifth scores of the structure and the layers of the first neural network model by measuring the capacity of the structure and the layers of the first neural network model; and (Definition 12 gives the equation for calculating the capacity score.)
obtaining complexity scores of the structure and the layers of the first neural network model based on the fourth scores and the fifth scores. (Section 3.3 states "We propose to measure the capacity of the concept space by summarising the similarities between all pairs of concepts in the space." As the similarity (complexity) scores are used to obtain the capacity scores, the complexity score is the fourth score and is also based on the fifth score.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi and Tensorboard with the algorithms taught by Syzmanski for the reasons given above in regards to claim 1.
Regarding claim 19, Konishi teaches
an input device configured to receive first model information about a first neural network model and device information about a first target device that is used to execute the first neural network model, the first neural network comprising layers; ([0006] states "A neural network construction device according to one aspect of the present disclosure which solves the aforementioned problem includes: an obtainer which obtains a first condition and a second condition, the first condition being used to determine a candidate hyperparameter that is a candidate of a hyperparameter of a neural network to be constructed, the second condition being related to required performance of a model of the neural network;". The obtainer is interpreted as the input device. [0049] states "For example, the first condition may include a resource condition related to a computational resource of an embedded device, and the setting unit may calculate the upper limit of the candidate hyperparameter based on the resource condition, and determine, as the candidate hyperparameter, at least one of hyperparameters less than or equal to the upper limit." [0051] states "For example, the resource condition may include information of a memory size of the embedded device, and the setting unit may calculate, as the upper limit of the candidate hyperparameter, an upper limit of the hyperparameter of the neural network that fits within the memory size, and determine, as the candidate hyperparameter, at least one of hyperparameters less than or equal to the upper limit." The candidate hyperparameter is interpreted as the first model information and the resource condition is interpreted as the device information. Fig. 1 shows that the model comprises layers.)
a storage device configured to store information about program routines; ([0069] states "Furthermore, for example, a recording medium according to one aspect of the present disclosure is a non-transitory computer-readable recording medium having recorded thereon a program to be executed by an arithmetic processing device included in a neural network construction device including the arithmetic processing device and a storage device." The non-transitory computer-readable recording medium is interpreted as the storage device.)
a processor configured to read and execute the program routines, which cause the processor to: ([0069] further states "The program is executed by the arithmetic processing device to cause the neural network construction device to execute:".)
perform an analysis on the first target device, based on the first model information, the device information, and at least one of a plurality of suitability determination algorithms; and ([0069] further states "obtaining resource information related to a computational resource of an embedded device and a performance constraint related to processing performance of the embedded device; setting a scale constraint of a neural network based on the resource information; generating a model of the neural network based on the scale constraint; determining whether or not the model generated meets the performance constraint; and outputting data based on a result of the determining." The determination is interpreted as performing the analysis, and the performance constraints are interpreted as the suitability determination algorithms.)
generate a result of the analysis; and (As stated above, [0069] states "outputting data based on a result of the determining".)
an output device configured to visually output the result of the analysis, wherein the output device is further configured to: ([0088], in regards to FIG. 3, states "Output device 4 is, for example, a display such as a monitor, and displays text and graphics on a screen to prompt a user to input data or to present the progress or the result of a process performed by arithmetic processing device 3.")
Konishi does not appear to explicitly teach
display a graphical representation on a graphical user interface (GUI), the graphical representation showing layer boxes corresponding to the layers, a connection between the layer boxes corresponding to a connection between the layers in the neural network model, and a structure of the layers between an input and an output of the first neural network model;
display, on the GUI that displays the graphical representation, a menu including a plurality of selectable GUI elements respectively corresponding to the plurality of suitability determination algorithms, the plurality of selectable GUI elements including a first GUI element corresponding to a first algorithm for a performance score, a second GUI element corresponding to a complexity score, and a third algorithm for a memory footprint score, and a fourth GUI element corresponding to a total score based on the performance score, the complexity score, and the memory footprint score, wherein a user input to select one of the first to the fourth GUI elements is received via the GUI; and
display, based on a corresponding suitability determination algorithm of the selected a GUI element, displaying the layer boxes of the graphical representation on the GUI in a manner such that a first layer box that corresponds to a layer having a higher score is displayed according to a first color, and a second layer box that corresponds to a layer having a lower score according to the corresponding suitability determination algorithm is displayed according to a second color different from the first color,
wherein, for any one of other GUI elements among the first through fourth GUI elements that is selected by the user input via the GUI, the GUI displays the layer boxes of the graphical representation in the same manner of displaying a layer box of a layer having a higher score based on a corresponding suitability determination algorithm of a selected one of other GUI elements according to the first color and displaying a layer box of a layer having a lower score based on the corresponding suitability determination algorithm of the selected one of other GUI elements according to the second color.
However, Tensorboard—directed to analogous art—teaches
wherein the outputting comprises: (Pages 1-5 shows output of the system.)
display a graphical representation on a graphical user interface (GUI), the graphical representation showing layer boxes corresponding to the layers, a connection between the layer boxes corresponding to a connection between the layers in the neural network model, and a structure of the layers between an input and an output of the first neural network model; (Page 1 shows the graph, with nodes labeled with operations such as conv, referring to a convolutional layer of a neural network. Therefore, these nodes are interpreted as layer boxes. The layer boxes are connected by connections. Page 2 states “Tensorflow graphs have two kinds of connections: data dependencies and control dependencies. Data dependencies show the flow of tensors between two ops and are shown as solid arrows, while control dependencies use dotted lines.” The data dependency connections are interpreted as the connection between the layers. Therefore, the connection corresponds to connections between the layers in the neural network model. As the connections show the flow of tensors, meaning a flow from an input to an output, the graph shows a structure of the layers between an input and an output of the first neural network model. )
display, on the GUI that displays the graphical representation, a menu including a plurality of selectable GUI elements respectively corresponding to the plurality of suitability determination algorithms, the plurality of selectable GUI elements including a first GUI element corresponding to a first algorithm for a performance score, a second GUI element corresponding to a complexity score, and a third algorithm for a memory footprint score, and a fourth GUI element corresponding to a total score based on the performance score, the complexity score, and the memory footprint score, wherein a user input to select one of the first to the fourth GUI elements is received via the GUI; and (Page 5 shows the Color menu, interpreted as the menu, which includes four radio buttons, interpreted as the four selectable GUI elements. It includes a first algorithm for a performance score (compute time) and a third algorithm for a memory footprint (memory). One of ordinary skill in the art would realize that the selection must be received in order for the application to change the graph based on the user input.)
receiving, via the GUI, a user input to select one of the plurality of GUI elements from the menu on the GUI, and based on a corresponding suitability determination algorithm of the selected GUI element, displaying the layer boxes of the graphical representation on the GUI in a manner such that a first layer box that corresponds to a layer having a higher score is displayed according to a first color, and a second layer box that corresponds to a layer having a lower score according to the corresponding suitability determination algorithm is displayed according to a second color different from the first color (Page 5 shows that, when a color button is selected (a user input is received), layer boxes are colored closer to a first color, in this case white, when the layer box has a lower score and colored in another color, in this case red, when the score is higher.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi and Tensorboard because as stated by Tensorboard on page 1, “TensorFlow computation graphs are powerful but complicated. The graph visualization can help you understand and debug them.”
The combination of Konishi and OpenVINO does not appear to explicitly teach
a second algorithm for a complexity score and a third algorithm for a memory footprint score; and
However, Szymanski—directed to analogous art—teaches
a second algorithm for a complexity score (Algorithm 1 on page 8 shows the computation of conceptual capacity. Section 5 discusses how effective the conceptual capacity is as a measure of effective complexity. Section 5.1 states "It is quite evident that conceptual capacity rates the SPIRAL-trained neural network as the most complex, then XOR, and then LIN, inline with the expectations based on the visualisation of their respective function maps." As the conceptual capacity is correlated with the complexity of the neural network model, the algorithm is used to analyze both capacity and complexity.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi and Tensorboard with the complexity and capacity algorithm of Szymanski because, as stated by Szymanski, "Empirical evaluations show that this new measure is correlated with the complexity of the mapping function and thus the generalisation capabilities of the corresponding network. It captures the effective, as oppose to the theoretical, complexity of the network function." The combination of Konishi, Tensorboard, and Szymanski does not appear to explicitly teach
a fourth algorithm for a total score based on [other scores] and the memory footprint score
However, Cassimon—directed to analogous art—teaches
a fourth algorithm for a total score based on [other scores] and the memory footprint score (Section 3.1, Equation 2 provides an equation for a hard memory constraint. The result of this equation is interpreted as the memory footprint score. The equation is based on the "the amount of available memory on the target device" (Section 3.1). Equation (1) provides a reward function for the neural network model. The equation includes hard constraints and soft constraints. According to Section 3.1, the memory calculation and the latency calculation are hard constraints, and the soft constraints include a possible compression calculation, a cache size calculation, and a network performance calculation. The result of the reward function is interpreted as the total score of the neural network model.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi, OpenVINO, and Szymanski with the memory footprint and combination score taught by Cassimon because as Cassimon states in reference to the memory footprint score, "We chose to take the parameter size into account, because this allows us to compare this to the available memory of real platforms, and check the feasibility of running our networks on embedded devices." Additionally, Cassimon states, in reference to the extra constraints in the reward function, "We will design these constraints to maximize the generated networks’ performance, while keeping resource requirements low."
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Konishi (US 2021/0056420 A1), Tensorboard (“TensorBoard: Graph Visualization”, 2019), Szymanski (“Conceptual Capacity and Efffective Complexity of Neural Networks”, March 2021), and Cassimon (“Designing Resource-Constrained Neural Networks Using Neural Architecture Search Targeting Embedded Devices”, 2020). as applied to claims 1 and 3 above, and further in view of Benmeziane ("A Comprehensive Survey on Hardware-Aware Neural Architecture Search", January 2021) and Li ("HW-NAS-Bench: HardWare-aware Neural Architectecture Search Benchmark", March 2021).
Regarding claim 4, the rejection of claim 3 is incorporated herein. Further, Konishi teaches
obtaining second scores of the structure and the layers of the first neural network model by predicting a processing time of the structure and the layers of the first neural network model using a performance estimator; ([0156] states "Next, generator 13 generates a source code for temporary use by converting a portion corresponding to the inference process of the neural network (S515)." [0157] states "Next, generator 13 calculates a duration of the inference process using the source code obtained by the conversion in Step S515 (S516)." As the inference process of the neural network is a result of the structure and layers of the neural network, calculating the duration of the inference process is predicting a processing time of the structure and layers of the neural network. The portion of the generator that generates a source code for temporary use and calculates a duration of the inference process is interpreted as the performance estimator, as it provides an estimate (not a direct measurement) of the performance.)
[obtaining performance score of the structure and the layers of the first neural network based on] the second scores, (As taught above, the second scores are obtained by the generator.)
The combination of Konishi, Tensorboard, Syzmanski, and Cassimon does not appear to explicitly teach
obtaining first scores of the structure and the layers of the first neural network model using a pre-listed table for the first target device;
obtaining third scores of the structure and the layers of the first neural network model using a pre-trained deep learning model for the first target device; and
obtaining performance scores of the structure and the layers of the first neural network model based on4 the first scores, … and the third scores.
However, Benmeziane—directed to analogous art—teaches
obtaining first scores of the structure and the layers of the first neural network model using a pre-listed table for the first target device; (Section XIII. Hardware Cost Estimation Models states "That’s why many works tend to use a prediction model [53], [72], [118], [119], [26], [133] or a pre-collected lookup table [23], [120], [122] or computing an analytical estimation [26], [100].” The pre-collected lookup table is interpreted as the pre-listed table.)
obtaining third scores of the structure and the layers of the first neural network model using a pre-trained deep learning model for the first target device; and (Section XIII. Hardware Cost Estimation Models states "That’s why many works tend to use a prediction model [53], [72], [118], [119], [26], [133] or a pre-collected lookup table [23], [120], [122] or computing an analytical estimation [26], [100].” The prediction model is interpreted as the pre-trained deep learning model.)
[obtaining performance score of the structure and the layers of the first neural network based on] the first scores, (Fig. 14 shows the performance of the measurement methods on search time speedups. Section VIII. Hardware Cost Estimation, referring to the experiment with results shown in Fig. 14, states "The search algorithm used to calculate the search time is an evolutionary algorithm based on the validation accuracy given by the benchmark and the latency measured by the different methods." Section VII. Search Strategies, subsection A. Search Algorithm, 2) Evolutionary Algorithm states "Generally, neuro-evolutionary NAS evolves a population of models, sample some models to generate offsprings by applying some mutations (recombination is not used in neuro-evolutionary NAS), and finally evaluate the fitness of the offsprings and update the new generation by adding the best ones to the population." As an evolutionary algorithm evaluates the offsprings (models), the performance scores, from the first scores (look-up table) and third scores (deep learning model) are obtained for the structure and the layers of the neural network.)
[obtaining performance score of the structure and the layers of the first neural network based on] the third scores. (See previous limitation for explanation.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi, Tensorboard, Syzmanski, and Cassimon and the methods taught by Benmeziane because, as taught by Benmeziane in Table V, using lookup tables and multilayer perceptrons as measurement estimators is faster way to evaluate models than real-time measurements.
The combination of Konishi, Tensorboard, Syzmanski, Cassimon, and Benmeziane does not appear to explicitly teach
obtaining performance scores of the structure and the layers of the first neural network model based on [scores from a variety of methods.]
However, Li—directed to analogous art—teaches
obtaining performance scores of the structure and the layers of the first neural network model based on [scores from a variety of methods.] (Table 1 on page 5 shows the hardware-cost, interpreted as performance scores, collection methods for six different hardware devices for each neural network architecture that is analyzed. Two are estimated, while four are measured.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi, Tensorboard, Syzmanski, Cassimon, and Benmeziane with the combination of collection methods taught by Li because, as one of ordinary skill in the art would realize, there are different sources of data for different pieces of hardware. For example, as taught by Li on page 5, the ASIC-Eyeriss has pre-existing performance simulators while the Pixel 3 has an official benchmark binary file.
Claim(s) 8-13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Konishi (US 2021/0056420 A1), OpenVINO (“Visualize Model”, OpenVINO Toolkit Documentation, v. 2021.2, December 2020), Szymanski (“Conceptual Capacity and Efffective Complexity of Neural Networks”, March 2021), and Cassimon (“Designing Resource-Constrained Neural Networks Using Neural Architecture Search Targeting Embedded Devices”, 2020) as applied to claim 1 above, further in view of Marchisio (“NASCaps: A Framework for Neural Architecture Search to Optimize the Accuracy and Hardware Efficiency of Convolutional Capsule Networks”, 2020)
Regarding claim 8, the rejection of claim 1 is incorporated herein. The combination of Konishi, Tensorboard, Syzmanski, and Cassimon does not appear to explicitly teach
wherein the third algorithm is used to determine a memory efficiency of a structure and the layers of the first neural network model associated with the first target device.
However, Marchisio—directed to analogous art—teaches
wherein the third algorithm is used to determine a memory efficiency of a structure and the layers of the first neural network model associated with the first target device. (Section 3.2 states "Our models estimate the latency and the energy consumption of the inference of one input, for a given CapsNet, while the memory footprint is computed as the sum of the number of weights for each layer.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi, Tensorboard, Syzmanski, and Cassimon with the algorithms taught by Marchisio because the algorithms estimate additional measures that can indicate information about the neural network. Having additional measures, as one of ordinary skill in the art would understand, would allow a user to better understand the performance of the neural network that is being analyzed, and better optimize a structure, as used by Marchisio to find an optimal neural network architecture.
Regarding claim 9, the rejection of claim 8 is incorporated herein. Further, Konishi does not appear to explicitly teach
performing, based on a GUI element corresponding to the third algorithm being selected by the user input, a third analysis on the first neural network model based on the third algorithm.
However, Tensorboard—directed to analogous art—teaches
a GUI element corresponding to [an algorithm] being selected by the user input (Page 5 shows the Color menu, interpreted as the menu, which includes four radio buttons, interpreted as the four selectable GUI elements. It includes a first algorithm for a performance score (compute time) and a third algorithm for a memory footprint (memory). Page 5 states that “In the controls on the left hand side, you will be able to color the nodes by total memory or total compute time.”)
performing a third analysis on the first neural network model based on the third algorithm. (As the results of the analysis are shown when the ‘Memory’ GUI element is chosen, the analysis must have been performed based on the algorithm.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi and Tensorboard for the reasons given above in regards to claim 1.
Regarding claim 10, the rejection of claim 9 is incorporated herein. Further, the combination of Konishi, Tensorboard, and Syzmanski does not appear to explicitly teach
obtaining memory footprint scores of the structure and the layers of the first neural network model based on a memory limitation of the first target device.
However, Cassimon—directed to analogous art—teaches
obtaining memory footprint scores of the structure and the layers of the first neural network model based on a memory limitation of the first target device. (Section 3.1, Equation 2 provides an equation for a hard memory constraint. The result of this equation is interpreted as the memory footprint score. The equation is based on the "the amount of available memory on the target device" (Section 3.1).)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi, Tensorboard, and Syzmanski with the memory footprint taught by Cassimon because, as stated by Cassimon in Section 3.1 "We chose to take the parameter size into account, because this allows us to compare this to the available memory of real platforms, and check the feasibility of running our networks on embedded devices."
Regarding claim 11, the rejection of claim of claim 10 is incorporated herein. Further, Konishi teaches
changing the first neural network model based on the first neural network model being unavailable within [a resource constraint] (Step S517 in Figure 10 determines if inference time (the resource constraint) is met. If not, Step S517 states that the model is discarded, and the process goes back to generating a new model (interpreted as changing the model).)
[changing a neural network model based on] the memory limitation. (Section 3 states "If the system fails to meet one of the hard constraints, its reward will be zero. If the hard constraints are met, the system will be rewarded with the sum of its soft constraints." Section 3.1 explains the hard constraints, including the memory limitation. The reward function is therefore impacted by the memory limitation, and, as one of ordinary skill in the art would understand, the neural architecture search model will change its design based on the reward function.)
Regarding claim 12, the rejection of claim 1 is incorporated herein. Further, Konishi teaches
obtaining performance scores of the structure and the layers of the first neural network model associated with the first target device, by performing a first analysis on the first neural network model based on the first algorithm; ([0156] states "Next, generator 13 generates a source code for temporary use by converting a portion corresponding to the inference process of the neural network (S515)." [0157] states "Next, generator 13 calculates a duration of the inference process using the source code obtained by the conversion in Step S515 (S516)." As the inference process of the neural network is a result of the structure and layers of the neural network, calculating the duration of the inference process is predicting a processing time of the structure and layers of the neural network. As processing time is a measure of performance, as would be understood by one of ordinary skill in the art, calculating the duration of the inference process is obtaining performance scores.)
the performance scores, (See above explanation.)
The combination of Konishi and Tensorboard does not appear to explicitly teach
obtaining complexity scores of the structure and the layers of the first neural network model, by performing a second analysis on the first neural network model based on the second algorithm;
obtaining memory footprint scores of the structure and the layers of the first neural network model associated with the first target device, by performing a third analysis on the first neural network model based on the third algorithm; and
obtaining total scores of the structure and the layers of the first neural network model based on … the complexity scores, and the memory footprint scores.
However, Szymanski—directed to analogous art—teaches
obtaining complexity scores of the structure and the layers of the first neural network model, by performing a second analysis on the first neural network model based on the second algorithm; (Algorithm 1 on page 8 shows the computation of conceptual capacity. Section 5 discusses how effective the conceptual capacity is as a measure of effective complexity. Section 5.1 states "It is quite evident that conceptual capacity rates the SPIRAL-trained neural network as the most complex, then XOR, and then LIN, inline with the expectations based on the visualisation of their respective function maps." As the conceptual capacity is correlated with the complexity of the neural network model, the algorithm is used to analyze both capacity and complexity. Section 6.1 states "We can evaluate the conceptual capacity of a model during training, to see what happens to the architecture as it is learning a task. Figure 8 shows such analysis of a VGG16 network at different stages of training on the CIFAR10 dataset (for details of the training see Appending A.7).")
the complexity scores, (See above explanation.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi and Tensorboard with the complexity and capacity algorithm of Szymanski for the reasons given above in regards to claim 1.
The combination of Konishi, Tensorboard, and Syzmanski does not appear to teach
obtaining memory footprint scores of the structure and the layers of the first neural network model associated with the first target device, by performing a third analysis on the first neural network model based on the third algorithm; and
obtaining total scores of the structure and the layers of the first neural network model based on [other scores] and the memory footprint scores.
However, Cassimon—directed to analogous art—teaches
obtaining memory footprint scores of the structure and the layers of the first neural network model associated with the first target device, by performing the third analysis on the first neural network model based on a third algorithm; and (Section 3.1, Equation 2 provides an equation for a hard memory constraint. The result of this equation is interpreted as the memory footprint score. The equation is based on the "the amount of available memory on the target device" (Section 3.1).)
obtaining total scores of the structure and the layers of the first neural network model based on [other scores] and the memory footprint scores. (Equation (1) provides a reward function for the neural network model. The equation includes hard constraints and soft constraints. According to Section 3.1, the memory calculation and the latency calculation are hard constraints, and the soft constraints include a possible compression calculation, a cache size calculation, and a network performance calculation. The result of the reward function is interpreted as the total score of the neural network model.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi, Tensorboard, and Syzmanski with the memory footprint and combination score taught by Cassimon because as Cassimon states in reference to the memory footprint score, "We chose to take the parameter size into account, because this allows us to compare this to the available memory of real platforms, and check the feasibility of running our networks on embedded devices." Additionally, Cassimon states, in reference to the extra constraints in the reward function, "We will design these constraints to maximize the generated networks’ performance, while keeping resource requirements low."
Regarding claim 13, the rejection of claim 12 is incorporated herein. Further, Konishi teaches
changing at least one of the layers of the first neural network model based on the result of the analysis. (Step S517 in Figure 10 determines if inference time (the resource constraint) is met. If not, Step S517 states that the model is discarded, and the process goes back to generating a new model (interpreted as changing the model).)
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Konishi (US 2021/0056420 A1), OpenVINO (“Visualize Model”, OpenVINO Toolkit Documentation, v. 2021.2, December 2020), Szymanski (“Conceptual Capacity and Efffective Complexity of Neural Networks”, March 2021), and Cassimon (“Designing Resource-Constrained Neural Networks Using Neural Architecture Search Targeting Embedded Devices”, 2020) as applied to claim 1 above, further in view of Marchisio (“NASCaps: A Framework for Neural Architecture Search to Optimize the Accuracy and Hardware Efficiency of Convolutional Capsule Networks”, 2020) as applied to claims 12 and 13 above, further in view of Zheng (US 2022/0044094 A1).
Regarding claim 14, the rejection of claim 13 is incorporated herein. Further, the combination of Konishi, Tensorboard, Syzmanski, Cassimon, and Marchisio does not appear to explicitly teach
based on selecting of a first layer having a lowest score from among the layers of the first neural network model;
providing at least one second layer that has a score higher than that of the first layer as a candidate for replacing the first layer; and
changing the first layer based on the at least one second layer.
However, Zheng—directed to analogous art—teaches
based on selecting of a first layer having a lowest score from among the layers of the first neural network model; ([0173] states "To resolve the foregoing problems, an optimization problem is converted into a Markov decision process (MDP), and an operation with higher calculation efficiency (such as a skip connection or a direct removal connection) is calculated to replace a redundant operation." The redundant operation, as it is redundant, would have the lowest calculation efficiency, which is interpreted as the score. [0194] states "In FIG. 14, O represents other calculation operations than a skip connection operation and a null connection operation, and the operations include a convolution operation, a pooling operation, and the like; S represents a skip connection, that is, the skip connection operation; and N represents a null connection, that is, no calculation operation. A magnitude relationship between calculation costs in FIG. 14 is c(O)>c(S)>c(N), where c(.) represents a function for measuring the calculation costs. The calculation operations O may be changed into the skip connection S or the null connection N, and the skip connection S may be changed into the null connection N.” The operations, as layers have operations, is interpreted as layers.)
providing at least one second layer that has a score higher than that of the first layer as a candidate for replacing the first layer; and (As stated above, a layer is replaced with a layer with a lower calculation cost. The opposite of that, the calculation efficiency, is interpreted as the score. Therefore, when a layer is replaced with a layer having a lower calculation cost, the layer is replaced with a layer having a higher calculation efficiency.)
changing the first layer based on the at least one second layer. (As stated above, the layer is replaced with another layer.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi, Tensorboard, Syzmanski, Cassimon, and Marchisio with the teachings of Zheng because, as stated by Zheng in [0046], "redundant calculation units or calculation operations are removed from the network structure of the image recognition neural network, thereby saving a subsequent calculation amount, so that the model performance of the image recognition neural network can be improved."
Claim(s) 15 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Konishi (US 2021/0056420 A1), Tensorboard (“TensorBoard: Graph Visualization”, 2019), Szymanski (“Conceptual Capacity and Efffective Complexity of Neural Networks”, March 2021), and Cassimon (“Designing Resource-Constrained Neural Networks Using Neural Architecture Search Targeting Embedded Devices”, 2020) as applied to claim 1 above, further in view of Wu (“Mixed Precision Quantization of ConvNet Differentiable Neural Architecture Search”, 2018).
Regarding claim 15, the rejection of claim 1 is incorporated herein. Further, the combination of Konishi, Tensorboard, Syzmanski, and Cassimon does not appear to explicitly teach
applying different quantization schemes to at least some of the layers of the first neural network model.
However, Wu—directed to analogous art—teaches
applying different quantization schemes to at least some of layers of the first neural network model. (Section 3 states "For mixed precision quantization, we assume that we have the flexibility to choose different precisions for different layers of a network.")
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi, Tensorboard, Syzmanski, and Cassimon with the quantization taught by Wu because, as taught by Wu in Table 1, mixed precision quantization provided high compression with better accuracy than full precision quantization.
Regarding claim 16, the rejection of claim 15 is incorporated herein. The combination of Konishi, Tensorboard, Syzmanski, and Cassimon does not appear to explicitly teach
receiving second model information about the first neural network model, the second model information being obtained after a training on the first neural network model is completed;
changing a quantization scheme of a third layer, which is selected from among the layers of the first neural network model based on the second model information.
However, Wu—directed to analogous art—teaches
receiving second model information about the first neural network model, the second model information being obtained after a training on the first neural network model is completed; (Section 6 states "We only perform quantization on weights and use full-precision activations." In order to perform quantization on weights, the model that is quantized must be trained.)
changing a quantization scheme of a third layer, which is selected from among the layers of the first neural network model based on the second model information. (Section 6 continues with "We conduct mixed precision search at the block level – all layers in one block use the same precision. Following the convention, we do not quantize the first and the last layer. We construct a super net whose macro architecture is exactly the same as our target network. For each block, we can choose a precision from {0, 1, 2, 3, 4, 8, 32}.” As the super net is used for the search (see Section 4.2), the second model information is used for changing the quantization.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi, Tensorboard, Syzmanski, and Cassimon with the quantization taught by Wu because of the reasons given above in regards to claim 15.
Claim(s) 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Konishi (US 2021/0056420 A1), Tensorboard (“TensorBoard: Graph Visualization”, 2019), Yoshiyama (US 2021/0042453 A1), Szymanski (“Conceptual Capacity and Efffective Complexity of Neural Networks”, March 2021), and Marchisio (“NASCaps: A Framework for Neural Architecture Search to Optimize the Accuracy and Hardware Efficiency of Convolutional Capsule Networks”, 2020).
Regarding claim 21, Konishi teaches
A method of optimizing a neural network model, the method comprising: ([0005] states "In view of this, the present disclosure provides a neural network construction device that contributes to an improvement in the efficiency of obtaining an optimal neural improvement in the efficiency of obtaining an optimal neural network by narrowing candidate neural networks. Furthermore, the present disclosure provides a neural network construction method and a recording medium that are used by the neural network construction device.")
receiving first model information about a first neural network model, the first neural network model comprising layers; ([0056] states "The obtainer may further obtain learning data on the neural network, the determination unit may output data indicating a model generated by the generator and determined as meeting the second condition, the learning unit may perform, using the learning data, learning of the model indicated in the data output by the determination unit, and the outputter may output at least a part of the model that has already been learned." The data indicating a model generated by the generator is interpreted as the first model information. Fig. 1 shows that the model comprises layers.)
receiving device information about a first target device that is used to execute the first neural network model; (The abstract states "A neural network construction device includes: an obtainer which obtains resource information related to a computational resource of an embedded device and performance constraints related to processing performance of the embedded device;". The resource information is interpreted as the device information.)
performing an analysis on execution of the first neural network model on the first target device, based on the first model information, the device information, and at least one of a plurality of suitability determination algorithms; ([0157] states "Next, generator 13 calculates a duration of the inference process using the source code obtained by the conversion in Step S515 (S516)." As the calculation is of the inference process, this involves the model information. [0157] further states "Furthermore, generator 13 calculates a duration of a process with said number of execution cycles using information having an impact on the processing time such as the operating frequency, etc., of the arithmetic processing device included in the resource information obtained in Step S501." [0158] states "Next, determination unit 14 determines whether or not the duration calculated in Step S516 meets target processing time which is the second condition included in the condition information obtained in Step S501, in other words, the performance constraints (S517)." The determination is interpreted as performing the analysis, and the performance constraints are interpreted as the suitability determination algorithms.)
displaying … and a result of the analysis are displayed on a screen ([0088], in regards to FIG. 3, states "Output device 4 is, for example, a display such as a monitor, and displays text and graphics on a screen to prompt a user to input data or to present the progress or the result of a process performed by arithmetic processing device 3.")
Konishi does not appear to explicitly teach
a first graphical representation on a graphical user interface (GUI) such that the first model information [is displayed], the first graphical representation showing layer boxes corresponding to the layers, a connection between the layer boxes corresponding to a connection between the layers in the neural network model, and a structure of the layers between an input and an output of the first neural network model; and
displaying a second graphical representation on the GUI such that a result of changing at least one of layers of the first neural network model based on the result of the analysis is displayed.
wherein the displaying the first graphical representation comprises:
displaying, on the GUI that displays the first graphical representation, a menu including a plurality of selectable GUI elements, the plurality of selectable GUI elements including a first GUI element corresponding to a first algorithm for a performance score, a second GUI element corresponding to a complexity score, and a third algorithm for a memory footprint score, and a fourth GUI element corresponding to a total score based on the performance score, the complexity score, and the memory footprint score; and
receiving, via the GUI, a user input to select one of the plurality of GUI elements from the menu on the GUI, and based on a corresponding suitability determination algorithm of the selected GUI element, displaying the layer boxes of the first graphical representation on the GUI in a manner such that a first layer box that corresponds to a layer having a higher score is displayed according to a first color, and a second layer box that corresponds to a layer having a lower score according to the corresponding suitability determination algorithm is displayed according to a second color different from the first color,
wherein, for any one of other GUI elements among the first through fourth GUI elements that is selected by the user input via the GUI, the GUI displays the layer boxes of the graphical representation in the same manner of displaying a layer box of a layer having a higher score based on a corresponding suitability determination algorithm of a selected one of other GUI elements according to the first color and displaying a layer box of a layer having a lower score based on the corresponding suitability determination algorithm of the selected one of other GUI elements according to the second color.
However, OpenVINO—directed to analogous art—teaches
a first graphical representation on a graphical user interface (GUI) such that the first model information [is displayed], the first graphical representation showing layer boxes corresponding to the layers, a connection between the layer boxes corresponding to a connection between the layers in the neural network model, and a structure of the layers between an input and an output of the first neural network model; and (Page 1 shows the graph, with nodes labeled with operations such as conv, referring to a convolutional layer of a neural network. Therefore, these nodes are interpreted as layer boxes. The layer boxes are connected by connections. Page 2 states “Tensorflow graphs have two kinds of connections: data dependencies and control dependencies. Data dependencies show the flow of tensors between two ops and are shown as solid arrows, while control dependencies use dotted lines.” The data dependency connections are interpreted as the connection between the layers. Therefore, the connection corresponds to connections between the layers in the neural network model. As the connections show the flow of tensors, meaning a flow from an input to an output, the graph shows a structure of the layers between an input and an output of the first neural network model. )
displaying, on the GUI that displays the first graphical representation, a menu including a plurality of selectable GUI elements, the plurality of selectable GUI elements including a first GUI element corresponding to a first algorithm for a performance score, a second GUI element corresponding to a …, and a third algorithm for a memory footprint score, and a fourth GUI element corresponding to a …; and (Page 5 shows the Color menu, interpreted as the menu, which includes four radio buttons, interpreted as the four selectable GUI elements. It includes a first algorithm for a performance score (compute time) and a third algorithm for a memory footprint (memory).)
receiving, via the GUI, a user input to select one of the plurality of GUI elements from the menu on the GUI, and based on a corresponding suitability determination algorithm of the selected GUI element, displaying the layer boxes of the first graphical representation on the GUI in a manner such that a first layer box that corresponds to a layer having a higher score is displayed according to a first color, and a second layer box that corresponds to a layer having a lower score according to the corresponding suitability determination algorithm is displayed according to a second color different from the first color, (Page 5 shows that, when a color button is selected (a user input is received), layer boxes are colored closer to a first color, in this case white, when the layer box has a lower score and colored in another color, in this case red, when the score is higher.)
wherein, for any one of other GUI elements among the first through fourth GUI elements that is selected by the user input via the GUI, the GUI displays the layer boxes of the graphical representation in the same manner of displaying a layer box of a layer having a higher score based on a corresponding suitability determination algorithm of a selected one of other GUI elements according to the first color and displaying a layer box of a layer having a lower score based on the corresponding suitability determination algorithm of the selected one of other GUI elements according to the second color. (Page 5 states that “In the controls on the left hand side, you will be able to color the nodes by total memory or total compute time.” One of ordinary skill in the art would reason that, when selecting either memory or compute time, the layer boxes will be colored based on the score with higher scores closer to one color, while lower scores are closer to another color. Note that the other GUI elements in the menu do not necessarily have a score. However, it would be obvious to one of ordinary skill in the art to use the other suitability determination algorithms taught by Syzmanski and Marchisio, which do have scores, and color them in the same manner.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi and Tensorboard because as stated by Tensorboard on page 1, “TensorFlow computation graphs are powerful but complicated. The graph visualization can help you understand and debug them.”
The combination of Konishi and OpenVINO does not appear to explicitly teach
displaying a second graphical representation on the GUI such that a result of changing at least one of layers of the first neural network model based on the result of the analysis is displayed
a second algorithm for a complexity score and a third algorithm for a memory footprint score; and
However, Yoshiyama—directed to analogous art—teaches
displaying a second graphical representation on the GUI such that a result of changing at least one of layers of the first neural network model based on the result of the analysis is displayed ([0136] states "Note that the form control unit 210 allows the user to be presented with a network optimized by the generation unit 220 when the user approves the information presentation as described above. That is, the form control unit 210 may reflect the optimized network structure in a network may reflect the optimized network structure in a network may be performed by pressing a button or using voice, for example." The display of the optimized network structure is interpreted as the second graphical representation. Figures 6-17 show different optimization results where at least one of the layers of a neural network are changed.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi and Tensorboard with the teachings of Yoshiyama because as stated by Yoshiyama in [0051] – [0052], "That is, the information processing device according to the embodiment of the present disclosure allows a user to be presented with various proposals related to optimization of the network structure in which a software designer can design a neural network specialized for hardware in advance. In accordance with the aforementioned function of the information processing device according to the embodiment of the present disclosure, a software designer can easily design a neural network in consideration of hardware implementation, so that it is possible to eliminate double effort of a hardware designer and to significantly improve the efficiency of work for implementing the neural network as hardware."
The combination of Konishi, Tensorboard, and Yoshiyama does not appear to explicitly teach
a second algorithm for a complexity score and a third algorithm for a memory footprint score; and
However, Szymanski—directed to analogous art—teaches
a second algorithm for a complexity score (Algorithm 1 on page 8 shows the computation of conceptual capacity. Section 5 discusses how effective the conceptual capacity is as a measure of effective complexity. Section 5.1 states "It is quite evident that conceptual capacity rates the SPIRAL-trained neural network as the most complex, then XOR, and then LIN, inline with the expectations based on the visualisation of their respective function maps." As the conceptual capacity is correlated with the complexity of the neural network model, the algorithm is used to analyze both capacity and complexity.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi, Tensorboard, and Yoshiyama with the complexity and capacity algorithm of Szymanski because, as stated by Szymanski, "Empirical evaluations show that this new measure is correlated with the complexity of the mapping function and thus the generalisation capabilities of the corresponding network. It captures the effective, as oppose to the theoretical, complexity of the network function."
The combination of Konishi, Tensorboard, Yoshiyama, and Szymanski does not appear to explicitly teach
a fourth algorithm for a total score based on [other scores] and the memory footprint score
However, Cassimon—directed to analogous art—teaches
a fourth algorithm for a total score based on [other scores] and the memory footprint score (Section 3.1, Equation 2 provides an equation for a hard memory constraint. The result of this equation is interpreted as the memory footprint score. The equation is based on the "the amount of available memory on the target device" (Section 3.1). Equation (1) provides a reward function for the neural network model. The equation includes hard constraints and soft constraints. According to Section 3.1, the memory calculation and the latency calculation are hard constraints, and the soft constraints include a possible compression calculation, a cache size calculation, and a network performance calculation. The result of the reward function is interpreted as the total score of the neural network model.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Konishi, Tensorboard, Yoshiyama, and Szymanski with the memory footprint and combination score taught by Cassimon because as Cassimon states in reference to the memory footprint score, "We chose to take the parameter size into account, because this allows us to compare this to the available memory of real platforms, and check the feasibility of running our networks on embedded devices." Additionally, Cassimon states, in reference to the extra constraints in the reward function, "We will design these constraints to maximize the generated networks’ performance, while keeping resource requirements low."
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Kanit Wongsuphasawat et al., "Visualizing Dataflow Graphs of Deep Learning Models in TensorFlow", August 28, 2017, IEEE, IEEE Transactions on Visualization and Computer Graphics, Vol. 24, No. 1, January 2018 (Year: 2017) further describes the graphs in Tensorflow. Of particular relevance is Section 5.4 ‘Overlaying Additional Quantitative Data,’ which describes the coloring of the layer boxes.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JESSICA THUY PHAM whose telephone number is (571)272-2605. The examiner can normally be reached Monday - Friday, 9 A.M. - 5:00 P.M..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li Zhen can be reached at (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.T.P./ Examiner, Art Unit 2121
/Li B. Zhen/ Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights