Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Detailed Action
This Final Office Action is responsive to Applicants’ Reply dated 1/29/26. Claims 1-20 remain pending, of which claims 1, 11, and 17 are independent.
Responsive to the Reply, the Examiner has withdrawn the prior rejections made under 35 U.S.C. 101.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office Action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Non-Patent Literature “FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search” (“Wu”) in view of CN 113269312 A (“Guo”).
Regarding claim 1, WU teaches “A computing apparatus, the apparatus comprising: one or more processors; and a memory storing instructions configured to, when executed by the one or more processors, cause the computing apparatus to: generate a neural network loss based on parameters of a candidate architecture for the neural network”:
([Abstract] “Designing accurate and efficient ConvNets (convolutional neural networks) for mobile devices is challenging because the design space is combinatorially large. Due to this, previous neural architecture search (NAS) methods are computationally expensive. ConvNet architecture optimality depends on factors such as input resolution and target devices (parameters). However, existing approaches are too resource demanding for case-by-case redesigns. Also, previous work focuses primarily on reducing FLOPs, but FLOP count does not always reflect actual latency. To address these, we propose a differentiable neural architecture search (DNAS) framework (containing candidate architectures) that uses gradient-based methods to optimize ConvNet architectures, avoiding enumerating and training individual architectures separately as in previous methods. FBNets (Facebook-Berkeley-Nets), a family of models discovered by DNAS surpass state-of-the-art models both designed manually and generated automatically. FBNet-B achieves 74.1% top-1 accuracy on ImageNet with 295M FLOPs and 23.1 ms latency on a Samsung S8 phone, 2.4x smaller and 1.5x faster than MobileNetV2-1.3[17] with similar accuracy. Despite higher accuracy and lower latency than MnasNet[20], we estimate FBNet-B’s search cost is 420x smaller than MnasNet’s, at only 216 GPU-hours. Searched for different resolutions and channel sizes, FBNets achieve 1.5% to 6.4% higher accuracy than MobileNetV2. The smallest FBNet achieves 50.2% accuracy and 2.9 ms latency (345 frames per second) on a Samsung S8. Over a Samsung-optimized FBNet, the iPhone-X-optimized model achieves a 1.4x speedup on an iPhone X. FBNet models are open sourced at https://github.com/facebookresearch/mobile-vision.”) Additionally, “a differentiable neural architecture search framework” for optimizing neural network architectures inherently requires a computer, which is widely understood to be an apparatus with processor and memory functions to function as understood.
And further:
PNG
media_image1.png
359
432
media_image1.png
Greyscale
In the above figure’s description, optimal architectures are sampled from a trained distribution, and the latency of each operator is measured on target devices and used “to compute the loss for the super net” which is equivalent to the generation of a neural network loss based on parameters of a candidate architecture for the neural network.
Further, WU teaches “measure first hardware resources … used to perform an operation of the neural network with the candidate architecture”:
([Page 10736, 3. Method] “In this paper, we use differentiable neural architecture search (DNAS) to solve the problem of ConvNet design. We formulate the neural architecture search problem as
PNG
media_image2.png
29
284
media_image2.png
Greyscale
. Given an architecture space
PNG
media_image3.png
20
20
media_image3.png
Greyscale
, we seek to find an optimal architecture
PNG
media_image4.png
20
55
media_image4.png
Greyscale
such that after training its weights wa, it can achieve the minimal loss L(a, wa) (the neural network loss). In our work, we focus on three factors of the problem: a) the search space
PNG
media_image3.png
20
20
media_image3.png
Greyscale
. b) The loss function L(a, wa) that considers actual latency (a hardware resource loss). c) An efficient search algorithm.”)
And further:
([Page 10737, 3.2 Latency-Aware Loss Function, Paragraphs 1-2] “The loss function used in (1) has to reflect not only the accuracy of a given architecture but also the latency on the target hardware (thus acknowledging the need to optimize based on hardware resources). To achieve this goal, we define the following loss function:
PNG
media_image5.png
26
367
media_image5.png
Greyscale
The first term CE(a, wa) denotes the cross-entropy loss of architecture a with parameter wa (this is the neural network loss). The second term LAT(a) denotes the latency of the architecture on the target hardware measured in micro-second (a measurement of hardware resources used in operation of the neural network with the candidate architecture). The coefficient
PNG
media_image6.png
13
14
media_image6.png
Greyscale
controls the overall magnitude of the loss function. The exponent coefficient " modulates the magnitude of the latency term.”)
Further, WU teaches “generate a prediction, using a hardware resource prediction model, of second hardware resources that would be used for operating the neural network with the candidate architecture”:
([Page 10737, 3.2 Latency-Aware Loss Function, Paragraphs 1-4] “The loss function used in (1) has to reflect not only the accuracy of a given architecture but also the latency on the target hardware (thus acknowledging the need to optimize based on hardware resources. Here, the Latency-Aware Loss Function is the hardware resource prediction model, and its output is the prediction). To achieve this goal, we define the following loss function:
PNG
media_image5.png
26
367
media_image5.png
Greyscale
The first term CE(a, wa) denotes the cross-entropy loss of architecture a with parameter wa. The second term LAT(a) denotes the latency of the architecture on the target hardware measured in micro-second (a measurement of hardware resources used in operation of the neural network with the candidate architecture). The coefficient
PNG
media_image6.png
13
14
media_image6.png
Greyscale
controls the overall magnitude of the loss function. The exponent coefficient " modulates the magnitude of the latency term.
The cross-entropy term can be easily computed. However, the latency term is more difficult, since we need to measure the actual runtime of an architecture on a target device (hardware resources that would be used for operating the neural network with the candidate architecture). To cover the entire search space, we need to measure about 1021 architectures, which is an impossible task.
To solve this problem, we use a latency lookup table model to estimate the overall latency of a network based on the runtime of each operator. More formally, we assume
PNG
media_image7.png
45
307
media_image7.png
Greyscale
where
PNG
media_image8.png
26
29
media_image8.png
Greyscale
denotes the block at layer-l from architecture a. This assumes that on the target processor, the runtime of each operator is independent of other operators. The assumption is valid for many mobile CPUs and DSPs, where operators are computed sequentially one by one. This way, by benchmarking the latency of a few hundred operators used in the search space, we can easily estimate the actual runtime of the 1021 architectures in the entire search space (generating a prediction of the hardware resources that would be used for operating the neural network with the candidate architecture). More importantly, as will be explained in section 3.3, using the lookup table model makes the latency term in the loss function (2) differentiable with respect to layer-wise block choices, and this allows us to use gradient-based optimization to solve problem (1).”)
Further, WU teaches “determine a hardware resource loss based on the first hardware resources and the second hardware resources”:
([Page 10738, 3.3 Search Algorithm] “Solving the problem (1) through brute-force enumeration of the search space is very infeasible. The inner problem of optimizing wa involves training a neural network. For ImageNet classification, training a ConvNet typically takes several days or even weeks. The outer problem of optimizing
PNG
media_image4.png
20
55
media_image4.png
Greyscale
has a combinatorially large search space.
Most of the early works on NAS [30, 31, 20] follow the paradigm above. To reduce the computational cost, the inner problem is replaced by training candidate architectures on an easier proxy dataset. For example, [30, 31] trains the architecture on the CIFAR10 dataset, and [20] trains on ImageNet but only for 5 epochs. The learned architectures are then transferred to the target dataset. To avoid exhaustively iterating through the search space, [30, 31, 20] use reinforcement learning to guide the exploration. Despite these improvements, solving problem (1) is still prohibitively expensive – training a network on the proxy dataset is still time-consuming, and thousands of architectures need to be trained before reaching the optimal solution.
We adopt a different paradigm of solving problem (1). We first represent the search space by a stochastic super net. The super net has the same macro-architecture as described in Table 1, and each layer contains 9 parallel blocks as described in Table 2. During the inference of the super net, only one candidate block is sampled and executed with the sampling probability of
PNG
media_image9.png
46
409
media_image9.png
Greyscale
0l contains parameters that determine the sampling probability of each block at layer-l (the per-layer latency is the first hardware resouce). Equivalently, the output of layer-l can be expressed as
PNG
media_image10.png
42
309
media_image10.png
Greyscale
where ml,I is a random variable in {0, 1} and is evaluated to 1 if block bl,i is sampled. The sampling probability is determined by equation (4). bl,i(xl) denotes the output of block-i at layer l given the input feature map xl. We let each layer sample independently, therefore, the probability of sampling an architecture a can be described as
PNG
media_image11.png
41
314
media_image11.png
Greyscale
where 0 denotes the a vector consists of all the 0l,i for each block-i at layer-l.
PNG
media_image12.png
29
27
media_image12.png
Greyscale
denotes that in the sampled architecture a, block-i is chosen at layer-l.
PNG
media_image13.png
365
442
media_image13.png
Greyscale
PNG
media_image14.png
245
416
media_image14.png
Greyscale
The above tables were placed here by the examiner for convenience since they are referenced in the text.
Instead of solving for the optimal architecture
PNG
media_image4.png
20
55
media_image4.png
Greyscale
, which has a discrete search space, we relax the problem to optimize the probability P0 of the stochastic super net to achieve the minimum expected loss. Formally, we re-write the discrete optimization problem (1) as
PNG
media_image15.png
30
317
media_image15.png
Greyscale
It is obvious the loss function in (7) is differentiable with respect to the architecture weights wa and therefore can be optimized by stochastic gradient descent (SGD). However, the loss is not directly differentiable to the sampling parameter 0, since we cannot pass the gradient through the discrete random variable ml,i to 0l,i. To sidestep this, we relax the discrete mask variable ml,i to be a continuous random variable computed by the Gumbel Softmax function [9, 14]
PNG
media_image16.png
76
334
media_image16.png
Greyscale
where gl,i ~ Gumbel(0, 1) is a random noise following the Gumbel distribution. The Gumbel Softmax function is controlled by a temperature parameter
PNG
media_image17.png
11
12
media_image17.png
Greyscale
. As
PNG
media_image17.png
11
12
media_image17.png
Greyscale
approaches 0, it approximates the discrete categorical sampling following the distribution in (6). As
PNG
media_image17.png
11
12
media_image17.png
Greyscale
becomes larger, ml,i becomes a continuous random variable. Regardless of the value of
PNG
media_image17.png
11
12
media_image17.png
Greyscale
, the mask ml,i is directly differentiable with respect to the parameter 0l,i. The technique of using Gumbel Softmax for neural architecture search is also proposed in [25, 1].
As a result, it is clear that the cross-entropy term from the loss function (2) is differentiable with respect to the mask ml,i and therefore 0l,i. For the latency term, since we use the lookup table based model for efficiency estimation, equation (3) can be written as
PNG
media_image18.png
42
341
media_image18.png
Greyscale
The latency of each operator LAT(bl,i) is a constant coefficient, so the overall latency of architecture-a is differentiable with respect to the mask ml,i, therefore 0l,i. (The predicted architecture latency is the second hardware resource.)
As a result, the loss function (2) is fully differentiable with respect to both weights wa and the architecture distribution parameter 0. This allows us to use SGD to efficiently solve problem (1). Here, we see a hardware resource loss being determined based on the previously cited algorithms that determined the first and second hardware resources, which are the actual per-layer latency and the predicted architecture latency (via summing lookup table values).”)
Further, WU teaches “determine a target architecture of the neural network based on the neural network loss and the hardware resource loss” and “output a target neural network comprising the target architecture determined for a given purpose”:
([Page 10738, 3.3 Search Algorithm, final paragraph] “Our search process is now equivalent to training the stochastic super net. During the training, we compute
PNG
media_image19.png
20
65
media_image19.png
Greyscale
to train each operator’s weight in the super net (The first half before the “/” being the neural network loss, and the second half being the hardware resource loss.). This is no different from training an ordinary ConvNet. After operators get trained, different operators can have a different contribution to the accuracy and the efficiency of the overall network (optimal features). Therefore, we compute
PNG
media_image20.png
19
55
media_image20.png
Greyscale
to update the sampling probability P0 for each operator. This step selects operators with better accuracy and lower latency and suppresses the opposite ones (“better accuracy” reflects the optimal neural network loss while “lower latency” represents the optimal hardware resource loss.). After the super net training finishes, we can then obtain the optimal architectures by sampling from the architecture distribution P0.” Here we see the above methods concluding with the target architecture of the neural network being determined based on the loss functions, and any use/provision of the network as determined is akin to an output step).
Applicants’ claim has been amended to clarify that the first hardware resources are measured as they are actually used to perform the operation of the neural network with the candidate architecture, and to now include further additional recitations to determine a difference between the first hardware resources and the second hardware resources output from the hardware prediction model and determine the hardware resource loss by a weighted sum between the difference and a factor for optimizing the hardware resource and determine an optimization loss by a weighted sum between neural network loss and the hardware resource loss and determine the target architecture by searching for the candidate architecture that minimizes the optimization loss and parameters of the candidate architecture.
As had been noted previously in relation to claim 3 as originally filed, Wu teaches some version of determining a difference/loss between separate hardware resource specifications, essentially to arrive at an optimal configuration via iterative search. See, e.g., pages 33-36 of the Non-Final Office Action dated 9/29/25. The Examiner notes Applicants’ arguments that Wu merely teaches the use of a table of values to ascertain at least one of the specifications in the comparison and hence cannot be fairly construed to actually measure as the claim now reads. Rather, the Examiner relies upon GUO to teach what Wu lacks, see e.g., Guo’s comparable hardware/architecture search and optimization process as outlined via steps S1-S4 on pages 3-4 as provided, and where the determination of a hardware loss is discussed in relation to step S3, just prior to the discussion of step S4 on page 4: “Because the search model is adapted to resource constraints of different hardware platforms, therefore, adding loss of calculation cost in the classification verification loss function. according to the number of floating point operation of the filter to describe the calculation cost of the single network, then calculating the weighted sum of all candidate network cost, namely the network total calculation cost lcost, λ cost is the weight of the calculation cost. The loss function is as follows: L=lc + λ costlcost, wherein lc represents the cross entropy loss of the search network structure.” The Examiner believes this loss determination taught in relation to Guo’s S1-S4 reads on Applicants’ clarification of determining a hardware resource loss specifically, and is understood to arrive at an optimal hardware configuration for performing the model, i.e., involving the determination and minimization of an optimization loss. In particular, Guo’s step S3, found on page 2, appears to involve actual values that would be understood to be measured as part of its search and evaluation aspect, rather than Wu’s dependency at least in part on theoretical values from a table.
Wu and Guo are similarly directed in relating to hardware configuration search in pursuit of optimization. Hence, they are analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Guo’s hardware resource search and evaluation aspects into Wu’s framework, with a reasonable expectation of success, as to provide a more accurate evaluation that relies more directly on measured values of greater relevance per Guo as an improvement over Wu’s lookup table.
Regarding claim 2, WU in view of GUO teaches the limitations of claim 1, as discussed above. Further, WU teaches “wherein the hardware resource prediction model comprises a neural network configured to accept the parameters of the candidate architecture as inputs”:
(Wu: [Page 10736-10737, 3.1 search Space, Paragraphs 2-3] “In this work, we construct a layer-wise search space with a fixed macro-architecture, and each layer can choose a different block. The macro architecture is described in Table 1. The macro architecture defines the number of layers and the input/output dimensions of each layer. The first and the last three layers of the network have fixed operators. For the rest of the layers, their block type needs to be searched. The filter numbers for each layer are hand-picked empirically. We use relatively small channel sizes for early layers, since the input resolution at early layers is large, and the computational cost (FLOP count) is quadratic to input size.
Each searchable layer in the network can choose a different block from the layer-wise search space. The block structure is inspired by MobileNetV2 [17] and ShiftNet [23], and is illustrated in Figure 3. It contains a point-wise (1x1) convolution, a K-by-K depthwise convolution where K denotes the kernel size, and another 1x1 convolution. “ReLU” activation functions follow the first 1x1 convolution and the depthwise convolution, but there are no activation functions following the last 1x1 convolution. If the output dimension stays the same as the input dimension, we use a skip connection to add the input to the output. Following [17, 23], we use a hyperparameter, the expansion ratio e, to control the block. It determines how much do we expand the output channel size of the first 1x1 convolution compared with its input channel size. Following [20], we also allow choosing a kernel size of 3 or 5 for the depthwise convolution. In addition, we can choose to use group convolution for the first and the last 1x1 convolution to reduce the computation complexity. When we use group convolution, we follow [29] to add a channel shuffle operation to mix the information between channel groups.
In our experiments, our layer-wise search space contains 9 candidate blocks, with their configurations listed in Table 2. Note we also have a block called “skip”, which directly feed the input feature map to the output without actual computations. This candidate block essentially allows us to reduce the depth of the network.
In summary, our overall search space contains 22 layers and each layer can choose from 9 candidate blocks from Table 2, so it contains 922
PNG
media_image21.png
13
15
media_image21.png
Greyscale
1021 possible architectures.”) Here, we see various parameters of the candidate architecture such as K (the kernel size) and the expansion ratio e. We see the candidate blocks which form the architecture / the candidate architecture’s parameters being used as inputs between each layer of the neural network.
And further:
[Figure 3]
PNG
media_image22.png
371
426
media_image22.png
Greyscale
In the above figure, the inputs denoted as H x W x Cin & H x W x (e x Cin) & (H/s) x (W/s) x (e x Cin) are fed as parameters of candidate architectures within each block which choose different expansion rates, kernel sizes, and numbers of groups.
Further, WU teaches “wherein the hardware resource prediction model comprises a neural network configured to… predict the second hardware resource of the neural network of the candidate architecture based on the parameters”:
([Page 10737, 3.2 Latency-Aware Loss Function, Paragraphs 1-4] “The loss function used in (1) has to reflect not only the accuracy of a given architecture but also the latency on the target hardware (thus acknowledging the need to optimize based on hardware resources. Here, the Latency-Aware Loss Function is the hardware resource prediction model, and its output is the prediction of the second hardware resources). To achieve this goal, we define the following loss function:
PNG
media_image5.png
26
367
media_image5.png
Greyscale
The first term CE(a, wa) denotes the cross-entropy loss of architecture a with parameter wa. The second term LAT(a) denotes the latency of the architecture on the target hardware measured in micro-second (a measurement of hardware resources used in operation of the neural network with the candidate architecture, which is based on parameters wa). The coefficient
PNG
media_image6.png
13
14
media_image6.png
Greyscale
controls the overall magnitude of the loss function. The exponent coefficient " modulates the magnitude of the latency term.
The cross-entropy term can be easily computed. However, the latency term is more difficult, since we need to measure the actual runtime of an architecture on a target device (hardware resources that would be used for operating the neural network with the candidate architecture). To cover the entire search space, we need to measure about 1021 architectures, which is an impossible task.
To solve this problem, we use a latency lookup table model to estimate the overall latency of a network based on the runtime of each operator. More formally, we assume
PNG
media_image7.png
45
307
media_image7.png
Greyscale
where
PNG
media_image8.png
26
29
media_image8.png
Greyscale
denotes the block at layer-l from architecture a. This assumes that on the target processor, the runtime of each operator is independent of other operators. The assumption is valid for many mobile CPUs and DSPs, where operators are computed sequentially one by one. This way, by benchmarking the latency of a few hundred operators used in the search space, we can easily estimate the actual runtime of the 1021 architectures in the entire search space (generating a prediction of the hardware resources that would be used for operating the neural network with the candidate architecture). More importantly, as will be explained in section 3.3, using the lookup table model makes the latency term in the loss function (2) differentiable with respect to layer-wise block choices, and this allows us to use gradient-based optimization to solve problem (1).”)
Further, WU teaches “wherein the hardware resource prediction model comprises a neural network configured to… output a hardware resource prediction value”:
[Page 10741, Table 5]
PNG
media_image23.png
114
753
media_image23.png
Greyscale
In the above table, you can see the FLOPS and the Latency (a hardware resource prediction value) calculated based on target devices (iPhone X & Samsung Galaxy S8)
The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 3, WU in view of GUO teaches the limitations of claim 1, as discussed above. Further, WU teaches “wherein the processor is further configured to: “update the parameters of the candidate architecture to minimize the hardware resource loss”:
([Page 10738, 3.3 Search Algorithm, final paragraph] “Our search process is now equivalent to training the stochastic super net. During the training, we compute
PNG
media_image19.png
20
65
media_image19.png
Greyscale
to train each operator’s weight in the super net. This is no different from training an ordinary ConvNet. After operators get trained, different operators can have a different contribution to the accuracy and the efficiency of the overall network. Therefore, we compute
PNG
media_image20.png
19
55
media_image20.png
Greyscale
to update the sampling probability P0 for each operator. This step selects operators with better accuracy and lower latency and suppresses the opposite ones (thus updating the parameters to minimize the hardware resource loss). After the super net training finishes, we can then obtain the optimal architectures by sampling from the architecture distribution P0.” Here we see the above methods concluding with the target architecture of the neural network being determined based on the loss functions.)
Regarding claim 4, WU in view of GUO teaches the limitations of claim 1, as discussed above. Further, WU teaches “wherein the processor is further configured to determine a weighted sum of the neural network loss and the hardware resource loss as an optimization loss and to determine the target architecture to minimize the optimization loss”:
([Page 10737, 3.2 Latency-Aware Loss Function, Paragraphs 1-2] “The loss function used in (1) has to reflect not only the accuracy of a given architecture but also the latency on the target hardware (thus acknowledging the need to optimize based on hardware resources). To achieve this goal, we define the following loss function:
PNG
media_image5.png
26
367
media_image5.png
Greyscale
The first term CE(a, wa) denotes the cross-entropy loss of architecture a with parameter wa (a neural network loss). The second term LAT(a) denotes the latency of the architecture on the target hardware measured in micro-second (hardware resource loss). The coefficient
PNG
media_image6.png
13
14
media_image6.png
Greyscale
controls the overall magnitude of the loss function (a weight applied to the hardware resource loss). The exponent coefficient " modulates the magnitude of the latency term.”) In the above equation, the selection of the architecture is clearly based on a Least sum (a sum of the losses, which is weighted using the coefficient
PNG
media_image6.png
13
14
media_image6.png
Greyscale
) of a neural network loss, as evidenced by the cross-entropy loss of architecture a with parameter wa (neural network loss) and the weight
PNG
media_image6.png
13
14
media_image6.png
Greyscale
(making it a weighted sum) applied to the hardware resource loss, LAT(a), resulting in a weighted sum of them. This total loss is aimed to be minimized for the optimization of the neural network on the target device (an optimization loss).
The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 5, WU in view of GUO teaches the limitations claim 1, as discussed above. Further, WU teaches “wherein the processor is further configured to determine the target architecture and target parameters to reduce the neural network loss and the hardware resource loss”:
([Page 10736-10737, 3.1 search Space, Paragraphs 2-3] “In this work, we construct a layer-wise search space with a fixed macro-architecture, and each layer can choose a different block. The macro architecture is described in Table 1. The macro architecture defines the number of layers and the input/output dimensions of each layer. The first and the last three layers of the network have fixed operators. For the rest of the layers, their block type needs to be searched. The filter numbers for each layer are hand-picked empirically. We use relatively small channel sizes for early layers, since the input resolution at early layers is large, and the computational cost (FLOP count) is quadratic to input size.
Each searchable layer in the network can choose a different block from the layer-wise search space. The block structure is inspired by MobileNetV2 [17] and ShiftNet [23], and is illustrated in Figure 3. It contains a point-wise (1x1) convolution, a K-by-K depthwise convolution where K denotes the kernel size, and another 1x1 convolution. “ReLU” activation functions follow the first 1x1 convolution and the depthwise convolution, but there are no activation functions following the last 1x1 convolution. If the output dimension stays the same as the input dimension, we use a skip connection to add the input to the output. Following [17, 23], we use a hyperparameter, the expansion ratio e, to control the block. It determines how much do we expand the output channel size of the first 1x1 convolution compared with its input channel size. Following [20], we also allow choosing a kernel size of 3 or 5 for the depthwise convolution. In addition, we can choose to use group convolution for the first and the last 1x1 convolution to reduce the computation complexity. When we use group convolution, we follow [29] to add a channel shuffle operation to mix the information between channel groups.
In our experiments, our layer-wise search space contains 9 candidate blocks, with their configurations listed in Table 2. Note we also have a block called “skip”, which directly feed the input feature map to the output without actual computations. This candidate block essentially allows us to reduce the depth of the network.
In summary, our overall search space contains 22 layers and each layer can choose from 9 candidate blocks from Table 2, so it contains 922
PNG
media_image21.png
13
15
media_image21.png
Greyscale
1021 possible architectures.”) Here, we see various parameters of the candidate architecture such as K (the kernel size) and the expansion ratio e. We see the candidate blocks which form the architecture / the candidate architecture’s parameters being used as inputs between each layer of the neural network.
And further:
[Figure 3]
PNG
media_image22.png
371
426
media_image22.png
Greyscale
In the above figure, the inputs denoted as H x W x Cin & H x W x (e x Cin) & (H/s) x (W/s) x (e x Cin) are fed as parameters of candidate architectures within each block which choose different expansion rates, kernel sizes, and numbers of groups, thus changing parameters while determining the architecture.
And further:
([Page 10736, 3. Method] “In this paper, we use differentiable neural architecture search (DNAS) to solve the problem of ConvNet design. We formulate the neural architecture search problem as
PNG
media_image2.png
29
284
media_image2.png
Greyscale
. Given an architecture space
PNG
media_image3.png
20
20
media_image3.png
Greyscale
, we seek to find an optimal architecture
PNG
media_image4.png
20
55
media_image4.png
Greyscale
such that after training its weights wa, it can achieve the minimal loss L(a, wa) Here, we see that the goal is to minimize the loss of the neural network. In our work, we focus on three factors of the problem: a) the search space
PNG
media_image3.png
20
20
media_image3.png
Greyscale
. b) The loss function L(a, wa) that considers actual latency (here we also see the goal of minimizing hardware resource loss). c) An efficient search algorithm.”)
And further:
([Page 10737, 3.2 Latency-Aware Loss Function, Paragraphs 1-4] “The loss function used in (1) has to reflect not only the accuracy of a given architecture but also the latency on the target hardware (thus acknowledging the need to minimize both the neural network loss (accuracy of the given architecture) and the hardware resource loss (latency on the target hardware)).
The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 6, WU in view of GUO teaches the limitations of claim 1, as discussed above. Further, WU teaches “wherein, for each layer of the neural network, a corresponding candidate architecture is determined by selecting a respective candidate operation from among candidate operations of a corresponding layer”:
([Page 10736-10737, 3.1 search Space, Paragraphs 2-3] “In this work, we construct a layer-wise search space (for each layer of the neural network…) with a fixed macro-architecture, and each layer can choose a different block. The macro architecture is described in Table 1. The macro architecture defines the number of layers and the input/output dimensions of each layer. The first and the last three layers of the network have fixed operators (fixed operations). For the rest of the layers, their block type needs to be searched (candidate operations). The filter numbers for each layer are hand-picked empirically. We use relatively small channel sizes for early layers, since the input resolution at early layers is large, and the computational cost (FLOP count) is quadratic to input size.
Each searchable layer in the network can choose a different block from the layer-wise search space. The block structure is inspired by MobileNetV2 [17] and ShiftNet [23], and is illustrated in Figure 3. It contains a point-wise (1x1) convolution, a K-by-K depthwise convolution where K denotes the kernel size, and another 1x1 convolution. “ReLU” activation functions follow the first 1x1 convolution and the depthwise convolution, but there are no activation functions following the last 1x1 convolution. If the output dimension stays the same as the input dimension, we use a skip connection to add the input to the output. Following [17, 23], we use a hyperparameter, the expansion ratio e, to control the block. It determines how much do we expand the output channel size of the first 1x1 convolution compared with its input channel size. Following [20], we also allow choosing a kernel size of 3 or 5 for the depthwise convolution. In addition, we can choose to use group convolution for the first and the last 1x1 convolution to reduce the computation complexity. When we use group convolution, we follow [29] to add a channel shuffle operation to mix the information between channel groups.
In our experiments, our layer-wise search space contains 9 candidate blocks, with their configurations listed in Table 2. Note we also have a block called “skip”, which directly feed the input feature map to the output without actual computations. This candidate block essentially allows us to reduce the depth of the network.
In summary, our overall search space contains 22 layers and each layer can choose from 9 candidate blocks from Table 2, so it contains 922
PNG
media_image21.png
13
15
media_image21.png
Greyscale
1021 possible architectures.”) Here, we see various parameters of the candidate architecture such as K (the kernel size) and the expansion ratio e. We see the candidate blocks which form the architecture / the candidate architecture’s parameters being used as inputs between each layer of the neural network.
And further:
[Figure 3]
PNG
media_image22.png
371
426
media_image22.png
Greyscale
In the above figure, the inputs denoted as H x W x Cin & H x W x (e x Cin) & (H/s) x (W/s) x (e x Cin) are fed as parameters of candidate architectures within each block which choose different expansion rates, kernel sizes, and numbers of groups, thus changing parameters while determining the architecture via layer-wise candidate operations.
The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 7, WU in view of GUO teaches the limitations of claim 6, as discussed above. Further, WU teaches “wherein information associated with the selected candidate operation is input to the hardware resource prediction model”:
[Figure 3]
PNG
media_image22.png
371
426
media_image22.png
Greyscale
In the above figure, the inputs denoted as H x W x Cin & H x W x (e x Cin) & (H/s) x (W/s) x (e x Cin) are fed as parameters of candidate architectures within each block which choose different expansion rates, kernel sizes, and numbers of groups, thus changing parameters while determining the architecture via layer-wise candidate operations, which is input to the hardware resource prediction model (The Latency-Aware Loss Function) shown in section 3.2.
The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 8, WU in view of GUO teaches the limitations of claim 1, as discussed above. Further, WU teaches “wherein the first hardware resources comprises any one or any combination of a measured power consumption, a memory demand, a number of operations, and a processing time to operate the neural network with the candidate architecture”:
[Page 10741, Table 4]
PNG
media_image24.png
424
913
media_image24.png
Greyscale
In the above table, CPU Latency can be seen which is equivalent to “a processing time to operate the neural network with the candidate architecture”, as is FLOPs, which equivalates to a “measured power consumption”.
The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 9, WU in view of GUO teaches the limitations of claim 1, as discussed above. Further, WU teaches “wherein the processor is further configured to: determine an optimization loss comprising the neural network loss and the hardware resource loss”:
([Page 10738, 3.3 Search Algorithm] “Solving the problem (1) through brute-force enumeration of the search space is very infeasible. The inner problem of optimizing wa (optimizing based on the hardware resource loss) involves training a neural network. For ImageNet classification, training a ConvNet typically takes several days or even weeks. The outer problem of optimizing
PNG
media_image4.png
20
55
media_image4.png
Greyscale
(Optimizing based on the neural network loss, meaning that the optimal architecture is selected based on minimization of loss of accuracy of the neural network.) has a combinatorially large search space.
Most of the early works on NAS [30, 31, 20] follow the paradigm above. To reduce the computational cost, the inner problem is replaced by training candidate architectures on an easier proxy dataset. For example, [30, 31] trains the architecture on the CIFAR10 dataset, and [20] trains on ImageNet but only for 5 epochs. The learned architectures are then transferred to the target dataset. To avoid exhaustively iterating through the search space, [30, 31, 20] use reinforcement learning to guide the exploration. Despite these improvements, solving problem (1) is still prohibitively expensive – training a network on the proxy dataset is still time-consuming, and thousands of architectures need to be trained before reaching the optimal solution.
We adopt a different paradigm of solving problem (1) (optimization based on the neural network loss and the hardware resource loss, meaning that the actual hardware latency is considered between the architecture and target hardware, thus calculation of the hardware resource loss). We first represent the search space by a stochastic super net. The super net has the same macro-architecture as described in Table 1, and each layer contains 9 parallel blocks as described in Table 2. During the inference of the super net, only one candidate block is sampled and executed with the sampling probability of
PNG
media_image9.png
46
409
media_image9.png
Greyscale
0l contains parameters that determine the sampling probability of each block at layer-l. Equivalently, the output of layer-l can be expressed as
PNG
media_image10.png
42
309
media_image10.png
Greyscale
where ml,I is a random variable in {0, 1} and is evaluated to 1 if block bl,i is sampled. The sampling probability is determined by equation (4). bl,i(xl) denotes the output of block-i at layer l given the input feature map xl. We let each layer sample independently, therefore, the probability of sampling an architecture a can be described as
PNG
media_image11.png
41
314
media_image11.png
Greyscale
where 0 denotes the a vector consists of all the 0l,i for each block-i at layer-l.
PNG
media_image12.png
29
27
media_image12.png
Greyscale
denotes that in the sampled architecture a, block-i is chosen at layer-l.
PNG
media_image13.png
365
442
media_image13.png
Greyscale
PNG
media_image14.png
245
416
media_image14.png
Greyscale
The above tables were placed here by the examiner for convenience since they are referenced in the text.
Instead of solving for the optimal architecture
PNG
media_image4.png
20
55
media_image4.png
Greyscale
, which has a discrete search space, we relax the problem to optimize the probability P0 of the stochastic super net to achieve the minimum expected loss (minimizing the neural network loss for optimization). Formally, we re-write the discrete optimization problem (1) as
PNG
media_image15.png
30
317
media_image15.png
Greyscale
It is obvious the loss function in (7) is differentiable with respect to the architecture weights wa and therefore can be optimized by stochastic gradient descent (SGD). However, the loss is not directly differentiable to the sampling parameter 0, since we cannot pass the gradient through the discrete random variable ml,i to 0l,i. To sidestep this, we relax the discrete mask variable ml,i to be a continuous random variable computed by the Gumbel Softmax function [9, 14]
PNG
media_image16.png
76
334
media_image16.png
Greyscale
where gl,i ~ Gumbel(0, 1) is a random noise following the Gumbel distribution. The Gumbel Softmax function is controlled by a temperature parameter
PNG
media_image17.png
11
12
media_image17.png
Greyscale
. As
PNG
media_image17.png
11
12
media_image17.png
Greyscale
approaches 0, it approximates the discrete categorical sampling following the distribution in (6). As
PNG
media_image17.png
11
12
media_image17.png
Greyscale
becomes larger, ml,i becomes a continuous random variable. Regardless of the value of
PNG
media_image17.png
11
12
media_image17.png
Greyscale
, the mask ml,i is directly differentiable with respect to the parameter 0l,i. The technique of using Gumbel Softmax for neural architecture search is also proposed in [25, 1].
As a result, it is clear that the cross-entropy term from the loss function (2) is differentiable with respect to the mask ml,i and therefore 0l,i. For the latency term, since we use the lookup table based model for efficiency estimation, equation (3) can be written as
PNG
media_image18.png
42
341
media_image18.png
Greyscale
The latency of each operator LAT(bl,i) is a constant coefficient, so the overall latency of architecture-a is differentiable with respect to the mask ml,i, therefore 0l,i.
As a result, the loss function (2) is fully differentiable with respect to both weights wa and the architecture distribution parameter 0 (Meaning that the optimization loss is calculated based on both the neural network loss and the hardware resource loss). This allows us to use SGD to efficiently solve problem (1).”)
And further:
([Page 10738, 3.3 Search Algorithm, final paragraph] “Our search process is now equivalent to training the stochastic super net. During the training, we compute
PNG
media_image19.png
20
65
media_image19.png
Greyscale
to train each operator’s weight in the super net. This is no different from training an ordinary ConvNet. After operators get trained, different operators can have a different contribution to the accuracy and the efficiency of the overall network. Therefore, we compute
PNG
media_image20.png
19
55
media_image20.png
Greyscale
to update the sampling probability P0 for each operator. This step selects operators with better accuracy and lower latency and suppresses the opposite ones (“better accuracy” represents minimization of the neural network loss while “lower latency” represents minimization of the hardware resource loss, which together, form a combined optimization loss.). After the super net training finishes, we can then obtain the optimal architectures by sampling from the architecture distribution P0.” Here we see the above methods concluding with the target architecture of the neural network being determined based on the loss functions.)
Further, WU teaches “determine the target architecture by selecting a target operation that minimizes the optimization loss from among candidate operations of each layer comprised in the neural network of the candidate architecture”:
([Page 10736-10737, 3.1 search Space, Paragraphs 2-3] “In this work, we construct a layer-wise search space (for each layer of the neural network…) with a fixed macro-architecture, and each layer can choose a different block. The macro architecture is described in Table 1. The macro architecture defines the number of layers and the input/output dimensions of each layer. The first and the last three layers of the network have fixed operators (fixed operations). For the rest of the layers, their block type needs to be searched (candidate operations). The filter numbers for each layer are hand-picked empirically. We use relatively small channel sizes for early layers, since the input resolution at early layers is large, and the computational cost (FLOP count) is quadratic to input size.
Each searchable layer in the network can choose a different block from the layer-wise search space. The block structure is inspired by MobileNetV2 [17] and ShiftNet [23], and is illustrated in Figure 3. It contains a point-wise (1x1) convolution, a K-by-K depthwise convolution where K denotes the kernel size, and another 1x1 convolution. “ReLU” activation functions follow the first 1x1 convolution and the depthwise convolution, but there are no activation functions following the last 1x1 convolution. If the output dimension stays the same as the input dimension, we use a skip connection to add the input to the output. Following [17, 23], we use a hyperparameter, the expansion ratio e, to control the block. It determines how much do we expand the output channel size of the first 1x1 convolution compared with its input channel size. Following [20], we also allow choosing a kernel size of 3 or 5 for the depthwise convolution. In addition, we can choose to use group convolution for the first and the last 1x1 convolution to reduce the computation complexity. When we use group convolution, we follow [29] to add a channel shuffle operation to mix the information between channel groups.
In our experiments, our layer-wise search space contains 9 candidate blocks, with their configurations listed in Table 2. Note we also have a block called “skip”, which directly feed the input feature map to the output without actual computations. This candidate block essentially allows us to reduce the depth of the network.
In summary, our overall search space contains 22 layers and each layer can choose from 9 candidate blocks from Table 2, so it contains 922
PNG
media_image21.png
13
15
media_image21.png
Greyscale
1021 possible architectures.”) Here, we see various parameters of the candidate architecture such as K (the kernel size) and the expansion ratio e. We see the candidate blocks which form the architecture / the candidate architecture’s parameters being used as inputs between each layer of the neural network.
And further:
[Figure 3]
PNG
media_image22.png
371
426
media_image22.png
Greyscale
In the above figure, the inputs denoted as H x W x Cin & H x W x (e x Cin) & (H/s) x (W/s) x (e x Cin) are fed as parameters of candidate architectures within each block which choose different expansion rates, kernel sizes, and numbers of groups, thus changing parameters while determining the architecture via layer-wise candidate operations. This citation further explains how the process works on a layer-by-layer level. In combination with the aforementioned citation of section 3.2, it is clear that the chosen operations are based on minimizing loss.
The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 10, WU in view of GUO teaches the limitations of claim 1, as discussed above. Further, WU teaches “wherein the processor is further configured to determine the neural network loss based on a difference between validation data and result data output by the neural network of the candidate architecture processing training data”:
([Page 10739, 4.1 ImageNet Classification, Paragraph 3] “Our experiment results are summarized in Table 3. We compare our searched models with state-of-the-art efficient models both designed automatically and manually. The primary metrics we care about are top-1 accuracy on the ImageNet validation set (top-1 accuracy in the table is used to show a difference between validation data and result data output by the neural network of the candidate architecture processing training data) and the latency. If the latency is not available, we use FLOP as the secondary efficiency metric. For baseline models, we directly cite the parameter size, FLOP count, and top-1 accuracy from the original paper. Since our network is deployed with caffe2 with highly efficient in8 implementation, we have an unfair latency advantage against other baselines. Therefore, we implement the baseline models ourselves and measure their latency under the same environment for a fair comparison. For automatically designed models, we also compare the search method, search space, and search cost.”)
PNG
media_image25.png
492
913
media_image25.png
Greyscale
Table 3, as referenced, was placed here by the examiner for convenience.
Regarding claim 11, the claim teaches essentially the same limitations as claim 1 discussed above, and is therefore rejected under the same rationale.
The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claims 12-15, WU in view of GUO teaches the limitations of claim 11. Further, claims 12-15 comprise similar additional limitations as claims 2, 4, 6, & 8, respectively, and are rejected under the same rationale.
Regarding claim 16, WU in view of GUO teaches the limitations of claim 11, as discussed above. Further, WU teaches “A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method”:
([Abstract] “Designing accurate and efficient ConvNets (convolutional neural networks) for mobile devices is challenging because the design space is combinatorially large. Due to this, previous neural architecture search (NAS) methods are computationally expensive. ConvNet architecture optimality depends on factors such as input resolution and target devices (parameters). However, existing approaches are too resource demanding for case-by-case redesigns. Also, previous work focuses primarily on reducing FLOPs, but FLOP count does not always reflect actual latency. To address these, we propose a differentiable neural architecture search (DNAS) framework (containing candidate architectures) that uses gradient-based methods to optimize ConvNet architectures, avoiding enumerating and training individual architectures separately as in previous methods. FBNets (Facebook-Berkeley-Nets), a family of models discovered by DNAS surpass state-of-the-art models both designed manually and generated automatically. FBNet-B achieves 74.1% top-1 accuracy on ImageNet with 295M FLOPs and 23.1 ms latency on a Samsung S8 phone, 2.4x smaller and 1.5x faster than MobileNetV2-1.3[17] with similar accuracy. Despite higher accuracy and lower latency than MnasNet[20], we estimate FBNet-B’s search cost is 420x smaller than MnasNet’s, at only 216 GPU-hours. Searched for different resolutions and channel sizes, FBNets achieve 1.5% to 6.4% higher accuracy than MobileNetV2. The smallest FBNet achieves 50.2% accuracy and 2.9 ms latency (345 frames per second) on a Samsung S8. Over a Samsung-optimized FBNet, the iPhone-X-optimized model achieves a 1.4x speedup on an iPhone X. FBNet models are open sourced at https://github.com/facebookresearch/mobile-vision.”) Additionally, “a differentiable neural architecture search framework” for optimizing neural network architectures inherently requires a computer with a processor and at least some form of physical memory, which is, under broadest reasonable interpretation, a non-transitory computer-readable storage medium, in order to function.
The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 17, the claim teaches essentially the same limitations as claim 1 discussed above, and is therefore rejected under the same rationale.
Regarding claim 18, WU in view of GUO teaches the limitations of claim 17, as discussed above. Further, WU teaches “wherein the selecting of the target architecture of the neural network comprises determining the target architecture of the neural network with a least sum of the neural network loss and a weight applied to the hardware resource loss”:
([Page 10737, 3.2 Latency-Aware Loss Function, Paragraphs 1-2] “The loss function used in (1) has to reflect not only the accuracy of a given architecture but also the latency on the target hardware (thus acknowledging the need to optimize based on hardware resources). To achieve this goal, we define the following loss function:
PNG
media_image5.png
26
367
media_image5.png
Greyscale
The first term CE(a, wa) denotes the cross-entropy loss of architecture a with parameter wa (a neural network loss). The second term LAT(a) denotes the latency of the architecture on the target hardware measured in micro-second (hardware resource loss). The coefficient
PNG
media_image6.png
13
14
media_image6.png
Greyscale
controls the overall magnitude of the loss function (a weight applied to the hardware resource loss). The exponent coefficient " modulates the magnitude of the latency term.”) In the above equation, the selection of the architecture is clearly based on a Least sum of a neural network loss, as evidenced by the cross-entropy loss of architecture a with parameter wa and the weight
PNG
media_image6.png
13
14
media_image6.png
Greyscale
applied to the hardware resource loss, LAT(a).
The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 19, WU in view of GUO teaches the limitations of claim 17, as discussed above. Further, WU teaches “wherein the generating of the hardware resource loss comprises generating the hardware resource loss based on applying a loss function that considers the difference between the first hardware resource and the second hardware resource”:
([Page 10736-10737, 3.1 search Space, Paragraphs 2-3] “In this work, we construct a layer-wise search space with a fixed macro-architecture, and each layer can choose a different block. The macro architecture is described in Table 1. The macro architecture defines the number of layers and the input/output dimensions of each layer. The first and the last three layers of the network have fixed operators. For the rest of the layers, their block type needs to be searched. The filter numbers for each layer are hand-picked empirically. We use relatively small channel sizes for early layers, since the input resolution at early layers is large, and the computational cost (FLOP count) is quadratic to input size.
Each searchable layer in the network can choose a different block from the layer-wise search space. The block structure is inspired by MobileNetV2 [17] and ShiftNet [23], and is illustrated in Figure 3. It contains a point-wise (1x1) convolution, a K-by-K depthwise convolution where K denotes the kernel size, and another 1x1 convolution. “ReLU” activation functions follow the first 1x1 convolution and the depthwise convolution, but there are no activation functions following the last 1x1 convolution. If the output dimension stays the same as the input dimension, we use a skip connection to add the input to the output. Following [17, 23], we use a hyperparameter, the expansion ratio e, to control the block. It determines how much do we expand the output channel size of the first 1x1 convolution compared with its input channel size. Following [20], we also allow choosing a kernel size of 3 or 5 for the depthwise convolution. In addition, we can choose to use group convolution for the first and the last 1x1 convolution to reduce the computation complexity. When we use group convolution, we follow [29] to add a channel shuffle operation to mix the information between channel groups.
In our experiments, our layer-wise search space contains 9 candidate blocks, with their configurations listed in Table 2. Note we also have a block called “skip”, which directly feed the input feature map to the output without actual computations. This candidate block essentially allows us to reduce the depth of the network.
In summary, our overall search space contains 22 layers and each layer can choose from 9 candidate blocks from Table 2, so it contains 922
PNG
media_image21.png
13
15
media_image21.png
Greyscale
1021 possible architectures.”) The actual measured per-layer latency is the first hardware resource. The predicted architecture latency is the second hardware resource, via summing lookup table values. The “difference” is inherently minimized because the lookup table is built to match the measured values, meaning that the prediction error is the gap between measurement and table-based estimates.
And further:
[Figure 3]
PNG
media_image22.png
371
426
media_image22.png
Greyscale
In the above figure, the inputs denoted as H x W x Cin & H x W x (e x Cin) & (H/s) x (W/s) x (e x Cin) are fed as parameters of candidate architectures within each block which choose different expansion rates, kernel sizes, and numbers of groups.
And further:
([Page 10737, 3.2 Latency-Aware Loss Function, Paragraphs 1-2] “The loss function used in (1) has to reflect not only the accuracy of a given architecture but also the latency on the target hardware (thus acknowledging the need to optimize based on hardware resources). To achieve this goal, we define the following loss function:
PNG
media_image5.png
26
367
media_image5.png
Greyscale
The first term CE(a, wa) denotes the cross-entropy loss of architecture a with parameter wa (a neural network loss). The second term LAT(a) denotes the latency of the architecture on the target hardware measured in micro-second (hardware resource loss). The coefficient
PNG
media_image6.png
13
14
media_image6.png
Greyscale
controls the overall magnitude of the loss function (a weight applied to the hardware resource loss). The exponent coefficient " modulates the magnitude of the latency term.”) You can directly see in the equation that the latency term LAT(a), which is based on the predictor, is added to the overall weighted loss.
The motivation for combining the references is as discussed above in relation to claim 1.
Regarding claim 20, WU in view of GUO teaches the limitations of claim 17, as discussed above. Further, WU teaches “wherein the candidate architecture comprises a neural network structure including a set of candidate operations for each layer of the neural network”:
([Page 10736-10737, 3.1 search Space, Paragraphs 2-3] “In this work, we construct a layer-wise search space with a fixed macro-architecture, and each layer can choose a different block. The macro architecture is described in Table 1. The macro architecture defines the number of layers and the input/output dimensions of each layer. The first and the last three layers of the network have fixed operators. For the rest of the layers, their block type needs to be searched. The filter numbers for each layer are hand-picked empirically. We use relatively small channel sizes for early layers, since the input resolution at early layers is large, and the computational cost (FLOP count) is quadratic to input size.
Each searchable layer in the network can choose a different block from the layer-wise search space. The block structure is inspired by MobileNetV2 [17] and ShiftNet [23], and is illustrated in Figure 3. It contains a point-wise (1x1) convolution, a K-by-K depthwise convolution where K denotes the kernel size, and another 1x1 convolution. “ReLU” activation functions follow the first 1x1 convolution and the depthwise convolution, but there are no activation functions following the last 1x1 convolution. If the output dimension stays the same as the input dimension, we use a skip connection to add the input to the output. Following [17, 23], we use a hyperparameter, the expansion ratio e, to control the block. It determines how much do we expand the output channel size of the first 1x1 convolution compared with its input channel size. Following [20], we also allow choosing a kernel size of 3 or 5 for the depthwise convolution. In addition, we can choose to use group convolution for the first and the last 1x1 convolution to reduce the computation complexity. When we use group convolution, we follow [29] to add a channel shuffle operation to mix the information between channel groups.
In our experiments, our layer-wise search space contains 9 candidate blocks, with their configurations listed in Table 2. Note we also have a block called “skip”, which directly feed the input feature map to the output without actual computations. This candidate block essentially allows us to reduce the depth of the network.
In summary, our overall search space contains 22 layers and each layer can choose from 9 candidate blocks from Table 2, so it contains 922
PNG
media_image21.png
13
15
media_image21.png
Greyscale
1021 possible architectures.”)
And further:
[Figure 3]
PNG
media_image22.png
371
426
media_image22.png
Greyscale
In the above figure, the inputs denoted as H x W x Cin & H x W x (e x Cin) & (H/s) x (W/s) x (e x Cin) are fed as parameters of candidate architectures within each block which choose different expansion rates, kernel sizes, and numbers of groups.
Conclusion
Applicants’ amendment necessitated the new ground(s) of rejection presented in this Office Action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicants are reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHOURJO DASGUPTA whose telephone number is (571)272-7207. The examiner can normally be reached M-F 8am-5pm CST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tamara Kyle can be reached at 571 272 4241. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHOURJO DASGUPTA/Primary Examiner, Art Unit 2144
Prosecution Insights