ADAPTATION OF TASK PERFORMABLE BY PRE-TRAINED MODEL INTO PARALLEL HARDWARE

§103 §112

DETAILED ACTION This communication is in response to the amendment filed 8/4/26 in which claims 1, 3-5, 16-18, 20 were amended. Claims 1-20 are pending. Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Response to Arguments Applicant’s arguments with respect to the amended features have been considered but are moot in light of the new reference (Taylor) used to teach those features. Applicant further argues: In Bhardwaj, knowledge partitioning splits the teaching model based on the characteristics for the entire distributed system and the teacher model. Characterization is instead done on the teacher model, to distribute network partitions to student models and find a total system memory and FLOPS budget. For example, Bhardwaj's Fig. 1 shows the NoNN student model split from a 500 KB total budget. Each student merely satisfies a memory or FLOP budget allocation from the combined resources. Bhardwaj teaches assigning students to devices, so they are not created based on characteristics of the hardware. There, the system is designed to break up a large machine learning model into several partitions based on patterns of activation of filters as the teacher trains. Bhardwaj states, "the sizes of partitions of [the final convolution layer of the teacher model] should be almost equal so that the resulting students networks fit within a memory budget, while their computation costs fit a FLOP budget." (Bhardwaj 7). This is shown in problem (2), where Bhardwaj explains, "the last two constraints [The memory budget and FLOP budget] specify the per-device memory- and FLOP-budgets that must be satisfied by each student (which will be deployed on individual device)." (Bhardwaj 8, Fig. 4). The memory and FLOPS budgets are not based on specific characteristics of each compute resource, but a total, fixed system budget to hold an equal portion of the knowledge partition; while some characteristics could contribute to the budget, each learner model is not individually selected for adaptation to a specific hardware. Applicant’s arguments have been considered but are unpersuasive. Bhardwaj teaches specifying a per-device memory and FLOP budget constraint that must be satisfied by each student. The per-device memory and FLOP budget is interpreted as the claimed “characteristics of the corresponding compute resource on which the learner model is to run,” at least because the per-device budgets are with respect to the devices on which the models will be deployed. In particular, the claim does not indicate how the claimed characteristics of the compute resources are determined with respect to specific hardware. Applicant further argues: Furthermore, because of the knowledge partitioning, "a NoNN consists of multiple, memory-limited student networks which individually learn only a specific part of teacher's function." (Bhardwaj 3). As shown in Figure 4, The NoNN students communicate at the last layer before output of the result, in this case a classification of "cat." This is distinct from the current claims, particularly as amended, which require each of the plurality of learner models generates a task result, and then the plurality of task results are aggregated to generate a different, aggregated task result. Applicant’s arguments have been considered but are unpersuasive. Figure 4 of Bhardwaj teaches that each student learns a part of teacher’s function. The term “task result” may reasonably be interpreted to be the output of whatever part of a teacher’s function that is performed by the student. Claim Rejections - 35 USC § 112 The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph: The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention. Claims 1, 16, and 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 1 recites the limitation "the hardware" in line 9. There is insufficient antecedent basis for this limitation in the claim. Claim 1 recites the limitation “the corresponding compute resource” in line 13. There is insufficient antecedent basis for this limitation in the claim. Claim 1 recites the limitation “the task” in line 14. There is insufficient antecedent basis for this limitation in the claim. Claim 1 recites the limitation “the specific configuration of compute resources” in line 15-16. There is insufficient antecedent basis for this limitation in the claim. Claim 1 recites the limitation “the pre-trained machine learning model” in line 17-18. There is insufficient antecedent basis for this limitation in the claim. Claims 16 and 20 recite the same limitations and, therefore, are similarly rejected. Claim Rejections - 35 USC § 103 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or non-obviousness. Claims 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Bhardwaj, Kartikeya, et al. "Memory-and communication-aware model compression for distributed deep learning inference on IoT." ACM Transactions on Embedded Computing Systems (TECS) 18.5s (2019): 1-22 (“Bhardwaj”) in view of Taylor, Ben, et al. "Adaptive deep learning model selection on embedded systems." ACM Sigplan Notices 53.6 (2018): 31-43 (“Taylor”). Regarding claim 16, Bhardwaj discloses [a] method performed by a computing system to parallelize a task capable of being accomplished by a pre-trained machine learning model, (Bhardwaj, p. 11 (“We compress various deep networks for five well-known image classification tasks: CIFAR-10, CIFAR-100, Scene [18], Caltech-UCSD-Birds (CUB) [22], and Imagenet.”)) the parallelization performed in a manner that is adapted to hardware, (Bhardwaj, p. 3 (“Since there has been limited research on performing efficient distributed inference on highly memory-constrained IoT devices, we propose a new NoNN paradigm to compress a large teacher into a set of independent and highly-parallel student networks.”)) the method comprising: characterizing a plurality of parallel compute resources of the hardware; and (Bhardwaj, p. 7 (“Moreover, the sizes of partitions of Tfconv should be almost equal so that the resulting student networks fit within a memory budget Bmem, while their computation costs fit a FLOP budget BFLOP.”), p. 8 (“The last two constraints in problem (2) specify the per-device memory- and FLOP-budgets [parallel compute resources] that must be satisfied by each student (which will be deployed on individual devices).”), problem (2) is reproduced here: PNG media_image1.png 264 948 media_image1.png Greyscale ) creating and operating a plurality of learner models by, for each of the parallel compute resources, performing the following: (Bhardwaj, p. 7 (“We first formulate our problem (Section 4.1), followed by our proposed solution which consists of two stages: (1) Network science-based knowledge partitioning of the teacher: We find disjoint partitions of teacher’s knowledge which lead to independent students with minimal communication costs (Section 4.2, Figure 4(b)). (2) Proposed NoNN architecture and the training process: We first select an efficient model architecture for each student that reduces its memory and FLOPS, and then jointly train individual NoNN students on disjoint partitions from teacher (Section 4.3; Figure 4(c)).”)) selecting one or more characteristics of a learner model based on one or more characteristics of the corresponding compute resource on which the learner model is to run (Bhardwaj, p. 7 (“Let h: R → R compute the memory of student i depending on its partition size |Pi|, and f: R → R compute the resulting FLOPS.”), p. 8 (“The first constraint in (2) aims to keep the sizes of all partitions (except P0) [e.g., characteristics of a learner model] almost equal since the sizes of student models must be below the fixed memory budget Bmem [e.g., characteristics of the corresponding compute resource]. The objective function and the second constraint aim to minimize change in teacher’s validation accuracy while removing as many useless filters from Tfconv as possible. The last two constraints in problem (2) specify the per-device memory- and FLOP-budgets that must be satisfied by each student (which will be deployed on individual devices).”), p. 10 (“Similar to prior art in KD [6, 25], we select our individual student networks to be significantly reduced versions of the teacher (i.e., far fewer layers and lower width). Specifically, we pick our individual student models based on user-defined memory and FLOP-constraints for IoT-devices.”)). Bhardwaj does not expressly disclose selecting one or more characteristics of a learner model…further based on one or more characteristics of the task, such that the learning model is optimally configured to run on the specific configuration of compute resources to perform the task; (but see Taylor p. 32 1st column (“This paper seeks to offer an alternative to enable efficient deep inference1 on embedded devices. Our goal is to design an adaptive scheme to determine, at runtime, which of the available DNN models is the best fit for the input and the precision requirement. This is motivated by the observation that the optimum model for inference depends on the input data and the precision requirement. For example, if the input image is taken under good lighting conditions with a simple background, a simple but fast model would be sufficient for object identification - otherwise, a more sophisticated but slower model will have to be employed; in a similar vein, if we want to detect objects with a high confidence, an advanced model should be used - otherwise, a simple model would be good enough. Given that DNN models are becoming increasingly diverse together with the evolving application workload and user requirements, the right strategy for model selection is likely to change over time. This ever-evolving nature makes automatic heuristic design highly attractive because the heuristic can be easily updated to adapt to the changing application context. This paper presents a novel runtime approach for DNN model selection on embedded devices, aiming to minimize the inference time while meeting the user requirement. We achieve this by employing machine learning to automatically construct predictors to select at runtime the optimum model to use. Our predictor is first trained off-line. Then, using a set of automatically tuned features of the DNN model input, the predictor determines the optimum DNN model for a new, unseen input, by taking into consideration the precision constraint and the characteristics of the input. We show that our approach can automatically derive high-quality heuristics for different precision requirements. The learned strategy can effectively leverage the prediction capability and runtime overhead of candidate DNN models, leading to an overall better accuracy when compared with the most capable DNN model, but with significantly less runtime overhead. Using our approach, one can also first apply model compression techniques to generate DNN models of different capabilities and inference time, and then choose a model to use at runtime. This is a new way for optimizing deep inference on embedded devices.”)). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bhardwaj to incorporate the teachings of Taylor to determine, at runtime, which of the available models is the best fit for the input and the precision requirement, at least because doing so would minimize the inference time while meeting the user requirement. Bhardwaj further discloses: tuning the learner model based on the pre-trained machine learning model; and (Bhardwaj, p. 7 (“Proposed NoNN architecture and the training process: We first select an efficient model architecture for each student [learner model] that reduces its memory and FLOPS and then jointly train individual NoNN students [learner model] on disjoint partitions from teacher.”), p. 8, Fig. 4(c) (“Train individual students in NoNN to mimic each partition.”), p. 5 (“KD, as shown in Figure 2(a), consists of two deep networks: (i) Teacher is the large deep network which we want to compress, and (ii) Student is a significantly smaller neural network which is trained to mimic the output of the teacher network. To mimic the teacher, the simplest way is to directly train a student model on teacher’s logits (see Figure 2(a))– the unnormalized outputs of the teacher model before the softmax– instead of training on true labels from the dataset [1]. The basic idea here is that while training the student model, using logits instead of direct classification probabilities can transfer more information about the teacher to the student.”)) providing input to the learner model so that the learner model generates a task result, such that the plurality of learning models generates a plurality of task results; and (Bhardwaj, p. 8, Fig. 4c indicates student networks receiving an input image to generate a classification result: PNG media_image2.png 270 644 media_image2.png Greyscale ) aggregating the task results from at least some of the plurality of learner models to generate an aggregated task result (Bhardwaj, pp. 10-11 (“Once the common group is replicated, the individual students are completely independent and do not communicate until the final fc layer (see Figure 5(b)). Hence, we obtain a final NoNN architecture which consists of multiple disjoint students. The output from all students is concatenated and passed through a fc to yield logits [aggregated task result].”)). Regarding claim 17, Bhardwaj, in view of Taylor, discloses the invention of claim 16 as discussed above. Bhardwaj further discloses the creation being performed for at least one of the plurality of learner models, creating the learner model as a student model by initializing the student model, and applying distillation from a teacher model to tune the student model (Bhardwaj, p. 5 (“KD, as shown in Figure 2(a), consists of two deep networks: (i) Teacher is the large deep network which we want to compress, and (ii) Student is a significantly smaller neural network which is trained to mimic the output of the teacher network. To mimic the teacher, the simplest way is to directly train a student model on teacher’s logits (see Figure 2(a))– the unnormalized outputs of the teacher model before the softmax– instead of training on true labels from the dataset [1]. The basic idea here is that while training the student model, using logits instead of direct classification probabilities can transfer more information about the teacher to the student. Hinton et al. [6] argued that we must not only train the student model on the correct predictions made by the teacher, but must also quantify how wrong the teacher was about the incorrect classes. Hence, Hinton et al. introduced KD [6] which uses both the hard-label loss (based on true labels from the dataset), as well as the soft-label loss (based on logits) to train a significantly smaller student model.”)). Regarding claim 18, Bhardwaj, in view of Taylor, discloses the invention of claim 16 as discussed above. Bhardwaj further discloses the creation being performed for at least one of the plurality of learner models, creating the learner model as a student model by sparcifying a teacher model to create the student model (Bhardwaj, p. 2 (“To this end, several deep learning model compression techniques exist in the literature. The most common approaches are pruning [5, 23], quantization [8, 12], and Knowledge Distillation (KD) [1, 6] and its variants such as Attention Transfer (AT) [25]. Since deep models can learn a large number of redundant or useless weights/channels, pruning aims to remove such parameters without sacrificing accuracy [5, 23]. Moreover, quantization reduces the number of bits required to represent weights and/or activations in deep networks [8, 12]. Also, KD-based approaches rely on teacher-student training, where a teacher model is a large deep network we want to compress. KDtrains a significantly smaller student model with far fewer layers/width, to mimic the teacher network (see Figure 1(a)). Further, most prior art on model compression refers to pruning and quantization.”)). Regarding claim 19, Bhardwaj, in view of Taylor, discloses the invention of claim 16 as discussed above. Bhardwaj further discloses at least one of the plurality of learner models comprising a neural network, (Bhardwaj, p. 3 (“To address this question, in this paper, we propose Network of Neural Networks (NoNN), a paradigm to derive low memory-, computation-, as well as communication-cost student architectures from a single (powerful) but much larger teacher model.”)) the pre-trained machine learning model also being a neural network, (Bhardwaj, p. 3 (“To address this question, in this paper, we propose Network of Neural Networks (NoNN), a paradigm to derive low memory-, computation-, as well as communication-cost student architectures from a single (powerful) but much larger teacher model.”)) a number of layers of the neural network of the learner model being less than a number of layers of the pre-trained machine learning model (Bhardwaj, p. 3 (“To address this question, in this paper, we propose Network of Neural Networks (NoNN), a paradigm to derive low memory-, computation-, as well as communication-cost student architectures from a single (powerful) but much larger teacher model.”), p. 2 (“To this end, several deep learning model compression techniques exist in the literature. The most common approaches are pruning [5, 23], quantization [8, 12], and Knowledge Distillation (KD) [1, 6] and its variants such as Attention Transfer (AT) [25]. Since deep models can learn a large number of redundant or useless weights/channels, pruning aims to remove such parameters without sacrificing accuracy [5, 23]. Moreover, quantization reduces the number of bits required to represent weights and/or activations in deep networks [8, 12]. Also, KD-based approaches rely on teacher-student training, where a teacher model is a large deep network we want to compress. KDtrains a significantly smaller student model with far fewer layers/width, to mimic the teacher network (see Figure 1(a)). Further, most prior art on model compression refers to pruning and quantization.”)). Regarding claim 20, Bhardwaj discloses [a] method performed by a computing system to parallelize a task capable of being accomplished by a pre-trained machine learning model, (Bhardwaj, p. 11 (“We compress various deep networks for five well-known image classification tasks: CIFAR-10, CIFAR-100, Scene [18], Caltech-UCSD-Birds (CUB) [22], and Imagenet.”)) the parallelization performed in a manner that is adapted to hardware (Bhardwaj, p. 3 (“Since there has been limited research on performing efficient distributed inference on highly memory-constrained IoT devices, we propose a new NoNN paradigm to compress a large teacher into a set of independent and highly-parallel student networks.”)) the method comprising: characterizing a plurality of parallel compute resources of the hardware; and (Bhardwaj, p. 7 (“Moreover, the sizes of partitions of Tfconv should be almost equal so that the resulting student networks fit within a memory budget Bmem, while their computation costs fit a FLOP budget BFLOP.”), p. 8 (“The last two constraints in problem (2) specify the per-device memory- and FLOP-budgets [parallel compute resources] that must be satisfied by each student (which will be deployed on individual devices).”), problem (2) is reproduced here: PNG media_image1.png 264 948 media_image1.png Greyscale ) creating and operating a plurality of learner models by, for each of at least some of the parallel compute resources, performing the following: (Bhardwaj, p. 7 (“We first formulate our problem (Section 4.1), followed by our proposed solution which consists of two stages: (1) Network science-based knowledge partitioning of the teacher: We find disjoint partitions of teacher’s knowledge which lead to independent students with minimal communication costs (Section 4.2, Figure 4(b)). (2) Proposed NoNN architecture and the training process: We first select an efficient model architecture for each student that reduces its memory and FLOPS, and then jointly train individual NoNN students on disjoint partitions from teacher (Section 4.3; Figure 4(c)).”)) selecting one or more characteristics of a learner model based on one or more characteristics of the corresponding compute resource on which the learner model is to run (Bhardwaj, p. 7 (“Let h: R → R compute the memory of student i depending on its partition size |Pi|, and f: R → R compute the resulting FLOPS.”), p. 8 (“The first constraint in (2) aims to keep the sizes of all partitions (except P0) [e.g., characteristics of a learner model] almost equal since the sizes of student models must be below the fixed memory budget Bmem [e.g., characteristics of the corresponding compute resource]. The objective function and the second constraint aim to minimize change in teacher’s validation accuracy while removing as many useless filters fromTfconv as possible. The last two constraints in problem (2) specify the per-device memory- and FLOP-budgets that must be satisfied by each student (which will be deployed on individual devices).”), p. 10 (“Similar to prior art in KD [6, 25], we select our individual student networks to be significantly reduced versions of the teacher (i.e., far fewer layers and lower width). Specifically, we pick our individual student models based on user-defined memory and FLOP-constraints for IoT-devices.”)). Bhardwaj does not expressly disclose selecting one or more characteristics of a learner model…further based on one or more characteristics of the task, such that the learner model is optimally configured to run on the specific configuration of compute resources to perform the task; (but see Taylor p. 32 1st column (“This paper seeks to offer an alternative to enable efficient deep inference1 on embedded devices. Our goal is to design an adaptive scheme to determine, at runtime, which of the available DNN models is the best fit for the input and the precision requirement. This is motivated by the observation that the optimum model2 for inference depends on the input data and the precision requirement. For example, if the input image is taken under good lighting conditions with a simple background, a simple but fast model would be sufficient for object identification - otherwise, a more sophisticated but slower model will have to be employed; in a similar vein, if we want to detect objects with a high confidence, an advanced model should be used ś otherwise, a simple model would be good enough. Given that DNN models are becoming increasingly diverse ś together with the evolving application workload and user requirements, the right strategy for model selection is likely to change over time. This ever-evolving nature makes automatic heuristic design highly attractive because the heuristic can be easily updated to adapt to the changing application context. This paper presents a novel runtime approach for DNN model selection on embedded devices, aiming to minimize the inference time while meeting the user requirement. We achieve this by employing machine learning to automatically construct predictors to select at runtime the optimum model to use. Our predictor is first trained off-line. Then, using a set of automatically tuned features of the DNN model input, the predictor determines the optimum DNN model for a new, unseen input, by taking into consideration the precision constraint and the characteristics of the input. We show that our approach can automatically derive high-quality heuristics for different precision requirements. The learned strategy can effectively leverage the prediction capability and runtime overhead of candidate DNN models, leading to an overall better accuracy when compared with the most capable DNN model, but with significantly less runtime overhead. Using our approach, one can also first apply model compression techniques to generate DNN models of different capabilities and inference time, and then choose a model to use at runtime. This is a new way for optimizing deep inference on embedded devices.”)). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bhardwaj to incorporate the teachings of Taylor to determine, at runtime, which of the available models is the best fit for the input and the precision requirement, at least because doing so would minimize the inference time while meeting the user requirement. Bhardwaj further discloses: tuning the learner model based on the pre-trained machine learning model; and (Bhardwaj, p. 7 (“Proposed NoNN architecture and the training process: We first select an efficient model architecture for each student [learner model] that reduces its memory and FLOPS and then jointly train individual NoNN students [learner models] on disjoint partitions from teacher.”), p. 8, Fig. 4(c) (“Train individual students in NoNN to mimic each partition.”), p. 5 (“KD, as shown in Figure 2(a), consists of two deep networks: (i) Teacher is the large deep network which we want to compress, and (ii) Student is a significantly smaller neural network which is trained to mimic the output of the teacher network. To mimic the teacher, the simplest way is to directly train a student model on teacher’s logits (see Figure 2(a))– the unnormalized outputs of the teacher model before the softmax– instead of training on true labels from the dataset [1]. The basic idea here is that while training the student model, using logits instead of direct classification probabilities can transfer more information about the teacher to the student.”)) providing input to the learner model so that the learner model generates a task result, such that the plurality of learner models generates a plurality of task results; and (Bhardwaj, p. 8, Fig. 4c indicates student networks [e.g., plurality of learner models] receiving an input image [e.g., input] to generate a classification result [e.g., task result]: PNG media_image2.png 270 644 media_image2.png Greyscale ) aggregating the task results from at least some of the at least some of the plurality of learner models to generate an aggregated task result, (Bhardwaj, pp. 10-11 (“Once the common group is replicated, the individual students are completely independent and do not communicate until the final fc layer (see Figure 5(b)). Hence, we obtain a final NoNN architecture which consists of multiple disjoint students. The output from all students [task results] is concatenated and passed through a fc to yield logits [aggregated task result].”)) the creation being performed for at least one of the plurality of learner models, creating the learner model as a student model by initializing the student model, and applying distillation from a teacher model to tune the student model, (Bhardwaj, p. 5 (“KD, as shown in Figure 2(a), consists of two deep networks: (i) Teacher is the large deep network which we want to compress, and (ii) Student is a significantly smaller neural network which is trained to mimic the output of the teacher network. To mimic the teacher, the simplest way is to directly train a student model on teacher’s logits (see Figure 2(a))– the unnormalized outputs of the teacher model before the softmax– instead of training on true labels from the dataset [1]. The basic idea here is that while training the student model, using logits instead of direct classification probabilities can transfer more information about the teacher to the student. Hinton et al. [6] argued that we must not only train the student model on the correct predictions made by the teacher, but must also quantify how wrong the teacher was about the incorrect classes. Hence, Hinton et al. introduced KD [6] which uses both the hard-label loss (based on true labels from the dataset), as well as the soft-label loss (based on logits) to train a significantly smaller student model.”), p. 10 (“A NoNN consisting of two students is shown in Figure 5(b). As evident, wherever possible, we make the initial few layers from the students common (since the knowledge learned at initial few layers will be common for all students).”), p. 11 (“To train the overall NoNN, we use the KD loss (see Equation (1)) and, as given below, we additionally propose a new loss function called activation-transfer for transferring partitioned knowledge from the teacher to individual students.”)) the creation being performed for at least another of the plurality of learner models, creating the learner model as a student model by sparcifying a teacher model to create the student model, (Bhardwaj, p. 2 (“To this end, several deep learning model compression techniques exist in the literature. The most common approaches are pruning [5, 23], quantization [8, 12], and Knowledge Distillation (KD) [1, 6] and its variants such as Attention Transfer (AT) [25]. Since deep models can learn a large number of redundant or useless weights/channels, pruning aims to remove such parameters without sacrificing accuracy [5, 23]. Moreover, quantization reduces the number of bits required to represent weights and/or activations in deep networks [8, 12]. Also, KD-based approaches rely on teacher-student training, where a teacher model is a large deep network we want to compress. KD trains a significantly smaller student model with far fewer layers/width, to mimic the teacher network (see Figure 1(a)). Further, most prior art on model compression refers to pruning and quantization.”)) at least one of the plurality of learner models comprising a neural network, the pre-trained machine learning model also being a neural network, a number of layers of the neural network of the learner model being less than a number of layers of the pre-trained machine learning model (Bhardwaj, p. 3 (“To address this question, in this paper, we propose Network of Neural Networks (NoNN), a paradigm to derive low memory-, computation-, as well as communication-cost student architectures from a single (powerful) but much larger teacher model.”), p. 2 (“To this end, several deep learning model compression techniques exist in the literature. The most common approaches are pruning [5, 23], quantization [8, 12], and Knowledge Distillation (KD) [1, 6] and its variants such as Attention Transfer (AT) [25]. Since deep models can learn a large number of redundant or useless weights/channels, pruning aims to remove such parameters without sacrificing accuracy [5, 23]. Moreover, quantization reduces the number of bits required to represent weights and/or activations in deep networks [8, 12]. Also, KD-based approaches rely on teacher-student training, where a teacher model is a large deep network we want to compress. KDtrains a significantly smaller student model with far fewer layers/width, to mimic the teacher network (see Figure 1(a)). Further, most prior art on model compression refers to pruning and quantization.”)). Claims 1-7, 9, and 11-15 are rejected under 35 U.S.C. 103 as being unpatentable over Bhardwaj in view of Taylor and Abdelfattah (US 2021/0081763 A1; published Mar. 18, 2021). Regarding claim 1, Bhardwaj discloses [a] computing system comprising: one or more processors; and (Bhardwaj, p. 12 (“Finally, our entire framework is implemented in Pytorch which is run on NVIDIA GTX 1080-Ti and Titan Xp GPUs [processors].”)) one or more computer-readable media having thereon computer-executable instructions that are executable by the one or more processors, (Bhardwaj, p. 12, (“Finally, our entire framework is implemented in Pytorch [e.g., computer-executable instructions] which is run on NVIDIA GTX 1080-Ti and Titan Xp GPUs.”)) to cause the computing system to: characterize a plurality of parallel compute resources of the hardware; and (Bhardwaj, p. 7 (“Moreover, the sizes of partitions of Tfconv should be almost equal so that the resulting student networks fit within a memory budget Bmem, while their computation costs fit a FLOP budget BFLOP.”), p. 8 (“The last two constraints in problem (2) specify the per-device memory- and FLOP-budgets [parallel compute resources of the hardware] that must be satisfied by each student (which will be deployed on individual devices).”), problem (2) is reproduced here: PNG media_image1.png 264 948 media_image1.png Greyscale create and operate a plurality of learner models by, for each of the parallel compute resources, performing the following: (Bhardwaj, p. 7 (“We first formulate our problem (Section 4.1), followed by our proposed solution which consists of two stages: (1) Network science-based knowledge partitioning of the teacher: We find disjoint partitions of teacher’s knowledge which lead to independent students with minimal communication costs (Section 4.2, Figure 4(b)). (2) Proposed NoNN architecture and the training process: We first select an efficient model architecture for each student that reduces its memory and FLOPS, and then jointly train individual NoNN students on disjoint partitions from teacher (Section 4.3; Figure 4(c)).”)) select one or more characteristics of a learner model based on one or more characteristics of the corresponding compute resource on which the learner model is to run (Bhardwaj, p. 7 (“Let h: R → R compute the memory of student i depending on its partition size |Pi|, and f: R → R compute the resulting FLOPS.”), p. 8 (“The first constraint in (2) aims to keep the sizes of all partitions (except P0) [e.g., characteristics of a learner model] almost equal since the sizes of student models must be below the fixed memory budget Bmem [e.g., characteristics of the corresponding compute resource]. The objective function and the second constraint aim to minimize change in teacher’s validation accuracy while removing as many useless filters from Tfconv as possible. The last two constraints in problem (2) specify the per-device memory- and FLOP-budgets that must be satisfied by each student (which will be deployed on individual devices).”), p. 10 (“Similar to prior art in KD [6, 25], we select our individual student networks to be significantly reduced versions of the teacher (i.e., far fewer layers and lower width). Specifically, we pick our individual student models based on user-defined memory and FLOP-constraints for IoT-devices.”) (the computed memory and FLOPS of a student model correspond to the claimed characteristics of a learner model and the per-device memory and FLOP constraints correspond to the claimed characteristics of the compute resource)). Bhardwaj does not expressly disclose select one or more characteristics of a learner model…further based on one or more characteristics of the task, such that the learner model is optimally configured to run on the specific configuration of compute resources to perform the task; (but see Taylor p. 32 1st column (“This paper seeks to offer an alternative to enable efficient deep inference1 on embedded devices. Our goal is to design an adaptive scheme to determine, at runtime, which of the available DNN models is the best fit for the input and the precision requirement. This is motivated by the observation that the optimum model2 for inference depends on the input data and the precision requirement. For example, if the input image is taken under good lighting conditions with a simple background, a simple but fast model would be sufficient for object identification - otherwise, a more sophisticated but slower model will have to be employed; in a similar vein, if we want to detect objects with a high confidence, an advanced model should be used ś otherwise, a simple model would be good enough. Given that DNN models are becoming increasingly diverse ś together with the evolving application workload and user requirements, the right strategy for model selection is likely to change over time. This ever-evolving nature makes automatic heuristic design highly attractive because the heuristic can be easily updated to adapt to the changing application context. This paper presents a novel runtime approach for DNN model selection on embedded devices, aiming to minimize the inference time while meeting the user requirement. We achieve this by employing machine learning to automatically construct predictors to select at runtime the optimum model to use. Our predictor is first trained off-line. Then, using a set of automatically tuned features of the DNN model input, the predictor determines the optimum DNN model for a new, unseen input, by taking into consideration the precision constraint and the characteristics of the input. We show that our approach can automatically derive high-quality heuristics for different precision requirements. The learned strategy can effectively leverage the prediction capability and runtime overhead of candidate DNN models, leading to an overall better accuracy when compared with the most capable DNN model, but with significantly less runtime overhead. Using our approach, one can also first apply model compression techniques to generate DNN models of different capabilities and inference time, and then choose a model to use at runtime. This is a new way for optimizing deep inference on embedded devices.”)). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bhardwaj to incorporate the teachings of Taylor to determine, at runtime, which of the available models is the best fit for the input and the precision requirement, at least because doing so would minimize the inference time while meeting the user requirement. Bhardwaj further discloses: tune the learner model based on the pre-trained machine learning model; and (Bhardwaj, p. 7 (“Proposed NoNN architecture and the training process: We first select an efficient model architecture for each student [learner model] that reduces its memory and FLOPS and then jointly train individual NoNN students [learner model] on disjoint partitions from teacher.”), p. 8, Fig. 4(c) (“Train individual students in NoNN to mimic each partition.”), p. 5 (“KD, as shown in Figure 2(a), consists of two deep networks: (i) Teacher is the large deep network which we want to compress, and (ii) Student is a significantly smaller neural network which is trained to mimic the output of the teacher network. To mimic the teacher, the simplest way is to directly train a student model on teacher’s logits (see Figure 2(a))– the unnormalized outputs of the teacher model before the softmax– instead of training on true labels from the dataset [1]. The basic idea here is that while training the student model, using logits instead of direct classification probabilities can transfer more information about the teacher to the student.”)) provide input to the learner model so that the learner model generates a task result, such that the plurality of learner models generate a plurality of task results; and (Bhardwaj, p. 8, Fig. 4c indicates student networks receiving an input image to generate a classification result: PNG media_image2.png 270 644 media_image2.png Greyscale ) aggregate the plurality of task results to generate an aggregated task result (Bhardwaj, pp. 10-11 (“Once the common group is replicated, the individual students are completely independent and do not communicate until the final fc layer (see Figure 5(b)). Hence, we obtain a final NoNN architecture which consists of multiple disjoint students. The output from all students is concatenated and passed through a fc to yield logits [aggregated task result].”)). Bhardwaj does not expressly disclose one or more computer-readable media (but see Abdelfattah ¶ 196 (“Meanwhile, various embodiments of the disclosure may be implemented in software, including instructions stored on machine-readable storage media readable by a machine (e.g., a computer). An apparatus may call instructions from the storage medium, and execute the called instruction, including an electronic device (for example, electronic device 100) according to the disclosed embodiments. When the instructions are executed by a processor, the processor may perform a function corresponding to the instructions directly or using other components under the control of the processor.”)). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bhardwaj to incorporate the teachings of Abdelfattah to store the PyTorch implementation on a computer-readable medium, at least because doing so would enable an apparatus to call instructions from the storage medium to execute the knowledge distillation process taught by Bhardwaj. Regarding claim 2, Bhardwaj, in view of Taylor and Abdelfattah, discloses the invention of claim 1 as discussed above. Bhardwaj teaches computing the FLOPS (floating point operations per second) of a student model so that the resulting student networks fit within a FLOP budget, but does not expressly disclose the one or more characteristics of the learner model comprising a time estimate for the learner model to perform the task using the corresponding compute resource (but see Abdelfattah ¶ 174 (“FIG. 20 is a diagram illustrating an example of a more sophisticated version of the system of FIG. 19 in which the cut-off model 1412 comprises a hardware runtime estimator 1430 and a validator 1432 according to an embodiment. The hardware runtime estimator 1430 is used to predict the hardware performance, e.g. latency, [time estimate] of a proposed model by the processor on a target hardware platform(s) [compute resource]. This is not a trivial task because the relationship between the total number of FLOPS needed for running a proposed model architecture or its parameter size has a non-linear relationship with latency on a specific hardware platform due to variations in on/off chip memory utilization, memory footprint, degree of parallelism, area usage, clock speed or any other relevant task or hardware metric.”), ¶ 177 (“Once the validator 1432 has received the model with its estimated hardware value(s) or measured hardware value(s), the validator 1432 checks if the proposed architecture meets all the hardware metrics. In other words, the validator 1432 may compare the hardware value(s) to the defined thresholds to determine if the hardware constraints are met. If the proposed model does meet the hardware criteria, the model is sent to the evaluation model 1422 for a more detailed evaluation, e.g. to generate a reward function, as described above. Accordingly, it is clear that in this arrangement, the processor 1400 sends all proposed model architectures for the CNN to the hardware runtime estimator 1430. Specifically, as shown in the Figure, the proposed model architectures are sent to the statistical model module 1440 and the discriminator 1442.”)). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bhardwaj to incorporate the teachings of Abdelfattah to determine the latency of the learner models to determine if the hardware constraints are met, at least because doing so would enable more accurately tailored use of IoT compute resources. Regarding claim 3, Bhardwaj, in view of Taylor and Abdelfattah, discloses the invention of claim 1 as discussed above. Bhardwaj further discloses the creation being performed for at least one of the plurality of learner models, creating the learner model as a student model by initializing the student model, and applying distillation from a teacher model to tune the student model (Bhardwaj, p. 5 (“KD, as shown in Figure 2(a), consists of two deep networks: (i) Teacher is the large deep network which we want to compress, and (ii) Student is a significantly smaller neural network which is trained to mimic the output of the teacher network. To mimic the teacher, the simplest way is to directly train a student model on teacher’s logits (see Figure 2(a))– the unnormalized outputs of the teacher model before the softmax– instead of training on true labels from the dataset [1]. The basic idea here is that while training the student model, using logits instead of direct classification probabilities can transfer more information about the teacher to the student. Hinton et al. [6] argued that we must not only train the student model on the correct predictions made by the teacher, but must also quantify how wrong the teacher was about the incorrect classes. Hence, Hinton et al. introduced KD [6] which uses both the hard-label loss (based on true labels from the dataset), as well as the soft-label loss (based on logits) to train a significantly smaller student model.”)). Regarding claim 4, Bhardwaj, in view of Taylor and Abdelfattah, discloses the invention of claim 3 as discussed above. Bhardwaj further discloses the learner model being a first learner model, the creation being performed for at least one of the plurality of learner models, creating the learner model as a second student model by sparcifying the teacher model to create the second student model (Bhardwaj, p. 2 (“To this end, several deep learning model compression techniques exist in the literature. The most common approaches are pruning [5, 23], quantization [8, 12], and Knowledge Distillation (KD) [1, 6] and its variants such as Attention Transfer (AT) [25]. Since deep models can learn a large number of redundant or useless weights/channels, pruning aims to remove such parameters without sacrificing accuracy [5, 23]. Moreover, quantization reduces the number of bits required to represent weights and/or activations in deep networks [8, 12]. Also, KD-based approaches rely on teacher-student training, where a teacher model is a large deep network we want to compress. KDtrains a significantly smaller student model with far fewer layers/width, to mimic the teacher network (see Figure 1(a)). Further, most prior art on model compression refers to pruning and quantization.”)). Regarding claim 5, Bhardwaj, in view of Taylor and Abdelfattah, discloses the invention of claim 1 as discussed above. Bhardwaj further discloses the creation being performed for at least one of the plurality of learner models, creating the learner model as a student model by sparcifying a teacher model to create the student model (Bhardwaj, p. 2 (“To this end, several deep learning model compression techniques exist in the literature. The most common approaches are pruning [5, 23], quantization [8, 12], and Knowledge Distillation (KD) [1, 6] and its variants such as Attention Transfer (AT) [25]. Since deep models can learn a large number of redundant or useless weights/channels, pruning aims to remove such parameters without sacrificing accuracy [5, 23]. Moreover, quantization reduces the number of bits required to represent weights and/or activations in deep networks [8, 12]. Also, KD-based approaches rely on teacher-student training, where a teacher model is a large deep network we want to compress. KDtrains a significantly smaller student model with far fewer layers/width, to mimic the teacher network (see Figure 1(a)). Further, most prior art on model compression refers to pruning and quantization.”)). Regarding claim 6, Bhardwaj, in view of Taylor and Abdelfattah, discloses the invention of claim 1 as discussed above. Bhardwaj further discloses at least one of the plurality of learner models comprising a neural network (Bhardwaj, p. 3 (“To address this question, in this paper, we propose Network of Neural Networks (NoNN), a paradigm to derive low memory-, computation-, as well as communication-cost student architectures from a single (powerful) but much larger teacher model.”)). Regarding claim 7, Bhardwaj, in view of Taylor and Abdelfattah, discloses the invention of claim 6 as discussed above. Bhardwaj further discloses the pre-trained machine learning model also being a neural network, a number of layers of the neural network of the learner model being less than a number of layers of the pre-trained machine learning model (Bhardwaj, p. 3 (“To address this question, in this paper, we propose Network of Neural Networks (NoNN), a paradigm to derive low memory-, computation-, as well as communication-cost student architectures from a single (powerful) but much larger teacher model.”), p. 2 (“To this end, several deep learning model compression techniques exist in the literature. The most common approaches are pruning [5, 23], quantization [8, 12], and Knowledge Distillation (KD) [1, 6] and its variants such as Attention Transfer (AT) [25]. Since deep models can learn a large number of redundant or useless weights/channels, pruning aims to remove such parameters without sacrificing accuracy [5, 23]. Moreover, quantization reduces the number of bits required to represent weights and/or activations in deep networks [8, 12]. Also, KD-based approaches rely on teacher-student training, where a teacher model is a large deep network we want to compress. KDtrains a significantly smaller student model with far fewer layers/width, to mimic the teacher network (see Figure 1(a)). Further, most prior art on model compression refers to pruning and quantization.”)). Regarding claim 9, Bhardwaj, in view of Taylor and Abdelfattah, discloses the invention of claim 1 as discussed above. Bhardwaj further discloses the task being a classification task in which input is classified (Bhardwaj, Fig. 4: PNG media_image3.png 282 673 media_image3.png Greyscale ). Regarding claim 11, Bhardwaj, in view of Taylor and Abdelfattah, discloses the invention of claim 1 as discussed above. Bhardwaj further discloses at least one of the parallel compute resources being a processor core (Bhardwaj, p. 16 (“We use eight Raspberry Pi-3 Model-B and eight Odroid-XU4S boards. Each RPi has an Arm Cortex-A53 quad-core processor with nominal frequency of 1.2GHz and 1GB of RAM.”)). Regarding claim 12, Bhardwaj, in view of Taylor and Abdelfattah, discloses the invention of claim 1 as discussed above. Bhardwaj further discloses at least one of the parallel compute resources being a central processing unit (Bhardwaj, p. 16 (“We use eight Raspberry Pi-3 Model-B and eight Odroid-XU4S boards. Each RPi has an Arm Cortex-A53 quad-core processor with nominal frequency of 1.2GHz and 1GB of RAM.”)). Regarding claim 13, Bhardwaj, in view of Taylor and Abdelfattah, discloses the invention of claim 1 as discussed above. Bhardwaj further discloses at least one of the parallel compute resources being a graphical processing unit (Bhardwaj, p. 12 (“Finally, our entire framework is implemented in Pytorch which is run on NVIDIA GTX 1080-Ti and Titan Xp GPUs.”)). Regarding claim 14, Bhardwaj, in view of Taylor and Abdelfattah, discloses the invention of claim 1 as discussed above. Bhardwaj further discloses at least one of the parallel compute resources being a compute node (Bhardwaj, p. 16 (“We use eight Raspberry Pi-3 Model-B and eight Odroid-XU4S boards. Each RPi has an Arm Cortex-A53 quad-core processor with nominal frequency of 1.2GHz and 1GB of RAM.”) (Each RPi is interpreted as a compute node)). Regarding claim 15, Bhardwaj, in view of Taylor and Abdelfattah, discloses the invention of claim 1 as discussed above. Bhardwaj further discloses at least one of the parallel compute resources being a compute cluster (Bhardwaj, p. 16 (“We use eight Raspberry Pi-3 Model-B and eight Odroid-XU4S boards. Each RPi has an Arm Cortex-A53 quad-core processor with nominal frequency of 1.2GHz and 1GB of RAM.”) (the eight RPi may be interpreted as a cluster)). Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Bhardwaj, Taylor, and Abdelfattah as applied to claim 1 above, and further in view of Lu, Xuetao, and J. Jack Lee. "Distillation Decision Tree." arXiv preprint arXiv:2206.04661 (2022) (“Lu”). Regarding claim 8, Bhardwaj, in view of Taylor and Abdelfattah, discloses the invention of claim 1 as discussed above. Bhardwaj does not expressly disclose at least one of the plurality of learner models comprising a decision tree (but see Lu Abstract (“Black-box machine learning models are criticized as lacking interpretability, although they tend to have good prediction accuracy. Knowledge Distillation (KD) is an emerging tool to interpret the black-box model by distilling its knowledge into a transparent model. With well-known advantages in interpretation, decision tree is a competitive candidate of the transparent model. However, theoretical or empirical understanding for the decision tree generated from KD process is limited. In this paper, we name this kind of decision tree the distillation decision tree (DDT) and lay the theoretical foundations for tree structure stability which determines the validity of DDT’s interpretation. We prove that the structure of DDT can achieve stable (convergence) under some mild assumptions. Meanwhile, we develop algorithms for stabilizing the induction of DDT, propose parallel strategies for improving algorithm’s computational efficiency, and introduce a marginal principal component analysis method for overcoming the curse of dimensionality in sampling. Simulated and real data studies justify our theoretical results, validate the efficacy of algorithms, and demonstrate that DDT can strike a good balance between model’s prediction accuracy and interpretability.”)). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bhardwaj to incorporate the teachings of Lu to apply its knowledge distillation process for a decision tree teacher model, at least because doing so would strike a good balance between the decision tree model’s prediction accuracy and interpretability. Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Bhardwaj, Taylor, and Abdelfattah as applied to claim 1 above, and further in view of Liu, Xiaodong, et al. "Improving multi-task deep neural networks via knowledge distillation for natural language understanding." arXiv preprint arXiv:1904.09482 (2019) (“Liu”). Regarding claim 10, Bhardwaj, in view of Taylor and Abdelfattah, discloses the invention of claim 1 as discussed above. Bhardwaj describes knowledge distillation in the context of image classification by convolutional neural networks but does not expressly disclose the task being a natural language task in which output is generated based on input text (but see Liu Introduction (“Knowledge distillation is a process of distilling or transferring the knowledge from a (set of) large, cumbersome model(s) to a lighter, easier to-deploy single model, without significant loss in performance (Bucilu et al., 2006; Hinton et al., 2015; Balan et al., 2015; Ba et al., 2016; Chen et al., 2015; Tan et al., 2019). In this paper, we explore the use of knowledge distillation to improve a Multi-Task Deep Neural Network (MT-DNN)(Liuet al., 2019) for learning text representations across multiple NLU tasks. Since MT-DNN incorporates a pre-trained BERT model, its ensemble is expensive to serve at runtime.”)). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bhardwaj to incorporate the teachings of Liu to apply the knowledge distillation process taught by Bhardwaj to natural language understanding tasks, at least because doing so would enable large ensemble models that are otherwise prohibitively expensive due to their computational cost, to be deployed online. See Liu Introduction. Conclusion The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Lu, Bingqian, et al. "Automating deep neural network model selection for edge inference." 2019 IEEE First International conference on cognitive machine intelligence (CogMI). IEEE, 2019. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHAHID KHAN whose telephone number is (571)270-0419. The examiner can normally be reached M-F, 9-5 est. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached at (571)272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /SHAHID K KHAN/ Primary Examiner, Art Unit 2146