DETAILED ACTION
This action is written in response to the remarks and amendments filed 4/7/26. This action is made final. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
In view of the Applicant’s arguments, the Examiner withdraws all outstanding rejections under §101.
The Applicants argue that the previous art of record does not anticipate or render obvious the claims as currently amended. The Examiner provides updated prior art rejections below necessitated by the current amendments.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
The following are the references relied upon in the rejections below:
Aytekin (EP 3,767,548 A1, cited by Applicant in IDS dated 10/5/25)
Benmeziane (Benmeziane H, Maghraoui KE, Ouarnoughi H, Niar S, Wistuba M, Wang N. A comprehensive survey on hardware-aware neural architecture search. arXiv preprint arXiv:2101.09336. 2021 Jan 22. Cited by Applicant in IDS dated 5/8/24.)
Chai (US 2021/0241108 A1, cited by Applicant in IDS dated 5/8/24)
Roziere (Roziere, Baptiste, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. "Unsupervised translation of programming languages." Advances in neural information processing systems 33 (2020): 20601-20611.)
Trofimov (Trofimov, Ilya, et al. "Multi-fidelity neural architecture search with knowledge distillation." IEEE Access 11 (2023): 59217-59225. Published 5 Jan 2023.)
Claims 1-3, 5-6, 8-10 and 17-21 are rejected under 35 U.S.C. 103 as being unpatentable over Chai and Trofimov.
Regarding claim 1, Chai discloses a method comprising:
obtaining a plurality of context-specific machine learning models, each context-specific machine learning model being derived from a base machine learning model adapted to a plurality of contexts and each context-specific machine learning model being adapted to a different context of the plurality of contexts;
[0137] “an original model 1902, which was developed for the deep cloud, cannot run within constraints of the target runtime parameters 1904 for edge devices.”
[0138]: "In order to run on the smartphone, the model can go through a build process 1906 and a run process 1910 to condition it for dynamic runtime execution. The build process 1906 includes workflows for distill, compress, and compile operations, to optimize the ONN model. The result of the build process 1906 is context-specific models 1908 and associated runtime engines (not shown) that are able to run on a smartphone within constraints of the target runtime parameters.” (Emphasis added.)
[0141]: “the original model 1902 may classify all objects in a retail store using processors in the deep cloud. The generated context-specific model 1908 may only classify objects that are located on a single aisle (e.g. face/hair products, or cleaning supplies).”
detecting a particular context of a particular device;
[0143] “locating beacons (e.g. GPS or Wi-Fi maps) may be used to provide context”.
selecting a particular context-specific machine learning model from the plurality of context-specific machine learning models based at least on the particular context of the particular device …
[0143] “provide context to switch among a number of different context-specific models 1908”.
providing the particular context-specific machine learning model to the particular device.
[0142] “While in the face/hair aisle in the retail store, only the context-specific model 1908 for face/hair is used, and while in the cleaning supplies aisle in the retail store, only the context-specific model 1908 for cleaning supplies is used."
[0158] “The context-specific models may be detecting key audio signatures or trigger words. Once detected, a remote service is enabled to process subsequent collection of sensor data. Then, a new context-specific model may be moved and loaded into the device to setup a new trigger event." (Emphasis added.)
Trofimov discloses the following further limitation which Chai does not disclose:
wherein the particular context-specific machine learning model has a particular neural architecture that has been determined via a neural architecture search based at least on distillation loss of the particular context-specific machine learning model with respect to the base machine learning model;
P. 59218, second col., “we propose a new approach to the low-fidelity evaluation of neural architectures—training for a few epochs with a knowledge distillation loss (Section IV);”
P. 59219, second col., “The KD [knowledge distillation] loss is a linear combination of the logistic loss and cross-entropy between predictions of the teacher and the student”. See equation 2, reproduced below.
PNG
media_image1.png
118
618
media_image1.png
Greyscale
See also p. 59219, second col., “In this paper, we will apply a particular variant of Knowledge Distillation: the NST loss[15].The NST loss matches the distributions of neuron selectivity patterns (feature maps) between teacher and student networks.”
At the time of filing, it would have been obvious to a person of ordinary skill to apply distillation loss as a criterion for a neural architecture search (as taught by Trofimov) in combination with the Chai system. The particular benefits of this criterion are set forth by Tofimov:
P. 59222, first col., “In this work, we have proposed the new MF-KD method tailored to neural architecture search. By doing experiments, we have proved that the MF-KD method is efficient. It leads to a better architecture selection than several state-of-the art baselines given the same computational budget.”
Regarding claim 2, Chai discloses the further limitation comprising:
detecting that the particular device has switched to another context; and
[0142] “These context-specific models 1908 are available to run on the smartphone edge device. While in the face/hair aisle in the retail store, only the context-specific model 1908 for face/hair is used, and while in the cleaning supplies aisle in the retail store, only the context-specific model 1908 for cleaning supplies is used. The constraints of the target runtime parameters 1904 are met because the context-specific models 1908 are much smaller sized and run faster than the larger original model 1902, which is more comprehensive.”
responsive to detecting that the particular device has switched to the another context:
Id.
selecting another context-specific machine learning model from the plurality of context-specific machine learning models based at least on the another context; and
Id.
providing the another context-specific machine learning model to the particular device.
Id.
Regarding claim 3, Chai discloses the further limitation comprising:
determining the particular context and the another context using an automated context prediction algorithm based at least on context data received from the particular device.
[0142] “These context-specific models 1908 are available to run on the smartphone edge device. While in the face/hair aisle in the retail store, only the context-specific model 1908 for face/hair is used, and while in the cleaning supplies aisle in the retail store, only the context-specific model 1908 for cleaning supplies is used. The constraints of the target runtime parameters 1904 are met because the context-specific models 1908 are much smaller sized and run faster than the larger original model 1902, which is more comprehensive.”
Regarding claim 5, Chai discloses the further limitation wherein the base machine learning model is adapted to recognize a plurality of object types in images and the particular context-specific machine learning model is adapted to recognize a subset of the plurality of object types.
[0095] “During runtime operation of the neural network, our framework can perform a profiling operation to keep track of all pathways the neural network activates while making an inference, such as classifying a car. This information can be used to gain insight into how the neural network makes a specific inference. For example, say we have a neural network that recognizes objects in an image, such as a car, a dog or a bicycle.”
See also [0142], describing product-specific context detection based on the user’s location with respect to a shopping isle.
Regarding claim 6, Chai discloses the further limitation comprising:
compressing the particular context-specific machine learning model to obtain a compressed version and sending the compressed version to the particular device over a network.
[0049] DNN quantization and pruning.
Regarding claim 8, Chai discloses a method comprising:
obtaining a base machine learning model adapted for a plurality of contexts;
[0137] “an original model 1902, which was developed for the deep cloud, cannot run within constraints of the target runtime parameters 1904 for edge devices.”
[0138]: "In order to run on the smartphone, the model can go through a build process 1906 and a run process 1910 to condition it for dynamic runtime execution. The build process 1906 includes workflows for distill, compress, and compile operations, to optimize the ONN model. The result of the build process 1906 is context-specific models 1908 and associated runtime engines (not shown) that are able to run on a smartphone within constraints of the target runtime parameters.” (Emphasis added.)
[0141]: “the original model 1902 may classify all objects in a retail store using processors in the deep cloud. The generated context-specific model 1908 may only classify objects that are located on a single aisle (e.g. face/hair products, or cleaning supplies).”
deriving, from the base machine learning model, multiple context-specific machine learning models adapted for different contexts of the plurality of contexts; ….
Id.
outputting the multiple context-specific machine learning models for use in the different contexts.
[0142] “While in the face/hair aisle in the retail store, only the context-specific model 1908 for face/hair is used, and while in the cleaning supplies aisle in the retail store, only the context-specific model 1908 for cleaning supplies is used."
[0158] “The context-specific models may be detecting key audio signatures or trigger words. Once detected, a remote service is enabled to process subsequent collection of sensor data. Then, a new context-specific model may be moved and loaded into the device to setup a new trigger event." (Emphasis added.)
Trofimov discloses the following further limitation which Chai does not disclose:
wherein the deriving includes performing a neural architecture search for a particular neural architecture used by a particular context-specific machine learning model and the neural architecture search involves selecting the particular neural architecture based at least on an evaluation of distillation loss of the particular context-specific machine learning model with respect to the base machine learning model;
P. 59218, second col., “we propose a new approach to the low-fidelity evaluation of neural architectures—training for a few epochs with a knowledge distillation loss (Section IV);”
P. 59219, second col., “The KD [knowledge distillation] loss is a linear combination of the logistic loss and cross-entropy between predictions of the teacher and the student”. See equation 2, reproduced below.
PNG
media_image1.png
118
618
media_image1.png
Greyscale
See also p. 59219, second col., “In this paper, we will apply a particular variant of Knowledge Distillation: the NST loss[15].The NST loss matches the distributions of neuron selectivity patterns (feature maps) between teacher and student networks.”
The obviousness analysis of claim 1 applies equally here.
Regarding claim 9, Chai discloses the further limitation comprising:
employing the base machine learning model as a teacher and the multiple context-specific machine learning models as students when training the multiple context-specific machine learning models.
[0139] “In the distill workflow, the original model, which was developed for the cloud, serves as a teacher model, while the context-specific models 1908 are student models that learn from the teacher model.” (Emphasis added.)
Regarding claim 10, Trofimov discloses the further limitation comprising:
training the particular context-specific machine learning model for a particular context by adjusting parameters of the particular context-specific model, the training being performed using a loss function that is based at least on the distillation loss.
P. 59218, second col., “we propose a new approach to the low-fidelity evaluation of neural architectures—training for a few epochs with a knowledge distillation loss (Section IV);”
P. 59219, second col., “The KD [knowledge distillation] loss is a linear combination of the logistic loss and cross-entropy between predictions of the teacher and the student”. See equation 2, reproduced below.
PNG
media_image1.png
118
618
media_image1.png
Greyscale
See also p. 59219, second col., “In this paper, we will apply a particular variant of Knowledge Distillation: the NST loss[15].The NST loss matches the distributions of neuron selectivity patterns (feature maps) between teacher and student networks.”
Regarding claim 17, Chai discloses a computing device comprising:
a hardware processing unit; and
[0004] “computing cores”.
a storage resource storing computer-readable instructions which, when executed by the hardware processing unit, cause the hardware processing unit to:
[0004] “memory”
receive a particular context-specific machine learning model adapted for a particular context, the particular context-specific machine learning model having been derived from a base machine learning model adapted for a plurality of contexts; …
[0158] “The context-specific models may be detecting key audio signatures or trigger words. Once detected, a remote service is enabled to process subsequent collection of sensor data. Then, a new context-specific model may be moved and loaded into the device to setup a new trigger event." (Emphasis added.)
[0137] “an original model 1902, which was developed for the deep cloud, cannot run within constraints of the target runtime parameters 1904 for edge devices.”
[0138]: "In order to run on the smartphone, the model can go through a build process 1906 and a run process 1910 to condition it for dynamic runtime execution. The build process 1906 includes workflows for distill, compress, and compile operations, to optimize the ONN model. The result of the build process 1906 is context-specific models 1908 and associated runtime engines (not shown) that are able to run on a smartphone within constraints of the target runtime parameters.”
[0141]: “the original model 1902 may classify all objects in a retail store using processors in the deep cloud. The generated context-specific model 1908 may only classify objects that are located on a single aisle (e.g. face/hair products, or cleaning supplies).”
execute the particular context-specific machine learning model on the computing device when the computing device is in the particular context.
[0142] “While in the face/hair aisle in the retail store, only the context-specific model 1908 for face/hair is used, and while in the cleaning supplies aisle in the retail store, only the context-specific model 1908 for cleaning supplies is used."
[0143] “provide context to switch among a number of different context-specific models 1908”.
Trofimov discloses the following further limitation which Chai does not disclose:
wherein the particular context- specific machine learning model has a particular neural architecture that has been determined via a neural architecture search based at least on distillation loss of the particular context-specific machine learning model with respect to the base machine learning model;
P. 59218, second col., “we propose a new approach to the low-fidelity evaluation of neural architectures—training for a few epochs with a knowledge distillation loss (Section IV);”
P. 59219, second col., “The KD [knowledge distillation] loss is a linear combination of the logistic loss and cross-entropy between predictions of the teacher and the student”. See equation 2, reproduced below.
PNG
media_image1.png
118
618
media_image1.png
Greyscale
See also p. 59219, second col., “In this paper, we will apply a particular variant of Knowledge Distillation: the NST loss[15].The NST loss matches the distributions of neuron selectivity patterns (feature maps) between teacher and student networks.”
The obviousness analysis of claim 1 applies equally here.
Regarding claim 18, Chai discloses the further limitation wherein the computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to:
receive another context-specific machine learning model derived from the base machine learning model and adapted for another context; and
[0142] “These context-specific models 1908 are available to run on the smartphone edge device. While in the face/hair aisle in the retail store, only the context-specific model 1908 for face/hair is used, and while in the cleaning supplies aisle in the retail store, only the context-specific model 1908 for cleaning supplies is used. The constraints of the target runtime parameters 1904 are met because the context-specific models 1908 are much smaller sized and run faster than the larger original model 1902, which is more comprehensive.”
execute the another context-specific machine learning model on the computing device when the computing device is in the another context.
Id.
Regarding claim 19, Chai discloses the further limitation the hardware processing unit comprising a central processing unit, the computing device further comprising an inference processing unit and an inference processing unit memory, wherein the computer-readable instructions, when executed by the central processing unit, cause the central processing unit to:
retrieve compressed slices of the particular context-specific machine learning model;
[0079]-[0082] describing DNN compression.
decompress the slices; and
[0123] “decompression module”.
load the decompressed slices into the inference processing unit memory for execution by the inference processing unit.
See generally [0122]-[0123] describing deployment and decompression procedure.
Regarding claim 20, Chai discloses the further limitation wherein the compressed slices include parameters of individual layers of the particular context-specific machine learning model.
[0094] “FIG. 7 illustrates an example of layer wise delivery of neural network(s) from a server 110 to a client 120, with an optional performance evaluation step performed at a third-party device 130, according to an embodiment. It is noted that even though the embodiments of FIG. 7 are described using layers as an example of a subset, the embodiments may be applied to any other subsets of neural networks. For example, convolutional layers of the neural network could be delivered filter-by-filter. Transmitting or updating at least a subset may refer to transmitting or updating a subset of the neural network or the whole neural network, for example all layers of the neural network.”
[0096]-[0097] “At 704, server 110 may transmit a first layer of the first compressed neural network to client 120. …. At 705, the client 120 may receive the first layer of the compressed neural network from the server 110.”
The obviousness analysis of claim 7 applies equally here.
Regarding claim 21, Trofimov discloses the further limitation wherein the deriving comprises comparing the particular neural architecture to other neural architectures based at least on the distillation loss of the particular neural architecture and distillation loss of the other neural architectures with respect to the base machine learning model.
Trofimov is directed to the problem of neural architecture search, ie comparing heuristics for different models in order to choose the best one. See abstract and passim.
P. 59218, second col., “we propose a new approach to the low-fidelity evaluation of neural architectures—training for a few epochs with a knowledge distillation loss (Section IV);”
P. 59219, second col., “The KD [knowledge distillation] loss is a linear combination of the logistic loss and cross-entropy between predictions of the teacher and the student”. See equation 2, reproduced below.
PNG
media_image1.png
118
618
media_image1.png
Greyscale
See also p. 59219, second col., “In this paper, we will apply a particular variant of Knowledge Distillation: the NST loss[15].The NST loss matches the distributions of neuron selectivity patterns (feature maps) between teacher and student networks.”
Claims 4 is rejected under 35 U.S.C. 103 as being unpatentable over Chai, Trofimov and Roziere.
Regarding claim 4, Roziere discloses the following further limitation which Chai/Trofimov do not disclose wherein the base machine learning model is adapted to generate code in a plurality of programming languages, the particular context relates to a particular programming language, and the particular context-specific machine learning model is adapted to generate code in the particular programming language.
P. 1, introduction, “A transcompiler, transpiler, or source-to-source compiler, is a translator which converts between programming languages that operate at a similar level of abstraction. …. Initially, transcompilers were developed to port source code between different platforms (e.g. convert source code designed for the Intel 8080 processor to make it compatible with the Intel 8086).”
At the time of filing, it would have been obvious to a person of ordinary skill to apply the transcomplier technique disclosed by Roziere to the Chai/Trofimov system so that models could be adapted for heterogeneous user devices.
Claims 7 is rejected under 35 U.S.C. 103 as being unpatentable over Chai, Trofimov and Aytekin.
Regarding claim 7, Aytekin discloses the following further limitation which Chai/Trofimov do not disclose: the compressed version having respective slices corresponding to individual layers of the particular context-specific machine learning model.
[0094] “FIG. 7 illustrates an example of layer wise delivery of neural network(s) from a server 110 to a client 120, with an optional performance evaluation step performed at a third-party device 130, according to an embodiment. It is noted that even though the embodiments of FIG. 7 are described using layers as an example of a subset, the embodiments may be applied to any other subsets of neural networks. For example, convolutional layers of the neural network could be delivered filter-by-filter. Transmitting or updating at least a subset may refer to transmitting or updating a subset of the neural network or the whole neural network, for example all layers of the neural network.”
[0096]-[0097] “At 704, server 110 may transmit a first layer of the first compressed neural network to client 120. …. At 705, the client 120 may receive the first layer of the compressed neural network from the server 110.”
At the time of filing, it would have been obvious to a person of ordinary skill to apply the technique disclosed by Aytekin to the Chai system in order to provide a device-appropriate workload to a particular edge device.
Claims 11-16 are rejected under 35 U.S.C. 103 as being unpatentable over Chai, Trofimov and Benmeziane.
Regarding claim 11, Benmeziane discloses the further limitation which Chai does not disclose comprising:
wherein the particular neural architecture is shared by each of the multiple context-specific machine learning models.
P. 1, second col. “In general, the neural network architecture can be formalized as a Directed Acyclic Graph (DAG) where each node corresponds to an operator applied to the set of its parent nodes [13]. Example operators are convolution, pooling, activation, and self-attention. Linking these operators together gives rise to different architectures. A key aspect of designing a well-performing deep neural network is deciding the type and number of nodes and how to compose and link them.”
At the time of filing, it would have been obvious to a person of ordinary skill to apply the neural architecture search techniques of Benmeziane to the Chai/Trofimov system because—as noted by Benmeziane— “[a] key aspect of designing a well-performing deep neural network is deciding the type and number of nodes and how to compose and link them.” (P. 1.)
Regarding claim 12, Benmeziane discloses the further limitation wherein the neural architecture search starting with a seed model and iteratively selecting new parent models from a Pareto frontier according to two or more criteria.
P. 11, second col. “2) solve the multi-objective optimization problem using dedicated heuristics or meta-heuristics such as genetic algorithms or tabu search [137]. In general, this second approach provides not one optimal solution but a set of solutions that form the optimal Pareto front of the multi-objective optimization problem.” (Emphasis added.)
Regarding claim 13, Benmeziane discloses the further limitation the search being constrained based on a hardware constraint for an inference processing unit.
P. 4, “constrained optimization”.
P. 5, “Edge devices pose several challenges in this context, as they are constrained with limited energy and computational power”
Regarding claim 14, Benmeziane discloses the further limitation: the Pareto frontier including being defined by a cost axis and a loss axis…
P. 11, second col. “2) solve the multi-objective optimization problem using dedicated heuristics or meta-heuristics such as genetic algorithms or tabu search [137]. In general, this second approach provides not one optimal solution but a set of solutions that form the optimal Pareto front of the multi-objective optimization problem.” (Emphasis added.)
See also p. 11, second col., discussing loss function.
P. 12, first col, “Thus, it is important to cast the problem as a multi-objective optimization problem, where a series of models is found along the Pareto front of multiple objectives such as accuracy, computational cost or inference time, and number of parameters.”
Trofimov discloses the use of distillation loss as a loss heuristic, see mapping supra.
Regarding claim 15, Trofimov discloses the further limitation comprising the deriving comprising:
wherein the loss axis is based on a weighted combination of the distillation loss and standard loss.
P. 59219, second col., “The KD [knowledge distillation] loss is a linear combination of the logistic loss and cross-entropy between predictions of the teacher and the student”. See equation 2, reproduced below.
PNG
media_image1.png
118
618
media_image1.png
Greyscale
Regarding claim 16, Chai discloses the further limitation: the cost axis being based on one or more of latency, power consumption, or computing resource consumption.
P. 12, first col., “Thus, it is important to cast the problem as a multi-objective optimization problem, where a series of models is found along the Pareto front of multiple objectives such as accuracy, computational cost or inference time, and number of parameters.”
Additional Relevant Prior Art
The following references were identified by the Examiner as being relevant to the disclosed invention, but are not relied upon in any particular prior art rejection:
Zhang discloses an autocomplete system with a context-specific model component. (US 2021/0319178 A1)
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Vincent Gonzales whose telephone number is (571) 270-3837. The examiner can normally be reached on Monday-Friday 7 a.m. to 4 p.m. MT. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang, can be reached at (571) 270-7092.
Information regarding the status of an application may be obtained from the USPTO Patent Center.
/Vincent Gonzales/Primary Examiner, Art Unit 2124
Prosecution Insights