DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu et al. (“Latency-Aware Differentiable Neural Architecture Search”, arXiv 2020) in view of Swan et al. (US 20200311617 A1).
Regarding claim 1.
Xu teaches a method for evaluating latency of a machine learning model during the model training process, the method comprising: performing, during the model training process, model training for the machine learning model using training data (see page 2, figure 1, proxy dataset includes training data inputted into LADNAS for training, also see page 4, “DARTS defines an over parameterized network h(x; ω,α) where ω and α denote the network and architectural parameters. With a bi-level optimization process, ω and α are trained in a proxy dataset and α is used to determine the final architecture.”);
in response to performing the model training, generating see page 7, section 3.3, “we can easily transplant the learning-based approach to other device without much expertise which eases the deployment of NAS on a wide range of hardware. We will show an example in Section 4.3.”);
providing, see page 2, figure 1, proxy dataset includes training data inputted into LADNAS for training, also see page 4, “DARTS defines an over parameterized network h(x; ω,α) where ω and α denote the network and architectural parameters. With a bi-level optimization process, ω and α are trained in a proxy dataset and α is used to determine the final architecture.”);
obtaining, based on processing of the portion of the training data by the temporarily deployed machine learning model, response data indicating output of the temporarily deployed machine learning model (see page 4, “
PNG
media_image1.png
384
806
media_image1.png
Greyscale
”);
determining a latency value indicating a processing time for the temporarily deployed machine learning model to generate the response data (see page 4, “Despite the satisfying performance of the searched architecture, we are not sure if the architecture is also optimized in terms of efficiency, e.g., latency. In particular, DARTS involves many inter-layer connections (e.g., each cell receives input from two previous cells) which may bring memory access issues and slow down the architecture. More importantly, such a complex architecture brings uncertainty in latency estimation, because the cost of memory access is often difficult to measure, unlike that of a specific operator.”);
and optimizing the machine learning model using the latency value (see page 5, “The key of LA-DARTS is to design a differentiable loss function that can predict the latency of the architecture parameter, a, so that it can be integrated into the over-parameterized network optimization process. We denote this function as LAT(a), which is the expectation of latency when an architecture is sampled according to the weights of a”).
Xu do not specifically teach generating a temporary deployment of the machine learning model and providing, as input to the temporarily deployed machine learning model, a portion of the training data including one or more elements.
Swan teaches generating a temporary deployment of the machine learning model and providing, as input to the temporarily deployed machine learning model, a portion of the training data including one or more elements (see ¶ 55, “Upon receiving the deployment request, the model hosting system 140 initializes ones or more ML scoring containers 150 in one or more hosted virtual machine instance 142. In embodiments in which the deployment request includes an identification of one or more container images, the model hosting system 140 forms the ML scoring container(s) 150 from the identified container image(s). For example, a container image identified in a deployment request can be the same container image used to form an ML training container 130 used to train the machine learning model corresponding to the deployment request.”).
Both Xu and Swan pertain to the problem of Neural Architecture Search, thus being analogous. It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine Xu and Swan to teach the above limitations. The motivation for doing so would be “packaging and deploying algorithms using containers for flexible machine learning. In some embodiments, users can create or utilize relatively simple containers adhering to a specification of a provider network, where the containers include code for how a machine learning model is to be trained and/or executed. The provider network can automatically train a model and/or host a model using the containers. The containers can use a wide variety of algorithms and use a variety of types of languages, libraries, data types, etc. Accordingly, users can simply perform machine learning training and hosting with extremely minimal knowledge of how the overall training and/or hosting is actually performed.” (see Swan abstract).
Regarding claim 2.
Xu and Swan teaches the method of claim 1,
Xu further teaches wherein optimizing the machine learning model using the latency value comprises: adjusting one or more of an amount of features, quantization of feature values, or a number of processing layers (see page 9, section 4.1, ”the training set is partitioned into two parts, with the first part used for optimizing network parameters, e.g., convolutional weights, and the second part used for optimizing architectural parameters.”).
Swan also teaches adjusting one or more of an amount of features, quantization of feature values, or a number of processing layers (¶ 45, “the user can then use the model metrics to determine whether to adjust the training process and/or to stop the training process… the user, via the user device 102, can transmit a request to the model training system 120 to modify the machine learning model being trained (e.g., transmit a modification request). The request can include a new or modified container image, a new or modified algorithm, new or modified hyperparameter(s), and/or new or modified information describing the computing machine on which to train a machine learning model. The model training system 120 can modify the machine learning model accordingly.”)
The motivation utilized in the combination of claim 1, super, applies equally as well to claim 2.
Regarding claim 3.
Xu and Swan teaches the method of claim 1,
Swan further teaches wherein providing, as input to the temporarily deployed machine learning model, the portion of the training data comprises: generating the training data as a set of serialized machine readable input in a format supported by the temporarily deployed machine learning model (see ¶ 42, “the algorithm included in the container image can be in a format that allows for the parallelization of the training process.”, also see ¶ 23, “machine learning models can be trained and executed irrespective of the type of machine learning model, the programming language in which the machine learning model is defined, the data input format of the machine learning model, and/or the data output format of the machine learning model.”).
The motivation utilized in the combination of claim 1, super, applies equally as well to claim 3.
Regarding claim 4.
Xu and Swan teaches the method of claim 1,
Xu further teaches comprising: performing evaluation of the machine learning model subsequent to optimizing the machine learning model using the latency value (see page 4, section 3.1, “The goal of DARTS is to search for the robust cell architectures to construct the evaluation network. Specifically, a cell is represented by a directed acyclic graph (DAG) of N nodes, {x0,x1,...,xN−1}, where each node represents a set of feature maps.”, also see page 7, “we randomly sample 100K architectures from the DARTS space, and evaluate the latency of each architecture with randomized network weights.”, also see page 8-9, section 4, experiment and evaluation, “Firstly, we evaluate our LADNAS on CIFAR10 [15]. The CIFAR10 dataset con sists of 60k colored natural images with 32×32 resolution of 10 categories, which is split into 50K training and 10K testing images.”).
Swan also teaches (¶ 45, “The ML model evaluator 128 periodically generates model metrics during the training process and stores the model metrics in the training metrics data store 165…The user can then use the model metrics to determine whether to adjust the training process and/or to stop the training process. For example, the model metrics can indicate that the machine learning model is performing poorly (e.g., has an error rate above a threshold value, has a statistical distribution that is not an expected or desired distribution (e.g., not a binomial distribution, a Poisson distribution, a geometric distribution, a normal distribution, Gaussian distribution, etc.), has an execution latency above a threshold value, has a confidence level below a threshold value)) and/or is performing progressively worse (e.g., the quality metric continues to worsen over time).”)
The motivation utilized in the combination of claim 1, super, applies equally as well to claim 4.
Regarding claim 5.
Xu and Swan teaches the method of claim 4,
Swan further teaches wherein performing the evaluation of the machine learning model comprises: performing (i) accuracy evaluation on the machine learning model, including generating one or more model performance metrics, and (ii) latency evaluation on the machine learning model (see ¶ 45, “While the machine learning model is being trained, a user, via the user device 102, can access and retrieve the model metrics from the training metrics data store 165. The user can then use the model metrics to determine whether to adjust the training process and/or to stop the training process. For example, the model metrics can indicate that the machine learning model is performing poorly (e.g., has an error rate above a threshold value, has a statistical distribution that is not an expected or desired distribution (e.g., not a binomial distribution, a Poisson distribution, a geometric distribution, a normal distribution, Gaussian distribution, etc.), has an execution latency above a threshold value, has a confidence level below a threshold value)) and/or is performing progressively worse (e.g., the quality metric continues to worsen over time). In response, in some embodiments, the user, via the user device 102, can transmit a request to the model training system 120 to modify the machine learning model being trained (e.g., transmit a modification request).”).
The motivation utilized in the combination of claim 1, super, applies equally as well to claim 5.
Regarding claim 6.
Xu and Swan teaches the method of claim 5,
Swan further teaches wherein performing latency evaluation on the machine learning model comprises: performing latency evaluation on the machine learning model within a processing framework that includes an interface for website input or output and data retrieval from one or more data sources communicably connected to one or more computers operating a website (see ¶ 48, “an input/output device interface, all of which can communicate with one another by way of a communication bus. The network interface can provide connectivity to one or more networks or computing systems. The processing unit can thus receive information and instructions from other computing systems or services (e.g., user devices 102, the model hosting system 140, etc.). The processing unit can also communicate to and from a memory of a virtual machine instance 122 and further provide output information for an optional display via the input/output device interface. The input/output device interface can also accept input from an optional input device. The memory can contain computer program instructions (grouped as modules in some embodiments) that the processing unit executes in order to implement one or more aspects of the present disclosure.”, also see ¶ 131).
The motivation utilized in the combination of claim 1, super, applies equally as well to claim 6.
Regarding claim 7.
Xu and Swan teaches the method of claim 1,
Swan further teaches wherein determining the latency value indicating the processing time for the temporarily deployed machine learning model to generate the response data comprises: determining one or more values each indicating a processing time required by the temporarily deployed machine learning model to process an element of the one or more elements of the training data; generating a distribution from the one or more values each indicating a processing time; and determining the latency value as a percentile of the distribution (see ¶ 44, “the model metrics can include quality metrics, such as an error rate of the machine learning model being trained, a statistical distribution of the machine learning model being trained, a latency of the machine learning model being trained, a confidence level of the machine learning model being trained (e.g., a level of confidence that the accuracy of the machine learning model being trained is known, etc. The ML model evaluator 128 can obtain the model data for a machine learning model being trained and evaluation data from the training data store 160.”, also see ¶ 45, “The ML model evaluator 128 periodically generates model metrics during the training process and stores the model metrics in the training metrics data store 165 in some embodiments. While the machine learning model is being trained, a user, via the user device 102, can access and retrieve the model metrics from the training metrics data store 165. The user can then use the model metrics to determine whether to adjust the training process and/or to stop the training process. For example, the model metrics can indicate that the machine learning model is performing poorly (e.g., has an error rate above a threshold value, has a statistical distribution that is not an expected or desired distribution (e.g., not a binomial distribution”).
The motivation utilized in the combination of claim 1, super, applies equally as well to claim 7.
Regarding claim 8.
Xu and Swan teaches the method of claim 7,
Swan further teaches wherein the percentile includes the 99th percentile of the distribution (see ¶ 44, “the model metrics can include quality metrics, such as an error rate of the machine learning model being trained, a statistical distribution of the machine learning model being trained, a latency of the machine learning model being trained, a confidence level of the machine learning model being trained (e.g., a level of confidence that the accuracy of the machine learning model being trained is known, etc. The ML model evaluator 128 can obtain the model data for a machine learning model being trained and evaluation data from the training data store 160.”, also see ¶ 45, “The ML model evaluator 128 periodically generates model metrics during the training process and stores the model metrics in the training metrics data store 165 in some embodiments. While the machine learning model is being trained, a user, via the user device 102, can access and retrieve the model metrics from the training metrics data store 165. The user can then use the model metrics to determine whether to adjust the training process and/or to stop the training process. For example, the model metrics can indicate that the machine learning model is performing poorly (e.g., has an error rate above a threshold value, has a statistical distribution that is not an expected or desired distribution (e.g., not a binomial distribution”, i.e. percentile includes the 99th percentile is obvious metric selection since distribution is taught).
The motivation utilized in the combination of claim 1, super, applies equally as well to claim 8.
Regarding claim 9.
Xu and Swan teaches the method of claim 7,
Swan further teaches wherein optimizing the machine learning model using the latency value comprises: comparing the latency value as the percentile of the distribution to a threshold latency; determining that the latency value satisfies the threshold latency; and in response to determining that the latency value satisfies the threshold latency, optimizing the machine learning model using the latency value (see ¶ 45, “The ML model evaluator 128 periodically generates model metrics during the training process and stores the model metrics in the training metrics data store 165…The user can then use the model metrics to determine whether to adjust the training process and/or to stop the training process. For example, the model metrics can indicate that the machine learning model is performing poorly (e.g., has an error rate above a threshold value, has a statistical distribution that is not an expected or desired distribution (e.g., not a binomial distribution, a Poisson distribution, a geometric distribution, a normal distribution, Gaussian distribution, etc.), has an execution latency above a threshold value, has a confidence level below a threshold value)) and/or is performing progressively worse (e.g., the quality metric continues to worsen over time).”).
The motivation utilized in the combination of claim 1, super, applies equally as well to claim 9.
Regarding claim 10.
Xu and Swan teaches the method of claim 9,
Swan further teaches wherein the threshold latency is adjustable by a user (see ¶ 45, “The ML model evaluator 128 periodically generates model metrics during the training process and stores the model metrics in the training metrics data store 165…The user can then use the model metrics to determine whether to adjust the training process and/or to stop the training process. For example, the model metrics can indicate that the machine learning model is performing poorly (e.g., has an error rate above a threshold value, has a statistical distribution that is not an expected or desired distribution (e.g., not a binomial distribution, a Poisson distribution, a geometric distribution, a normal distribution, Gaussian distribution, etc.), has an execution latency above a threshold value, has a confidence level below a threshold value)) and/or is performing progressively worse (e.g., the quality metric continues to worsen over time).”).
The motivation utilized in the combination of claim 1, super, applies equally as well to claim 10.
Regarding claim 11.
Xu and Swan teaches the method of claim 10,
Swan further teaches wherein the threshold latency is less than or equal to 100 milliseconds (see ¶ 45, “The ML model evaluator 128 periodically generates model metrics during the training process and stores the model metrics in the training metrics data store 165…The user can then use the model metrics to determine whether to adjust the training process and/or to stop the training process. For example, the model metrics can indicate that the machine learning model is performing poorly (e.g., has an error rate above a threshold value, has a statistical distribution that is not an expected or desired distribution (e.g., not a binomial distribution, a Poisson distribution, a geometric distribution, a normal distribution, Gaussian distribution, etc.), has an execution latency above a threshold value, has a confidence level below a threshold value)) and/or is performing progressively worse (e.g., the quality metric continues to worsen over time).”, i.e. threshold could be any numerical value therefore is obvious to include threshold latency is less than or equal to 100 milliseconds).
The motivation utilized in the combination of claim 1, super, applies equally as well to claim 11.
Regarding claim 12.
Xu and Swan teaches the method of claim 1,
Swan further teaches wherein optimizing the machine learning model using the latency value comprises: providing a user interface that (i) visualizes the latency value on a display of a user device (see ¶ 48, “an input/output device interface, all of which can communicate with one another by way of a communication bus. The network interface can provide connectivity to one or more networks or computing systems. The processing unit can thus receive information and instructions from other computing systems or services (e.g., user devices 102, the model hosting system 140, etc.). The processing unit can also communicate to and from a memory of a virtual machine instance 122 and further provide output information for an optional display via the input/output device interface. The input/output device interface can also accept input from an optional input device. The memory can contain computer program instructions (grouped as modules in some embodiments) that the processing unit executes in order to implement one or more aspects of the present disclosure.”, also see ¶ 131) and (ii) accepts input from a user to adjust one or more features of the machine learning model (see ¶ 45, “The ML model evaluator 128 periodically generates model metrics during the training process and stores the model metrics in the training metrics data store 165 in some embodiments. While the machine learning model is being trained, a user, via the user device 102, can access and retrieve the model metrics from the training metrics data store 165. The user can then use the model metrics to determine whether to adjust the training process and/or to stop the training process. For example, the model metrics can indicate that the machine learning model is performing poorly (e.g., has an error rate above a threshold value, has a statistical distribution that is not an expected or desired distribution (e.g., not a binomial distribution”).
The motivation utilized in the combination of claim 1, super, applies equally as well to claim 12.
Regarding claim 13.
Xu and Swan teaches the method of claim 1,
Swan further teaches wherein the machine learning model is configured to provide a ranked list of search output based on a user query (see ¶ 131, “It should be understood that there can be many other aspects that may need to be stored in the data store, such as page image information and access rights information, which can be stored in any of the above listed mechanisms as appropriate or in additional mechanisms in the data store 1210. The data store 1210 is operable, through logic associated therewith, to receive instructions from the application server 1208 and obtain, update, or otherwise process data in response thereto. In one example, a user might submit a search request for a certain type of item. In this case, the data store 1210 might access the user information 1216 to verify the identity of the user and can access a production data 1212 to obtain information about items of that type. The information can then be returned to the user, such as in a listing of results on a web page that the user is able to view via a browser on the user device 1202. Information for a particular item of interest can be viewed in a dedicated page or window of the browser.”).
The motivation utilized in the combination of claim 1, super, applies equally as well to claim 13.
Claims 14-21 recites a non-transitory computer-readable medium to perform the method recited in claims 1-8. Therefore the rejection of claims 1-8 above applies equally here. Swan also teaches the addition elements of claim 14 not recited in claim 1 comprising non-transitory computer-readable medium storing one or more instructions executable by a computer system (see ¶ 115, “The memory 810 generally includes RAM, ROM, or other persistent or non-transitory memory.”).
Claim 22 recites a system to perform the method recited in claim 1. Therefore the rejection of claims 1 above applies equally here. Swan also teaches the addition elements of claim 22 not recited in claim 1 system, comprising: one or more processors; and machine-readable media interoperably coupled with the one or more processors and storing one or more instructions that, when executed by the one or more processors (see ¶ 115, “The memory 810 generally includes RAM, ROM, or other persistent or non-transitory memory. The memory 810 can store an operating system 814 that provides computer program instructions for use by the processing unit 804 in the general administration and operation of the functionality implemented by the model training system 120 and/or the model hosting system 140.”).
Prior arts:
Park et al. (“CondNAS: Neural Architecture Search for Conditional CNNs”, Conditional CNNs. Electronics 2022) teaches see page 4, section 3.1, “The GA in CondNAS adopts machine learning-based accuracy and latency prediction models to evaluate the score of an arbitrary conditional CNN encountered in the search. To this end, before the GA starts, the training dataset generation module extracts a number of various conditional CNNs from the SuperNet and generates their accuracy and latency label for the given target platform ( 2) in a similar way to BPNet: it actually runs and profiles the conditional CNNs, feeding the calibration dataset. The generated datasets are used for training the prediction models, which are then used in the search of (near-)optimal conditional CNNs for the given target platform ( 3). The SuperNet and the accuracy prediction model are trained only once for a new input image dataset, whereas the latency prediction model should be trained whenever the target platform changes.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IMAD M KASSIM whose telephone number is (571)272-2958. The examiner can normally be reached 10:30AM-5:30PM, M-F (E.S.T.).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J. Huntley can be reached at (303) 297 - 4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/IMAD KASSIM/Primary Examiner, Art Unit 2129
Prosecution Insights