DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the application and claims filed 5/17/2023. Claims 1-20 are pending and have been examined. Claims 1-20 are rejected.
Priority
The examiner acknowledges the priority benefit to U.S. Provisional Application No. 63/343,014, filed on 5/17/2022. The present application claims priority to U.S. Provisional Application No. 63/343,014, filed 5/17/2022.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/21/2023 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement has been considered by the examiner.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character not mentioned in the description:
604 in FIG. 6.
Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities:
Reference character 604 shown in Figure 6 is not described in applicant’s specification (see, e.g., paragraphs 66-71 describing FIG. 6).
The use of the terms Linux®, Windows® and MacOS®, which are trade names or marks used in commerce, has been noted in this application. They should be capitalized wherever they appear and be accompanied by the generic terminology. For instance, the above-noted terms appear in paragraph 25 of the specification.
Although the use of trade names and marks used in commerce (i.e., trademarks, service marks, certification marks, and collective marks) are permissible in patent applications, the proprietary nature of the marks should be respected and every effort made to prevent their use in any manner which might adversely affect their validity as commercial marks. Appropriate correction is required.
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed. In particular, the title of the invention is “PROXY SYSTEMS AND METHODS FOR MULTIPROCESSING ARCHITECTURES”. The following title is suggested: “PROXY SYSTEMS AND METHODS FOR LOAD BALANCING NEURAL NETWORK INFERENCING REQUESTS ACROSS MULTIPROCESSING ARCHITECTURES”. A descriptive title indicative of the invention will help in proper indexing, classifying, searching, etc. See, MPEP § 606.01. However, the title of the invention should be limited to 500 characters. The examiner suggests including the aspect(s) of the claims which Applicant believes to be novel or nonobvious over the prior art.
Claim Objections
Claims 4-7 and 14-17 are objected to because of the following informalities:
Claims 4 and 14 both recite “an input tensor” in line 1. Applicant previously introduced “an input tensor” in base claims 1 and 11 (see, line 3 of claim 1 and line 8 of claim 11). As such, it appears the subsequent recitations of “an input tensor” should recite “[[an]] the input tensor.” Appropriate correction is required.
Claims 6 and 16 are objected to under 37 CFR 1.75 as being substantial duplicates of claims 2 and 12, respectively. When two claims in an application are duplicates or else are so close in content that they both cover the same thing, despite a slight difference in wording, it is proper after allowing one claim to object to the other as being a substantial duplicate of the allowed claim. See MPEP § 608.01(m). Appropriate correction is required.
Claims 7 and 17 are objected to under 37 CFR 1.75 as being substantial duplicates of claims 5 and 151, respectively. When two claims in an application are duplicates or else are so close in content that they both cover the same thing, despite a slight difference in wording, it is proper after allowing one claim to object to the other as being a substantial duplicate of the allowed claim. See MPEP § 608.01(m). Appropriate correction is required.
Claims 5 and 15, which depend directly from claims 4 and 14, respectively, are objected to based on their respective dependencies from claim 4 and 14.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 3, 9-10, 13 and 19-20 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claims 3 and 13 both recite “the neural network” in line 1. There is insufficient antecedent basis for this limitation in these claims. No “neural network” was previously introduced in these claims or in their respective base claims, claims 1 and 11. Applicant previously introduced “a neural network model” in claims 1 and 11 (see, line 3 of claim 1 and line 4 of claim 11). However, it is unclear if the subsequent recitation of “the neural network” refers to the previously-introduced “neural network model” or to some other, distinct “neural network”. For the purposes of determining patent eligibility and comparison with the prior art, the examiner is interpreting the term “the neural network” as the previously-introduced “neural network model”. Appropriate correction is required.
Claims 9 and 19 both recite “the selected model ID” in line 2. There is insufficient antecedent basis for this limitation in these claims. No “selected model ID” was previously introduced in these claims or in their respective base claims, claims 1 and 11. Applicant previously introduced “a model ID associated with a neural network model” in claims 1 and 11 (see, line 3 of claim 1 and line 7 of claim 11). However, it is unclear if the subsequent recitation of “the selected model ID” refers to a selected ID of the previously-introduced “model ID associated with a neural network model” or to some distinct “neural network”. For examination purposes, “the selected model ID” is being interpreted as “a selected model ID.” Appropriate correction is required.
Claims 10 and 20, which depend directly from claims 9 and 19, respectively, are rejected under 35 U.S.C. 112(b) as being indefinite under the same rationale as claims 9 and 19.
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claims 4 and 14 are rejected under 35 U.S.C. 112(d) as being of improper dependent form for failing to further limit the subject matter of the claims upon which they depend, or for failing to include all the limitations of the claims upon which they depend.
The limitations of claims 4 and 14 do not further limit the receiving “an inference request …” limitations in base claims 1 and 11. In particular, claims 4 and 14, which depend directly from claims 1 and 11, respectively, both recite “wherein the inference request includes an input tensor.” However, claims 1 and 11 recite, using respective similar language, “receiving an inference request from a client computing system, the inference request comprising a model ID associated with a neural network model and an input tensor.” Thus, claims 4 and 14 fail to further limit claims 1 and 11, upon which these claims respectively depend.
Applicant may cancel the claims, amend the claims to place the claims in proper dependent form, rewrite the claims in independent form, or present a sufficient showing that the dependent claims comply with the statutory requirements.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. because the claimed invention is directed to an abstract idea without significantly more. The analysis below of the claims’ subject matter eligibility follows the 2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50-57 (January 7, 2019) (“2019 PEG”) and the 2024 Guidance Update on Patent Subject Matter Eligibility, Including on Artificial Intelligence, 89 Fed. Reg. 58128-58138 (July 17, 2024) (“2024 AI SME Update”).
When considering subject matter eligibility under 35 U.S.C. 101, it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter (Step 1). If the claim does fall within one of the statutory categories, the second step in the analysis is to determine whether the claim is directed to a judicial exception (Step 2A). The Step 2A analysis is broken into two prongs. In the first prong (Step 2A, Prong 1), it is determined whether or not the claims recite a judicial exception (e.g., mathematical concepts, mental processes, certain methods of organizing human activity). If it is determined in Step 2A, Prong 1 that the claims recite a judicial exception, the analysis proceeds to the second prong (Step 2A, Prong 2), where it is determined whether or not the claims integrate the judicial exception into a practical application. If it is determined at step 2A, Prong 2 that the claims do not integrate the judicial exception into a practical application, the analysis proceeds to determining whether the claim is a patent-eligible application of the exception (Step 2B). If an abstract idea is present in the claim, any element or combination of elements in the claim must be sufficient to ensure that the claim integrates the judicial exception into a practical application, or else amounts to significantly more than the abstract idea itself.
Regarding independent claims 1 and 11, these claims are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 1 is directed to a method, corresponding to a process, and claim 1 is directed to an apparatus comprising a proxy computing system; a client computing system communicatively coupled to the proxy computing system; and a set of processing devices preloaded with a neural network model and communicatively coupled to the proxy computing system, corresponding to a system, which are both one of the statutory categories.
Step 2A Prong One Analysis: The claims are directed to an abstract idea. In particular, the claims recite mental processes that are concepts performed in the human mind (including an observation, evaluation, judgment, opinion) combined with mathematical concepts (i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations).
The claims recite, using respective similar language, the following limitations:
selecting a target processing device from the subset based on the load states;
monitoring an execution of the inference request by the target processing device based on the neural network model; - under their broadest reasonable interpretation (BRI), in light of the specification, the selecting and monitoring limitations encompass mental processes of selecting/choosing a generically-recited target processing device based on received/observed load states and observation of execution of a request by the target device based on a generically-recited neural network model (i.e., evaluation/judgement/opinion to select a target device based on observed/received load data and observation to monitor execution of the inference request by the target processing device based on the neural network model).
computing the average inference time for the inference request execution based on the monitoring - under its BRI, in light of the specification, this is a mathematical concept (mathematical calculation – computing the average inference time for the inference request execution based on the monitoring data).
MPEP 2106.04(a)(2)(II) provides “The mathematical concepts grouping is defined as mathematical relationships, mathematical formulas or equations, and mathematical calculations.”
MPEP 2106.04(a)(2)(II) further provides “It is important to note that a mathematical concept need not be expressed in mathematical symbols, because "[w]ords used in a claim operating on data to solve a problem can serve the same purpose as a formula." In re Grams, 888 F.2d 835, 837 and n.1, 12 USPQ2d 1824, 1826 and n.1 (Fed. Cir. 1989). See, e.g., SAP America, Inc. v. InvestPic, LLC, 898 F.3d 1161, 1163, 127 USPQ2d 1597, 1599 (Fed. Cir. 2018) (holding that claims to a ‘‘series of mathematical calculations based on selected information’’ are directed to abstract ideas); Digitech Image Techs., LLC v. Elecs. for Imaging, Inc., 758 F.3d 1344, 1350, 111 USPQ2d 1717, 1721 (Fed. Cir. 2014) (holding that claims to a ‘‘process of organizing information through mathematical correlations’’ are directed to an abstract idea); and Bancorp Servs., LLC v. Sun Life Assurance Co. of Can. (U.S.), 687 F.3d 1266, 1280, 103 USPQ2d 1425, 1434 (Fed. Cir. 2012) (identifying the concept of ‘‘managing a stable value protected life insurance policy by performing calculations and manipulating the results’’ as an abstract idea).”
Therefore, the claims recite mental processes based on mathematical concepts.
These steps and operations cover performance of the limitations in the mind (i.e., selecting a target processing device based on observed load states and observation of execution of a request by the target device based on a neural network model combined with a mathematical concept (mathematical calculations to compute the average inference time for the inference request execution based on the monitoring).
Regarding the “neural network model” limitations, no details of the neural network or its training are recited and the neural network is recited at a high level of generality. Aside from repeating the claim language in paragraphs 4, 26, 29-30, 47-48, 58, 62-65 69-70 and 81, and providing general examples in paragraphs 33, 35, 38, 43, 45 and 56 in stating “Model 304 may be a neural network model. In an aspect, model 304 may include input tensor space 306 (e.g., an image or a video frame), and sets of weight tensors 308 and 310.” and “Model 424 may be a neural network model comprised of input tensor space 426, set of weight tensors 428, and output tensor space 430. Model 424 is output as model data 432.”, applicants’ specification does not define the “neural network model”. Thus, the claimed “neural network model”, under the BRI, in light of the specification, could be any neural network model, which could be constructed by hand with pen and paper.
Also, the neural network model is recited at a high level of generality and therefore is being interpreted as performing a mental process on a generic computer. See MPEP 2106.04(a)(2) § III.C which states that “a concept that is performed in the human mind and applicant is merely claiming that concept performed 1) on a generic computer, or 2) in a computer environment, or 3) is merely using a computer as a tool to perform the concept” still recite a mental process.
Regarding the “computing the average inference time for the inference request execution based on the monitoring” limitation, under the BRI, in light of the specification, this encompasses a mathematical concept (i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations). Such computations/calculations can be carried out by hand with pen and paper, as suggested by the discussion of the “average inference time” in paragraphs 61 and 65 of applicants’ specification, which merely repeat the claim language (these paragraphs contain the only mentions of any “average inference time” in the entire specification). Given a sufficiently small set of observed inference execution time data, nothing in the claims prohibit this computation/calculation from being performed with pen and paper.
The above-noted selecting and monitoring limitations, as drafted, are a process that, under its BRI, covers performance of the limitations in the mind in combination with mathematical concepts – the calculating limitation.
If the claim limitations, under their broadest reasonable interpretations, cover performance of the limitations in the mind and mathematical relationships, mathematical formulas or equations, or mathematical calculations but for the recitation of generic computer components (i.e., the generically-recited “neural network model”, the “client computing system” of claim 1, and “An apparatus comprising: a proxy computing system; a client computing system communicatively coupled to the proxy computing system; and a set of processing devices” of claim 11) then they fall within the “Mental Processes” and “Mathematical concepts” groupings of abstract ideas. Accordingly, claims 1 and 11 recite an abstract idea.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claims recite, using respective similar language, these additional elements receiving an inference request from a client computing system, the inference request comprising a model ID associated with a neural network model and an input tensor;
receiving a statistics request from the client computing system, the statistics request including the model ID and an average inference time request;
accessing a load state of each processing device in a subset of processing devices preloaded with the neural network model;
transmitting the inference request to the target processing device;
receiving an inference result generated by the target processing device after executing the inference request; and
transmitting the inference result and the average inference time to the client computing system - These are insignificant extra-solution activities that do not add a meaningful limitation to the above-noted abstract idea (mental processes based on mathematical concepts) specified in these claims because “receiving an inference request”, “receiving a statistics request”, “accessing a load state of each processing device” and “receiving an inference result” is mere data gathering, and “transmitting the inference request” and “transmitting the inference result and the average inference time” amount to necessary data gathering (the inference request) and outputting (transmitting the result and data) (See MPEP § 2106.05(g)).
Claim 11 also recites “An apparatus comprising: a proxy computing system; a client computing system communicatively coupled to the proxy computing system; and a set of processing devices preloaded with a neural network model and communicatively coupled to the proxy computing system, wherein: the proxy computing system” <performs the above-noted receiving, accessing, selecting, computing and transmitting operations/steps>. The above-noted additional elements in the claim amount to recitation of the words "apply it" (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer, which does not integrate a judicial exception into a practical application. See MPEP 2106.05(f). Merely asserting that a judicial exception is to be carried out on a generic computer (i.e., with the generically-recited “apparatus comprising: a proxy computing system; a client computing system … and a set of processing devices preloaded with a neural network model”) cannot meaningfully integrate the judicial exception into a practical application. See MPEP § 2106.05(f).
Accordingly, the additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. See MPEP 2106.04(d).
The claims are directed to an abstract idea.
Step 2B Analysis: The claims do not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
Receiving and communicating data are insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP2106.05(d)(II) (“The courts have recognized the following computer functions as well‐understood, routine, and conventional functions… i. Receiving or transmitting data over a network…iv. Storing and retrieving information in memory”) (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)). Therefore, recitations of “receiving an inference request”, “receiving a statistics request”, “accessing a load state of each processing device”. “receiving an inference result”, “transmitting the inference request” and “transmitting the inference result and the average inference time” are the well-understood, routine, conventional activities of receiving or transmitting data over a network, as discussed in MPEP § 2106.05(d).
Also ere instructions to apply the mental process electronically (i.e., with the recited “neural network model” and “apparatus comprising: a proxy computing system; a client computing system … and a set of processing devices” of claim 11 do not amount to significantly more than the judicial exception. As noted above, merely asserting that a judicial exception is to be carried out on a generic computer cannot provide significantly more than the judicial exception. See MPEP § 2106.05(f).
These claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, there are no additional elements recited that impose any meaningful limits on practicing the abstract idea. Therefore, the additional elements of these dependent claims are not sufficient to amount to significantly more than the abstract idea. These claims are not patent eligible.
Regarding claims 2, 6, 12 and 16 these claims are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claims 2 and 6 are directed to a method as depending from claim 1 and claims 12 and 16 are directed to an apparatus as depending from claim 11, thus the analysis for patent eligibilities of claims 1 and 11 are incorporated herein.
Step 2A Prongs 1-2: The claims each recite “wherein the inference result is an output tensor2.” This limitation does nothing to alter the fundamental nature of the claims as a mental process based on or combined with a mathematical concept. This is because the above-noted wherein clause merely limits the invention to a narrower abstract idea by further narrowing what receiving and transmitting the inference result includes, i.e., receiving and transmitting “an output tensor.” The additional limitation added by these claims merely require that a received inference result includes an output tensor (i.e., a data object/structure populated with output data).
Dependent claims 2, 6, 12 and 16, when analyzed as a whole, are not patent eligible under 35 U.S.C. 101 because the additional recited limitation fails to establish that the claims are not directed to an abstract idea.
Thus, this limitation does nothing to alter the analysis of claims 1 and 11.
Step 2B Analysis: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception electronically (i.e., with the recited “neural network model” and “apparatus comprising: a proxy computing system; a client computing system … and a set of processing devices” of base claim 11) cannot provide an inventive concept. The claims are not patent eligible.
Regarding claims 3 and 13, these claims are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 3 is directed to a method as depending from claim 1 and claim 13 is directed to an apparatus as depending from claim 11, thus the analysis for patent eligibilities of claims 1 and 11 are incorporated herein.
Step 2A Prong 1: The claims both recite “wherein the neural network3 is a convolutional neural network or a neural network comprised of one or more linear algebra operators.” This limitation does nothing to alter the fundamental nature of the claims as a mental process based on or combined with a mathematical concept. This is because the above-noted wherein clause merely limits the invention to a narrower abstract idea by further narrowing what the generically-recited “neural network” can be, i.e., a generically-recited “convolutional neural network or a neural network comprised of one or more linear algebra operators”.
Also, the additional limitation of “a neural network comprised of one or more linear algebra operators” added by these claims is directed to a mathematical concept and encompasses algebraic operators (i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations).
Dependent claims 3 and 13, when analyzed as a whole, are not patent eligible under 35 U.S.C. 101 because the additional recited limitation fails to establish that the claims are not directed to an abstract idea. The additional limitation added by these claims covers a mathematical concept.
Thus, this limitation does nothing to alter the analysis of claims 1 and 11.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The “convolutional neural network or a neural network comprised of one or more linear algebra operators” are recited at a high level of generality as mere instructions to implement an abstract idea on a computer and amounts to the recitation of the words “apply it” (or an equivalent) or amount to no more than mere instructions to implement an abstract idea or other exception on a computer or merely use a computer as a tool to perform an abstract idea (i.e., as generic computer components performing generic computer functions). See MPEP 2106.05(f).
There are no additional elements to integrate the abstract idea into a practical application or to impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
Step 2B Analysis: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception electronically (i.e., with the recited “neural network model” and “apparatus comprising: a proxy computing system; a client computing system … and a set of processing devices” of base claim 11) cannot provide an inventive concept. The claims are not patent eligible.
Regarding claims 4 and 14, these claims are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 4 is directed to a method as depending from claim 1 and claim 14 is directed to an apparatus as depending from claim 11, thus the analysis for patent eligibilities of claims 1 and 11 are incorporated herein.
Step 2A Prongs 1-2: The claims both recite “wherein the inference request includes an input tensor4.” This limitation does nothing to alter the fundamental nature of the claims as a mental process based on or combined with a mathematical concept. This is because the above-noted wherein clause merely limits the invention to a narrower abstract idea by further narrowing what receiving and transmitting the inference request includes, i.e., receiving and transmitting “an input tensor.” The additional limitation added by these claims merely require that a received inference request includes an input tensor (i.e., a data object/structure populated with input data).
Dependent claims 4 and 14, when analyzed as a whole, are not patent eligible under 35 U.S.C. 101 because the additional recited limitation fails to establish that the claims are not directed to an abstract idea.
Thus, this limitation does nothing to alter the analysis of claims 1 and 11.
Step 2B Analysis: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception electronically (i.e., with the recited “neural network model” and “apparatus comprising: a proxy computing system; a client computing system … and a set of processing devices” of claim 11) cannot provide an inventive concept. The claims are not patent eligible.
Regarding claims 5 and 15, these claims are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 5 is directed to a method as depending from claim 4 and claim 15 is directed to an apparatus as depending from claim 14, thus the analysis for patent eligibilities of claims 4 and 14, and of base claims 1 and 11 are incorporated herein.
Step 2A Prongs 1-2: The claims both recite “wherein the input tensor is an image generated by an image sensor.” This limitation does nothing to alter the fundamental nature of the claims as a mental process based on or combined with a mathematical concept. This is because the above-noted wherein clause merely limits the invention to a narrower abstract idea by further narrowing what the input tensor includes, i.e., “an image generated by an image sensor.” The additional limitation added by these claims merely require that a received inference request includes an input tensor populated with an image generated by an image sensor.
Dependent claims 5 and 15, when analyzed as a whole, are not patent eligible under 35 U.S.C. 101 because the additional recited limitation fails to establish that the claims are not directed to an abstract idea.
Thus, this limitation does nothing to alter the analysis of claims 1 and 11.
Step 2B Analysis: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception electronically (i.e., with the recited “neural network model”, “image sensor” and “apparatus comprising: a proxy computing system; a client computing system … and a set of processing devices” of claim 11) cannot provide an inventive concept. The claims are not patent eligible.
Regarding claims 7 and 17, these claims are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 7 is directed to a method as depending from claim 1 and claim 17 is directed to an apparatus as depending from claim 11, thus the analysis for patent eligibilities of claims 1 and 11 are incorporated herein.
Step 2A Prongs 1-2: The claims both recite “wherein the input tensor is an image generated by an image sensor.” This limitation does nothing to alter the fundamental nature of the claims as a mental process based on or combined with a mathematical concept. This is because the above-noted wherein clause merely limits the invention to a narrower abstract idea by further narrowing what the input tensor includes, i.e., “an image generated by an image sensor.” The additional limitation added by these claims merely require that a received inference request includes an input tensor populated with an image generated by an image sensor.
Dependent claims 7 and 17, when analyzed as a whole, are not patent eligible under 35 U.S.C. 101 because the additional recited limitation fails to establish that the claims are not directed to an abstract idea.
Thus, this limitation does nothing to alter the analysis of claims 1 and 11.
Step 2B Analysis: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception electronically (i.e., with the recited “neural network model”, “image sensor” and “apparatus comprising: a proxy computing system; a client computing system … and a set of processing devices” of claim 11) cannot provide an inventive concept. The claims are not patent eligible.
Regarding claims 8 and 18, these claims are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 8 is directed to a method as depending from claim 1 and claim 18 is directed to an apparatus as depending from claim 11, thus the analysis for patent eligibilities of claims 1 and 11 are incorporated herein.
Step 2A Prongs 1-2: The claims both recite “wherein the load state includes an endpoint execution queue associated with each processing device.” This limitation does nothing to alter the fundamental nature of the claims as a mental process based on or combined with a mathematical concept. This is because the above-noted wherein clause merely limits the invention to a narrower abstract idea by further narrowing what receiving/accessing the load state includes, i.e., receiving/accessing “an endpoint execution queue”/data structure that is associated with each generically-recited “processing device”. The additional limitation added by these claims merely require that a received load state includes an execution queue (i.e., a data object/structure/list populated with processing device data/information).
Dependent claims 8 and 18, when analyzed as a whole, are not patent eligible under 35 U.S.C. 101 because the additional recited limitation fails to establish that the claims are not directed to an abstract idea.
Thus, this limitation does nothing to alter the analysis of claims 1 and 11.
Step 2B Analysis: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception electronically (i.e., with the recited “neural network model”, “each processing device” and “apparatus comprising: a proxy computing system; a client computing system … and a set of processing devices” of claim 11) cannot provide an inventive concept. The claims are not patent eligible.
Regarding claims 9 and 19, these claims are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 9 is directed to a method as depending from claim 1 and claim 19 is directed to an apparatus as depending from claim 11, thus the analysis for patent eligibilities of claims 1 and 11 are incorporated herein.
Step 2A Prong 1: The claims recite, using respective similar language, “selecting the target processing device based on a model occupancy for the selected model ID5.” This limitation does nothing to alter the fundamental nature of the claims as a mental process based on or combined with a mathematical concept. This is because the above-noted wherein clause merely limits the invention to a narrower abstract idea by further narrowing what selecting a generically-recited target processing device includes, i.e., selecting the device based on an observed model occupancy for a selected model ID.
Also, the additional limitation of “selecting the target processing device based on a model occupancy for the selected model ID” added by these claims encompass a mental process of selecting/choosing a generically-recited target processing device based on received/observed model occupancy data for a selected model identifier/ID (i.e., evaluation/judgement/opinion to select a target device based on observed model data).
Dependent claims 9 and 19, when analyzed as a whole, are not patent eligible under 35 U.S.C. 101 because the additional recited limitation fails to establish that the claims are not directed to an abstract idea. The additional limitation added by these claims covers a mental process.
Thus, this limitation does nothing to alter the analysis of claims 1 and 11.
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
The “target processing device” is recited at a high level of generality as mere instructions to implement an abstract idea on a computer and amounts to the recitation of the words “apply it” (or an equivalent) or amount to no more than mere instructions to implement an abstract idea or other exception on a computer or merely use a computer as a tool to perform an abstract idea (i.e., as generic computer components performing generic computer functions). See MPEP 2106.05(f).
There are no additional elements to integrate the abstract idea into a practical application or to impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
Step 2B Analysis: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception electronically (i.e., with the recited “target processing device” and “apparatus comprising: a proxy computing system; a client computing system … and a set of processing devices” of base claim 11) cannot provide an inventive concept. The claims are not patent eligible.
Regarding claims 10 and 20, these claims are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 10 is directed to a method as depending from claim 9 and claim 20 is directed to an apparatus as depending from claim 19, thus the analysis for patent eligibilities of claims 9 and 19, and of base claims 1 and 11 are incorporated herein.
Step 2A Prongs 1-2: The claims both recite “wherein the model occupancy is stored in a device library.” This limitation does nothing to alter the fundamental nature of the claims as a mental process based on or combined with a mathematical concept. This is because the above-noted wherein clause merely limits the invention to a narrower abstract idea by further narrowing what the mental process of selecting a generically-recited target processing device includes, i.e., selecting the device based on an obtained/retrieved model occupancy stored in a generically-recited “device library.”
Additionally, the above recitation of “the model occupancy is stored in a device library” recites the insignificant extra-solution activity of mere data storage. That is, adding a step of storing data to the above-noted mental process does not add a meaningful limitation to the process. See MPEP § 2106.05(g).
Dependent claims 10 and 20, when analyzed as a whole, are not patent eligible under 35 U.S.C. 101 because the additional recited limitation fails to establish that the claims are not directed to an abstract idea.
Thus, this limitation does nothing to alter the analysis of claims 9 and 19.
The “device library” is recited at a high level of generality as mere instructions to implement an abstract idea on a computer and amounts to the recitation of the words “apply it” (or an equivalent) or amount to no more than mere instructions to implement an abstract idea or other exception on a computer or merely use a computer as a tool to perform an abstract idea (i.e., as generic computer components performing generic computer functions). See MPEP 2106.05(f).
There are no additional elements to integrate the abstract idea into a practical application or to impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
Step 2B Analysis: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception electronically (i.e., with the recited “device library” and “apparatus comprising: a proxy computing system; a client computing system … and a set of processing devices” of base claim 11) cannot provide an inventive concept.
Moreover, receiving, communicating, and storing data are insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP2106.05(d)(II) (“The courts have recognized the following computer functions as well‐understood, routine, and conventional functions… i. Receiving or transmitting data over a network…iv. Storing and retrieving information in memory”) (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)). Therefore, the recitations of “wherein the model occupancy is stored in a device library” is the well-understood, routine, conventional activity of storing information in memory, as discussed in MPEP § 2106.05(d).
The claims are not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-9 and 11-19 are rejected under 35 U.S.C. 103 as being unpatentable over Diamant et al. (U.S. Patent No. 10,846,201 B1, hereinafter “Diamant”) in view of non-patent literature Prasad et al. ("A Joint Model Provisioning and Request Dispatch Solution for Low-Latency Inference Services on Edge." Sensors 21.19 (2021): 6594, hereinafter “Prasad”).
With respect to claim 1, Diamant discloses the invention as claimed including a method (see, e.g., Column 20, lines 22-23, “method for debugging and improving the performance of a neural network”) comprising:
receiving an inference request from a client computing system (see, e.g., Column 7, lines 30-34, “an unseen sample (e.g., a test sample or a new sample) is input into the CNN, the CNN may go through the forward propagation step and output a probability for each class using the trained weights and parameters, which may be referred to as an inference (or prediction)”; Column 8, lines 14-27, “Host interface 214 may enable communications between the host device and neural network processor 202. … host interface 214 may be configured to transmit the memory descriptors including the memory addresses of the stored data (e.g., input data, weights, results of computations, etc.) between the host device and neural network processor 202. … processor 202 may provide the computing resources to support the neural network computations for inference, such as image classification.”; column 10, lines 9-23, “neural network processors 202 may be used to implement a deep neural network that may include multiple sets of convolution, activation, and pooling layers. … neural network processor 202 may first receive input data and instructions for implementing a first set of convolution, activation, and/or pooling layers. The input data may include the network parameters for the first set of network layers, such as the number of nodes, the weights, or the parameters of the filters, etc. The input data may also include the external input data to be processed by the neural network or intermediate output data from previous layers of the neural network. The instructions may include instructions for computing engine 224, activation engine 228a, and/or pooling engine 228b.” [i.e., input data and instructions received include data to be processed by a neural network which is interpreted as receiving an inference request] and col. 21, lines 31-34, “hardware and software resources of computing device 1300 (e.g., the hardware and software resources associated with provision of an image recognition service) can be allocated to a client” [i.e., a method including receiving an image recognition/inference request from a client computing system or host device]), the inference request comprising a model ID associated with a neural network model (see, e.g., Column 20, lines 55-61, “the notification packet may also include an identification of the instruction, and an identification of the processing engine that executes the instruction.” [i.e., notification packet including an identification/ID an inference request/instruction, and of an identification/ID of processing engine that executes the instruction is a model ID]) and an input tensor (see, e.g., Column 3, lines 40-44 and 54-61, “specialized hardware circuits, such as … tensor processing units (TPUs), neural network processing units (NPUs) … may be used for the training and/or inference.” and “An object 110 to be classified, such as an input image, may be represented by a matrix of pixel values. The input image may include multiple channels, each channel representing a certain component of the image. For example, an image from a digital camera may have a red channel, a green channel, and a blue channel. Each channel may be represented by a 2-D matrix of pixels having pixel values in the range of, for example, 0 to 255 (i.e., 8-bit).” [i.e., Image/object to be classified is an inference request, inference/classification request includes 2D matrices of pixel values, an input tensor]); …
accessing a load state of each processing device in a subset (see, e.g., Column 10, lines 58-63, “compiler may maintain a list of available hardware resources and the functions and usage of the hardware resources of the neural network, and assign operations of the neural network to appropriate hardware resources based on the functions and usage of the hardware resources.” [i.e., accessing a load state is recording or viewing system resource usage or available hardware resources, maintaining a list of available hardware resources includes accessing a load state of each processing device in a subset of devices]) of processing devices preloaded with the neural network model (see, e.g., Column 8, lines 14-20, “Host interface 214 may enable communications between the host device and neural network processor 202. … interface 214 may be configured to transmit the memory descriptors including the memory addresses of the stored data (e.g., input data, weights, results of computations, etc.) between the host device and neural network processor 202.”; column 10, lines 9-23, “One or more neural network processors 202 may be used to implement a deep neural network … a neural network processor 202 may first receive input data and instructions for implementing a first set of convolution, activation, and/or pooling layers. The input data may include the network parameters for the first set of network layers, such as the number of nodes, the weights, or the parameters of the filters, etc. The input data may also include the external input data to be processed by the neural network or intermediate output data from previous layers of the neural network. The instructions may include instructions for computing engine 224, activation engine 228a, and/or pooling engine 228b.” [i.e., processing devices are loaded with received input data, instructions, and parameters of the neural network - the neural network model, transmitting such data and having it be received by each neural processor is loading the neural network model into each processing device]);
selecting a target processing device from the subset based on the load states (see, e.g., Column 10, lines 58-63, “the compiler may maintain a list of available hardware resources and the functions and usage of the hardware resources of the neural network, and assign operations of the neural network to appropriate hardware resources based on the functions and usage of the hardware resources.” [i.e., selecting a target hardware resource/target processing device from a subset of available processing devices based on the usage/load states by assigning operations of a neural network based on usage/load states of hardware resources]);
transmitting the inference request to the target processing device (see, e.g., Column 8, lines 14-20, “Host interface 214 may enable communications between the host device and neural network processor 202. … host interface 214 may … transmit … the stored data (e.g., input data, weights, results of computations, etc.) between the host device and neural network processor 202.”; column 10, lines 9-23, “neural network processors 202 may be used to implement a deep neural network … neural network processor 202 may first receive input data and instructions for implementing a first set of convolution, activation, and/or pooling layers. The input data may include the network parameters for the first set of network layers, such as the number of nodes, the weights, or the parameters of the filters, etc. The input data may also include the external input data to be processed by the neural network … instructions may include instructions for computing engine 224, activation engine 228a, and/or pooling engine 228b.” [i.e., input data and instructions transmitted to and received by the processing device includes data to be processed, the data and instructions to be processed is considered an inference request that is transmitted to the target processing device]);
monitoring an execution of the inference request by the target processing device based on the neural network model (see, e.g., Column 15, lines 41-42, “FIG. 7 illustrates an example process 700 for monitoring and debugging the performance of a neural network”, col. 16, lines 35-40, “debug outputs, such as the timestamps, may be included in notification packets. … the timestamps may be generated based on a global reference clock or a global clock counter that counts the number of clock cycles since a reference point (e.g., the start of an inference”, col. 17, line 66-col. 18, line 12, “FIG. 9 illustrates an example of a notification packet 900 for monitoring the performance of a neural network … in FIG. 9, notification packet 900 includes 16 bytes. Notification packet 900 may include a "Notification Type" field that specifies the type of notification. … Notification packet 900 may also include a "Block ID" field that indicates the processing engine associated with the debugging circuit that generates the notification packet or the processing engine that executes the instruction, such as the convolution engine” and col. 18, lines 29-34, “Notification packet 900 may include a 64-bit timestamp that indicates the time associated with the execution of the instruction … the starting or completion time of the instruction since a reference time point (e.g., … a start of an inference).” [i.e., monitoring performance/execution of the inference request by the target processing device/engine based on the neural network model]);
receiving an inference result generated by the target processing device after executing the inference request (see, e.g., Column 8, lines 9-14, “Neural network processor 202 may also store the results of computations (e.g., one or more image recognition decisions or intermediary data) at memory 212, and provide the memory addresses for the stored results to the host device.” [i.e., neural network processor storing the results of computations in memory and providing memory addresses for the same is the target processing device transmitting the inference result/image recognition decision after executing the request]); … and
transmitting the inference result and the average inference time to the client computing system (see, e.g., Column 8, lines 9-14, “Neural network processor 202 may also store the results of computations (e.g., one or more image recognition decisions or intermediary data) at memory 212, and provide the memory addresses for the stored results to the host device.” [i.e., providing memory addresses for stored results to a host device is transmitting the results to the client computing system]).
Although Diamant substantially discloses the claimed invention, Diamant is not relied on to explicitly disclose receiving a statistics request from the client computing system, the statistics request including the model ID and an average inference time request; and
computing the average inference time for the inference request execution based on the monitoring.
However, in the same field, analogous art Prasad teaches receiving a statistics request from the client computing system, the statistics request including the model ID and an average inference time request (see, e.g., pages 3, Sect 2.2, “an edge network has been deployed, with multiple mobile users requesting inference services. We can formally define the model provisioning and request dispatch problem based on the system model. … Given: – ϴi,j: execution time of request ri’s inference job on edge node s j … mi: memory requirement of the model requested by ri … objective U represents the average serving latency of all requests”, 5-6, Sect. 3.2.1, “endpoints were implemented to allow the back-end to acquire statistics, such as when and how often a given model is requested”, “Request Stats Endpoint. The front-end records the model requested, the access latency … Node Memory Stats Endpoint. Similar to the request stats endpoint, this endpoint is used by the back-end to acquire the current RAM usage of each node in the cluster”, 8, sect. 3.2.2, “number of requests, latency values, the execution time of a model on a node”, 9, Sect. 4.1.2, “obtaining empirical data (such as the execution time of inference jobs)”, 11, Sect. 4.2.1, “experiments focused on assessing the average serving latency of inference requests”, “our solution can record and provide real-time and statistical information about requests and communication latency” and 12, Sect. 4.2.3, “we evaluated the average latency under a various number of edge nodes.” [i.e., receiving a stats request to obtain statistics, the request including the model ID of the requested/given model and average execution time of inference job/inference time and average inference serving latency time]);
and computing the average inference time for the inference request execution based on the monitoring (see, e.g., pages 3, Sect. 2.2, “the optimization objective U represents the average serving latency of all requests.”, 11, Sect. 4.2.1, “experiments focused on assessing the average serving latency of inference requests … The ranges of inference execution time and inference instance size are set based on the testbed experiments”, “our solution can record and provide real-time and statistical information about requests and communication latency”,12, Sect. 4.2.3, “we evaluated the average latency under a various number of edge nodes.”, 13-14, Sect. 4.3, “the primary performance metric is the average serving latency … we use the average serving latency as the primary performance metric” and 15, Sect. 6, “We formalized the core problem as a non-linear integer programming problem to minimize the average inference serving latency” [i.e., assessing/computing average execution time of inference job/inference time and average inference serving latency time based on recording/monitoring real-time information about inference requests]).
Diamant and Prasad are analogous art because they are both directed to techniques and systems for improving neural network performance for tasks such as inferencing (see, e.g., Diamant, Abstract, col. 3, lines 34-46 and col. 20, lines 22-26 and Prasad, Abstract and page 16, Sect. 6).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Diamant with Prasad to provide “a novel solution that jointly provisions machine learning models and dispatches inference requests to reduce inference latency on edge nodes” where “The proposed solution provisions each edge node with the optimal number and type of inference instances under a holistic consideration of networking, computing, and memory resources.” (See, e.g., Prasad Abstract). Doing so would have allowed Diamant to use Prasad’s solution for dispatching inference requests that “provisions each edge node with the optimal number and type of inference instances under a holistic consideration of networking, computing, and memory resources” where “Optimization decisions were used to define how and where to place computing resources to minimize the latency experienced by users proactively” in order “to minimize the average inference serving latency”, as suggested by Prasad (See, e.g., Prasad, Abstract, and page 16, Sect 6).
With respect to independent claim 11, claim 11 is substantially similar to claim 1 and therefore is rejected on the same ground as claim 1, discussed above. In particular, claim 1 is an apparatus claim with operations that correspond to the method steps of claim 1.
Diamant further discloses an apparatus comprising: a proxy computing system (see, e.g., FIG. 2 – depicting an apparatus 200 including processor 202, memory 212 and other components of a computing system; column 7, lines 45-47, “Apparatus 200 may include a neural network processor 202 coupled to memory 212, a direct memory access (DMA) controller 216, and a host interface 214 via an interconnect 218.”; column 10, lines 45-55; column 17, lines 25-30, “According to certain embodiments, a notification 812 may be generated when state machine 800 leaves idle state 810, for example, when a new instruction is read from the instruction buffer by an instruction decoder. The notification may be generated by a debugging circuit in a control unit, such as control unit 305.” [i.e., a combination of memory, a direct memory access controller, a compiler, a control unit and a host interface is a proxy computing system]);
a client computing system communicatively coupled to the proxy computing system (see, e.g., FIG. 2 – depicting an apparatus 200 including interface 214 and interconnect 218 to communicatively couple a host computing system to neural network processor 202 and other components of the proxy computing system; column 7, lines 45-47; column 8, lines 14-20, “Host interface 214 may enable communications between the host device and neural network processor 202. For example, host interface 214 may be configured to transmit the memory descriptors including the memory addresses of the stored data (e.g., input data, weights, results of computations, etc.) between the host device and neural network processor 202.” [i.e., a host device is considered a client computing system, the host device communicates with a neural network processor through a host interface coupled to a proxy computing system, and is turn coupled to the proxy computing system]); and
a set of processing devices preloaded with a neural network model and communicatively coupled to the proxy computing system (see, e.g., FIG. 2 – depicting an apparatus 200 including neural network processor 202, interface 214 and interconnect 218 to communicatively couple a host/proxy computing system to processor 202; Column 7, lines 45-47; column 10, lines 9-11, “One or more neural network processors 202 may be used to implement a deep neural network that may include multiple sets of convolution, activation, and pooling layers.” and column 8, lines 14-20, “Host interface 214 may enable communications between the host device and neural network processor 202. For example, host interface 214 may be configured to transmit … memory addresses of the stored data (e.g., input data, weights, results of computations, etc.) between the host device and neural network processor 202.” [i.e., neural network processors are a set of processing devices preloaded with a neural network model]).
Regarding claims 2, 6, 12 and 166, as discussed above, Diamant in view of Prasad teaches the method of claim 1 and the apparatus of claim 11.
Diamant further discloses wherein the inference result is an output tensor (see, e.g., Column 5, lines 6-17, “The convolution operations in a CNN may be used to extract features from the input image. The convolution operations may preserve the spatial relationship between pixels by extracting image features using small regions of the input image. In a convolution, a matrix (referred to as a filter, a kernel, or a feature detector) may slide over the input image (or a feature map) at a certain step size (referred to as the stride). For every position (or step), element-wise multiplications between the filter matrix and the overlapped matrix in the input image may be calculated and summed to get a final value that represents a single element of an output matrix” [i.e., an inference result is an output of the CNN, matrices of output values are an output tensor]).
Regarding claims 3 and 13, as discussed above, Diamant in view of Prasad teaches the method of claim 1 and the apparatus of claim 11.
Diamant further discloses wherein the neural network7 is a convolutional neural network or a neural network comprised of one or more linear algebra operators (see, e.g., Column 3, lines 12-17, “Techniques disclosed herein may be used to debug any neural network or any other computing system that may include multiple processing engines or may perform a large number of calculations before yielding a final result, such as a convolutional neural network (also referred to as ConvNets or CNNs).” and Column 5, lines 6-17, “The convolution operations in a CNN may be used to extract features from the input image” [i.e., the neural network is a convolutional neural network/CNN]).
Regarding claims 4 and 14, as discussed above, Diamant in view of Prasad teaches the method of claim 1 and the apparatus of claim 11.
Diamant further discloses wherein the inference request includes an input tensor8 (see, e.g., Column 3, lines 54-61, “An object 110 to be classified, such as an input image, may be represented by a matrix of pixel values. The input image may include multiple channels, each channel representing a certain component of the image. For example, an image from a digital camera may have a red channel, a green channel, and a blue channel. Each channel may be represented by a 2-D matrix of pixels having pixel values in the range of, for example, 0 to 255 (i.e., 8-bit).” [i.e., an image/object to be classified is an inference request, inference/classification request includes 2D matrices of pixel values, an input tensor]).
Regarding claims 5 and 15, as discussed above, Diamant in view of Prasad teaches the method of claim 4 and the apparatus of claim 14.
Diamant further discloses wherein the input tensor is an image generated by an image sensor (see, e.g., Column 3, lines 54-61, “An object 110 to be classified, such as an input image, may be represented by a matrix of pixel values. The input image may include multiple channels, each channel representing a certain component of the image. For example, an image from a digital camera may have a red channel, a green channel, and a blue channel. Each channel may be represented by a 2-D matrix of pixels having pixel values in the range of, for example, 0 to 255 (i.e., 8-bit).” [i.e., a digital camera is an image sensor]).
Regarding claims 7 and 17, as discussed above, Diamant in view of Prasad teaches the method of claim 1 and the apparatus of claim 11.
Diamant further discloses wherein the input tensor is an image generated by an image sensor (see, e.g., Column 3, lines 54-61, “An object 110 to be classified, such as an input image, may be represented by a matrix of pixel values. The input image may include multiple channels, each channel representing a certain component of the image. For example, an image from a digital camera may have a red channel, a green channel, and a blue channel. Each channel may be represented by a 2-D matrix of pixels having pixel values in the range of, for example, 0 to 255 (i.e., 8-bit).” [i.e., a digital camera is an image sensor]).
Regarding claims 8 and 18, as discussed above, Diamant in view of Prasad teaches the method of claim 1 and the apparatus of claim 11.
Although Diamant substantially discloses the claimed invention, Diamant is not relied on to explicitly disclose wherein the load state includes an endpoint execution queue associated with each processing device.
However, in the same field, analogous art Prasad teaches wherein the load state includes an endpoint execution queue associated with each processing device (see, e.g., pages 4, Sect 3.1.1, “module has two primary functions that handle user requests. Firstly, it responds to the user’s inference requests with the optimal endpoint (IP address and port number). Secondly, it is responsible for collecting metrics related to inference request patterns. The metrics are made accessible through back-end facing REST endpoints.” and 5-6, Sect. 3.2, “accepting user requests and returning either a specific endpoint or a list of endpoints, proxying requests, and collecting metrics … two user-facing endpoints (the service query and service proxy endpoints, respectively). Additionally, three back-end facing endpoints were implemented to allow the back-end to acquire statistics, such as when and how often a given model is requested … Service Query Endpoint … for returning the user a serving endpoint for one specific model or a list of endpoints for all available models in the system.”, “Service Proxying Endpoint Proxying inference requests for some models is a more involved process than simply returning an endpoint. The user informs the front-end service that it wants a request proxied by including the TensorFlow Serving model path from the request endpoint [18]. This path is provided to the function within the request endpoint. … The service will then route the request to a function that can handle proxying. … Back-End Facing Endpoints … collect metrics about the inference request patterns. … These endpoints allow each module to remain semi-disconnected and avoid concurrency problems. Request Stats Endpoint. The front-end records the model requested, the access latency, and the chosen edge node for every request. The back-end uses this data to make provisioning decisions. … Node Memory Stats Endpoint … is used by the back-end to acquire the current RAM usage of each node in the cluster. This information is critical for deciding which node to place a model on” [i.e., load state metrics/statistics/stats include an endpoint execution list/queue associated with each processing device to be loaded with the requested neural network model]).
Diamant and Prasad are analogous art because they are both directed to techniques and systems for improving neural network performance for tasks such as inferencing (see, e.g., Diamant, Abstract, col. 3, lines 34-46 and col. 20, lines 22-26 and Prasad, Abstract and page 16, Sect. 6).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Diamant with Prasad to provide “a novel solution that jointly provisions machine learning models and dispatches inference requests to reduce inference latency on edge nodes” where “The proposed solution provisions each edge node with the optimal number and type of inference instances under a holistic consideration of networking, computing, and memory resources.” (See, e.g., Prasad Abstract). Doing so would have allowed Diamant to use Prasad’s solution for dispatching inference requests that “provisions each edge node with the optimal number and type of inference instances under a holistic consideration of networking, computing, and memory resources” where “Optimization decisions were used to define how and where to place computing resources to minimize the latency experienced by users proactively” in order “to minimize the average inference serving latency”, as suggested by Prasad (See, e.g., Prasad, Abstract, and page 16, Sect 6).
Regarding claims 9 and 19, as discussed above, Diamant in view of Prasad teaches the method of claim 1 and the apparatus of claim 11.
Although Diamant substantially discloses the claimed invention, Diamant is not relied on to explicitly disclose selecting the target processing device based on a model occupancy for the selected model ID.
However, in the same field, analogous art Prasad teaches selecting the target processing device based on a model occupancy for the selected model ID9 (see, e.g., pages 3, sect, 2.1, “The provisioner determines what machine learning models will be provisioned on an edge node.”, 6, sect. 3.2, “Node Memory Stats Endpoint … is used by the back-end to acquire the current RAM usage of each node in the cluster. This information is critical for deciding which node to place a model on and from which node to remove a model.” and 14, Sect. 5, “Service placement or model instance provisioning is the core function of the proposed solution. … The models can be quickly moved between nodes as users move within the coverage area of the system … Our solution can reduce the serving time by placing model instances at the optimized edge nodes” [i.e., select a target node/processing device based on model occupancy - whether the model instance is placed on the node or not]).
The motivation to combine Diamant and Prasad is the same as discussed above with respect to claims 8 and 18.
Conclusion
The prior art made of record, listed on form PTO-892, and not relied upon, is considered pertinent to applicant's disclosure.
The references listed on form PTO-892 are all generally related to techniques, methods and systems for allocating resources in proxy systems, neural networks, image processing, and multiprocessing architectures and load balancing machine learning and neural network model operations such as inferencing, prediction and classification operations.
For instance, Lee et al. (U.S. Patent Application Pub. No. 2023/0072337 A1, hereinafter “Lee”) discloses that “a first function (e.g., a classification function in a preview image acquired through a front single camera) by a camera application may operate based on a first neural network model 311 (e.g., a classification model” where “In operation 203, according to various embodiments, the electronic device 101 (e.g., the processor 120 of FIG. 1) may identify an available bandwidth of the memory 130 through a resource management unit included in the processor 120.” [i.e., analyzing a processing unit memory state of a processing device to identify available bandwidth/load state of memory in a processor] and “If failing to identify an available bandwidth of a memory in real time, a processor of an electronic device may not accurately calculate the memory bandwidth required to process the neural network model” [i.e., failing to identify available bandwidth may result in not accurately calculating the required bandwidth/load state/capacity to process a neural network] (see, e.g., Paragraphs [0051], [0054] and [0007]).
Also, for example, of non-patent literature Yu et al. (“Efficient architecture paradigm for deep learning inference as a service." 2020 IEEE 39th International Performance Computing and Communications Conference (IPCCC). IEEE, 2020, hereinafter “Yu”) discloses “a novel architecture paradigm on cloud for DIaaS” [Deep Learning Inference as a Service] that “use[s] the average single inference time to measure system performance. … under the three different models of darknet19, darknet53, and resnet152, the average single inference time of the unoptimized locality-aware architecture system” and “Fig.5 shows that, under the darknet19 and darknet53 model, the average single inference time of the detached architecture system was a little bit less than the locality-aware architecture system … Fig.5 shows that under the resnet152 model the results were the same as described above, except that the average single inference time of locality-aware OPT1 was a little bit more than the detached architecture system.” (see, e.g., Abstract and pages 6-7).
The examiner requests, in response to this office action, support be shown for language added to any original claims on amendment and any new claims. That is, indicate support for newly added claim language by specifically pointing to page(s) and line no(s) in the specification and/or drawing figure(s). This will assist the examiner in prosecuting the application.
When responding to this office action, Applicant is advised to clearly point out the patentable novelty which he or she thinks the claims present, in view of the state of the art disclosed by the reference cited or the objections made. He or she must also show how the amendments avoid such references or objections See 37 CFR 1.111 (c).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RANDY K BALDWIN whose telephone number is (571)270-5222. The examiner can normally be reached on Mon - Fri 9:00-6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RANDALL K. BALDWIN/Primary Examiner, Art Unit 2125
1 Examiner notes that claims 5 and 15 depend from intervening claims 4 and 14, respectively, whereas claims 7 and 17 depend directly from base, independent claims 1 and 11. However, as discussed in the 112(d) rejections of claims 4 and 14 below, claims 4 and 14 fail to further limit claims 1 and 11, upon which these claims respectively depend. As such, claims 7 and 17 are substantial duplicates of claims 5 and 15.
2 As indicated in the objections to claims 6 and 16 above, these claims are duplicates of claims 2 and 12, respectively.
3 As indicated above in the section 112(b) rejections of these claims, “the neural network” has been interpreted as the previously-introduced “neural network model”.
4 As noted above in the objections to these claims, it appears “an input tensor” should recite “[[an]] the input tensor.” As further noted above in the section 112(d) rejections of these claims, these claims fail to further limit the claims upon which they depend.
5 As discussed above in the section 112(b) rejections of these claims “the selected model ID” has been interpreted as “a selected model ID.”
6 As indicated in the objections to claims 6 and 16 above, these claims are duplicates of claims 2 and 12, respectively.
7 As indicated above in the section 112(b) rejections of these claims, “the neural network” has been interpreted as the previously-introduced “neural network model”.
8 As noted above in the objections to these claims, it appears “an input tensor” should recite “[[an]] the input tensor.” As further noted above in the section 112(d) rejections of these claims, these claims fail to further limit the claims upon which they depend.
9 As discussed above in the section 112(b) rejections of these claims “the selected model ID” has been interpreted as “a selected model ID.” Regarding the “model occupancy for the selected model”, paragraph 62 of applicant’s specification states “Model occupancy for selected model ID 520 may provide a listing of one or more PUs (e.g., unit 1 216, unit 2 218, and so on) that are loaded with a neural network model associated with model ID”. Therefore, “model occupancy for the selected model” under the BRI, in view of the specification, is any listing, list, or indication of processing units/devices/nodes that are loaded with a selected/given model.
Prosecution Insights