1\7Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on August 19, 2024, December 10, 2024, April 3, 2025 and August 14, 2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 12 recites the limitation "the inference output" in line 3. There is insufficient antecedent basis for this limitation in the claim. For purposes of clarity, it will be understood as “the first inference output” when examined.
Claim Rejections - 35 USC § 101
Claims 18-20 are rejected under 35 U.S.C. 101 because the claimed invention is not directed to a
statutory category of invention. All cite signals per se (computer-readable storage media) with no structural limitations and thus fail Step 1 of the eligibility analysis.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 10, 12-14, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Fang et al.
(“Serving Deep Neural Networks at the Cloud Edge for Vision Applications on Mobile Platforms”, 2019), in view of Croxford et al. (“US 20200184320”).
Regarding claim 1, Fang teaches a method performed by a system including a host and one or more hardware processing units configured to perform inference operations of a plurality of machine learning models (Page 36, Abstract, “We present DeepQuery, a new mobile offloading system that is capable to serve DNNs… on shared GPUs”, Page 40, Figure 3) comprising:
receiving, at the host, data representing a first machine learning model (Page 40, Figure 3), wherein the first machine learning model comprises inference operations for processing an input output (Page 37, Column 1, Paragraph 4, “DeepQuery… employs queries to process data using DNNs”) to generate a first inference output (Page 39, Column 2, Paragraph 1, “The output are the bounding boxes of tracked objects and the tracking states with updated target feature and location”);
obtaining a first estimated duration for the system to perform the inference operations of the first machine learning model for processing the input to generate the first inference output (Page 43, Column 1, Paragraph 1, “Dmb is the delay for running the DNN model m with a batchsize b”1);
identifying a priority time period reserved for performing priority inference operations (Page 41, Column 1, Paragraph 1, “Streaming RT queries are generated from continuous delay-critical mobile vision applications, which are considered as the primary workloads in many mobile offloading frameworks [11, 16]”) of a priority machine learning model (Page 43, Column 1, Paragraph 4, “For example, in Figure 1, object tracking and scene graph detecting tasks depend on the output of object detection”, Page 43, Column 2, Paragraph 1, “for instance, the Faster Region-based CNN (FRCNN) [40], a popular object detection model”2) during each occurrence of a recurring time window3 (Page 37, Column 1, Paragraph 2, “These applications would contain real-time (RT) work loads, such as object tracking in a live video… Particularly, cognitive tasks would have stringent timing requirements and usually target at sub-second end-to end delays to provide similar experiences to tasks conducted by humans [16]. For example, a previous study shows that on average it takes a human 370ms to recognize familiar faces and 620ms to recognize unfamiliar ones [39]. As a result, we would need a system that is capable of scheduling workloads with different timing requirements”) in which the one or more hardware processing units perform at least a portion of the inference operations of the plurality of machine learning models (Page 36, Figure 2)
determining a first remaining time period of each occurrence of the recurring time window that remains after reserving the priority time period for performing the priority inference operations (Page 44, Column 1, Paragraph 2, “this work solves the preemption problem by predicting future RT jobs and accordingly limiting the time budget for running non-RT jobs, in order to have them end before RT jobs start… The time budget for run a new non-RT job is given as Dbudget = tstartk −tnow if k ≤ Nquery”);
determining whether the first estimated duration is greater than the first remaining time period (Page 44, Column 1, Paragraph 2, “The scheduler then reduces the batch size bk to satisfy Dbk ≤ DBudget”4);
in response to determining that the first estimated duration is greater than the first remaining time period [reducing the batch bk]5; and
performing, by the one or more hardware processing units, inference operations of [the first machine learning model with he reduced batch size bk] (Page 44, Column 1, Paragraph 2, “…and then runs the non-RT job”)
Fang fails to teach the partitioning the first machine learning model into a first group of sub- models that have respective estimated durations that are less than or equal to the first remaining time period, wherein each sub-model of the first group of sub-models includes a respective portion of the inference operations of the first machine learning model and performing, by the one or more hardware processing units, inference operations of a sub-model in the first group of sub-models during the first remaining time period of an occurrence of the recurring time window.
However, Croxford teaches the partitioning a first machine learning model into a first group of sub-models (Paragraph 65, “that it is possible to assign portions corresponding to one or more (entire) layers of neural network”) that have respective estimated durations that are less than or equal to the first remaining time period (Paragraph 75, “The compute capability may correspond to… an amount of computation that the processor is capable of performing within a specified period of time”, Paragraph 100, “With regards to compute capability, in an embodiment, a processor may be assigned a part of a layer which is less than an entire layer when it is determined that the processor cannot perform the computations required for the layer of processing at a particular (predetermined) rate or within a particular (predetermined) time period”), wherein each sub-model of the first group of sub-models includes a respective portion of the inference operations of the first machine learning model (Paragraph 41, “performing the neural network processing by the two or more of the processors, wherein each of the processors performs neural network processing according to the portion of the neural network processing which it has been assigned, to thereby provide a neural network processing output”).
Fang and Croxford are considered analogous to the invention because all are directed towards
machine learning systems. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified Fang to incorporate the teachings of Croxford, and partition the model into multiple sub-models for further processing. Doing so can improve the efficiency of neural network processing (see Paragraphs 51-52 of Croxford).6
Regarding claim 2, Fang fails to teach the further limitations of the claim. However, Croxford teaches generating instructions at the host for assigning each of the first group of sub-models to respective hardware processing units of the one or more hardware processing units (see claim 1 analysis) , and performing, according to the instructions and a sequence of the first group of sub- models arranged in the first machine learning model, respective inference operations of the first group of sub-models for processing the input on the assigned hardware processing units (Paragraph 120, “different layers are assigned to different processors. In such an embodiment, the different layers may be processed in sequence such that the output from one layer of processing performed by a processor is used as an input by a next processor for a next layer of processing”).
Fang and Croxford are considered analogous to the invention because all are directed towards
machine learning systems. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified Fang to incorporate the teachings of Croxford, and perform inference operations for the respective sub-models. Doing so is a logical implementation for working with models partitioned into sub-models, which, as laid out in the claim 1 analysis, can improve the efficiency of neural network processing.
Regarding claim 3, Fang fails to teach the further limitations of the claim. However, Croxford teaches wherein performing the respective inference operations of the first group of sub-models further comprises: scheduling, by the host, the one or more hardware processing units to perform the respective inference operations of the first group of sub-models each assigned to corresponding hardware processing units to generate the first inference output. (Figure 12A-12C)
Fang and Croxford are considered analogous to the invention because all are directed towards machine learning systems. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified Fang to incorporate the teachings of Croxford, and scheduling how the inference operations for the respective sub-models are processed. Doing so is a logical implementation for working with models partitioned into sub-models, which, as laid out in the claim 1 analysis, can improve the efficiency of neural network processing.
Regarding claim 10, Fang teaches the use of performing inferences operations associated with a first [batch of the first group of batches] according to the sequence in the first remaining time period of a first recurring time window (see claim 1 analysis) and performing inferences operations with a second [batch of the first group of batches] that succeeds the first [batch] according to the sequence in the first remaining time period (see claim 1 analysis) of a second recurring time window of the plurality of recurring time windows (Page 42, Column 1, Paragraph 1, “As shown in Figure 4, runnable jobs are buffered in a ready job queue, and each job runner fetches jobs from the queue to execute. A job runner executes one job at a time, which means the GPU device is exclusively used by a model. RT jobs are prioritized in the ready queue”; Page 42, Figure 4)7.
Fang fails to teach the use of sub-models. However, Croxford teaches the use of sub-models (see claim 1 analysis).
Fang and Croxford are considered analogous to the invention because all are directed towards
machine learning systems. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified Fang to incorporate the teachings of Croxford, and partition the model into multiple sub-models for further processing. Doing so can improve the efficiency of neural network processing (see Paragraphs 51-52 of Croxford).8
Regarding claim 12, Fang teaches wherein the input comprises a sequence of inputs including a first input and a second input received at the host according to an order (Page 36, Abstract, “…processes live video streams using DNNs on server-class GPU accelerators”). Fang fails to teach wherein generating the inference output further comprises:
performing inference operations associated with a first sub-model according to the sequence for processing the first input to generate a first intermediate output;
providing the first intermediate output as a first intermediate input, through a fill layer for a second sub-model, to the second sub-model that succeeds the first sub-model according to the sequence; and
performing inference operations associated with the first sub-model according to the sequence for processing the second input to generate a second intermediate output, meanwhile, performing inference operations associated with the second sub-model for processing the first intermediate input.
However, Croxford teaches performing inference operations associated with a first sub-model according to the sequence for processing the first input to generate a first intermediate output (see claim 11 analysis);
providing the first intermediate output as a first intermediate input, through a fill layer for a second sub-model, to the second sub-model that succeeds the first sub-model according to the sequence (see claim 11 analysis); and
performing inference operations associated with the first sub-model according to the sequence for processing the second input to generate a second intermediate output, meanwhile, performing inference operations associated with the second sub-model for processing the first intermediate input (Paragraph 120, “different layers may be processed in sequence such that the output from one layer of processing performed by a processor is used as an input by a next processor for a next layer of processing (i.e. a “pipelined” arrangement). Once a processor has completed its assigned layer of processing, it will be free to perform other processing. Hence, the data processing system can be used to perform neural network processing for plural input data arrays (e.g. plural image frames) simultaneously, such that a processor may begin layer processing for a next input data array (e.g. from a next frame of video data) once it has completed layer processing for a previous input data array (e.g. previous frame of video data).”; Figure 13)
Fang and Croxford are considered analogous to the invention because all are directed towards machine learning systems. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified Fang to incorporate the teachings of Croxford, and work with sequential input within the sub-model system. Doing so is a logical implementation for working with models partitioned into sub-models, which, as laid out in the claim 1 analysis, can improve the efficiency of neural network processing.
Regarding claim 13, Fang fails to teach the further limitations of the claim. However, Croxford teaches wherein generating the first inference output further comprises:
storing an intermediate output generated by a sub-model of the first group of sub- models for processing the input at a memory unit in the system; and
retrieving, from the memory unit in the system, the intermediate output as an intermediate input for another sub-model that succeeds the sub-model according to the sequence (Paragraph 176-179, “This passing of data between processors that are processing consecutive layers may be referred to as “pipelining” of the processing… in embodiments, data may be exchanged (transferred) between processors. The data may be transferred directly between the processors (e.g. via a direct interface), or the data may be transferred between processors via a storage (e.g. a local, on-chip storage, such as shared buffer which is accessible to plural processors). The processors (and storage, if used) may exchange signals as a “handshake” to control data transfer”, Figure 17B).
Fang and Croxford are considered analogous to the invention because all are directed towards machine learning systems. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified Fang to incorporate the teachings of Croxford, and store the output of a preceding sub-model in memory before inputting it into a succeeding sub-model. Doing so provides a safer method of data transfer than a direct interface.
Claim 14 is a system claim corresponding to method claim 1 and is rejected for the same reasons as given in the rejection of that claim.
Claim 18 is a computer readable storage media claim corresponding to method claim 1 and is rejected for the same reasons as given in the rejection of that claim.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Fang et al. (“Serving Deep Neural Networks at the Cloud Edge for Vision Applications on Mobile Platforms”, 2019), in view of Croxford et al. (“US 20200184320”) and in further view of Joshi et al. (“US 20130041706”).
Regarding claim 4, Fang fails to teach the further limitations of the claim. However, Joshi teaches compiling, by a compiler of the host, the first group of sub-models and deploying each of the compiled sub-models to the one or more hardware processing units (Paragraph 66, “The model executor 134 further compiles the one or more sub-models before processing and sending the compiled one or more sub-models to the optimization engine (not shown) via a parallelization engine (not shown) for processing”)
Fang, Croxford, and Joshi are considered analogous to the invention because all are directed towards machine learning systems. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified Fang to incorporate the teachings of Joshi, and compiling and deploying the sub-models. Doing so better prepares the model for faster execution.
Claims 5, 7, 15, 17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Fang et al.
(“Serving Deep Neural Networks at the Cloud Edge for Vision Applications on Mobile Platforms”, 2019), in view of Croxford et al. (“US 20200184320”) and in further view of Yuan et al. (“Comprehensive and Efficient Data Labeling via Adaptive Model Scheduling”, 2020).
Regarding claim 5, Fang teaches wherein receiving data representing the first machine learning model further comprises: receiving data representing a plurality of machine learning models (Page 40, Figure 3), each of the plurality of machine learning models being configured to implement a respective task (Page 40, Section 3.49, “A feature extractor has the same architecture as a CNN based object classifier [24, 28, 53]. The difference is that a feature extractor removes the last fully connected classification layer and outputs a feature vector instead of a classification label… A scene graph detection task (SG)… We consider the model proposed in [47]… A video description task generates a sentence in natural language from a video clip. We adopt the sequence to sequence model[44]”) and including respective inference operations to be performed by the system for processing the input.
Fang fails to teach measuring a respective level of priority for each of the plurality of machine learning models based on characteristics of the respective tasks; and selecting, as the first machine learning model, one machine learning model from the plurality of machine learning models based on the respective levels of priority.
However, Yuan teaches measuring a respective measuring a respective level of priority for each of the plurality of machine learning models based on characteristics of the respective tasks (Page 5, Column 1, Paragraph 1, “considering the different requirements of diverse applications for model priorities, we introduce a parameter θm as the user defined priority for a model m”); and
selecting, as the first machine learning model, one machine learning model from the plurality of machine learning models based on the respective levels of priority (Page 9, Column 2, Paragraph 3, “To study the effect of θ, we train DRL agents by setting the parameter of one “face detection model” to 1.0, 2.0, 5.0 and 10.0 to increase its priority, while keeping θ of other models as 1.0”, Page 10, Figure 9a10).
Fang and Yuan are considered analogous to the invention because all are directed towards machine learning model scheduling systems. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified Fang to incorporate the teachings of Yuan, and measure and select models based on a measured level of priority. Doing so allows you to select more important and impactful model tasks when computing resources are limited (see E. Model Priority Section of Yuan).
Regarding claim 7, Fang teaches wherein the input includes an image frame of a plurality of image frames captured by a sensor (Page 37, Figure 1);
wherein each occurrence of the recurring time window corresponds to the image frame of the plurality of image frames (Page 37, Figure 111);
wherein the respective tasks includes at least one of background detection, focal point detection, object detection, or human face recognition (Page 37, Figure 1); and
wherein the characteristics of the respective tasks include at least a dependency of the respective tasks and respective estimated durations for performing the respective tasks by the one or more processing units in the system (Page 39, Figure 2).
Claim 15 is a system claim corresponding to method claim 5 and is rejected for the same reasons as given in the rejection of that claim.
Claim 17 is a system claim corresponding to method claim 7 and is rejected for the same reasons as given in the rejection of that claim.
Claim 19 is a computer readable storage media claim corresponding to method claim 5 and is rejected for the same reasons as given in the rejection of that claim.
Claims 6, 16, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Fang et al. (“Serving Deep Neural Networks at the Cloud Edge for Vision Applications on Mobile Platforms”, 2019), in view of Croxford et al. (“US 20200184320”) and in further view of Pasko et al. (“US 12437204”, 202112).
Regarding claim 6, the limitations of the claim essentially apply the method as taught in claim 1 to a second machine learning model. The method as applied to a first machine learning model is taught by Fang and Croxford (see claim 1 analysis). Fang fails to explicitly teach the application to a second machine learning model.
However, Pasko teaches a system of scheduling neural network tasks (Paragraph 17, “This disclosure describes systems and methods for scheduling neural inference tasks”), in which a scheduler can assign tasks to a second machine learning model (Paragraph 19, “The scheduler may inspect such a neural task to determine, for example, that the task involves using a first model to process an input to generate intermediate data, and using a second model to process the intermediate data to generate an output”).13
Fang and Pasko are considered analogous to the invention because all are directed towards machine learning model scheduling systems that aim at maximizing throughput. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified Fang to incorporate the teachings of Pasko, and applied the scheduling system of identifying priority times and partitioning a first machine learning model, to a second machine learning model. Doing so allows the system to be applied for a wider array of situations.
Claim 16 is a system claim corresponding to method claim 6 and is rejected for the same reasons as given in the rejection of that claim.
Claim 20 is a computer readable storage media claim corresponding to method claim 6 and is rejected for the same reasons as given in the rejection of that claim.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Fang et al. (“Serving Deep Neural Networks at the Cloud Edge for Vision Applications on Mobile Platforms”, 2019), in view of Croxford et al. (“US 20200184320”) and in further view of Ren et al. (‘US 20210099737”14).
Regarding claim 8, Fang teaches wherein the system is configured to perform inference operations of one or more machine learning models for processing a sequence of inputs (see claim 7 analysis). Fang fails to teach wherein each of the sequence of inputs is received at the host according to an order at a particular frequency, and wherein a time period of the recurring time window is determined based on the particular frequency.
However, Ren teaches wherein each of the sequence of inputs is received at the host according to an order at a particular frequency, and wherein a time period of the recurring time window is determined based on the particular frequency (Paragraph 96, “In the embodiment, the check interval and the total number of checks are also determined by the image frame rate of the video stream to be checked and the data processing performance parameters of the current device. It can be understood that the more times the second positioning results are checked, the more time it takes to complete the object positioning method in the present embodiment, and the greater the amount of computation is. Therefore, if the data processing performance of the current device is good and the image frame rate of the video stream to be detected is not very high, the check interval may be set relatively small, and the total number of checks may be set relatively large; if the data processing performance of the current device is average or the image frame rate of the video stream to be detected is high, it is not advisable to set the total number of checks too large. Otherwise, frame drop can easily occur, and the check interval should be set relatively large.”)
Fang and Ren are considered analogous to the invention because all are directed towards machine learning models that process video streaming data. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified Fang to incorporate the teachings of Yuan, and adjust the recurring time period based on the frequency of sequential inputs ingested. Doing so allows sample time to process the inputs, and reduces likelihood of latency, lag and frame rate drop (see Paragraph 96 of Ren).
Claims 9, 11 are rejected under 35 U.S.C. 103 as being unpatentable over Fang et al. (“Serving Deep Neural Networks at the Cloud Edge for Vision Applications on Mobile Platforms”, 2019), in view of Croxford et al. (“US 20200184320”) and in further view of Cai et al. (‘ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware”, 2019).
Regarding claim 9, Fang teaches wherein the first machine learning model comprises a neural network including multiple network layers arranged in a sequence (see claim 1 analysis15). Fang fails to teach wherein obtaining the first estimated duration comprises:
for each layer of the network layers, determining a respective estimated layer duration for the system to perform respective layer operations specified in the layer for processing the input; and
aggregating the respective estimated layer durations for all network layers to obtain the first estimated duration.
However, Cai teaches for each layer of the network layers, determining a respective estimated layer duration for the system to perform respective layer operations specified in the layer for processing the input; and
aggregating the respective estimated layer durations for all network layers to obtain the first estimated duration (Page 5, Paragraph 7, “For the whole network with a sequence of mixed operations (Figure 3 left), since these operations are executed sequentially during inference, the expected latency of the network can be expressed with the sum of these mixed operations’ expected latencies: …”, Page 5, Equation 6).
Fang and Cai are considered analogous to the invention because all are directed towards machine learning models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified Fang to incorporate the teachings of Yuan and calculate the duration time of a machine learning model layer-by-layer, and then aggregate it. Doing so gives one more granular data to understand the latency of the neural network as a whole.
Regarding claim 11, Fang fails to teach the further limitations of the claim. However, Croxford teaches wherein partitioning the first machine learning model comprising the neural network further comprises:
partitioning the neural network into the first group of sub-models (see claim 1 analysis), each including a respective number of network layers arranged according to the sequence (Paragraph 136, “…performing layers of the neural network processing by the processors to which they have been assigned. The layers of processing may be performed in sequence…”); and
determining a respective fill layer for each sub-model of the first group of sub-models except for the first sub-model so that an intermediate output generated from another sub- model preceding the sub-model is provided to the sub-model as an input by the respective fill layer;
wherein the respective fill layers each are the first layer of network layers included in a corresponding sub-model of the first group of sub-models (Paragraphs 136-137, “…such that an output from a layer of processing is used as an input for a subsequent layer of processing. In the case that a processor is assigned an entire layer to process, the processor may access input data (e.g. an (entire) input feature map) needed to perform the layer of processing, and may generate corresponding layer output data (e.g. an (entire) output feature map) when performing the layer of processing. The layer output data may be made accessible to subsequent processor(s) that have been assigned a subsequent layer to process.”).16
Fang and Croxford are considered analogous to the invention because all are directed towards
machine learning systems. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified Fang to incorporate the teachings of Croxford, and have the sub-model outputs be inputs to succeeding sub-models. Doing so is a logical implementation for working with models partitioned into sub-models, which, as laid out in the claim 1 analysis, can improve the efficiency of neural network processing.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEWOS MESFIN whose telephone number is (571)270-0782. The examiner can normally be reached Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571) 272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEWOS MESFIN/Examiner, Art Unit 2145
/CESAR B PAULA/Supervisory Patent Examiner, Art Unit 2145
1 The section Estimating DNN Delays on the following column explains the exact procedure for how the delay value is estimated
2 Object detection is a priority task (has dependent actions), and FRCNN is the model used for it, thus a priority model
3 STREAMING RT queries are generated from continuous digital input, and that input can be classified as a recurring time window, where the window repeats for every streaming frame input (which corresponds to the frame rate).
4 Means it calculates the duration of the non-RT job
5 See reference in above limitation
6 It should be noted that both portioning input into batches and partitioning models into sub-models are two different attempts at solving for the same thing (latency/delay reduction). Thus, substituting one for the other would have been obvious.
7 In a queue with both Streaming RT tasks and a Non-RT Batched query, the streaming tasks are prioritized. Thus, it is implied that a batch from a task in one time window can run in a succeeding (or second) time window.
8 See footnote 6
9 This section lists out 3 subtasks used in the system, and the models used for them, which it borrows from existing references, indicated by the numbers in brackets
10 Adjusting the value of theta allows the system to select the face detection model first
11 The recurring time window is indicated by a RT query (object detection)
12 US Patent with filing date of June 29, 2021, and thus earlier than this application’s earliest filing date
13 The earlier references fail to explicitly show the method applied to a second model. The Pasko reference is evidence that a neural network scheduling system having a second model is well-known in the art and would’ve been an obvious option.
14 US PGPUB with publishing date of June 29, 2021, and thus earlier than this application’s earliest filing date
15 Fang teaches the use of DNNs, which, by definition, are a type of neural network where multiple layers are arranged in a sequence.
16 Fill layer is implied in this structure
Prosecution Insights