DETAILED ACTION
Statement of claims
The present application includes:
Claims 10, 12-17, 27 and 29-34 are cancelled.
Claims 1, 18 are pending independent claims and claims 2-9, 11, 19-26, 28 are pending dependent claims in this application.
Claims 1-9, 11, 18-26 and 28 are being considered on the merits.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 09/30/2024. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed on 09/30/2024.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-9, 11, 18-26 and 28 are rejected under 35 U.S.C. 103 as being unpatentable over Lee at al. (US 2023/0289291, Lee hereinafter) in view of Kumar et al. (US 2022/0318603, Kumar hereinafter).
Claim 1, A task scheduler arranged in an artificial intelligence processor (e.g., see FIG. 3, para [0049] Referring to FIG. 3, an example neural processor circuit 218 may include, among other components, neural task manager 310) , wherein the artificial intelligence processor further comprises an executing circuit configured to execute a task ( e.g., “Neural Engine 314”, FIG. 3, para [0050] Each of neural engines 314 performs computing operations for machine learning in parallel”, “ Each of neural engines 314 includes components for storing one or more kernels, for performing multiply-accumulate operations, and for post-processing to generate an output data 328, as described below in detail with reference to FIG. 4. Neural engines 314 may specialize in performing computation heavy operations such as convolution operations and tensor product operations. Convolution operations may include different kinds of convolutions, such as cross-channel convolutions (a convolution that accumulates values from different channels), channel-wise convolutions, and transposed convolutions.
Thus, “computing operations” represents the tasks ) , and the task scheduler comprises:
a first sending circuit (e.g., “314A”, FIG. 3) configured to send a prefetch task of a next task to the executing circuit during an execution of a real task of a current task by the executing circuit (e.g., see para 107 and 108, wherein for “or one or more tasks”, “The task descriptors are sent to the neural task manager 310 of neural processor circuit 218 to set the operations of the neural task manager 310 and other components of neural processor circuit 218.”, “[0108] Neural processor circuit 218, such as via neural task manager 310, may analyze 820 task descriptors to generate prefetch requests based on the task descriptors”, “The second task may be determined to be a high bandwidth task and may be associated with a task descriptor indicating that the second input data associated with the second task is to be prefetched to the cache circuit 240. “, “One or more neural engine circuits 314 of neural processor circuit 218 may also carry out operations according to the task descriptor. The prefetch operation may be scheduled to be carried out in the first set of operating cycles corresponding to the execution of the first task”) ,
wherein a task in the task scheduler is a prefetch task and a real task that are interrelated (e.g., para 108, “The prefetch operation may be scheduled to be carried out in the first set of operating cycles corresponding to the execution of the first task, but the timing of the prefetch operation may depend on various factors such as the availability of cache circuit 240 and the sieve factor associated with the second task. “ for “Neural task manager 310 may control the sequence and execution of various tasks associated with the neural network.” in para 86. Thus, wherein a task in the task scheduler is a prefetch task and a real task that are interrelated); and
a second sending circuit (e.g., “314B”, FIG. 3) configured to send a real task of the next task to the executing circuit after the executing circuit has completed an execution of the prefetch task of the next task (para 49, 53, “neural task manager 310, a plurality of neural engines 314A through 314N (hereinafter collectively referred to as “neural engines 314” and individually also referred to as “neural engine 314”),” , “ tasks in its task queues” , “ tasks in its task queues, choose a task to perform, and send task commands to other components of the neural processor circuit 218 for performing the chosen task” “to complete the prefetch operation when the cache circuit 240 becomes available” in para 113),
so that the executing circuit executes the real task of the next task after the executing circuit has completed the execution of the real task of the current task (e.g., para [0086] The second task may be scheduled for processing in a second set of operating cycles subsequent to the first set of operating cycles, whether the second set of operating cycles is immediately after the first set or there are other tasks between the two sets of operating cycles. Neural task manager 310 may control the sequence and execution of various tasks associated with the neural network
[0112] In some embodiments, while a prefetch request “, “ may only be partially completed when neural processor circuit 218 reaches the second set of operating cycles”, “ prefetching may be dynamically determined based on the availability of cache circuit 240, which may be shared by one or more processing circuits external to the neural processor circuit 218 (e.g., CPU 208, GPU 220, and image signal processor 206) for caching data.” , [0113] the prefetch of the second input data is completed”, “to attempt to complete the prefetch operation when the cache circuit 240 becomes available).
However, Lee does not explicitly teach wherein the task in the task scheduler is split into the prefetch task and the real task .
Kumar teaches a task in the task scheduler (e.g., 322, FIG. 3a, para 30, “, a prefetch controller 322 directs the prefetching of task B 324b from the memory 310.) ” is split into a prefetch task and a real task that are interrelated (e.g., para 31, “The graphical timeline 340 shows tasks completed over a period of time 342 (x-axis).” and “the compute die 420 has fetched task A 424a to deliver it to a compute core 428”, “prefetched task B 424b”, “wait until one task is completed before submitting another task”, “fetching a task while a processor is working on another task and is therefore not yet ready to accept the new task. Fetching a task before a processor is ready to compute the task may be referred to as a prefetch” in para 36-38, see FIG. 4B.
Thus, “another task” and the “new task” represents the real task , coupled with The “prefetched task”, “over a period of time 342 (x-axis)”, therefore wherein the task in the task scheduler is split into the prefetch task and the real task ).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Lee with those of Kumar because both references are directed to related systems addressing similar technical problems within the same field and seek to improve system performance, reliability, and efficiency.
Lee et al. disclose A task scheduler arranged in an artificial intelligence processor, wherein the artificial intelligence processor further comprises an executing circuit configured to execute a task, and the task scheduler comprises: a first sending circuit configured to send a prefetch task of a next task to the executing circuit during an execution of a real task of a current task by the executing circuit while Kumar et al. teach wherein a task in the task scheduler is split into a prefetch task and a real task that are interrelated.
Incorporating the teachings of Kumar et al. into the system of Lee et al. would have been a predictable and logical modification, yielding improved operational robustness and efficiency without requiring undue experimentation.
Such a combination would merely involve the substitution or integration of known elements performing their established functions, as taught by Kumar et al., into the system of Lee et al., consistent with design incentives and market demands for improved performance and scalability. Moreover, Kumar et al. explicitly recognize benefits to enable “the latency for using the AI model may be reduced by using fast low-density memory as buffer segments 120a, 120b, and 120c.” or “to reduce or hide latency” (see Kumar, in para 24 and 29) . —that would naturally be desirable in the system of Lee et al.
Accordingly, to one of ordinary skill in the art would have had a reasonable expectation of success in combining Lee et al. with Kumar et al., and the combination represents no more than the predictable use of prior art elements according to their known functions.
Claim 2 , Lee does not explicitly teach a first receiving circuit configured to receive the task that is split by a program instruction into a prefetch task and a real task that are interrelated; or a splitting circuit configured to split the received task into a prefetch task and a real task that are interrelated.
However, Kumar teaches a first receiving circuit configured to receive the task that is split by a program instruction into a prefetch task and a real task that are interrelated (see para 31 and 32, “The graphical timeline 340 shows tasks “, “the tasks the components perform are shown on the graph” for “The prefetch controller 322 performs prefetch controller work 358 “ for “task A 356a “, “ another task” in para 33); or a splitting circuit configured to split the received task into a prefetch task and a real task that are interrelated ( para 32, “Buffer work 352 may include fetching (or prefetching) and containing the data for task A 352a”, “Core work 356 includes performing task A 356a and performing task B 356b.” ) .
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Lee by adopting the teachings of Kumar to enable “the latency for using the AI model may be reduced by using fast low-density memory as buffer segments 120a, 120b, and 120c.” or “to reduce or hide latency” (see Kumar, in para 24 and 29).
Claim 3, Lee teaches further wherein in sending the prefetch task of the next task to the executing circuit during the execution of the real task of the current task by the executing circuit,
the second sending circuit is further configured to:
send the prefetch task of the next task to the executing circuit at a predetermined time before the execution of the real task of the current task is completed (e.g., para 110, “to perform a prefetch operation until a predetermined period of time has elapsed”, “during the first set of operating cycles corresponding to the first task, a portion of second input data of a second task of the neural network),
so that the executing circuit executes the prefetch task of the next task during the execution of the real task of the current task by the executing circuit (e.g., para 111 and 11, “to prefetch “, “during the first set of operating cycles corresponding to the first task, a portion of second input data of a second task of the neural network. Details of the prefetching operation are illustrated in FIG. 6A and FIG. 7A”).
Claim 4 , Lee teaches further a second receiving circuit (e.g., another one of “Neural engine 314A…314N”, FIG. 3) configured to receive a pre-finish indication of the real task of the current task from the executing circuit, wherein the first sending circuit is configured to send the prefetch task of the next task to the executing circuit in response to receiving the pre-finish indication (e.g., para 89, “to prefetch 614 from system memory 230 the second input data of the second task of the neural network. The second task may be scheduled for the second set of operating cycles that are subsequent to the first set of operating cycles”, “during the second set of operating cycles, the speed of completing the second task may be limited by the bandwidth of system memory 230”),
so that the executing circuit releases hardware resources to execute the prefetch task of the next task (e.g., para[0089]-[0090] determined as high bandwidth data that may cause the second task to be memory bound.” , “,”during the first set of operating cycles and remain unconsumed in cache circuit 240 until the second set of operating cycles.”, “ Since the speed of cache circuit 240 is higher than system memory 230, the overall completion speed of the second task may be increased. The second input data may be sent to one or more neural engine circuits 314 to perform convolution operations on the second input data in the second set of operating cycles” and [0112] , wherein “ while a prefetch request for the second input data of the second task is issued, the prefetch operation “, “completed when neural processor circuit 218 reaches the second set of operating cycles”.
Examiner note : the “the bandwidth of system memory 230.” Represents the “hardware resources” for “the overall completion speed of the second task may be increased” , “during the second set of operating cycles”, “determined as high bandwidth data that may cause the second task to be memory bound” .
Therefore, examiner interpretation of these teaching of the reference cited suggests a second receiving circuit configured to receive a pre-finish indication of the real task of the current task from the executing circuit, wherein the first sending circuit is configured to send the prefetch task of the next task to the executing circuit in response to receiving the pre-finish indication, so that the executing circuit releases hardware resources to execute the prefetch task of the next task).
Claim 5 , Lee teaches a third receiving circuit (e.g., another one of “Neural engine 314A…314N”, FIG. 3)) configured to receive a finish indication of the prefetch task of the next task from the executing circuit (e.g., para [0090] During the second set of operating cycles, the second input data is fetched 616 from cache circuit 240 to neural processor circuit 218 for processing. Since the speed of cache circuit 240 is higher than system memory 230, the overall completion speed of the second task may be increased. The second input data may be sent to one or more neural engine circuits 314 to perform convolution operations on the second input data in the second set of operating cycles.); and
a timer configured to time the execution of the real task of the current task by the executing circuit in response to receiving the finish indication of the prefetch task of the next task from the executing circuit (e.g., para 86, 87 , 110, and 112, wherein “The second task may be scheduled for processing in a second set of operating cycles subsequent to the first set of operating cycles, whether the second set of operating cycles is immediately after the first set or there are other tasks between the two sets of operating cycles. Neural task manager 310 may control the sequence and execution of various tasks associated with the neural network”, “ a prefetch operation until a predetermined period of time” “while a prefetch request for the second input data of the second task is issued, the prefetch operation may not be completed or may only be partially completed when neural processor circuit 218 reaches the second set of operating cycles”, “the prefetch operation is limited. In the case where the prefetch operation is not started at the beginning of the second set of operating cycles” , “ the prefetch may be partially completed based on the sieve factor described in the task descriptor”.
Examiner note that the “operating cycles” coupled with “a prefetch operation until a predetermined period of time” include the timer for “the sequence and execution of various tasks “ .
Therefore , theses teaching of Lee suggest , to receive a finish indication of the prefetch task of the next task from the executing circuit; and a timer configured to time the execution of the real task of the current task by the executing circuit in response to receiving the finish indication of the prefetch task of the next task from the executing circuit).
Claim 6 , Lee teaches a fourth receiving circuit (e.g., another one of “Neural engine 314A…314N”, FIG. 3)) configured to receive an unfinished indication used to indicate that the execution of the real task has not been completed from the executing circuit (e.g., para 108, “The prefetch operation may be scheduled to be carried out in the first set of operating cycles corresponding to the execution of the first task, but the timing of the prefetch operation may depend on various factors such as the availability of cache circuit 240 and the sieve factor associated with the second task. In some cases, due to the unavailability of cache circuit 240, the prefetch operation may be delayed. If the prefetch operation is not able to be carried out until the second set of operating cycles that are scheduled for the execution of the second task, the prefetch operation may be canceled and the second input data may be fetched directly from system memory 230”) ,
wherein the first sending circuit is configured to send the prefetch task of the next task to the executing circuit (e.g., para 111, “a prefetch operation may occur. For example, cache access circuit 360 may instruct 870 the cache circuit 240 to prefetch from system memory 230, during the first set of operating cycles corresponding to the first task, a portion of second input data of a second task of the neural network. Details of the prefetching operation are illustrated in FIG. 6A and FIG. 7A.”) or to another executing circuit in response to receiving the unfinished indication (e.g., para 111 and 112, wherein “during the first set of operating cycles corresponding to the first task, a portion of second input data of a second task of the neural network”, “while a prefetch request for the second input data of the second task is issued, the prefetch operation may not be completed or may only be partially completed when neural processor circuit 218 reaches the second set of operating cycles. For example, the prefetch operation may be delayed due to the telemetry data indicating that cache circuit 240 is unavailable or the bandwidth of the cache circuit 240 for the prefetch operation is limited. In the case where the prefetch operation is not started at the beginning of the second set of operating cycles, the second input data may be fetched from system memory 230.” ) .
Claim 7, Lee teaches wherein the first sending circuit is further configured to send the prefetch task of the next task to the executing circuit or another executing circuit in response to a case where the timing of the timer exceeds a preset threshold and no indication is received from the executing circuit (e.g., para 110, “ instructing cache circuit 240 to perform a prefetch operation until a predetermined period of time has elapsed” for the “[0110] Neural processor circuit 218, such as via cache access circuit 360, may receive 850 telemetry data indicating whether cache circuit 240 is available. “ and “for neural processor circuit 230 exceeds a threshold and the rate of operation associated with the computation of the input data may be bound by the bandwidth of system memory 230. The tasks associated with those input data may be referred to as high bandwidth tasks that are illustrated in FIG. 6A through FIG. 7B. For illustration purposes, the second task is described here as a high bandwidth task.” In para 106)
.
Claim 8, Lee teaches wherein in sending the prefetch task of the next task to the executing circuit or another executing circuit, the first sending circuit is further configured to: place the prefetch task of the next task into a priority sending queue, so that the prefetch task of the next task is re-sent to the executing circuit or another executing circuit with the highest sending permission (e.g., para 53, wherein “ Neural task manager 310 may receive a task list from a compiler executed by CPU 208, store tasks in its task queues, choose a task to perform, and send task commands to other components of the neural processor circuit 218 for performing the chosen task”, “Neural task manager 310 may also perform switching of tasks on detection of events such as receiving instructions from CPU 208” for “circuit 610 during the operating cycles corresponding to the low bandwidth task and the high bandwidth task.” In para 91.
Thus, wherein in sending the prefetch task of the next task to the executing circuit or another executing circuit, the first sending circuit is further configured to: place the prefetch task of the next task into a priority sending queue, so that the prefetch task of the next task is re-sent to the executing circuit or another executing circuit with the highest sending permission).
Claim 9 , Lee teaches a recording circuit configured to record an error that occurs during the execution of the prefetch tasks and an error reporting circuit configured to report the error when a real task that is interrelated with the prefetch task is executed (e.g., para 108, and 112, wherein “The prefetch operation may be scheduled to be carried out in the first set of operating cycles corresponding to the execution of the first task, but the timing of the prefetch operation may depend on various factors such as the availability of cache circuit 240 and the sieve factor associated with the second task. In some cases, due to the unavailability of cache circuit 240, the prefetch operation may be delayed. If the prefetch operation is not able to be carried out until the second set of operating cycles that are scheduled for the execution of the second task, the prefetch operation may be canceled”, “ the prefetch operation may be delayed due to the telemetry data indicating that cache circuit 240 is unavailable or the bandwidth of the cache circuit 240 for the prefetch operation is limited” for the “Neural processor circuit 218”.
The examiner notes that “ the prefetch operation may be delayed”, coupled with “the prefetch operation is not able to be carried out” suggest a recording circuit configured to record an error that occurs during the execution of the prefetch tasks and an error reporting circuit configured to report the error when a real task that is interrelated with the prefetch task is executed) .
Claim 11, Lee teaches wherein the executing circuit comprises a plurality of intelligent processing unit (IPU) cores for executing tasks in parallel (e.g., para [0070] Input data is typically split into smaller pieces of data for parallel processing at multiple neural engines 314 or neural engines 314 and planar engine 340.), wherein the task is split into a plurality of sub-tasks and each sub-task is executed by a corresponding IPU core, and the task scheduler is further configured to: interact with the plurality of IPU cores, so that the plurality of IPU cores execute prefetch sub-tasks and real sub-tasks of corresponding sub-tasks in parallel (e.g., para [0070] Input data is typically split into smaller pieces of data for parallel processing at multiple neural engines 314 or neural engines 314 and planar engine 340. A set of data used for a convolution operation may be referred to as a convolution group, which can be split into multiple smaller units. The hierarchy of smaller units (portions of data) may be convolution groups, slices, tiles, work units, output channel groups, input channels (Cin), sub-Cins for input stride, etc. For example, a convolution group may be split into several slices; a slice may be split into several tiles; a tile may be split into several work units; and so forth. In the context of neural engine 314, a work unit may be a portion of the input data, such as data processed by planar engine 340 or data processed a prior cycle of neural engines 314 having a size that produces output values that fit into accumulator 414 of neural engine 314 during a single cycle of the computation core 416. “
Thus, the “work units” coupled with “data” for parallel processing at multiple neural engines 314 include the sub-tasks for parallel processing at multiple neural engines 314 , therefore wherein the executing circuit comprises a plurality of intelligent processing unit (IPU) cores for executing tasks in parallel, wherein the task is split into a plurality of sub-tasks and each sub-task is executed by a corresponding IPU core, and the task scheduler is further configured to: interact with the plurality of IPU cores, so that the plurality of IPU cores execute prefetch sub-tasks and real sub-tasks of corresponding sub-tasks in parallel).
As to Claim 18, see rejection of claim 1 above.
As to claims 19 -26 and 28 , see rejection of claims 2-9 and 11 above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Lizawa (US 2021/0034415) discloses An information processing device includes; a plurality of threads, each of the plurality of threads being configured to process any of a plurality of tasks, the plurality of tasks being obtained by dividing a job; and a control circuit configured to execute processing when designating a next task in scheduling for the plurality of threads, the processing including inquiring of an assignment destination thread out of the plurality of threads as to whether the next task is to be completed by a scheduled time, and preferentially assigning a task supposed to be completed by the scheduled time in the assignment destination thread, as the next task from among the plurality of tasks.
Wang et al. (US 20210073169) discloses to provide an on-chip heterogeneous Artificial Intelligence (AI) processor comprising at least two different architectural types of computation units, wherein each of the computation units is associated with a respective task queue configured to store computation subtasks to be executed by the computation unit. The AI processor also comprises a controller configured to partition a received computation graph associated with a neural network into a plurality of computation subtasks according to a preset scheduling strategy and distribute the computation subtasks to the task queues of the computation units. The AI processor further comprises a storage unit configured to store data required by the computation units to execute their respective computation subtasks and an access interface configured to access an off-chip memory. Different application tasks are processed by managing and scheduling the different architectural types of computation units in an on-chip heterogeneous manner.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ABDOU K SEYE whose telephone number is (571)270-1062. The examiner can normally be reached M-F 9-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre Vital can be reached at 5712724215. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ABDOU K SEYE/Examiner, Art Unit 2198
/PIERRE VITAL/Supervisory Patent Examiner, Art Unit 2198
Prosecution Insights