DETAILED ACTION
Claims 1-20 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6, 8-14, and 16-19 are rejected under 35 U.S.C. 103 as being unpatentable over Nandan et al. (US 2021/0209041) in view of Mishra et al. (US 2016/0283240 A1).
Regarding claim 1, Nandan teaches the invention substantially as claimed including an integrated circuit ([0025] FIG. 2 is an example diagram 200 of a hardware acceleration subsystem 210 integrated into a System on a Chip (SoC) 220; [0061] the example hardware acceleration subsystem 310 of FIG. 3 could be implemented by one or more analog or digital circuit(s), logic circuits, programmable processor(s), programmable controller(s), graphics processing unit(s) (GPU(s)), digital signal processor(s) (DSP(s)), application specific integrated circuit(s) (ASIC(s)), programmable logic device(s) (PLD(s)) and/or field programmable logic device(s) (FPLD(s)).) comprising:
a set of hardware accelerators each configured to perform a respective task ([0019] Typically, hardware accelerators are configured to perform processing tasks on data elements in the form of blocks or lines. For example, in image processing where imaging/vision algorithms are often two-dimensional (2D) block-based, the hardware accelerators may be configured to process two-dimensional blocks from an image frame rather than processing the entire image frame as lines.; [0020] the hardware accelerator is implemented on a System on a Chip (SoC); [0023] Multiple hardware accelerators can be integrated into a hardware acceleration sub-system to form a hardware acceleration chain; [0027] In the example illustrated in FIG. 2, the hardware accelerators 250a-d (i.e., a set) are configured to consume data elements 231, 232 and/or intermediate data elements 233, 234 as input, perform a processing task on the data elements 231, 232 and/or intermediate data elements 233, 234, and produce the processed data elements 236, 238 as output to be consumed by another hardware accelerator 250a-d, (i.e., respective task));
a hardware accelerator thread scheduler coupled to the set of hardware accelerators (Fig. 2, Scheduler 280 is shown coupled to the set of hardware accelerators 250a-d)and configured to:
schedule execution of a plurality of pipelines, wherein each pipeline of the plurality of pipelines defines a series of tasks performed by one or more hardware accelerators of the set of hardware accelerators to complete a process, and wherein a first pipeline of the plurality of pipelines includes a hardware enable flag configuration setting that allows initiation of the first pipeline based on completion of a second pipeline of the plurality of pipelines ([0024]; [0034]; [0037] Further, the example hardware acceleration subsystem 310 may include different types of example hardware accelerators 350a-c and/or example hardware accelerators 350a-c that operate on different types of data, e.g., block or line, and/or perform different processing tasks, e.g., LDC, scaling, and noise filtering, thus allowing the user to customize the example hardware accelerator subsystem 310 for a variety of functions.; [0047] In some examples, the example first hardware accelerator 350a generates a done signal (e.g., a Tdone signal) in response to the example first hardware accelerator 350a completing the processing of an example data element 402, 404, 406, 408 (FIG. 4) and sends the Tdone signal to the example first scheduler 382a (FIG. 3). In some examples, in response to receiving a Tdone signal, the example first scheduler 382a instructs the example second hardware accelerator 350b to read the processed data element 402, 404, 406, 408 (FIG. 4) or aggregated data element 420a, 420b or 430a or 430b; [0048]; [0058] In the example hardware acceleration subsystem 710 of FIG. 7, the example LDC hardware accelerator 750 is configured to perform lens distortion correction operations on data to produce data elements 732. In the example hardware acceleration subsystem 710, the example MSC/NF hardware accelerator 750b is configured to consume the data elements 732, perform scaling and noise filtering operations on the data elements 732, and produce the data elements 734.; wherein the LDC, MSC and NF operations correspond to the claimed pipeline/series of tasks));
detect an end of pipeline event indicating completion of the second pipeline ([0038]; [0054] In some examples, the example scheduler, e.g., the example first scheduler 382a of FIG. 3, reads a Tdone signal from the example first hardware accelerator 350a and, in response, the example pattern adapter 588 logically converts the 24-line data element to a 24×32 B data block and the example producer socket 586 produces the processed data block as output data for consumption by another hardware accelerator); and
in response to the end of pipeline event indicating completion of the second pipeline and the hardware enable flag configuration setting in the first pipeline, initiate execution of the first pipeline ([0016]; [0038]; [0047] In some examples, the example first hardware accelerator 350a generates a done signal (e.g., a Tdone signal) in response to the example first hardware accelerator 350a completing the processing of an example data element 402, 404, 406, 408 (FIG. 4) and sends the Tdone signal to the example first scheduler 382a (FIG. 3).; [0048] the example first hardware accelerator 350a sends a Tdone signal to the example first scheduler 382a, at which point the example first scheduler 3 82a may instruct the example second hardware accelerator 350b or the example third hardware accelerator 350c to read the aggregated data element 420a, 420b from the example local memory 360.; [0054]; [0057] In the example of FIG. 6, the example MSC hardware accelerator 650b consumes a set of data lines based on the output of the example LDC hardware accelerator 650a, performs a scaling operation on the data lines, and produces the data line element 636 which is consumed by the example NF hardware accelerator 650c and the example third DMA controller 644.; [0058] pipeline/series of tasks).
While Nandan teaches performing operations in a hardware accelerator and upon completion notifying using a Tdone signal which allows for subsequent processing in a different hardware accelerator, Nandan does not explicitly teaches the signal being a hardware enable flag configuration setting.
However, Mishra teaches a hardware enable flag configuration setting (Fig. 3; [0047] In certain embodiments, on receipt by the accelerator complex 300 of the schema (e.g., prepared by the compiler and embedded in the application binary), the accelerator controller (e.g., control logic) and scheduler 304 in the accelerator complex prepare a set of virtual accelerator threads (VATs) to be scheduled on the hardware accelerator circuit(s) (e.g., accelerator threads (ATs). These tasks may be queued into an accelerator work queue from which each of the hardware accelerators may pull work and notify completion with a done flag. This is schematically shown in FIG. 3.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Mishra of utilizing a flag to notify completion with the Tdone signal as taught by Nandan. The modification would have been motivated by the desire of performing a simple substitution of a flag for a signal as both are utilized for the same purpose and therefore yield predictable results.
Regarding claim 2, Nandan teaches wherein the end of pipeline event comprises a hardware event from one of the one or more hardware accelerators indicating completion of a last task in the series of tasks defined in the second pipeline [0038]; [0047] In some examples, the example first hardware accelerator 350a generates a done signal (e.g., a Tdone signal) in response to the example first hardware accelerator 350a completing the processing of an example data element 402, 404, 406, 408 (FIG. 4) and sends the Tdone signal to the example first scheduler 382a (FIG. 3).; [0048] the example first hardware accelerator 350a sends a Tdone signal to the example first scheduler 382a, at which point the example first scheduler 3 82a may instruct the example second hardware accelerator 350b or the example third hardware accelerator 350c to read the aggregated data element 420a, 420b from the example local memory 360.; [0054]; [0057] In the example of FIG. 6, the example MSC hardware accelerator 650b consumes a set of data lines based on the output of the example LDC hardware accelerator 650a, performs a scaling operation on the data lines, and produces the data line element 636 which is consumed by the example NF hardware accelerator 650c and the example third DMA controller 644.; [0058] pipeline/series of tasks).
Regarding claim 3, Nandan teaches wherein at least one task of the series of tasks of the first pipeline comprises an instruction to access memory external to a chip comprising the hardware accelerator thread scheduler (Fig. 2; [0026] The hardware acceleration subsystem 210 illustrated in FIG. 2 includes a first direct memory access (DMA) controller 240 to facilitate transfer of the data elements 231, 232 from the external memory 230, e.g., from the input frame 237 stored in the external memory 230, to the local memory 260, a second DMA controller 242 to facilitate transfer of the processed data elements 236, 238 to the external memory 230, e.g., to the output frame 239 of the external memory 230, from the local memory 260 and/or hardware accelerators 250a, 250b, 250c, 250d, four hardware accelerators 250a, 250b, 250c, 250d to perform various processing tasks on the data elements 231, 232 to produce intermediate data elements 233, 234 and/or processed data elements 236, 238, a local memory 260 to store the data elements 231, 232 and/or intermediate data elements 233, 234 temporarily during processing, and a scheduler 280 to coordinate the workflow between the hardware accelerators 250a-d, the local memory 260, and the DMA controllers 240, 242.).
Regarding claim 4, Nandan teaches wherein the series of tasks for at least one of the plurality of pipelines comprises tasks to perform image processing ([0018] Hardware acceleration has various applications across many different fields including the automotive industry, advanced driver system (ADAS), manufacturing, high performance computing, robotics, drones, and other industries involving complex, high-speed processing, e.g., hardware-based encryption, computer generated graphics, artificial intelligence, and digital image processing, the latter of which involves various complex processing operations performed on a single image or video stream, for example, lens distortion correction, scaling, transformations, noise filtering, dense optical flow, pyramid representation, stereo screen door effect (SDE), and other processing operations.).
Regarding claim 5, Nandan teaches wherein a third pipeline of the plurality of pipelines includes a second hardware enable flag configuration setting that allows initiation of the third pipeline based on completion of the first pipeline, wherein the series of tasks for the second pipeline comprises tasks for restoring context information for image processing, wherein the series of tasks for the first pipeline comprises tasks for performing image processing using the restored context information, wherein the series of tasks for the third pipeline comprises tasks for saving resulting context information after the performing the image processing ([0016]; [0018]; [0026]), and wherein the hardware accelerator thread scheduler is further configured to:
detect a second end of pipeline event indicating completion of the first pipeline ([0016] Unless specifically stated otherwise, descriptors such as “first,” “second,” “third,” etc. are used herein without imputing or otherwise indicating any meaning of priority, physical order, arrangement in a list, and/or ordering in any way, but are merely used as labels and/or arbitrary names to distinguish elements for ease of understanding the disclosed examples. In some examples, the descriptor “first” may be used to refer to an element in the detailed description, while the same element may be referred to in a claim with a different descriptor such as “second” or “third.”; [0048] the example first hardware accelerator 350a sends a Tdone signal to the example first scheduler 382a, at which point the example first scheduler 3 82a may instruct the example second hardware accelerator 350b or the example third hardware accelerator 350c to read the aggregated data element 420a, 420b from the example local memory 360.;);
in response to the second end of pipeline event indicating completion of the first pipeline and the second hardware enable flag configuration setting in the third pipeline, initiate execution of the third pipeline ([0016]; [0038]; [0047] In some examples, the example first hardware accelerator 350a generates a done signal (e.g., a Tdone signal) in response to the example first hardware accelerator 350a completing the processing of an example data element 402, 404, 406, 408 (FIG. 4) and sends the Tdone signal to the example first scheduler 382a (FIG. 3).; [0048] the example first hardware accelerator 350a sends a Tdone signal to the example first scheduler 382a, at which point the example first scheduler 3 82a may instruct the example second hardware accelerator 350b or the example third hardware accelerator 350c to read the aggregated data element 420a, 420b from the example local memory 360.; [0054]; [0057] In the example of FIG. 6, the example MSC hardware accelerator 650b consumes a set of data lines based on the output of the example LDC hardware accelerator 650a, performs a scaling operation on the data lines, and produces the data line element 636 which is consumed by the example NF hardware accelerator 650c and the example third DMA controller 644.; [0058] pipeline/series of tasks).;
detect a third end of pipeline event indicating completion of the third pipeline ([0016]; [0048]); and receive an initiate signal from an external processor to initiate execution of the second pipeline ([0003] While central processing units (CPUs) have improved to meet the demands of modern applications, computer performance remains limited by the substantial amounts of data that must be processed simultaneously by the CPU. Hardware accelerator sub-systems may provide improved performance and/or power consumption by offloading tasks from a computer's central processing unit (CPU) to hardware components that specialize in performing those tasks.;[0016]; [0038]; [0047] In some examples, the example first hardware accelerator 350a generates a done signal (e.g., a Tdone signal) in response to the example first hardware accelerator 350a completing the processing of an example data element 402, 404, 406, 408 (FIG. 4) and sends the Tdone signal to the example first scheduler 382a (FIG. 3).; [0048] the example first hardware accelerator 350a sends a Tdone signal to the example first scheduler 382a, at which point the example first scheduler 3 82a may instruct the example second hardware accelerator 350b or the example third hardware accelerator 350c to read the aggregated data element 420a, 420b from the example local memory 360.; [0054]; [0057] In the example of FIG. 6, the example MSC hardware accelerator 650b consumes a set of data lines based on the output of the example LDC hardware accelerator 650a, performs a scaling operation on the data lines, and produces the data line element 636 which is consumed by the example NF hardware accelerator 650c and the example third DMA controller 644.; [0058] pipeline/series of tasks).
In addition, Mishra teaches a hardware enable flag configuration setting (Fig. 3; [0047] In certain embodiments, on receipt by the accelerator complex 300 of the schema (e.g., prepared by the compiler and embedded in the application binary), the accelerator controller (e.g., control logic) and scheduler 304 in the accelerator complex prepare a set of virtual accelerator threads (VATs) to be scheduled on the hardware accelerator circuit(s) (e.g., accelerator threads (ATs). These tasks may be queued into an accelerator work queue from which each of the hardware accelerators may pull work and notify completion with a done flag. This is schematically shown in FIG. 3.).
Regarding claim 6, Nandan teaches wherein each execution of the second pipeline, the first pipeline, and the third pipeline performs image processing on a different frame ([0019] For example, in image processing where imaging/vision algorithms are often two-dimensional (2D) block-based, the hardware accelerators may be configured to process two-dimensional blocks from an image frame rather than processing the entire image frame as lines. Various example hardware accelerators may operate on block sizes of 16×16 bytes, 32×32 bytes, and 64×32 bytes.; [0038] In some examples, the example first scheduler 382a sends an initiate signal to the example first hardware accelerator 350a to indicate start-of-frame processing to the example hardware accelerator 350a. In some examples, the example first hardware accelerator 350a sends an end-of-frame signal to the example first scheduler 382a to communicate end-of-frame processing, e.g., to communicate the example first hardware accelerator has finished processing a frame.).
Regarding claim 8, it is a system type claim having similar limitations as claim 1 above. Therefore, it is rejected under the same rationale above. Further Nandan teach the additional limitations including a system, comprising:
a memory having stored thereon instructions that, upon execution by one or more processors ([0084] The processor 1012 of the illustrated example includes a local memory 1013 (e.g., a cache). The processor 1012 of the illustrated example is in communication with a main memory including a volatile memory 1014 and a non-volatile memory 1016 via a bus 1018. The volatile memory 1014 may be implemented by Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), RAIVIBUS® Dynamic Random Access Memory (RDRAM®) and/or any other type of random access memory device. The non-volatile memory 1016 may be implemented by flash memory and/or any other desired type of memory device. Access to the main memory 1014, 1016 is controlled by a memory controller.), cause the one or more processors to:
send an initiate signal to a hardware accelerator thread scheduler to initiate execution of a first pipeline of a plurality of pipelines configured in the hardware thread scheduler ([0003] While central processing units (CPUs) have improved to meet the demands of modern applications, computer performance remains limited by the substantial amounts of data that must be processed simultaneously by the CPU. Hardware accelerator sub-systems may provide improved performance and/or power consumption by offloading tasks from a computer's central processing unit (CPU) to hardware components that specialize in performing those tasks.);
one or more hardware accelerators (Fig. 2 Hardware Accelerators 250a-d); and
the hardware accelerator thread scheduler (Fig. 2 Scheduler 280).
Regarding claim 9, it is a system type claim having similar limitations as claim 2 above. Therefore, it is rejected under the same rationale above.
Regarding claim 10, it is a system type claim having similar limitations as claim 3 above. Therefore, it is rejected under the same rationale above.
Regarding claim 11, it is a system type claim having similar limitations as claim 4 above. Therefore, it is rejected under the same rationale above.
Regarding claim 12, Nandan teaches further comprising: a camera for capturing images ([0087] camera).
Regarding claim 13, it is a system type claim having similar limitations as claim 5 above. Therefore, it is rejected under the same rationale above.
Regarding claim 14, it is a system type claim having similar limitations as claim 6 above. Therefore, it is rejected under the same rationale above.
Regarding claim 16, it is a method type claim having similar limitations as claim 8 above. Therefore, it is rejected under the same rationale above.
Regarding claim 17, it is a method type claim having similar limitations as claim 2 above. Therefore, it is rejected under the same rationale above.
Regarding claim 18, it is a method type claim having similar limitations as claim 3 above. Therefore, it is rejected under the same rationale above.
Regarding claim 19, it is a method type claim having similar limitations as claim 5 above. Therefore, it is rejected under the same rationale above.
Claims 7, 15, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Nandan and Mishra, as applied claim 1, in further view of Moyer et al. (US 2016/0011907 A1).
Regarding claim 7, Nandan teaches sockets ([0027-29] producer and consumer sockets of hardware scheduler). Nandan nor Mishra explicitly teach wherein a third pipeline of the plurality of pipelines includes a clear pend enable flag configuration setting that allows clearing of a pend block signal in a producer socket of a producer node in the third pipeline based on an internal event, and wherein the hardware accelerator thread scheduler is further configured to: detect the internal event; and in response to the internal event and the clear pend enable flag configuration setting, clear the pend block signal in the producer socket.
However, Moyer teaches wherein a third pipeline of the plurality of pipelines includes a clear pend enable flag configuration setting that allows clearing of a pend block signal in a producer socket of a producer node in the third pipeline based on an internal event, and wherein the hardware accelerator thread scheduler is further configured to: detect the internal event, and in response to the internal event and the clear pend enable flag configuration setting, clear the pend block signal in the producer socket (Moyer Table 1; [0011] task tracking; [0018-20] completion information shared by accelerator schedulers; [0036-40] release inhibit command based on task completion and multiple states used to track waiting, executing or inhibiting states between accelerators and tasks).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the teachings of Nandan, Mishra and Moyer to update execution signals to show current status of tasks. The combination would have been motivated by the desire of combining known elements to yield predictable results.
Regarding claim 15, it is a system type claim having similar limitations as claim 7 above. Therefore, it is rejected under the same rationale above.
Regarding claim 20, it is a method type claim having similar limitations as claim 7 above. Therefore, it is rejected under the same rationale above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JORGE A CHU JOY-DAVILA whose telephone number is (571)270-0692. The examiner can normally be reached Monday-Friday, 6:00am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Aimee J Li can be reached at (571)272-4169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JORGE A CHU JOY-DAVILA/Primary Examiner, Art Unit 2195