DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This Office Action is in response to Request for Continued Examination filed on 30 June, 2026 and Applicant Amendment and Arguments filed on 22 May, 2026.
Claims 1-35 are pending in this application. Claim 35 was newly added.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 22 May, 2026 has been entered.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-35 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
Claim 1 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1, Statutory Category: Yes, the claim 1 is a system that performs a series of steps and therefore falls in the statutory category of a machine.
Step 2A- Prong 1: Judicial Exception Recited: Yes, the claim recites: “identify that a first hardware accelerator is a pre-selected hardware accelerator to execute the one or more video frame transformations corresponding to a set of clients based, at least in part, on configuration information; assign the set of clients to cause the first hardware accelerator to execute the video frame transformations with respect to a batch of video frames to modify one or more individual frames in the batch of video frames; generate a determination that a first metric associated with each of the set of clients exceeds a threshold, wherein the metric is computed based, at least in part, on a type of the one or more video frame transformations; assign a subset of clients of the set of clients to a set of second hardware accelerators to cause the metric associated with each of the set of clients to be below the threshold in response to the determination; as a result of assigning the subset of clients, reassign at least one of the subset of clients to the first hardware accelerator to cause the metric associated with each of the set of clients to remain below the threshold” As drafted, the claim as a whole recites a system that performs a series of steps which could be performed in the human mind, but for the recitation of generic computing components. The human mind can easily judging/evaluating/identifying a first hardware accelerator as a pre-selected hardware accelerator which is to execute a set of clients based on configuration information, assigning/scheduling/planning the set of processes to the first hardware accelerator for processing, determining if the computed/determined metric associated with each of the set of clients exceeds a threshold, assigning/scheduling/planning a subset of the processes to another set of hardware accelerators in order to allow the metric to below the threshold, and then scheduling/planning/assigning/reassigning the at least one of the subset of clients/process back to the first hardware accelerator based on if the determined metric is remain below the threshold. Therefore, but for the recitation of generic computing components, these steps may be a Mental Processes that can be performed in the human mind (including an observation, evaluation, judgment, opinion).
Therefore, yes, the claims do recite judicial exceptions.
Step 2A- Prong 2: Integrated into a practical Application: No, this judicial exception is not integrated into a practical application. In particular, the claim recites additional elements “one or more processors; and memory storing instructions that, as a result of being executed by the one or more processors, cause the system to”, “wherein the configuration information comprises a mapping between the video frame transformations and hardware accelerators” are directed to generic computing components/functions (MPEP § 2106.05(b) merely applying the abstract idea (MPEP § 2106.05(f)). In addition, “receive one or more calls via an application programming interface (API) from a set of clients, the one or more calls comprising one or more requests to perform one or more video frame transformations” which is insignificant pre-solution data gathering (see MPEP § 2106.05(g)). Further, the limitation of “cause the first hardware accelerator to execute at least one of the one or more video frame transformations corresponding to the subset of clients” which is merely applying the judicial exception or abstract idea (See MPEP 2106.05(f)). (i.e., The claim does not define any particular machine to “cause” this “video frame transformation,” other than a generic machine such as the “hardware accelerator,” and no details what so ever on how the claimed function (i.e., video frame transformations) will occur). The combination of these additional elements is no more than mere instructions to apply the exception using a generic computer component (MPEP 2106.05(f)). Accordingly, even in combination, these additional elements do not integrate the abstract idea into a practical application because they not impose any meaningful limits on practicing the abstract idea. Therefore, the claim is directed to the abstract idea.
Step 2B: Claim provides an Inventive Concept: No. The additional element “one or more processors; and memory storing instructions that, as a result of being executed by the one or more processors, cause the system to” and “wherein the configuration information comprises a mapping between the video frame transformations and hardware accelerators” are directed to generic computing components/functions (MPEP § 2106.05(b) merely applying the abstract idea (MPEP § 2106.05(f)). Further, the limitation of “cause the first hardware accelerator to execute at least one of the one or more video frame transformations corresponding to the subset of clients” which is merely applying the judicial exception or abstract idea (See MPEP 2106.05(f)). (i.e., The claim does not define any particular machine to “cause” this “video frame transformation,” other than a generic machine such as the “hardware accelerator,” and no details what so ever on how the claimed function (i.e., video frame transformations) will occur). Further, “receive one or more calls via an application programming interface (API) from a set of clients, the one or more calls comprising one or more requests to perform one or more video frame transformations” which is insignificant pre-solution data gathering (see MPEP § 2106.05(g)) which are well understood, routine, conventional activity (see MPEP § 2106.05(d)). Courts have identified “receiving and transmitting data, storing and retrieving information”, et cetera as well understood, routine, conventional and mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(f))). These additional elements and combination of the elements does not amount to significant more than the exception itself or provide an inventive concept in Step 2B.
Under the 2019 PEG, a conclusion that an additional element is insignificant extra-solution activity in Step 2A should be re-evaluated in Step 2B. Here, the “receive” steps were considered to be extra-solution activity in Step 2A as insignificant data gathering and communication and are well understood, routine, conventional activity in the field. The “receive” steps are for the purpose of “communication” and “transmitting the data” and these can be reached on one of court case (Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) see MPEP § 2106.05(d) II). Accordingly, a conclusion that “receive” are well understood, routine, conventional activity is supported under Berkheimer options 2.
For these reasons, there is no inventive concept in the claim, and thus the claim is ineligible.
Independent claim 22 is rejected for the same reason as claim 1 above. In addition, claim 22 further recites “assigning a set of application clients to a video image compositor (VIC) engine”, “determining that an average time used by the VIC engine to execute the video frame transformations for the set of application clients exceeds a frame processing threshold”, “assigning, to a set of processing units, distinct from VIC engine, a subset of application clients of the set of application clients to cause an average time used…to be below the frame processing threshold…”, “reassigning the subset of application clients to cause the average time used by the VIC engine to remain below the frame processing threshold…” are being treated as part of abstract idea and is analogues to Mental processes, such that concept can be performed in the human mind. For example, the human mind can easily evaluating/planning/scheduling/assigning a set of application clients to a video image compositor (VIC) engine for processing the requests, judging/determining/comparing whether the average time of the processing is exceeding a threshold and based on the determination further planning/scheduling/assigning the processing load to different processing devices in order to improving the processing time and keep determining whether reassigning the subset of application clients causes the average time used by the VIC engine to remain below the frame processing threshold, and then scheduling/moving the processes back to the previous VIC for processing to improving the processing time. Therefore, but for the recitation of generic computing components, these steps may be a Mental Processes that can be performed in the human mind (including an observation, evaluation, judgment, opinion). Further, the claimed limitation of “a video image compositor (VIC) engine” and “processing unit” are directed to generic computing components/functions merely applying the abstract idea (MPEP § 2106.05(f)). And the limitation of “cause the VIC engine to execute the one or more portions of the video frame transformations as a result of reassigning the subset of application clients” which is merely applying the judicial exception or abstract idea (See MPEP 2106.05(f)). (i.e., The claim does not define any particular machine to “cause” this “video frame transformation,” other than a generic machine such as the “VIC engine,” and no details what so ever on how the claimed function (i.e., video frame transformations) will occur).
Independent claim 30 is rejected for the same reason as claims 1 and 22 above. In addition, the claim 30 further recites “A non-transitory computer readable storage medium storing thereon executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to” is directed to generic computing components/functions merely applying the abstract idea (MPEP § 2106.05(f)). In addition, the limitation of “obtain first performance data associated with a video image compositor (VIC)” which is insignificant pre-solution data gathering (see MPEP § 2106.05(g)) which are well understood, routine, conventional activity in the field. The “obtain” steps are for the purpose of “communication” and “transmitting the data” and these can be reached on one of court case (Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) see MPEP § 2106.05(d) II). Accordingly, a conclusion that “obtain” are well understood, routine, conventional activity is supported under Berkheimer options 2. Further, the limitation of “cause the set of processors to execute at least a portion of the video frame transformations corresponding to the subset of application clients to relieve load from the VIC” which is merely applying the judicial exception or abstract idea (See MPEP 2106.05(f)). (i.e., The claim does not define any particular machine to “cause” this “video frame transformation,” other than a generic machine such as the “VIC engine,” and no details what so ever on how the claimed function (i.e., video frame transformations) will occur).
With respect to the dependent claim 2, the claim elaborates that wherein the threshold is determined based, at least in part, on a framerate associated with the batch of video frames. (“threshold is determined based, at least in part, on a framerate associated with the batch of frames” as being treated as part of abstract idea and is analogues to Mental processes, such that concept can be performed in the human mind. In addition, the claim as a whole is a Mental Processes that can be performed in the human mind (including an observation, evaluation, judgment, opinion)).
With respect to the dependent claim 3, the claim elaborates that wherein the metric further comprises a value indicating a percentage of activity of the first hardware accelerator (“a value indicating a percentage of activity” is directed to generic computing components/functions merely applying the abstract idea (MPEP § 2106.05(f)).
With respect to the dependent claim 4, the claim elaborates that wherein the first hardware accelerator further comprises a video image compositor (VIC) and the set of second hardware accelerators further comprises a graphics processing unit (GPU). (“a video image compositor” and “GPU” are directed to generic computing components/functions merely applying the abstract idea (MPEP § 2106.05(f)).
With respect to the dependent claim 5, the claim elaborates that wherein the one or more calls comprise respective client identification numbers used to associate the one or more requests with the set of clients (“respective client identification numbers used to associate the one or more requests with the set of clients” is directed to generic computing components/functions merely applying the abstract idea (MPEP § 2106.05(f)).
With respect to the dependent claim 6, the claim elaborates that wherein the metric further comprises an amount of time at least one of the set of clients utilizes the first hardware accelerator to perform processing of the batch of video frames (“an amount of time at least one of the set of clients utilizes the first hardware accelerator to perform processing” is directed to generic computing components/functions merely applying the abstract idea (MPEP § 2106.05(f)).
With respect to the dependent claim 7, the claim elaborates that wherein the metric further comprises a percentage of processing capability the first hardware accelerator utilized to process the batch of video frames on behalf of at least one of the set of clients. (“a percentage of processing capability” is directed to generic computing components/functions merely applying the abstract idea (MPEP § 2106.05(f)).
With respect to the dependent claim 8, the claim elaborates that wherein the metric further comprises an average amount of load generated by at least processing the batch of video frames provided by at least one of the set of clients (“metric further comprises an average amount of load generated” is directed to generic computing components/functions merely applying the abstract idea (MPEP § 2106.05(f)).
With respect to the dependent claim 9, the claim elaborates that wherein the determination is generated during an interval of time (“first determination is generated” is directed to generic computing components/functions merely applying the abstract idea (MPEP § 2106.05(f)).
With respect to the dependent claim 10, the claim elaborates that wherein the determination is generated based at least in part on historical data (“the first determination is generated based at least in part on historical data” is directed to generic computing components/functions merely applying the abstract idea (MPEP § 2106.05(f)).
With respect to the dependent claim 11, the claim elaborates that wherein the configuration information is generated based, at least in part, on user input. (“wherein the configuration information is generated based, at least in part, on user input” as being treated as part of abstract idea and is analogues to Mental processes, such that concept can be performed in the human mind. In addition, the claim as a whole is a Mental Processes that can be performed in the human mind (including an observation, evaluation, judgment, opinion)).
With respect to the dependent claim 12, the claim elaborates that wherein the threshold is designated by a user (“wherein the threshold is designated by a user” is directed to generic computing components/functions merely applying the abstract idea (MPEP § 2106.05(f)).
With respect to the dependent claim 13, the claim elaborates that wherein the type comprises modifying resolutions of the batch of video frames, converting formats of the batch of frames, or modifying color of the batch of video frames. (“modifying resolutions of the batch of frames, converting formats of the batch of video frames, or modifying color of the batch of video frames” as being treated as part of abstract idea and is analogues to Mental processes, such that concept can be performed in the human mind. In addition, the claim as a whole is a Mental Processes that can be performed in the human mind (including an observation, evaluation, judgment, opinion)).
With respect to the dependent claim 14, the claim elaborates that wherein the metric is obtained at least from a hardware performance counter included in the first hardware accelerator (“obtained” which is insignificant pre-solution data gathering (see MPEP § 2106.05(g)) and can be reached on one of the court case (Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) see MPEP § 2106.05(d) II)).
With respect to the dependent claim 15, the claim elaborates that wherein the first hardware accelerator further comprises a field-programmable gate array (FPGA) (“field-programmable gate array” is directed to generic computing components/functions merely applying the abstract idea (MPEP § 2106.05(f)).
With respect to the dependent claim 16, the claim elaborates that wherein the memory further stores instructions that, as a result of being executed by the one or more processors, cause the system to obtain, through a system call, the metric (“obtain” which is insignificant pre-solution data gathering (see MPEP § 2106.05(g)) and can be reached on one of the court case (Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) see MPEP § 2106.05(d) II)).
With respect to the dependent claim 17, the claim elaborates that wherein the set of clients further comprise a set of components of an artificial intelligence pipeline (“artificial intelligence pipeline” is directed to generic computing components/functions merely applying the abstract idea (MPEP § 2106.05(f)).
With respect to the dependent claim 18, the claim elaborates that wherein the artificial intelligence pipeline includes one or more neural networks (“the artificial intelligence pipeline includes one or more neural networks” is directed to generic computing components/functions merely applying the abstract idea (MPEP § 2106.05(f)).
With respect to the dependent claim 19, the claim elaborates that wherein the memory further stores instructions that, as a result of being executed by the one or more processors, cause the system to cause the first hardware accelerator to process the batch of video frames by at least converting the batch of video frames from a first format to a second format (“process the batch of video frames by at least converting the batch of frames from a first format to a second format” as being treated as insignificant extra-solution activity and merely data manipulation (see MPEP § 2106.05(g) “Selecting a particular data source or type of data to be manipulated”).
With respect to the dependent claim 20, the claim elaborates that wherein the memory further stores instructions that, as a result of being executed by the one or more processors, cause the system to cause the first hardware accelerator to process the batch of video frames by at least scaling the batch of video frames (“scaling the batch of video frames” as being treated as insignificant extra-solution activity and merely data manipulation (see MPEP § 2106.05(g) “Selecting a particular data source or type of data to be manipulated:”).
With respect to the dependent claim 21, the claim elaborates that wherein the memory further stores instructions that, as a result of being executed by the one or more processors, cause the system to cause the first hardware accelerator to process the batch of video frames by at least modifying one or more color values associated with at least one frame of the batch of video frames (“modifying one or more color values” which is insignificant extra-solution activity and merely data manipulation (see MPEP § 2106.05(g) “Selecting a particular data source or type of data to be manipulated:”).
With respect to the dependent claim 23, the claim elaborates that wherein the assignment and the reassignment of the subset of application clients are repeated one or more times (“assignment and the reassignment” are repeated one or more times” as being treated as part of abstract idea and is analogues to Mental processes, such that concept can be performed in the human mind. In addition, the claim as a whole is a Mental Processes that can be performed in the human mind (including an observation, evaluation, judgment, opinion)).
With respect to the dependent claim 24, the claim elaborates that wherein the frame processing threshold is determined based at least in part on a framerate of video processing (“the frame processing threshold is determined” as being treated as part of abstract idea and is analogues to Mental processes, such that concept can be performed in the human mind. In addition, the claim as a whole is a Mental Processes that can be performed in the human mind (including an observation, evaluation, judgment, opinion)).
With respect to the dependent claim 25, the claim elaborates that wherein the assigning of the subset of application clients is executed using a load balancer. (“load balancer” is directed to generic computing components/functions merely applying the abstract idea (MPEP § 2106.05(f)).
With respect to the dependent claim 26, the claim elaborates that wherein the load balancer that assigns the subset of application clients comprises a processing thread (“assigns the subset of application clients” as being treated as part of abstract idea and is analogues to Mental processes, such that concept can be performed in the human mind. In addition, “wherein the load balancer comprises a processing thread” is directed to generic computing components/functions merely applying the abstract idea (MPEP § 2106.05(f)).
With respect to the dependent claim 27, the claim elaborates that wherein a load balancer that assigns the subset of application clients maintains a table of set of application clients (“assigns the subset of application clients” as being treated as part of abstract idea and is analogues to Mental processes, such that concept can be performed in the human mind. In addition, “maintains a table of set of application clients” as being treated as Insignificant Extra-Solution Activity (storing data) which is additionally well understood, routine, conventional activity (see MPEP § 2106.05(d)) and this can be reached on one of court case (Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93)).
With respect to the dependent claim 28, the claim elaborates that wherein the table of the set of application clients includes a processing engine assigned to each client and an average time taken by the client to compute the one or more video frame transformations (“the table of the set of application clients includes a processing engine assigned to each client and an average time taken by the client to compute video frame transformations” is directed to generic computing components/functions merely applying the abstract idea (MPEP § 2106.05(f)).
With respect to the dependent claim 29, the claim elaborates that wherein the set of processing units comprises a graphics processing unit (GPU). (“GPU” is directed to generic computing components/functions (MPEP § 2106.05(b) merely applying the abstract idea (MPEP § 2106.05(f)).
With respect to the dependent claim 31, the claim elaborates that wherein the threshold is determined based, at least in part, on one or more framerates associated with the set of application clients.. (“threshold is determined” as being treated as part of abstract idea and is analogues to Mental processes, such that concept can be performed in the human mind. In addition, the claim as a whole is a Mental Processes that can be performed in the human mind (including an observation, evaluation, judgment, opinion)).
With respect to the dependent claim 32, the claim elaborates that wherein the set of processors comprises a field-programmable gate array (FPGA) (“FPGA” is directed to generic computing components/functions merely applying the abstract idea (MPEP § 2106.05(f)).
With respect to the dependent claim 33, the claim elaborates that wherein the instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to maintain a table of the set of application clients of which each application client is a member (“cause the computer system to maintain a table” which is insignificant extra-solution activity and merely data storing (see MPEP § 2106.05(g)).
With respect to the dependent claim 34, the claim elaborates that wherein the table of the set of application clients includes information indicating that the set of processors is assigned to the subset of application clients, and an average time taken by the set of processors to compute video frame transformations for the subset of application clients (“the table of the set of application clients includes information” is directed to generic computing components/functions (MPEP § 2106.05(b)).
With respect to the dependent claim 35, the claim elaborates that wherein the instructions, if executed by the one or more processors, are to cause the system to: receive information from the set of clients related to execution of at least a portion of the one or more video frame transformations, and use the information to generate statistical data to use to perform load balancing with respect to a plurality of hardware accelerators that comprises the first and second hardware accelerators (“receive information” which is insignificant pre-solution data gathering (see MPEP § 2106.05(g)) and can be reached on one of the court case (Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) see MPEP § 2106.05(d) II)). In addition, “use the information to generate statistical data to use to perform load balancing” as being treated as part of abstract idea and is analogues to Mental processes, such that concept can be performed in the human mind).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 9, 13 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Bernat et al (US Pub. 2018/0027062 A1) in view of Dutta (US Patent. 9,313,134 B2) and further in view of Diard (US Patent. 7,075,541 B2), Lowry et al. (US Pub. 2018/0262684 A1), Shraer et al. (US Pub. 2017/0353536 A1) and Guo et al. (US Pub. 2011/0249086 A1).
Bernat, Dutta, Diard, Lowry and Shraer were cited in the previous Office Action.
As per claim 1, Bernat teaches the invention substantially as claimed including A system (Bernat, Fig. 12) comprising:
one or more processors; and memory storing instructions that, as a result of being executed by the one or more processors, cause the system to (Bernat, [0164] lines 3-6, compute device comprising one or more processors; one or more memory devices having stored therein a plurality of instructions that, when executed by the one or more processors):
identify that a first hardware accelerator is a pre-selected hardware accelerator to execute operations based, at least in part, on configuration information (Bernat, [0084] lines 18-32, workload assignor 1632 includes an acceleration assignor 1634 which is configured to assign certain workloads or portions thereof to corresponding accelerators (e.g., the accelerators 1250, 1260). In doing so, the acceleration assignor 1634 may identify a type of the workload based on a profile of resource utilizations of the workload over time or based on a tag, an analysis of the computer-executable instructions within the workload (as set of clients), a header of the workload, metadata indicative of the types operations to be executed in the workload (as set of clients, see specification [0057] “a client (e.g., application, processes, or other components), or from a request from a CPU (e.g., the CPU 1272) to offload a portion of a workload onto a particular accelerator 1250, 1260 or type of accelerator (e.g., FPGA, graphics accelerator, cryptography accelerator, compression accelerator, etc.); [0058] lines 2-4, an accelerator (e.g., the accelerator 1250) is configured to execute assigned workloads; [Examiner noted: the corresponding/particular hardware accelerator is identified (as first hardware accelerator/pre-selected hardware accelerator) for assignment based on the configuration]);
assign the set of clients to cause the first hardware accelerator to execute the operations (Bernat, [0058] lines 2-4, an accelerator (e.g., the accelerator 1250) is configured to execute assigned workloads; [0084] lines 28-32, metadata indicative of the types operations to be executed in the workload (as include set of clients), or from a request from a CPU (e.g., the CPU 1272) to offload a portion of a workload onto a particular accelerator 1250, 1260 or type of accelerator (e.g., FPGA, graphics accelerator, cryptography accelerator, compression accelerator, etc.), and assign the workload to a corresponding accelerator 1250, 1260 for execution);
generate a determination that a metric associated with each of the set of clients exceeds a threshold (Bernat, Fig. 18, 1726 execute the workload with corresponding logic portion of the accelerator, 1734, monitor utilization of the resources (as metric) associated with the resource utilization threshold, 1740 threshold satisfied? [0085] lines 1-10, the resource utilization analyzer, in the illustrative embodiment, is configured to receive the resource utilization data 1604 (as metric) and determine whether the present utilization of resources among the managed nodes 1230, such as the utilization by logic portions (e.g., the logic portions 1252, 1254) of shared resources within an accelerator (e.g., the accelerator 1250) is in violation of resource utilization threshold indicated in the resource utilization threshold data 1606; [0089] line 14, resource utilizations that exceed the corresponding resource utilization thresholds).
cause the first hardware accelerator to execute at least one of the operations corresponding to the subset of clients (Bernat, Fig. 18, 1740; Fig. 19, 1904; [0084] lines 18-32, workload assignor 1632 includes an acceleration assignor 1634 which is configured to assign certain workloads or portions thereof to corresponding accelerators (e.g., the accelerators 1250, 1260). In doing so, the acceleration assignor 1634 may identify a type of the workload based on a profile of resource utilizations of the workload over time or based on a tag, an analysis of the computer-executable instructions within the workload (as set of clients), a header of the workload, metadata indicative of the types operations to be executed in the workload or from a request from a CPU (e.g., the CPU 1272) to offload a portion of a workload onto a particular accelerator 1250, 1260 or type of accelerator (e.g., FPGA, graphics accelerator, cryptography accelerator, compression accelerator, etc.); also see [0002] accelerating the execution of a set of operations in a workload; [0091] determine to manage accelerator resource utilizations if at least one accelerator having the ability to execute workloads on separate logic portions 1252, 1254 is present in the system 1210…assigns a workload to be accelerated by a logic portions (e.g., the logic portion 1252) of an accelerator (e.g., the accelerator 1250)…the orchestrator server 1220 may make the determination to assign a workload in response to identifying the workload or a portion of the workload (as include the subset) as being amenable to a particular type of acceleration for which an accelerator (e.g., the accelerator 1250)).
Bernat fails to specifically teach when execute the operations, the operations are one or more video frame transformations, wherein the configuration information comprises a mapping between the one or more video frame transformations and hardware accelerators, execute the one or more video frame transformations with respect to a batch of video frames to modify one or more individual frames in the batch of video frames.
However, Dutta teaches when execute the operations, the operations are transformations (Dutta, Col 1, lines 56-60, An example method for leveraging hardware accelerators for scalable distributed streams in a network environment is provided and includes allocating a plurality of hardware accelerators to a corresponding plurality of bolts of a distributed stream in a network; Col 2, lines 11-14, the “bolt” implements processing logic to process (e.g., run functions, filter tuples, (as transformations) perform stream aggregations, talk to databases, etc.) the data elements in the stream);
wherein the configuration information comprises a mapping between the transformations and hardware accelerators, and execute the transformations (Dutta, Fig. 1, 16(1)-(N) hardware accelerators, 20(1)-(N) Bolt (i.e., which including the process of filter tuples (i.e., as transformations); Fig. 2, 44 input data to hardware accelerator for execution to generate output data 46; Col 4, lines 11-25, broker 14 may receive capability information from bolts 20(1)-20(N) and hardware accelerators 16(1)-16(N) and map hardware accelerators 16(1)-16(N) to corresponding bolts 20(1)-20(N). The capability information from bolts 20(1)-20(N) may include respective locations in distributed streams 17(1)-17(M) and identities; the capability information from hardware accelerators 16(1)-16(N) may include respective network locations (e.g., Internet Protocol (IP) address), and capabilities (e.g., RegEx processor, graphics processor, etc.). The mapping may be formatted into any suitable table, spreadsheet, memory mapping, etc. as suitable and based on particular needs. According to various embodiments, the mapping may be used to route the data elements of distributed streams 17(1)-17(M) to appropriate hardware accelerators 16(1)-16(N) for stream processing; Col 8, lines 23-27, distributed stream 17 may be processed by a set of computing devices called worker nodes 54. According to various embodiments, worker nodes 54 may include hardware accelerators 16(1)-16(N) and/or other computing devices).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat with Dutta because Dutta’s teaching of mapping the accelerators with the corresponding transformation operations for processing would have provided Bernat’s system with the advantage and capability to allow the system to easily assigning the operations to the corresponding hardware accelerators based on the mapping in order to improving the system performance and processing speed.
Although Bernat and Dutta teach processing transformations in the accelerator, Bernat and Dutta fail to specifically teach the transformation is one or more video frame transformations, execute the one or more video frame transformations with respect to a batch of video frames to modify one or more individual frames in the batch of video frames, wherein the metric is computed based, at least in part, on a type of the video frame transformations, assign a subset of clients of the set of clients to a set of second hardware accelerators to cause the metric associated with each of the set of clients to be below the threshold in response to the determination; as a result of assigning the subset of clients, reassign at least one of the subset of clients to the first hardware accelerator to cause the metric associated with each of the set of clients to remain below the threshold.
However, Diard teaches transformation is one or more frame transformations, execute the one or more frame transformations with respect to a batch of frames to modify one or more individual frames in the batch of frames. (Diard, Col 1, lines 24-38, Graphics processing subsystems are designed to render realistic animated images in real time, e.g., at 30 or more frames per second. These subsystems are most often implemented on expansion cards that can be inserted into appropriately configured slots on a motherboard of a computer system and generally include one or more dedicated graphics processing units (GPUs) and dedicated graphics memory. The typical GPU is a highly complex integrated circuit device optimized to perform graphics computations (e.g., matrix transformations, scan-conversion and/or other rasterization techniques, texture blending, etc.) and write the results to the graphics memory. The GPU is a "slave" processor that operates in response to commands received from a driver program executing on a "master" processor, generally the central processing unit (CPU) of the system; also see Col 2, lines 66-67, The graphics processors are instructed to render a number of frames, wherein the first and second graphics process; Col 5, lines 37-40, lighting transformations, coordinate transformations, scan-conversion of geometric primitives to rasterized data, shading computations, shadow rendering, texture blending);
wherein the metric is computed based, at least in part, on a type of the one or more frame transformations (Diard, Col 1, lines 31-38, The typical GPU is a highly complex integrated circuit device optimized to perform graphics computations (e.g., matrix transformations, scan-conversion and/or other rasterization techniques, texture blending, etc.) and write the results to the graphics memory. The GPU is a "slave" processor that operates in response to commands received from a driver program executing on a "master" processor; Col 3, lines 31-35, The load coefficient may be, e.g., an average of the recorded numeric values that can be compared to an arithmetic mean of the numeric values of the processor identifiers in order to determine whether an imbalance exists; Col 11, lines 18-27, determined whether the load coefficient exceeds a "high" threshold. The high threshold is preselected and may be exactly 0.5 or a somewhat higher value (e.g., 0.55 or 0.6). If the load coefficient exceeds the high threshold, then the loads are adjusted at step 512 by moving the boundary line P in FIG. 2 down by a preset amount (e.g., one line, five lines, ten lines). This reduces the fraction of the display area that is rendered by GPU-1, which will tend to reduce the load on GPU-1 and increase the load on GPU-0; Col 16, lines 8-23, In some multi-card embodiments used to render scenes in which foreground regions (most often but not always at the bottom of the display area) are consistently more complex than background regions, a performance advantage can be gained by assigning GPU 914a to process the background region of the scene and assigning GPU 914b to process the foreground region. For example, in FIG. 2, suppose that the foreground appears toward the bottom of display area 200. In that case, GPU 914a would be assigned to render top region 202 while GPU 914b would be assigned to render bottom region 204. The higher complexity of the foreground (bottom) region tends to increase the rendering time of GPU 914b. In response, the load-balancing processes described herein will tend to move the boundary line P toward the bottom of the display area [Examiner noted: the metric (i.e., load coefficient) is computed based, at least in part, on a type of the transformations (i.e., render scenes in which foreground regions are more complex than the background regions (as different types of transformations)]);
assign a subset of clients of the set of clients to a second hardware accelerator to cause the metric associated with each of the set of clients to be below the threshold in response to the determination (Diard, Fig. 5, 510 does average exceed high threshold, YES to 512 move boundary line down by preselected amount (as to cause the metric (i.e., load coefficient) below the threshold); Col 11, lines 11-27, determined whether the load coefficient exceeds a "high" threshold. The high threshold is preselected and may be exactly 0.5 or a somewhat higher value (e.g., 0.55 or 0.6). If the load coefficient exceeds the high threshold, then the loads are adjusted at step 512 by moving the boundary line P in FIG. 2 down by a preset amount (e.g., one line, five lines, ten lines) (as including a subset of clients of the set of clients). This reduces the fraction of the display area that is rendered by GPU-1, which will tend to reduce the load on GPU-1 and increase the load on GPU-0; see Fig. 2, 202 and 204, the load is adjusted between two GPUs; also see Col 10, lines 48-55, graphics driver may balance the load after some number (Q) of frames (as set of processes/frame renderings; i.e., set of clients); where Q might be, e.g., 1, 2, 5, 10, 20, etc…Alternatively, load balancing may be performed at regular time intervals (e.g., once per second) or according to any other criteria)
as a result of assigning the subset of clients, reassign at least one of the subset of clients to the first hardware accelerator to cause the metric associated with each of the set of clients to remain below the threshold (Diard, Abstract, lines 9-12, re-partitioned to increase a size of the portion assigned to the less heavily loaded processor and to decrease a size of the portion assigned to the more heavily loaded processor; Fig. 2, 202, 204, P and P’; Fig. 5, 510 does average exceed high threshold? NO to 514 is average below low threshold? YES to 516, move boundary line up by preselected amount (as reassigning to cause the metric associated with each of the set of clients to remain below the threshold); Col 8, lines 55-67, rendering commands 308 and associated rendering data for the next frame F1…At this point, the clip rectangles for each GPU may be modified by the graphics driver program based on the feedback data received in response to the various write notifier commands (e.g., commands 306, 310). For example, where the display area is divided as shown in FIG. 2, the value of P may be modified (e.g., to P') in response to feedback data: if the GPU that processes top portion 202 tends to finish its frames first, the value of P is increased, and if the GPU that processes bottom portion 204 tends to finish first, the value of P is decreased. Specific embodiments of re-partitioning a display area in response to feedback data are described below; Col 11, lines 18-27, at step 510 it is determined whether the load coefficient exceeds a "high" threshold… at step 514, it is determined whether the load coefficient is less than a "low" threshold. The low threshold is predefined and may be exactly 0.5 or a somewhat lower value (e.g., 0.45 or 0.4). If the load coefficient is below the low threshold, then the loads are adjusted at step 516 by moving the boundary line P in FIG. 2 up by a preset amount (e.g., one line, five lines, ten lines) (as reassign at least one of the subset of clients based on the dynamic load balancing between GPUs; also see Col 10, lines 48-55, graphics driver may balance the load after some number (Q) of frames (as set of processes/frame renderings).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat and Dutta with Diard because Diard’s teaching of dynamically adjusting the processing load among the GPUs based on the feedback data would have provided Bernat and Dutta’s system with the advantage and capability to allow the system to efficiently utilizing the resources based on the load which improving the processing speed and system efficiency.
Although Bernat, Dutta and Diard teach the frame transformations, Bernat, Dutta and Diard fail to specifically teach the frame transformations is one or more video frame transformations to a batch of video frames to modify one or more individual frames in the batch of video frames.
However, Lowry teaches the frame transformations is one or more video frame transformations with respect to a batch of video frames to modify one or more individual frames in the batch of video frames (Lowry, [0041] lines 3-9, Embodiments provide that a NVIDIA® K6000 (NVIDIA® is a registered trademark of NVIDIA) may be used as the GPU. Embodiments provide this as the first stage in the video pipeline (VP). In the GPU (14), video frames may be debayered, as well as other necessary video transformations, such as motion compensation, white balance, black level correction, etc.,).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta and Diard with Lowry because Lowry’s teaching of video frame transformations would have provided Bernat, Dutta and Diard’s system with the advantage and capability to allow the system to processing the video and image frames transformations (i.e., augmented reality) which improving the system performance and efficiency (see Lowry, [0004]).
Although Bernat, Dutta, Diard and Lowry teach assign a subset of clients of the set of clients to the second hardware accelerator, Bernat, Dutta, Diard and Lowry fail to specifically teach when assign a subset of clients of the set of clients, it is assign to a set of second hardware accelerators.
However, Shraer teaches when assign a subset of clients of the set of clients, it is assign to a set of second hardware accelerators (Shraer, [0016] lines 3-9, each set of partitions 132 may include multiple partitions...All of the partitions of all of the data sets 132 form the data set for the application job that is executed by the application system; [0033] lines 8-12, move operations can be considered for worker computers that are in a top subset of worker computers with high load measures, e.g., the top x %, or all worker computers with loads above a threshold load measure; [0062] lines 10-12, a partition may be migrated from one worker computer to two or more other worker computers, (as set of computers/hardware accelerators, please note: second hardware accelerator was taught by Bernat, Dutta and Diard).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Diard and Lowry with Shraer because Shraer’s teaching of assigning/migrating the operation/partition to two or more other worker computers for processing would have provided Bernat, Dutta, Diard and Lowry’s system with the advantage and capability to allow the system to re-balancing the load among different computers/accelerators in order to efficiently utilizing the resources which improving the system efficiency and performance.
Bernat, Dutta, Diard, Lowry and Shraer fail to specifically teach receive one or more calls via an application programming interface (API) from a set of clients, the one or more calls comprising one or more requests to perform one or more video frame transformations.
However, Guo teaches receive one or more calls via an application programming interface (API) from a set of clients, the one or more calls comprising one or more requests to perform one or more video frame transformations (Guo, [0513] one or more application programming interfaces (APIs) in an environment with calling program code interacting with other program code being called through the one or more interfaces; [0514] calling software component interacting with a called software component through an API. A method for operating through an API in this environment includes transferring one or more function calls, messages, other types of invocations or parameters via the API; [0515] Application Programming Interfaces (APIs) may be used in some embodiments. For example, some embodiments of the media exchange module 310 (or 910) provide a set of APIs to other software components for accessing various video processing and encoding functionalities described in FIGS. 3 and 6 such as the functionalities of the TNR module 1500 described in FIG. 15; also see [0091] The video processing module 325 performs image processing on the images and/or the videos captured by the cameras of the device. Examples of such operations include exposure adjustment operations, focus adjustment operations, perspective correction, dynamic range adjustment, image resizing, image compositing, etc. In some embodiments, some image processing operations can also be performed by the media exchange module 310. For instance, as shown in FIG. 3, the media exchange module 310 of some embodiments performs a temporal noise reduction (TNR) operation (e.g., by TNR 315) that reduces noise in video images captured by the cameras of the device. Further examples of such image processing operations of the video processing module 325 and the media exchange module 310 will be provided below; [0517] An API allows a developer of an API-calling component (which may be a third party developer) to leverage specified features provided by an API-implementing component. There may be one API-calling component or there may be more than one such component. An API can be a source code interface that a computer system or program library provides in order to support requests for services from an application; [0523] The API may allow multiple API-calling components written in different programming languages to communicate with the API-implementing component (thus the API may include features for translating calls and returns between the API-implementing component and the API-calling component); however the API may be implemented in terms of a specific programming language. An API-calling component can, in one embodiment, call APIs from different providers such as a set of APIs from an OS provider and another set of APIs from a plug-in provider and another set of APIs from another provider (e.g. the provider of a software library) or creator of the another set of APIs; also see [0284] encodes the image (e.g., using discrete cosine transform, quantization, entropy encoding, etc.) and returns a bitstream with the encoded image to the encoder driver 2200; please note: video frame transformation also taught by Lowry).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Diard, Lowry and Shraer with Guo because Guo’s teaching of using the API call for requesting video image transformations would have provided Bernat, Dutta, Diard, Lowry and Shraer’s system with the advantage and capability to allow the system to utilizing the different APIs from different providers for requesting different image/video processing which improving the system performance and efficiency.
As per claim 2, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Diard further teaches wherein the threshold is determined based, at least in part, on a framerate associated with the batch of video frames (Diard, Col 10, lines 64-Col 11, line 6, the graphics driver might not know whether the GPUs have finished a particular frame and the GPUs may be rendering a frame that is several frames earlier in the command stream than a current frame in the graphics driver. Where the feedback array is written in a circular fashion, as in process 400 described above, selecting Q to be equal to B provides an average over the B most recently rendered frames. In some embodiments, a weighted average may be used, e.g., giving a larger weight to more recently-rendered frames; Col 11, lines 7-17, The load coefficient is used to determine whether an adjustment to the clip rectangles for the GPUs needs to be made. If the GPUs are equally loaded, the likelihood of either GPU finishing a frame first is about 50%, and the average value over a suitable number of frames (e.g., 20) will be about 0.5 if identifier values of 0 and 1 are used. An average value in excess of 0.5 indicates that GPU-1 (which renders the bottom portion of the image) is more heavily loaded than GPU-0, and an average value below 0.5 indicates that GPU-0 (which renders the top portion of the image) is more heavily loaded than GPU-1; also see Col 12, lines 30-31, 30 frames are rendered per second).
As per claim 9, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Bernat further teaches wherein the determination is generated during an interval of time (Bernat, [0002] line 29, execution time of the workload; [0058] lines 7-9, during execution of the workloads (as during an interval of time; i.e., time for execution of the workload), the accelerator 1250 is to monitor the actual utilization of the resources allocated to the logic portions 1252, 1254).
As per claim 13, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Diard further teaches wherein the type comprises modifying resolutions of the batch of video frames, converting formats of the batch of video frames, or modifying color of the batch of video frames (Diard, Col 5, lines 36-40, These functions can include, for example, lighting transformations, coordinate transformations, scan-conversion of geometric primitives to rasterized data, shading computations, shadow rendering, texture blending, and so on).
As per claim 15, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Bernat further teaches wherein the first hardware accelerator further comprises a field-programmable gate array (FPGA) (Bernat, [0002] lines 1-2, Typical architectures for accelerator devices such as field programmable gate arrays (FPGAs)).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Bernat, Dutta, Diard, Lowry, Shraer and Guo, as applied to claim 1 above, and further in view of Fawcett (US Pub. 2020/0225989 A1).
Fawcett was cited in the previous Office Action.
As per claim 3, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Diard further teaches wherein the metric further comprises a value indicating a level of activity of the first hardware accelerator (Diard, Col 11, lines 11-27, determined whether the load coefficient exceeds a "high" threshold (as load level activity). The high threshold is preselected and may be exactly 0.5 or a somewhat higher value (e.g., 0.55 or 0.6). If the load coefficient exceeds the high threshold, then the loads are adjusted at step 512 by moving the boundary line P in FIG. 2 down by a preset amount (e.g., one line, five lines, ten lines). This reduces the fraction of the display area that is rendered by GPU-1, which will tend to reduce the load on GPU-1 and increase the load on GPU-0; see Fig. 2, 202 and 204, the load is adjusted between two GPUs; please notes: first hardware accelerator was taught by Bernat).
Bernat, Dutta, Diard, Lowry, Shraer and Guo fail to specifically teach wherein the first metric further comprises a value indicating a percentage of activity.
However, Fawcett teaches wherein the first metric further comprises a value indicating a percentage of activity (Fawcett, lines 1-4, A load metric is a measurement of the current usage level of a resource relative to the resource's total capacity. For example, a load metric of 75 percent means that that the resource is at 75 percent of its total capacity).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Diard, Lowry, Shraer and Guo with Fawcett because Fawcett’s teaching of load metric that indicating the percentage of load capacity would have provided Bernat, Dutta, Diard, Lowry, Shraer and Guo’s system with the advantage and capability to allow the system to easily identifying the current capacity of the load which enable the system to allocating the resource based on the load and improving the system performance.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Bernat, Dutta, Diard, Lowry, Shraer and Guo, as applied to claim 1 above, and further in view of Takemoto et al. (US Patent. 5,347,622).
Takemoto was cited in the previous Office Action.
As per claim 4, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Diard further teaches the set of second hardware accelerators further comprises a graphics processing unit (GPU) (Diard, Fig. 6, GPU 1, GPU 2).
Bernat, Dutta, Diard, Lowry, Shraer and Guo fail to specifically teach wherein the first hardware accelerator further comprises a video image compositor (VIC).
However, Takemoto teaches wherein the first hardware accelerator further comprises a video image compositor (VIC) (Takemoto, Col 2, lines 17-20, A first plurality of crosspoint switches connect the plurality of digital video signal inputs to the key processing subsystem and to the video image compositor).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Diard, Lowry, Shraer and Guo with Takemoto because Takemoto’s teaching of video image compositor would have provided Bernat, Dutta, Diard, Lowry, Shraer and Guo’s system with the advantage and capability to allow the system to processing the digital video images which improving the system performance and efficiency.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Bernat, Dutta, Diard, Lowry, Shraer and Guo, as applied to claim 1 above, and further in view of Zhao et al. (US Pub. 2021/0319056 A1).
As per claim 5, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Bernat, Dutta, Diard, Lowry, Shraer and Guo fail to specifically teach wherein the one or more calls comprise respective client identification numbers used to associate the one or more requests with the set of clients.
However, Zhao teaches wherein the one or more calls comprise respective client identification numbers used to associate the one or more requests with the set of clients (Zhao, [0052] media retrieval module 114 sends a media request 116 across the network 108 including a user ID of the user and the query. The media retrieval manager module 118 receives the media request 116, determines a user specific query embedding for a user associated with the user ID from user embeddings of the augmented graph and query embeddings of the entered query).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Diard, Lowry, Shraer and Guo with Zhao because Zhao’s teaching of request associated with user ID would have provided Bernat, Dutta, Diard, Lowry, Shraer and Guo’s system with the advantage and capability to allow the system to easily determining the specific user associated with the request via the ID which improving the system performance and efficiency.
Claims 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Bernat, Dutta, Diard, Lowry, Shraer and Guo, as applied to claim 1 above, and further in view of Ben ZEEV et al. (US Pub. 2017/0132046 A1; hereafter: ZEEV).
ZEEV was cited in the previous Office Action.
As per claim 6, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Bernat teaches the first hardware accelerator (Bernat, [0002] lines 1-2, Typical architectures for accelerator devices such as field programmable gate arrays (FPGAs)). In addition, Diard teaches perform processing of the batch of video frames (Diard, Col 10, lines 64-Col 11, line 6, the graphics driver might not know whether the GPUs have finished a particular frame and the GPUs may be rendering a frame that is several frames earlier in the command stream than a current frame in the graphics driver. Where the feedback array is written in a circular fashion, as in process 400 described above, selecting Q to be equal to B provides an average over the B most recently rendered frames. In some embodiments, a weighted average may be used, e.g., giving a larger weight to more recently-rendered frames; Col 11, lines 7-17, The load coefficient is used to determine whether an adjustment to the clip rectangles for the GPUs needs to be made. If the GPUs are equally loaded, the likelihood of either GPU finishing a frame first is about 50%, and the average value over a suitable number of frames (e.g., 20) will be about 0.5 if identifier values of 0 and 1 are used. An average value in excess of 0.5 indicates that GPU-1 (which renders the bottom portion of the image) is more heavily loaded than GPU-0, and an average value below 0.5 indicates that GPU-0 (which renders the top portion of the image) is more heavily loaded than GPU-1; also see Col 12, lines 30-31, 30 frames are rendered per second).
Bernat, Dutta, Diard, Lowry, Shraer and Guo fail to specifically teach wherein the metric further comprises an amount of time at least one of the set of clients utilizes the first hardware accelerator to perform processing of the batch of frames.
However, ZEEV teaches wherein the metric further comprises an amount of time at least one of the set of clients utilizes the first hardware accelerator to perform processing of the batch of frames (ZEEV, [0027] lines 1-7, Information related to usage of the resource for an individual time interval may comprise, for example, a number of requests from the tenant for the resource, a percentage of processing capability of the resource used by the tenant, an amount of time of the time interval during which processing capability was used by the resource, and/or other metric to measure usage of the resource by the tenant during the time interval).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Diard, Lowry, Shraer and Guo with ZEEV because ZEEV’s teaching of the metric data including the amount of time used for processing would have provided Bernat, Dutta, Diard, Lowry, Shraer and Guo’s system with the advantage and capability to allow the system to easily determining the processing time needed for task processing which improving the system performance and efficiency.
As per claim 7, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Diard teaches process the batch of video frames on behalf of at least one of the set of clients (Diard, Col 10, lines 64-Col 11, line 6, the graphics driver might not know whether the GPUs have finished a particular frame and the GPUs may be rendering a frame that is several frames earlier in the command stream than a current frame in the graphics driver. Where the feedback array is written in a circular fashion, as in process 400 described above, selecting Q to be equal to B provides an average over the B most recently rendered frames. In some embodiments, a weighted average may be used, e.g., giving a larger weight to more recently-rendered frames; Col 11, lines 7-17, The load coefficient is used to determine whether an adjustment to the clip rectangles for the GPUs needs to be made. If the GPUs are equally loaded, the likelihood of either GPU finishing a frame first is about 50%, and the average value over a suitable number of frames (e.g., 20) will be about 0.5 if identifier values of 0 and 1 are used. An average value in excess of 0.5 indicates that GPU-1 (which renders the bottom portion of the image) is more heavily loaded than GPU-0, and an average value below 0.5 indicates that GPU-0 (which renders the top portion of the image) is more heavily loaded than GPU-1; also see Col 12, lines 30-31, 30 frames are rendered per second). In addition, Bernat teaches the first hardware accelerator (Bernat, [0002] lines 1-2, Typical architectures for accelerator devices such as field programmable gate arrays (FPGAs)).
Bernat, Dutta, Diard, Lowry, Shraer and Guo fail to specifically teach wherein the metric further comprises a percentage of processing capability the first hardware accelerator utilized to processing.
However, ZEEV teaches wherein the metric further comprises a percentage of processing capability the first hardware accelerator utilized to processing (ZEEV, [0027] lines 1-7, Information related to usage of the resource for an individual time interval may comprise, for example, a number of requests from the tenant for the resource, a percentage of processing capability of the resource used by the tenant, an amount of time of the time interval during which processing capability was used by the resource, and/or other metric to measure usage of the resource by the tenant during the time interval).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Diard, Lowry, Shraer and Guo with ZEEV because ZEEV’s teaching of the metric data including the percentage of processing capability would have provided Bernat, Dutta, Diard, Lowry, Shraer and Guo’s system with the advantage and capability to allow the system to easily determining the processing capability of the device for task processing which improving the system performance and efficiency.
Claims 8 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Bernat, Dutta, Diard, Lowry, Shraer and Guo, as applied to claim 1 above, and further in view of Ueda (US Pub. 2012/0254443 A1).
Ueda was cited in the previous Office Action.
As per claim 8, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Diard teaches the batch of video frames provided by at least one of the set of clients (Diard, Fig. 1, 116b, 123a, 123b, 122b; Col 6, lines 41-43, a graphics driver program (or other program) executing on CPU 102 delivers rendering commands and associated data for processing by GPUs 114a, 114b; Col 10, lines 64-Col 11, line 6, the graphics driver might not know whether the GPUs have finished a particular frame and the GPUs may be rendering a frame that is several frames earlier in the command stream than a current frame in the graphics driver. Where the feedback array is written in a circular fashion, as in process 400 described above, selecting Q to be equal to B provides an average over the B most recently rendered frames. In some embodiments, a weighted average may be used, e.g., giving a larger weight to more recently-rendered frames; Col 11, lines 7-17, The load coefficient is used to determine whether an adjustment to the clip rectangles for the GPUs needs to be made. If the GPUs are equally loaded, the likelihood of either GPU finishing a frame first is about 50%, and the average value over a suitable number of frames (e.g., 20) will be about 0.5 if identifier values of 0 and 1 are used. An average value in excess of 0.5 indicates that GPU-1 (which renders the bottom portion of the image) is more heavily loaded than GPU-0, and an average value below 0.5 indicates that GPU-0 (which renders the top portion of the image) is more heavily loaded than GPU-1; also see Col 12, lines 30-31, 30 frames are rendered per second).
Bernat, Dutta, Diard, Lowry, Shraer and Guo fail to specifically teach wherein the metric further comprises an average amount of load generated by at least processing.
However, Ueda teaches wherein the metric further comprises an average amount of load generated by at least processing (Ueda, [0043] lines 1-7, the transfer condition for transferring to the alternate server may include threshold conditions for various metrics of the instances in the load distribution target server group for the designated load balancer 110, such as the average CPU utilization rate, the average memory utilization rate, the average degree of I/O utilization).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Diard, Lowry, Shraer and Guo with Ueda because Ueda’s teaching of the metric data including the average amount of load/utilization would have provided Bernat, Dutta, Diard, Lowry, Shraer and Guo’s system with the advantage and capability to allow the system to easily determining the average processing load for task processing in order to distributing the tasks based on the average processing load which improving the system performance and efficiency.
As per claim 10, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Bernat, Dutta, Diard, Lowry, Shraer and Guo fail to specifically teach wherein the determination is generated based at least in part on historical data.
However, Ueda teaches wherein the determination is generated based at least in part on historical data (Ueda, [0043] lines 1-7, the transfer condition for transferring to the alternate server may include threshold conditions for various metrics of the instances in the load distribution target server group for the designated load balancer 110, such as the average CPU utilization rate, the average memory utilization rate, the average degree of I/O utilization; also see [0060] in addition to the target server size based on the demands quantified by the load balancer, a predicted server size based on demand prediction using history information is determined. If the demands quantified by the load balancer are underestimated compared with the demands predicted with the history information, the server size based on the demand prediction may be selected).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Diard, Lowry, Shraer and Guo with Ueda because Ueda’s teaching of the utilizing the historical data for determining the target server for processing would have provided Bernat, Dutta, Diard, Lowry, Shraer and Guo’s system with the advantage and capability to allow the system to predicting the processing load based on the historical data for task assignment which improving the system performance and efficiency.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable Bernat, Dutta, Diard, Lowry, Shraer and Guo, as applied to claim 1 above, and further in view of IZENBERG et al. (US Pub. 2017/0195173 A1).
IZENBERG was cited in the previous Office Action.
As per claim 11, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Bernat, Dutta, Diard, Lowry, Shraer and Guo fail to explicitly teach wherein the configuration information is generated based, at least in part, on user input.
However, IZENBERG teaches wherein the configuration information is generated based, at least in part, on user input (IZENBERG, [0066] lines 5-10, the resource manager may recommend that the pre-existing instance 1053A be used for the second FPGA-utilizing application (assuming that the same FPGA can be used for the second application). In other embodiments, or based on the preferences of the client (as configuration is generated based on the user input), a new instance such as 1053B may be launched for the second application; [0067] in their programmatic interactions with the resource manager, and the resource manager may take the appropriate resource allocation choices based on the client's preferences)
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Diard, Lowry, Shraer and Guo with IZENBERG because IZENBERG’s teaching of providing/generating the resource allocation preference based on the user input would have provided Bernat, Dutta, Diard, Lowry, Shraer and Guo’s system with the advantage and capability to allow the system to allocating the different resource based on the user selection and preferences which improving the user experience and system performance.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Bernat, Dutta, Diard, Lowry, Shraer and Guo, as applied to claim 1 above, and further in view of Bernat et al (US Pub. 2020/0409748 A1; hereafter Bernat ‘748’).
Bernat ‘748’ was cited in the previous Office Action.
As per claim 12, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Bernat, Dutta, Diard, Lowry, Shraer and Guo fail to specifically teach wherein the threshold is designated by a user.
However, Bernat’748’ teaches wherein the threshold is designated by a user (Bernat, [0111] one or more power consumption thresholds specified in a policy (e.g., a SLA, QoS requirements, load balancing policies, application specifications, etc.); [0121] lines 1-4, as a function of a policy (e.g., a SLA, QoS requirements, a load balancing policy, user-defined specifications in an application associated with the workload, etc.)).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Diard, Lowry, Shraer and Guo with Bernat’748’ because Bernat’748’’s teaching of resource utilization threshold that is defined by user would have provided Bernat, Dutta, Diard, Lowry, Shraer and Guo’s system with the advantage and capability to allow the user to easily controlling the resource limit for running the workload which improving the user experience and system performance.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Bernat, Dutta, Diard, Lowry, Shraer and Guo, as applied to claim 1 above, and further in view of Bonebakker et al. (US Patent. 8,688,430 B1).
Bonebakker was cited in the previous Office Action.
As per claim 14, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Bernat, Dutta, Diard, Lowry, Shraer and Guo fail to specifically teach wherein the metric is obtained at least from a hardware performance counter included in the first hardware accelerator.
However, Bonebakker teaches wherein the metric is obtained from a hardware performance counter included in the first hardware accelerator (Bonebakker, Col 3, lines 29-31, one or more system metrics can be sampled values obtained from hardware counters within processor 102; please note: first hardware accelerator was taught by Bernat).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Diard, Lowry, Shraer and Guo with Bonebakker because Bonebakker’s teaching of obtaining metrics from the hardware counters would have provided Bernat, Dutta, Diard, Lowry, Shraer and Guo’s system with the advantage and capability to allow the system to obtaining the metrics from counters which improving the system performance and efficiency.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Bernat, Dutta, Diard, Lowry, Shraer and Guo, as applied to claim 1 above, and further in view of Joglekar et al. (US Pub. 2021/0374027 A1).
Joglekar was cited in the previous Office Action.
As per claim 16, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Bernat, Dutta, Diard, Lowry, Shraer and Guo fail to specifically teach wherein the memory further stores instructions that, as a result of being executed by the one or more processors, cause the system to obtain, through a system call, the metric.
However, Joglekar teaches wherein the memory further stores instructions that, as a result of being executed by the one or more processors, cause the system to obtain, through a system call, the metric (Joglekar, [0082] the metrics collecting agents 214 can retrieve and interpret metrics and their associated metric values from system calls made by the monitored services 210 and 212).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Diard, Lowry, Shraer and Guo with Joglekar because Joglekar’s teaching of obtaining metrics from the system call would have provided Bernat, Dutta, Diard, Lowry, Shraer and Guo’s system with the advantage and capability to allow the system to obtaining the metrics from system calls which improving the system performance and efficiency.
Claims 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Bernat, Dutta, Diard, Lowry, Shraer and Guo, as applied to claim 1 above, and further in view of Alvelda, VII et al. (US Pub. 2020/0106829 A1).
Alvelda was cited in the previous Office Action.
As per claim 17, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Bernat, Dutta, Diard, Lowry, Shraer and Guo fail to specifically teach wherein the set of clients further comprise a set of components of an artificial intelligence pipeline.
However, Alvelda teaches wherein the set of clients further comprise a set of components of an artificial intelligence pipeline (Alvelda, Abstract, lines 1-7, methods suitable for partitioning processes between devices. In particular, the present invention relates to partitioning client devices and server devices based on the performance and available resources of the respective devices to efficiently execute artificial intelligent, machine learning, and other processes; [0028] lines 1-3, The present disclosure makes use of a code base and library of executable files where machine learning and AI algorithms; [0030] lines 3-7, Specifically, FIG. 1 depicts an illustrative system 100 for partitioning data processing pipelines (e.g., for AI and machine learning algorithms) over a plurality of client devices).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Diard, Lowry, Shraer and Guo with Alvelda because Alvelda’s teaching of partitioning data machine learning processing pipelines over a plurality of client devices would have provided Bernat, Dutta, Diard, Lowry, Shraer and Guo’s system with the advantage and capability to allow the system to processing the AI driven applications which improving the system performance and efficiency.
As per claim 18, Bernat, Dutta, Diard, Lowry, Shraer, Guo and Alvelda teach the invention according to claim 17 above. Alvelda further teaches wherein the artificial intelligence pipeline includes one or more neural networks (Alvelda, [0028] lines 1-3, The present disclosure makes use of a code base and library of executable files where machine learning and AI algorithms; [0030] lines 3-7, Specifically, FIG. 1 depicts an illustrative system 100 for partitioning data processing pipelines (e.g., for AI and machine learning algorithms) over a plurality of client devices; [0027] encrypted machine-readable data in complex distributed data stores such as neural weight matrices).
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Bernat, Dutta, Diard, Lowry, Shraer and Guo, as applied to claim 1 above, and further in view of Galluzzi et al. (US Patent. 11,019,298 B2).
Galluzzi was cited in the previous Office Action.
As per claim 19, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Bernat teaches cause the first hardware accelerator to process the operations (Bernat, [0084] lines 28-32, metadata indicative of the types operations to be executed in the workload (as include operations), or from a request from a CPU (e.g., the CPU 1272) to offload a portion of a workload onto a particular accelerator 1250, 1260 or type of accelerator (e.g., FPGA, graphics accelerator, cryptography accelerator, compression accelerator, etc.), and assign the workload; Fig. 18, 1726 execute the workload with corresponding logic portion of the accelerator). In addition, Diard teaches when processing the operations, it is processing the batch of video frames (Diard, Col 1, lines 24-38, Graphics processing subsystems are designed to render realistic animated images in real time, e.g., at 30 or more frames per second. These subsystems are most often implemented on expansion cards that can be inserted into appropriately configured slots on a motherboard of a computer system and generally include one or more dedicated graphics processing units (GPUs) and dedicated graphics memory. The typical GPU is a highly complex integrated circuit device optimized to perform graphics computations (e.g., matrix transformations, scan-conversion and/or other rasterization techniques, texture blending, etc.) and write the results to the graphics memory. The GPU is a "slave" processor that operates in response to commands received from a driver program executing on a "master" processor, generally the central processing unit (CPU) of the system)).
Bernat, Dutta, Diard, Lowry, Shraer and Guo fail to explicitly teach when processing batch of frames it is by at least converting the batch of video frames from a first format to a second format.
However, Galluzzi teaches when processing batch of frames it is by at least converting the batch of video frames from a first format to a second format (Galluzzi, claim 1, grabbing the video frames in the first format and converting the video frames to a second format different from the first format).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Diard, Lowry, Shraer and Guo with Galluzzi because Galluzzi’s teaching of converting the first format of video frames to second format would have provided Bernat, Dutta, Diard, Lowry, Shraer and Guo’s system with the advantage and capability to allow the system to processing the different format of video frames which improving the system performance and efficiency.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Bernat, Dutta, Diard, Lowry, Shraer and Guo, as applied to claim 1 above, and further in view of Wu et al. (US Pub. 2017/0064313 A1).
Wu was cited in the previous Office Action.
As per claim 20, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Bernat teaches cause the first hardware accelerator to process operations (Bernat, [0084] lines 28-32, metadata indicative of the types operations to be executed in the workload (as include operations), or from a request from a CPU (e.g., the CPU 1272) to offload a portion of a workload onto a particular accelerator 1250, 1260 or type of accelerator (e.g., FPGA, graphics accelerator, cryptography accelerator, compression accelerator, etc.), and assign the workload; Fig. 18, 1726 execute the workload with corresponding logic portion of the accelerator). In addition, Diard teaches when processing the operations, it is processing the batch of video frames (Diard, Col 10, lines 64-Col 11, line 6, the graphics driver might not know whether the GPUs have finished a particular frame and the GPUs may be rendering a frame that is several frames earlier in the command stream than a current frame in the graphics driver. Where the feedback array is written in a circular fashion, as in process 400 described above, selecting Q to be equal to B provides an average over the B most recently rendered frames. In some embodiments, a weighted average may be used, e.g., giving a larger weight to more recently-rendered frames; also see Col 12, lines 30-31, 30 frames are rendered per second).
Bernat, Dutta, Diard, Lowry, Shraer and Guo fail to specifically teach when processing the batch of frames, it is by at least scaling the batch of video frames.
However, Wu teaches when processing the batch of frames, it is by at least scaling the batch of video frames (Wu, [0185] The accelerator scales the reference frames (2110), producing scaled versions (2130) of the reference frames at the second spatial resolution, which matches the spatial resolution of the current frame (2120). The scaled versions (2130) of the reference frames are labeled 088′, 089′, and 090′ for the sake of illustration, but the labelsneed not correspond to values tracked by the accelerator. Then, the accelerator performs motion compensation for blocks of the current frame (2120) using motion-compensated prediction values from the scaled versions (2130) of the reference frames).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Diard, Lowry, Shraer and Guo with Wu because Wu’s teaching of scaling the batch of frames for processing would have provided Bernat, Dutta, Diard, Lowry, Shraer and Guo’s system with the advantage and capability to allow the system to improve the frame processing speed which improving the system performance and efficiency.
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Bernat, Dutta, Diard, Lowry, Shraer and Guo, as applied to claim 1 above, and further in view of Saulters (US Pub. 2014/0125670 A1).
Saulters was cited in the previous Office Action.
As per claim 21, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Bernat teaches cause the first hardware accelerator to process operations (Bernat, [0084] lines 28-32, metadata indicative of the types operations to be executed in the workload, or from a request from a CPU (e.g., the CPU 1272) to offload a portion of a workload onto a particular accelerator 1250, 1260 or type of accelerator (e.g., FPGA, graphics accelerator, cryptography accelerator, compression accelerator, etc.), and assign the workload; Fig. 18, 1726 execute the workload with corresponding logic portion of the accelerator)). In addition, Diard teaches when processing the operations, it is processing the batch of video frames (Diard, Col 10, lines 64-Col 11, line 6, the graphics driver might not know whether the GPUs have finished a particular frame and the GPUs may be rendering a frame that is several frames earlier in the command stream than a current frame in the graphics driver. Where the feedback array is written in a circular fashion, as in process 400 described above, selecting Q to be equal to B provides an average over the B most recently rendered frames. In some embodiments, a weighted average may be used, e.g., giving a larger weight to more recently-rendered frames; also see Col 12, lines 30-31, 30 frames are rendered per second; Col 2, lines 66-67, The graphics processors are instructed to render a number of frames, wherein the first and second graphics process; Col 5, lines 37-40, lighting transformations, coordinate transformations, scan-conversion of geometric primitives to rasterized data, shading computations, shadow rendering, texture blending).
Bernat, Dutta, Diard, Lowry, Shraer and Guo fail to explicitly teach when processing, it is by at least modifying one or more color values associated with at least one frame of the batch of video frames.
However, Saulters teaches when processing, it is by at least modifying one or more color values associated with at least one frame of the batch of video frames (Saulters, [0046] lines 1-8, Graphics driver 140 loads a shader to GPU 122, so as to enable GPU 122 to adjust color values of one or more sample areas on the current rendered frame, based on at least the values of the frame transformation matrix for the current rendered frame and the previous rendered frame (as discussed in Step S01) and the depth values of the current rendered frame (as discussed in Step S02), whereby a motion blur effect is created or approximated in the current rendered frame. Also see [0049] With such information, GPU 122 can determine, for example, the locations and sizes of sample areas 252 and 254 on the rendered frame 220 and adjust the color values of the sample areas 252).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Diard, Lowry, Shraer and Guo with Saulters because Saulters’s teaching of adjusting color values of one or more sample areas on the current rendered frame would have provided Bernat, Dutta, Diard, Lowry, Shraer and Guo’s system with the advantage and capability to allow the system to adjusting the color values of the frame in order to improving the system performance and efficiency.
Claims 22-25 and 29 are rejected under 35 U.S.C. 103 as being unpatentable over Diard (US Patent. 7,075,541 B2) in view of Bernat et al (US Pub. 2018/0027062 A1) and further in view of Dutta (US Patent. 9,313,134 B2), Takemoto et al. (US Patent 5,347,622), Lowry et al. (US Pub. 2018/0262684 A1), Desai et al. (US Pub. 2011/0078318 A1), Shraer et al. (US Pub. 2017/0353536 A1), and Guo et al. (US Pub. 2011/0249086 A1).
Diard, Bernat, Dutta, Takemoto, Lowry, Desai and Shraer were cited in the previous Office Action.
As per claim 22, Diard teaches the invention substantially as claimed including A method comprising:
assigning the set of application clients to a GPU to execute frame transformations corresponding to the set of application clients (Diard, Col 1, lines 58-60, For example, each GPU may be instructed to render pixel data for a different portion of the displayable image, such as a number of lines of a raster-based display (as assigning the set of application clients to GPU for processing; see specs [0057] “a client (e.g., application, processes, or other components”), Col 1, lines 24-38, Graphics processing subsystems are designed to render realistic animated images in real time, e.g., at 30 or more frames per second. These subsystems are most often implemented on expansion cards that can be inserted into appropriately configured slots on a motherboard of a computer system and generally include one or more dedicated graphics processing units (GPUs) and dedicated graphics memory. The typical GPU is a highly complex integrated circuit device optimized to perform graphics computations (e.g., matrix transformations, scan-conversion and/or other rasterization techniques, texture blending, etc.) and write the results to the graphics memory. The GPU is a "slave" processor that operates in response to commands received from a driver program executing on a "master" processor, generally the central processing unit (CPU) of the system; Col 12, lines 30-31, 30 frames are rendered per second; Col 2, lines 66-67, The graphics processors are instructed to render a number of frames, wherein the first and second graphics process; Col 5, lines 37-40, lighting transformations, coordinate transformations, scan-conversion of geometric primitives to rasterized data, shading computations, shadow rendering, texture blending);
determining that time/load used by the GPU to execute the one or more frame transformations for the set of application clients exceeds a frame processing threshold based, at least in part, on a type of the one or more frame transformations (Diard, Col 13, lines 12-24, different feedback data may be used instead of or in addition to the GPU identifiers described above. For example, instead of providing one feedback array in system memory, with both GPUs writing feedback data to the same location for a given frame, each GPU may write to a corresponding entry of a different feedback array, and the feedback data may include timing information, e.g., a timestamp indicating when each GPU finished a particular frame. In this embodiment, the graphics driver is configured to use the timing information to determine whether one GPU is consistently using more time per frame than another and adjust the clip rectangles accordingly to balance the load; also see Col 11, lines 17-25, determined whether the load coefficient exceeds a "high" threshold. The high threshold is preselected and may be exactly 0.5 or a somewhat higher value (e.g., 0.55 or 0.6). If the load coefficient exceeds the high threshold, then the loads are adjusted at step 512 by moving the boundary line P in FIG. 2 down by a preset amount (e.g., one line, five lines, ten lines) (as determine the time/related load used exceed a frame processing threshold); Col 16, lines 8-23, In some multi-card embodiments used to render scenes in which foreground regions (most often but not always at the bottom of the display area) are consistently more complex than background regions, a performance advantage can be gained by assigning GPU 914a to process the background region of the scene and assigning GPU 914b to process the foreground region. For example, in FIG. 2, suppose that the foreground appears toward the bottom of display area 200. In that case, GPU 914a would be assigned to render top region 202 while GPU 914b would be assigned to render bottom region 204. The higher complexity of the foreground (bottom) region tends to increase the rendering time of GPU 914b. In response, the load-balancing processes described herein will tend to move the boundary line P toward the bottom of the display area [Examiner noted: as type of the transformations (i.e., render scenes in which foreground regions are more complex than the background regions (as different types of transformations)]);
assigning, to a processing unit distinct from the GPU , a subset of the application clients of the set of application clients to cause time/load used by the processing unit to be below the frame processing threshold, wherein assigning the subset of application clients causes the processing unit to execute one or more portions of the one or more frame transformations to modify one or more individual frames in a batch of frames (Diard, Col 13, lines 12-24, different feedback data may be used instead of or in addition to the GPU identifiers described above. For example, instead of providing one feedback array in system memory, with both GPUs writing feedback data to the same location for a given frame, each GPU may write to a corresponding entry of a different feedback array, and the feedback data may include timing information, e.g., a timestamp indicating when each GPU finished a particular frame. In this embodiment, the graphics driver is configured to use the timing information to determine whether one GPU is consistently using more time per frame than another and adjust the clip rectangles accordingly to balance the load; Col 11, lines 18-27, at step 510 it is determined whether the load coefficient exceeds a "high" threshold...then the loads are adjusted at step 512 by moving the boundary line P in FIG. 2 down by a preset amount (e.g., one line, five lines, ten lines). This reduces the fraction of the display area that is rendered by GPU-1, which will tend to reduce the load on GPU-1 and increase the load on GPU-0. (as assigning to a processing unit distinct from the original GPU, a subset of the application clients to cause time used by the set of processing units to be below the threshold); Col 2, lines 66-67, The graphics processors are instructed to render a number of frames, wherein the first and second graphics process; Col 5, lines 37-40, lighting transformations, coordinate transformations, scan-conversion of geometric primitives to rasterized data, shading computations, shadow rendering, texture blending); and
reassigning the subset of application clients to cause the time/load used by the GPU to remain below the frame processing threshold (Diard, Abstract, lines 9-12, re-partitioned to increase a size of the portion assigned to the less heavily loaded processor and to decrease a size of the portion assigned to the more heavily loaded processor; Fig. 2, 202, 204, P and P’; Fig. 5, 510 does average exceed high threshold? NO to 514 is average below low threshold? YES to 516, move boundary line up by preselected amount (as reassigning to cause the metric associated with each of the set of clients to remain below the threshold); Col 8, lines 55-67, rendering commands 308 and associated rendering data for the next frame F1…At this point, the clip rectangles for each GPU may be modified by the graphics driver program based on the feedback data received in response to the various write notifier commands (e.g., commands 306, 310). For example, where the display area is divided as shown in FIG. 2, the value of P may be modified (e.g., to P') in response to feedback data: if the GPU that processes top portion 202 tends to finish its frames first, the value of P is increased, and if the GPU that processes bottom portion 204 tends to finish first, the value of P is decreased. Specific embodiments of re-partitioning a display area in response to feedback data are described below; Col 11, lines 18-27, at step 510 it is determined whether the load coefficient exceeds a "high" threshold… at step 514, it is determined whether the load coefficient is less than a "low" threshold. The low threshold is predefined and may be exactly 0.5 or a somewhat lower value (e.g., 0.45 or 0.4). If the load coefficient is below the low threshold, then the loads are adjusted at step 516 by moving the boundary line P in FIG. 2 up by a preset amount (e.g., one line, five lines, ten lines) (as reassign at least one of the subset of clients based on the dynamic load balancing between GPUs; also see Col 10, lines 48-55, graphics driver may balance the load after some number (Q) of frames (as set of processes/frame renderings).
cause the GPU to execute the one or more portions of the one or more frame transformations as a result of reassigning the subset of application clients. (Diard, Abstract, lines 9-12, re-partitioned to increase a size of the portion assigned to the less heavily loaded processor and to decrease a size of the portion assigned to the more heavily loaded processor; Fig. 2, 202, 204, P and P’; Fig. 5, 510 issue initial clip rectangle command, 514 is average below low threshold? YES to 516, move boundary line up by preselected amount; Col 8, lines 55-67, rendering commands 308 and associated rendering data for the next frame F1…At this point, the clip rectangles for each GPU may be modified by the graphics driver program based on the feedback data received in response to the various write notifier commands (e.g., commands 306, 310). For example, where the display area is divided as shown in FIG. 2, the value of P may be modified (e.g., to P') in response to feedback data: if the GPU that processes top portion 202 tends to finish its frames first, the value of P is increased, and if the GPU that processes bottom portion 204 tends to finish first, the value of P is decreased. Specific embodiments of re-partitioning a display area in response to feedback data are described below; Col 11, lines 18-27, at step 510 it is determined whether the load coefficient exceeds a "high" threshold… at step 514, it is determined whether the load coefficient is less than a "low" threshold. The low threshold is predefined and may be exactly 0.5 or a somewhat lower value (e.g., 0.45 or 0.4). If the load coefficient is below the low threshold, then the loads are adjusted at step 516 by moving the boundary line P in FIG. 2 up by a preset amount (e.g., one line, five lines, ten lines); Col 5, lines 37-40, lighting transformations, coordinate transformations, scan-conversion of geometric primitives to rasterized data, shading computations, shadow rendering, texture blending).
Diard fails to specifically teach when assigning to a GPU, it is indicated as a pre-selected hardware accelerator to execute, based, at least in part, on configuration information.
However, Bernat teaches when assigning to a GPU, it is indicated as a pre-selected hardware accelerator to execute, based, at least in part, on configuration information (Bernat, [0084] lines 18-32, workload assignor 1632 includes an acceleration assignor 1634 which is configured to assign certain workloads or portions thereof to corresponding accelerators (e.g., the accelerators 1250, 1260). In doing so, the acceleration assignor 1634 may identify a type of the workload based on a profile of resource utilizations of the workload over time or based on a tag, an analysis of the computer-executable instructions within the workload, a header of the workload, metadata indicative of the types operations to be executed in the workload, or from a request from a CPU (e.g., the CPU 1272) to offload a portion of a workload onto a particular accelerator 1250, 1260 or type of accelerator (e.g., FPGA, graphics accelerator, cryptography accelerator, compression accelerator, etc.) and assign the workload (as including set of tasks/operations) to a corresponding accelerator 1250, 1260 for execution; [Examiner noted: the corresponding/particular hardware accelerator is identified (as first hardware accelerator/pre-selected hardware accelerator) for assignment based on the type of the operations within the workload because the type of the operation and that particular accelerator need to be corresponding with each other]);
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Diard with Bernat because Bernat’s teaching of assigning the processes based on the configuration of the processes to the corresponding accelerator would have provided Diard’s system with the advantage and capability to allow the system to easily determining the correct type of the accelerator for processing the corresponding type of the processes which improving the resource utilization and system efficiency.
Diard and Bernat fail to specifically teach wherein the configuration information that indicates a mapping between the one or more video frame transformations and hardware accelerators.
However, Dutta teaches wherein the configuration information that indicates a mapping between the one or more transformations and hardware accelerators (Dutta, Fig. 1, 16(1)-(N) hardware accelerators, 20(1)-(N) Bolt (i.e., which including the process of filter tuples (i.e., as transformations); Fig. 2, 44 input data to hardware accelerator for execution to generate output data 46; Col 4, lines 11-25, broker 14 may receive capability information from bolts 20(1)-20(N) and hardware accelerators 16(1)-16(N) and map hardware accelerators 16(1)-16(N) to corresponding bolts 20(1)-20(N). The capability information from bolts 20(1)-20(N) may include respective locations in distributed streams 17(1)-17(M) and identities; the capability information from hardware accelerators 16(1)-16(N) may include respective network locations (e.g., Internet Protocol (IP) address), and capabilities (e.g., RegEx processor, graphics processor, etc.). The mapping may be formatted into any suitable table, spreadsheet, memory mapping, etc. as suitable and based on particular needs. According to various embodiments, the mapping may be used to route the data elements of distributed streams 17(1)-17(M) to appropriate hardware accelerators 16(1)-16(N) for stream processing; Col 8, lines 23-27, distributed stream 17 may be processed by a set of computing devices called worker nodes 54. According to various embodiments, worker nodes 54 may include hardware accelerators 16(1)-16(N) and/or other computing devices).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Diard and Bernat with Dutta because Dutta’s teaching of mapping the accelerators with the corresponding transformation operations for processing would have provided Diard and Bernat’s system with the advantage and capability to allow the system to easily assigning the operations to the corresponding hardware accelerators based on the mapping in order to improving the system performance and processing speed.
Diard, Bernat and Dutta fail to specifically teach the GPU is video image compositor (VIC) engine.
However, Takemoto teaches the GPU is video image compositor (VIC) engine (Takemoto, Col 2, lines 17-20, A first plurality of crosspoint switches connect the plurality of digital video signal inputs to the key processing subsystem and to the video image compositor).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Diard, Bernat and Dutta with Takemoto because Takemoto’s teaching of video image compositor would have provided Diard, Bernat and Dutta’s system with the advantage and capability to allow the system to processing the digital video images which improving the system performance and efficiency.
Diard, Bernat, Dutta and Takemoto fail to specifically teach the frame transformations is one or more video frame transformations to a batch of video frames to modify one or more individual frames in the batch of video frames.
However, Lowry teaches the frame transformations is one or more video frame transformations to a batch of video frames to modify one or more individual frames in the batch of video frames (Lowry, [0041] lines 3-9, Embodiments provide that a NVIDIA® K6000 (NVIDIA® is a registered trademark of NVIDIA) may be used as the GPU. Embodiments provide this as the first stage in the video pipeline (VP). In the GPU (14), video frames may be debayered, as well as other necessary video transformations, such as motion compensation, white balance, black level correction, etc.,).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Diard, Bernat, Dutta and Takemoto with Lowry because Lowry’s teaching of video frame transformations would have provided Diard, Bernat, Dutta and Takemoto’s system with the advantage and capability to allow the system to processing the video and image frames transformations (i.e., augmented reality) which improving the system performance and efficiency (see Lowry, [0004]).
Diard, Bernat, Dutta, Takemoto and Lowry fail to specifically teach when comparing and determining the time/load used, it is an average time.
However, Desai teaches when comparing and determining the time used for load balancing, it is an average time (Desai, Fig. 3, 210 load balancing agent; [0003] load balancing can be determined based in part on an analysis of the current load on each server. Load in this instance can comprise at least the amount of resources allocated to executing user sessions. When a user requests access to an application or otherwise initiates the creation of a user session, the system can responsively determine which server has the least amount of load, and then establish the user session on that server; [0008] lines 1-7, determining the current load value can include evaluating any of the following: a number of page faults; an amount of memory used by the second computer; an average amount of time the second computer uses a central processing unit of the second computer).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Diard, Bernat, Dutta, Takemoto and Lowry with Desai because Desai’s teaching of determining the average amount of time used as load of each server for performing the load balancing would have provided Diard, Bernat, Dutta, Takemoto and Lowry’s system with the advantage and capability to allow the system to utilizing the average time used for processing as the load indicator for performing the load balancing in order to balancing the processing times among the different processing device which improving the system resource utilization and efficiency.
Although Diard, Bernat, Dutta, Takemoto, Lowry and Desai teach assign a subset of clients of the set of clients to a processing unit, Diard, Bernat, Dutta, Takemoto, Lowry and Desai fail to specifically teach when assign a subset of clients of the set of clients, it is assign to a set of processing units.
However, Shraer teaches when assign a subset of clients of the set of clients, it is assign to a set of processing units (Shraer, [0016] lines 3-9, each set of partitions 132 may include multiple partitions...All of the partitions of all of the data sets 132 form the data set for the application job that is executed by the application system; [0033] lines 8-12, move operations can be considered for worker computers that are in a top subset of worker computers with high load measures, e.g., the top x %, or all worker computers with loads above a threshold load measure; [0062] lines 10-12, a partition may be migrated from one worker computer to two or more other worker computers, (as set of computers/hardware accelerators, please note: second hardware accelerator was taught by Bernat and Diard).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Diard, Bernat, Dutta, Takemoto, Lowry and Desai with Shraer because Shraer’s teaching of assigning/migrating the operation/partition to two or more other worker computers for processing would have provided Diard, Bernat, Dutta, Takemoto, Lowry and Desai’s system with the advantage and capability to allow the system to re-balancing the load among different computers/accelerators in order to efficiently utilizing the resources which improving the system efficiency and performance.
Diard, Bernat, Dutta, Takemoto, Lowry, Desai and Shraer fail to specifically teach receiving one or more calls via an application programming interface (API) from a set of application clients, the one or more calls comprising one or more requests to perform one or more video frame transformations.
However, Guo teaches receiving one or more calls via an application programming interface (API) from a set of application clients, the one or more calls comprising one or more requests to perform one or more video frame transformations. (Guo, [0513] one or more application programming interfaces (APIs) in an environment with calling program code interacting with other program code being called through the one or more interfaces; [0514] calling software component interacting with a called software component through an API. A method for operating through an API in this environment includes transferring one or more function calls, messages, other types of invocations or parameters via the API; [0515] Application Programming Interfaces (APIs) may be used in some embodiments. For example, some embodiments of the media exchange module 310 (or 910) provide a set of APIs to other software components for accessing various video processing and encoding functionalities described in FIGS. 3 and 6 such as the functionalities of the TNR module 1500 described in FIG. 15; also see [0091] The video processing module 325 performs image processing on the images and/or the videos captured by the cameras of the device. Examples of such operations include exposure adjustment operations, focus adjustment operations, perspective correction, dynamic range adjustment, image resizing, image compositing, etc. In some embodiments, some image processing operations can also be performed by the media exchange module 310. For instance, as shown in FIG. 3, the media exchange module 310 of some embodiments performs a temporal noise reduction (TNR) operation (e.g., by TNR 315) that reduces noise in video images captured by the cameras of the device. Further examples of such image processing operations of the video processing module 325 and the media exchange module 310 will be provided below; [0517] An API allows a developer of an API-calling component (which may be a third party developer) to leverage specified features provided by an API-implementing component. There may be one API-calling component or there may be more than one such component. An API can be a source code interface that a computer system or program library provides in order to support requests for services from an application; [0523] The API may allow multiple API-calling components written in different programming languages to communicate with the API-implementing component (thus the API may include features for translating calls and returns between the API-implementing component and the API-calling component); however the API may be implemented in terms of a specific programming language. An API-calling component can, in one embodiment, call APIs from different providers such as a set of APIs from an OS provider and another set of APIs from a plug-in provider and another set of APIs from another provider (e.g. the provider of a software library) or creator of the another set of APIs; also see [0284] encodes the image (e.g., using discrete cosine transform, quantization, entropy encoding, etc.) and returns a bitstream with the encoded image to the encoder driver 2200; please note: video frame transformation also taught by Lowry).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Diard, Bernat, Dutta, Takemoto, Lowry, Desai and Shraer with Guo because Guo’s teaching of using the API call for requesting video image transformations would have provided Diard, Bernat, Dutta, Takemoto, Lowry, Desai and Shraer’s system with the advantage and capability to allow the system to utilizing the different APIs from different providers for requesting different image/video processing which improving the system performance and efficiency.
As per claim 23, Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer and Guo teach the invention according to claim 22 above. Diard further teaches wherein the assignment and the reassignment of the subset of application clients are repeated one or more times (Diard, Col 2, lines 28-31, It would, therefore, be desirable to provide a mechanism whereby the processing load on each GPU can be monitored and the division of the display area among the GPUs can be dynamically adjusted to balance the loads; also see Col 8, lines 17-25, each GPU provides feedback data to the graphics driver program (or another program executing on CPU 102). The feedback data provides information about the time taken by a particular GPU to render its portion of the image. The graphics driver program uses this feedback to dynamically balance the load among the GPUs by modifying the clip rectangle from time to time, e.g., by changing the dividing line to a different line P', based on the relative loads on the two GPUs).
As per claim 24, Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer and Guo teach the invention according to claim 22 above. Diard further teaches wherein the frame processing threshold is determined based at least in part on a framerate of processing (Diard, Col 10, lines 64-Col 11, line 6, the graphics driver might not know whether the GPUs have finished a particular frame and the GPUs may be rendering a frame that is several frames earlier in the command stream than a current frame in the graphics driver. Where the feedback array is written in a circular fashion, as in process 400 described above, selecting Q to be equal to B provides an average over the B most recently rendered frames. In some embodiments, a weighted average may be used, e.g., giving a larger weight to more recently-rendered frames; Col 11, lines 7-17, The load coefficient is used to determine whether an adjustment to the clip rectangles for the GPUs needs to be made. If the GPUs are equally loaded, the likelihood of either GPU finishing a frame first is about 50%, and the average value over a suitable number of frames (e.g., 20) will be about 0.5 if identifier values of 0 and 1 are used. An average value in excess of 0.5 indicates that GPU-1 (which renders the bottom portion of the image) is more heavily loaded than GPU-0, and an average value below 0.5 indicates that GPU-0 (which renders the top portion of the image) is more heavily loaded than GPU-1). In addition, Takemoto teaches video processing (Takemoto, Col 1, line 35, digital video image compositing; Col 2, lines 17-20, A first plurality of crosspoint switches connect the plurality of digital video signal inputs to the key processing subsystem and to the video image compositor).
As per claim 25, Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer and Guo teach the invention according to claim 22 above. Desai further teaches wherein the assigning of the subset of application clients is executed using a load balancer (Desai, Fig. 3, 210 load balancing agent; [0003] load balancing can be determined based in part on an analysis of the current load on each server. Load in this instance can comprise at least the amount of resources allocated to executing user sessions. When a user requests access to an application or otherwise initiates the creation of a user session, the system can responsively determine which server has the least amount of load, and then establish the user session on that server; [0008] lines 1-7, determining the current load value can include evaluating any of the following: a number of page faults; an amount of memory used by the second computer; an average amount of time the second computer uses a central processing unit of the second computer).
As per claim 29, Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer and Guo teach the invention according to claim 22 above. Desai further teaches wherein the set of processing units comprises a graphics processing unit (GPU) (Desai, Col 10, lines 64-Col 11, line 6, the graphics driver might not know whether the GPUs have finished a particular frame and the GPUs may be rendering a frame that is several frames earlier in the command stream than a current frame in the graphics driver. Where the feedback array is written in a circular fashion, as in process 400 described above, selecting Q to be equal to B provides an average over the B most recently rendered frames. In some embodiments, a weighted average may be used, e.g., giving a larger weight to more recently-rendered frames).
Claim 26 is rejected under 35 U.S.C. 103 as being unpatentable over Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer and Guo, as applied to claim 25 above, and further in view of Biran et al. (US Pub. 2018/0039516 A1).
Biran was cited in the previous Office Action.
As per claim 26, Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer and Guo teach the invention according to claim 25 above. Desai further teaches wherein the load balancer that assigns the subset of application clients (Desai, Fig. 3, 210 load balancing agent; [0003] load balancing can be determined based in part on an analysis of the current load on each server. Load in this instance can comprise at least the amount of resources allocated to executing user sessions. When a user requests access to an application or otherwise initiates the creation of a user session, the system can responsively determine which server has the least amount of load, and then establish the user session on that server; [0008] lines 1-7, determining the current load value can include evaluating any of the following: a number of page faults; an amount of memory used by the second computer; an average amount of time the second computer uses a central processing unit of the second computer; [0109] the load balancing agent 210 executes or otherwise assigns the user session to the selected server (Step 728)).
Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer and Guo fail to specifically teach the load balancer that assigns the subset of application clients comprises a processing thread.
However, Biran teaches the load balancer that assigns the subset of application clients comprises a processing thread (Biran, [0042] lines 1-2, load balancer 27 executes a request processing thread)
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer and Guo with Biran because Biran’s teaching of load balancer executing the processing thread (as comprising a processing thread) would have provided Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer and Guo’s system with the advantage and capability to allow the system to utilizing the load balancer for processing request processing thread in order to improving the system processing speed and efficiency.
Claim 27 is rejected under 35 U.S.C. 103 as being unpatentable over Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer and Guo, as applied to claim 25 above, and further in view of MALDANER (US Pub. 2011/0276695 A1).
MALDANER was cited in the previous Office Action.
As per claim 27, Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer and Guo teach the invention according to claim 25 above. Desai further teaches wherein a load balancer that assigns the subset of application clients (Desai, Fig. 3, 210 load balancing agent; [0003] load balancing can be determined based in part on an analysis of the current load on each server. Load in this instance can comprise at least the amount of resources allocated to executing user sessions. When a user requests access to an application or otherwise initiates the creation of a user session, the system can responsively determine which server has the least amount of load, and then establish the user session on that server; [0008] lines 1-7, determining the current load value can include evaluating any of the following: a number of page faults; an amount of memory used by the second computer; an average amount of time the second computer uses a central processing unit of the second computer; [0109] the load balancing agent 210 executes or otherwise assigns the user session to the selected server (Step 728)).
Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer and Guo fail to specifically teach the load balancer maintains a table of the set of application clients.
However, MALDANER teaches the load balancer maintains a table of the set of application clients (MALDANER, [0108] last two lines: the load balancer 255 may maintain a metrics table 420 for each service, device or application; [0114] lines 1-10, The managed objects or variables provided via the network management protocol may provide any type and form of metrics or operational characteristics of the service, server or device to be used by the appliance for load balancing, or any other function of the load balancer 255. In one embodiment, the device provided metrics 420 may include any of the metrics 410 collected by the load balancer 255 as described above. In another embodiment, the device provided metrics 420 may include any type and form of information on any resource usage of the managed device, service or system. In one embodiment, the metrics 410 include CPU, memory and/or disk usage of the device and/or service 270. In other embodiments, the metrics 420 may include information on a number of connections, sessions or clients of the service 270 (as load balancer having metrics table maintains a table of set of application clients)).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer and Guo with MALDANER because MALDANER’s teaching of load balancer having a metrics table that including the information about the clients of service would have provided Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer and Guo’s system with the advantage and capability to allow the system to easily determining and managing the different information regarding to the clients services which improving the system performance and efficiency.
Claim 28 is rejected under 35 U.S.C. 103 as being unpatentable over Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer, Guo and MALDANER, as applied to claim 27 above, and further in view of Kawai et al. (US Pub. 2006/0271700 A1) and Nakamura et al. (US Pub. 2012/0016994 A1).
Kawai and Nakamura were cited in the previous Office Action.
As per claim 28, Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer, Guo and MALDANER teach the invention according to claim 27 above. Lowry teaches compute the one or more video frame transformations (Lowry, [0041] lines 3-9, Embodiments provide that a NVIDIA® K6000 (NVIDIA® is a registered trademark of NVIDIA) may be used as the GPU. Embodiments provide this as the first stage in the video pipeline (VP). In the GPU (14), video frames may be debayered, as well as other necessary video transformations, such as motion compensation, white balance, black level correction, etc.,).
Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer, Guo and MALDANER fail to specifically teach wherein the table of the set of application clients includes a processing engine assigned to each client and an average time taken by the client to compute.
However, Kawai teaches wherein the table of the set of application clients includes a processing engine assigned to each client and a time taken by the client to compute (Kawai, Fig. 6, 112 user information table, users with assigned different services (as processing engines) and delay time; also see claim 3, the record medium with the load distribution program recorded thereon according to claim 2, wherein the delay time determination means refers to a delay time management table where communication delay time taken between the client connection server which provides an Internet connection service to the client and each data center is set in advance and determines the processing delay time taken between the client and each data center).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer, Guo and MALDANER with Kawai because Kawai’s teaching of providing a table which including the client and servers assignments and the communication delay times would have provided Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer, Guo and MALDANER’s system with the advantage and capability to allow the system to easily determining and managing the different clients and servers assignment/association for processing the workload which improving the system performance and efficiency.
Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer, Guo, MALDANER and Kawai fail to specifically teach the delay time is an average time taken by the client to compute.
However, Nakamura teaches the delay time is an average time taken by the client to compute (Nakamura, Fig. 3, processing time management table, distributed node ID, average processing time; [0021] lines 1-10, the client 200 has: a CPU 201, a memory 202; a storage medium 203; and a communication interface 210, and lying on the memory 202 are: an information request program 204 making a request of an information processing program of the distributed node for information processing; a node management table 206 holding network information of all distributed nodes connectable from the client; a processing time management table 207 measuring and holding time required for the information processing in each distributed node).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer, Guo, MALDANER and Kawai with Nakamura because Nakamura’s teaching of average processing time would have provided Diard, Bernat, Dutta, Takemoto, Lowry, Desai, Shraer, Guo, MALDANER and Kawai’s system with the advantage and capability to allow the system to easily determining and managing the different average processing times associated with different nodes which improving the system performance and efficiency.
Claims 30-32 are rejected under 35 U.S.C. 103 as being unpatentable over Bernat et al (US Pub. 2020/0409748 A1) in view of Dutta (US Patent. 9,313,134 B2) and further in view of Shraer et al. (US Pub. 2017/0353536 A1), Diard (US Patent. 7,075,541 B2), Lowry et al. (US Pub. 2018/0262684 A1), Takemoto et al. (US Patent 5,347,622) and Guo et al. (US Pub. 2011/0249086 A1).
Bernat, Dutta, Shraer, Diard, Lowry and Takemoto were cited in the previous Office Action.
As per claim 30, Bernat teaches the invention substantially as claimed including A non-transitory computer readable storage medium storing thereon executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to (Bernat, [0029] lines 1-12, The disclosed embodiments may be implemented, in some cases, in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried by or stored on a transitory or non-transitory machine-readable (e.g., computer-readable) storage medium, which may be read and executed by one or more processors… A machine-readable storage medium may be embodied as any storage device, mechanism, or other physical structure for storing or transmitting information in a form readable by a machine (e.g., a volatile or non-volatile memory):
obtain first performance data associated with an accelerator, the first performance data related to a set of application clients assigned to the accelerator to execute operations corresponding to the set of applications clients (Bernat, [0088] lines 8-16, offloading accelerator kernel tasks to the FPGA 1619, 1629, 1685. More particularly, a request going from a compute sled to a target accelerator device generally traverses one or more hops, such as the NIC 1618, 1628, 1684, prior to reaching the target accelerator device. In an embodiment, the NIC 1618, 1628, 1684 may include inline processing logic to identify one or more accelerator tasks (as application client) of a workload that can be performed by the FPGA 1619, 1629, 1685, respectively; [0089] lines 1-8, the orchestrator server 1616 includes a kernel analysis and decision logic unit 1617. The kernel analysis and decision logic unit 1617 may be embodied as any device or circuitry to obtain telemetry data indicative of resource usage and power consumption of the accelerator sleds; Fig. 23, 2302 monitor power consumption of a source accelerator sled having one or more accelerator devices executing a workload, 2304 monitor power consumption of the accelerator sled relative to one or more power consumption thresholds specified in a policy);
determine, that the first performance data exceeds a threshold (Bernat, Fig. 23 2306 power consumption threshold exceeded? [0111] In block 2306, the orchestrator server 1616 determines whether a power consumption threshold is exceeded. If not, then the method 2300 returns to block 2302. Otherwise, the orchestrator server 1616 determines whether accelerator resources in the system 1300 are available for a scale-out operation);
as a result of the determination that the first performance data exceeds the threshold, assign a subset of application clients of the set of application clients to a processor to cause second performance data related to the set of application clients to be below the threshold, cause the processor to execute at least a portion of the operations correspond to the subset of application clients to relieve load from the accelerator (Bernat, Fig. 23, 2312 Migrate the workload from one or more of the accelerator devices of the source accelerator sled to one or more accelerator devices of the target accelerator sled; [0112] lines 1-7, if accelerator resources are available, then in block 2310, the orchestrator server 1616 scales-out the workload to one or more accelerator devices on a target accelerator sled. For instance, to do so, in block 2312, the orchestrator server 1616 may migrate the workload from one or more of the accelerator devices of the source accelerator sled to one or more accelerator devices of the target accelerator sled (as second processor perform the task/application client which to relieve load from the source accelerator therefore, second performance data (after migration) related to the set of application clients to be below the threshold, i.e., the purpose of the migration is for make sure below the threshold); also see Fig. 23 2306 power consumption threshold exceeded? [0111] In block 2306, the orchestrator server 1616 determines whether a power consumption threshold is exceeded. If not, then the method 2300 returns to block 2302. Otherwise, the orchestrator server 1616 determines whether accelerator resources in the system 1300 are available for a scale-out operation).
Bernat fails to specifically teach the operations are one or more video frame transformations, wherein the accelerator is selected based, at least in part, on a mapping between the one or more video frame transformations and hardware accelerators, execute at least a portion of the one or more video frame transformations.
However, Dutta teaches when execute the operations, the operations are one or more transformations (Dutta, Col 1, lines 56-60, An example method for leveraging hardware accelerators for scalable distributed streams in a network environment is provided and includes allocating a plurality of hardware accelerators to a corresponding plurality of bolts of a distributed stream in a network; Col 2, lines 11-14, the “bolt” implements processing logic to process (e.g., run functions, filter tuples, (as transformations) perform stream aggregations, talk to databases, etc.) the data elements in the stream);
wherein the accelerator is selected based, at least in part, on a mapping between the one or more transformations and hardware accelerators, execute at least a portion of the one or more transformations (Dutta, Fig. 1, 16(1)-(N) hardware accelerators, 20(1)-(N) Bolt (i.e., which including the process of filter tuples (i.e., as transformations); Fig. 2, 44 input data to hardware accelerator for execution to generate output data 46; Col 4, lines 11-25, broker 14 may receive capability information from bolts 20(1)-20(N) and hardware accelerators 16(1)-16(N) and map hardware accelerators 16(1)-16(N) to corresponding bolts 20(1)-20(N). The capability information from bolts 20(1)-20(N) may include respective locations in distributed streams 17(1)-17(M) and identities; the capability information from hardware accelerators 16(1)-16(N) may include respective network locations (e.g., Internet Protocol (IP) address), and capabilities (e.g., RegEx processor, graphics processor, etc.). The mapping may be formatted into any suitable table, spreadsheet, memory mapping, etc. as suitable and based on particular needs. According to various embodiments, the mapping may be used to route the data elements of distributed streams 17(1)-17(M) to appropriate hardware accelerators 16(1)-16(N) for stream processing; Col 8, lines 23-27, distributed stream 17 may be processed by a set of computing devices called worker nodes 54. According to various embodiments, worker nodes 54 may include hardware accelerators 16(1)-16(N) and/or other computing devices).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat with Dutta because Dutta’s teaching of mapping the accelerators with the corresponding transformation operations for processing would have provided Bernat’s system with the advantage and capability to allow the system to easily assigning the operations to the corresponding hardware accelerators based on the mapping in order to improving the system performance and processing speed.
Although Bernat and Dutta teach assign a subset of clients of the set of clients to a processor, Bernat and Dutta fail to specifically teach when assign a subset of clients of the set of clients, it is assign to a set of processors.
However, Shraer teaches when assign a subset of clients of the set of clients, it is assign to a set of processors (Shraer, [0016] lines 3-9, each set of partitions 132 may include multiple partitions...All of the partitions of all of the data sets 132 form the data set for the application job that is executed by the application system; [0033] lines 8-12, move operations can be considered for worker computers that are in a top subset of worker computers with high load measures, e.g., the top x %, or all worker computers with loads above a threshold load measure; [0062] lines 10-12, a partition may be migrated from one worker computer to two or more other worker computers, (as set of computers/hardware accelerators).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat and Dutta with Shraer because Shraer’s teaching of assigning/migrating the operation/partition to two or more other worker computers for processing would have provided Bernat and Dutta’s system with the advantage and capability to allow the system to re-balancing the load among different computers/accelerators in order to efficiently utilizing the resources which improving the system efficiency and performance.
Bernat, Dutta and Shraer fail to specifically teach when determine that the first performance data exceeds a threshold, it is based, at least in part, on a type of the one or more video frame transformations, when execute, it is execute at least a portion of the one or more video frame transformations to a batch of video frames to modify one or more individual frames in the batch of video frames, wherein the one or more video frame transformations correspond to the subset of application clients to relive load form the VIC; reassign the subset of application clients to the VIC to cause the second performance data to remain below the threshold, wherein the reassignment causes the VIC to execute at least the portion of the one or more video frame transformations.
However, Diard teaches when determine that the first performance data exceeds a threshold, it is based, at least in part, on a type of the one or more frame transformations (Diard, Col 1, lines 31-38, The typical GPU is a highly complex integrated circuit device optimized to perform graphics computations (e.g., matrix transformations, scan-conversion and/or other rasterization techniques, texture blending, etc.) and write the results to the graphics memory. The GPU is a "slave" processor that operates in response to commands received from a driver program executing on a "master" processor; Col 3, lines 31-35, The load coefficient may be, e.g., an average of the recorded numeric values that can be compared to an arithmetic mean of the numeric values of the processor identifiers in order to determine whether an imbalance exists; Col 11, lines 18-27, determined whether the load coefficient exceeds a "high" threshold. The high threshold is preselected and may be exactly 0.5 or a somewhat higher value (e.g., 0.55 or 0.6). If the load coefficient exceeds the high threshold, then the loads are adjusted at step 512 by moving the boundary line P in FIG. 2 down by a preset amount (e.g., one line, five lines, ten lines). This reduces the fraction of the display area that is rendered by GPU-1, which will tend to reduce the load on GPU-1 and increase the load on GPU-0; Col 16, lines 8-23, In some multi-card embodiments used to render scenes in which foreground regions (most often but not always at the bottom of the display area) are consistently more complex than background regions, a performance advantage can be gained by assigning GPU 914a to process the background region of the scene and assigning GPU 914b to process the foreground region. For example, in FIG. 2, suppose that the foreground appears toward the bottom of display area 200. In that case, GPU 914a would be assigned to render top region 202 while GPU 914b would be assigned to render bottom region 204. The higher complexity of the foreground (bottom) region tends to increase the rendering time of GPU 914b. In response, the load-balancing processes described herein will tend to move the boundary line P toward the bottom of the display area; Col 2, lines 66-67, The graphics processors are instructed to render a number of frames, wherein the first and second graphics process; Col 5, lines 37-40, lighting transformations, coordinate transformations, scan-conversion of geometric primitives to rasterized data, shading computations, shadow rendering, texture blending [Examiner noted: as a type of the transformations (i.e., render scenes in which foreground regions are more complex than the background regions (as different types of transformations)]).
when execute, it is execute at least a portion of the one or more frame transformations to a batch of frames to modify one or more individual frames in the batch of frames, wherein the one or more frame transformations correspond to the subset of application clients to relive load form the GPU (Diard, Col 1, lines 58-60, For example, each GPU may be instructed to render pixel data for a different portion of the displayable image, such as a number of lines of a raster-based display (as frame transformations correspond the subset of application clients to GPU for processing; see specs [0057] “a client (e.g., application, processes, or other components”); Col 13, lines 12-24, different feedback data may be used instead of or in addition to the GPU identifiers described above. For example, instead of providing one feedback array in system memory, with both GPUs writing feedback data to the same location for a given frame, each GPU may write to a corresponding entry of a different feedback array, and the feedback data may include timing information, e.g., a timestamp indicating when each GPU finished a particular frame. In this embodiment, the graphics driver is configured to use the timing information to determine whether one GPU is consistently using more time per frame than another and adjust the clip rectangles accordingly to balance the load; Col 11, lines 18-27, at step 510 it is determined whether the load coefficient exceeds a "high" threshold...then the loads are adjusted at step 512 by moving the boundary line P in FIG. 2 down by a preset amount (e.g., one line, five lines, ten lines). This reduces (as relieve load) the fraction of the display area that is rendered by GPU-1, which will tend to reduce the load on GPU-1 and increase the load on GPU-0. (as assigning to a processing unit distinct from the original GPU, a subset of the application clients to cause time used by the set of processing units to be below the threshold); Col 2, lines 66-67, The graphics processors are instructed to render a number of frames, wherein the first and second graphics process; Col 5, lines 37-40, lighting transformations, coordinate transformations, scan-conversion of geometric primitives to rasterized data, shading computations, shadow rendering, texture blending);
reassign the subset of application clients to the GPU to cause the second performance data to remain below the threshold, wherein the reassignment causes the GPU to execute at least the portion of the one or more frame transformations (Diard, Abstract, lines 9-12, re-partitioned to increase a size of the portion assigned to the less heavily loaded processor and to decrease a size of the portion assigned to the more heavily loaded processor; Fig. 2, 202, 204, P and P’; Fig. 5, 514 is average below low threshold? YES to 516, move boundary line up by preselected amount; Col 8, lines 55-67, rendering commands 308 and associated rendering data for the next frame F1…At this point, the clip rectangles for each GPU may be modified by the graphics driver program based on the feedback data received in response to the various write notifier commands (e.g., commands 306, 310). For example, where the display area is divided as shown in FIG. 2, the value of P may be modified (e.g., to P') in response to feedback data: if the GPU that processes top portion 202 tends to finish its frames first, the value of P is increased, and if the GPU that processes bottom portion 204 tends to finish first, the value of P is decreased. Specific embodiments of re-partitioning a display area in response to feedback data are described below; Col 11, lines 18-27, at step 510 it is determined whether the load coefficient exceeds a "high" threshold… at step 514, it is determined whether the load coefficient is less than a "low" threshold. The low threshold is predefined and may be exactly 0.5 or a somewhat lower value (e.g., 0.45 or 0.4). If the load coefficient is below the low threshold, then the loads are adjusted at step 516 by moving the boundary line P in FIG. 2 up by a preset amount (e.g., one line, five lines, ten lines)).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta and Shraer with Diard because Diard’s teaching of dynamically adjusting the processing load among the GPUs based on the feedback data would have provided Bernat, Dutta and Shraer’s system with the advantage and capability to allow the system to efficiently utilizing the resources based on the load which improving the processing speed and system efficiency.
Bernat, Dutta, Shraer and Diard fail to specifically teach the frame transformations is one or more video frame transformations to a batch of video frames to modify one or more individual frames in the batch of video frames.
However, Lowry teaches the frame transformations is one or more video frame transformations to a batch of video frames to modify one or more individual frames in the batch of video frames (Lowry, [0041] lines 3-9, Embodiments provide that a NVIDIA® K6000 (NVIDIA® is a registered trademark of NVIDIA) may be used as the GPU. Embodiments provide this as the first stage in the video pipeline (VP). In the GPU (14), video frames may be debayered, as well as other necessary video transformations, such as motion compensation, white balance, black level correction, etc.,).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Shraer and Diard with Lowry because Lowry’s teaching of video frame transformations would have provided Bernat, Dutta, Shraer and Diard’s system with the advantage and capability to allow the system to processing the video and image frames transformations (i.e., augmented reality) which improving the system performance and efficiency (see Lowry, [0004]).
Bernat, Dutta, Shraer, Diard and Lowry fail to specifically teach the accelerator/GPU is video image compositor (VIC).
However, Takemoto teaches the accelerator/GPU is video image compositor (VIC) (Takemoto, Col 2, lines 17-20, A first plurality of crosspoint switches connect the plurality of digital video signal inputs to the key processing subsystem and to the video image compositor).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Shraer, Diard and Lowry with Takemoto because Takemoto’s teaching of video image compositor would have provided Bernat, Dutta, Shraer, Diard and Lowry’s system with the advantage and capability to allow the system to processing the digital video images which improving the system performance and efficiency.
Bernat, Dutta, Shraer, Diard, Lowry and Takemoto fail to specifically teach receive one or more calls via an application programming interface (API) from a set of application clients, the one or more calls comprising one or more requests to perform one or more video frame transformations.
However, Guo teaches receive one or more calls via an application programming interface (API) from a set of application clients, the one or more calls comprising one or more requests to perform one or more video frame transformations (Guo, [0513] one or more application programming interfaces (APIs) in an environment with calling program code interacting with other program code being called through the one or more interfaces; [0514] calling software component interacting with a called software component through an API. A method for operating through an API in this environment includes transferring one or more function calls, messages, other types of invocations or parameters via the API; [0515] Application Programming Interfaces (APIs) may be used in some embodiments. For example, some embodiments of the media exchange module 310 (or 910) provide a set of APIs to other software components for accessing various video processing and encoding functionalities described in FIGS. 3 and 6 such as the functionalities of the TNR module 1500 described in FIG. 15; also see [0091] The video processing module 325 performs image processing on the images and/or the videos captured by the cameras of the device. Examples of such operations include exposure adjustment operations, focus adjustment operations, perspective correction, dynamic range adjustment, image resizing, image compositing, etc. In some embodiments, some image processing operations can also be performed by the media exchange module 310. For instance, as shown in FIG. 3, the media exchange module 310 of some embodiments performs a temporal noise reduction (TNR) operation (e.g., by TNR 315) that reduces noise in video images captured by the cameras of the device. Further examples of such image processing operations of the video processing module 325 and the media exchange module 310 will be provided below; [0517] An API allows a developer of an API-calling component (which may be a third party developer) to leverage specified features provided by an API-implementing component. There may be one API-calling component or there may be more than one such component. An API can be a source code interface that a computer system or program library provides in order to support requests for services from an application; [0523] The API may allow multiple API-calling components written in different programming languages to communicate with the API-implementing component (thus the API may include features for translating calls and returns between the API-implementing component and the API-calling component); however the API may be implemented in terms of a specific programming language. An API-calling component can, in one embodiment, call APIs from different providers such as a set of APIs from an OS provider and another set of APIs from a plug-in provider and another set of APIs from another provider (e.g. the provider of a software library) or creator of the another set of APIs; also see [0284] encodes the image (e.g., using discrete cosine transform, quantization, entropy encoding, etc.) and returns a bitstream with the encoded image to the encoder driver 2200; please note: video frame transformation also taught by Lowry).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Shraer, Diard, Lowry and Takemoto with Guo because Guo’s teaching of using the API call for requesting video image transformations would have provided Bernat, Dutta, Shraer, Diard, Lowry and Takemoto’s system with the advantage and capability to allow the system to utilizing the different APIs from different providers for requesting different image/video processing which improving the system performance and efficiency.
As per claim 31, Bernat, Dutta, Shraer, Diard, Lowry, Takemoto and Guo teach the invention according to claim 30 above. Diard further teaches wherein the threshold is determined based, at least in part, on one or more framerates associated with the set of application clients (Diard, Col 10, lines 64-Col 11, line 6, the graphics driver might not know whether the GPUs have finished a particular frame and the GPUs may be rendering a frame (as task) that is several frames earlier in the command stream than a current frame in the graphics driver. Where the feedback array is written in a circular fashion, as in process 400 described above, selecting Q to be equal to B provides an average over the B most recently rendered frames. In some embodiments, a weighted average may be used, e.g., giving a larger weight to more recently-rendered frames; Col 11, lines 7-17, The load coefficient is used to determine whether an adjustment to the clip rectangles for the GPUs needs to be made. If the GPUs are equally loaded, the likelihood of either GPU finishing a frame first is about 50%, and the average value over a suitable number of frames (e.g., 20) will be about 0.5 if identifier values of 0 and 1 are used. An average value in excess of 0.5 indicates that GPU-1 (which renders the bottom portion of the image) is more heavily loaded than GPU-0, and an average value below 0.5 indicates that GPU-0 (which renders the top portion of the image) is more heavily loaded than GPU-1; also see Col 12, lines 30-31, 30 frames are rendered per second; Col 12, lines 36-46, Correspondingly, the high threshold and low threshold may have any values, and the two threshold values may be equal (e.g., both equal to 0.5), so long as the high threshold is not less than the low threshold. Both thresholds are advantageously set to values near or equal to the arithmetic mean of the two identifiers; an optimal selection of thresholds in a particular system may be affected by considerations such as the frequency of load rebalancing and any overhead associated with changing the clip rectangles assigned to each GPU. The threshold comparison is advantageously defined such that there is some condition for which the load is considered balanced).
As per claim 32, Bernat, Dutta, Shraer, Diard, Lowry, Takemoto and Guo teach the invention according to claim 30 above. Bernat further teaches wherein the set of second processors further comprises a field-programmable gate array (FPGA) (Bernat, Fig. 16, 1620 FPGA).
Claim 33 is rejected under 35 U.S.C. 103 as being unpatentable over Bernat, Dutta, Shraer, Diard, Lowry, Takemoto and Guo, as applied to claim 30 above, and further in view of MALDANER (US Pub. 2011/0276695 A1).
MALDANER was cited in the previous Office Action.
As per claim 33, Bernat, Dutta, Shraer, Diard, Lowry, Takemoto and Guo teach the invention according to claim 30 above. Bernat, Dutta, Shraer, Diard, Lowry, Takemoto and Guo fail to specifically teach maintain a table of set of application clients of which each application client is a member.
However, MALDANER teaches maintain a table of set of application clients of which each application client is a member (MALDANER, [0108] last two lines: the load balancer 255 may maintain a metrics table 420 for each service, device or application; [0114] lines 1-10, The managed objects or variables provided via the network management protocol may provide any type and form of metrics or operational characteristics of the service, server or device to be used by the appliance for load balancing, or any other function of the load balancer 255. In one embodiment, the device provided metrics 420 may include any of the metrics 410 collected by the load balancer 255 as described above. In another embodiment, the device provided metrics 420 may include any type and form of information on any resource usage of the managed device, service or system. In one embodiment, the metrics 410 include CPU, memory and/or disk usage of the device and/or service 270. In other embodiments, the metrics 420 may include information on a number of connections, sessions or clients of the service 270 (as load balancer having metrics table maintains a table of application clients)).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Shraer, Diard, Lowry, Takemoto and Guo with MALDANER because MALDANER’s teaching of load balancer having a metrics table that including the information about the clients of service would have provided Bernat, Dutta, Shraer, Diard, Lowry, Takemoto and Guo’s system with the advantage and capability to allow the system to easily determining and managing the different information regarding to the clients services which improving the system performance and efficiency.
Claim 34 is rejected under 35 U.S.C. 103 as being unpatentable over Bernat, Dutta, Shraer, Diard, Lowry, Takemoto, Guo and MALDANER, as applied to claim 33 above, and further in view of Kawai et al. (US Pub. 2006/0271700 A1) and Nakamura et al. (US Pub. 2012/0016994 A1).
Kawai and Nakamura were cited in the previous Office Action.
As per claim 34, Bernat, Dutta, Shraer, Diard, Lowry, Takemoto, Guo and MALDANER teach the invention according to claim 33 above. Bernat teaches compute operations for the subset of application clients (Bernat, [0088] lines 8-16, offloading accelerator kernel tasks to the FPGA 1619, 1629, 1685. More particularly, a request going from a compute sled to a target accelerator device generally traverses one or more hops, such as the NIC 1618, 1628, 1684, prior to reaching the target accelerator device. In an embodiment, the NIC 1618, 1628, 1684 may include inline processing logic to identify one or more accelerator tasks of a workload that can be performed by the FPGA 1619, 1629, 1685, respectively). In addition, Lowry teaches operations is video frame transformations (Lowry, [0041] lines 3-9, Embodiments provide that a NVIDIA® K6000 (NVIDIA® is a registered trademark of NVIDIA) may be used as the GPU. Embodiments provide this as the first stage in the video pipeline (VP). In the GPU (14), video frames may be debayered, as well as other necessary video transformations, such as motion compensation, white balance, black level correction, etc.,).
Bernat, Dutta, Shraer, Diard, Lowry, Takemoto, Guo and MALDANER fail to specifically teach wherein the table of set of application clients includes information indicating that the set of processor is assigned to the subset of application clients, and an average time taken by the set of processors to compute.
However, Kawai teaches wherein the table of set of application clients includes information indicating that the set of processor is assigned to the subset application clients and a delay time taken by the set of processors to compute (Kawai, Fig. 6, 112 user information table, users with assigned different services (as include set of second processors) and delay time; also see claim 3, the record medium with the load distribution program recorded thereon according to claim 2, wherein the delay time determination means refers to a delay time management table where communication delay time taken between the client connection server which provides an Internet connection service to the client and each data center is set in advance and determines the processing delay time taken between the client and each data center).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Shraer, Diard, Lowry, Takemoto, Guo and MALDANER with Kawai because Kawai’s teaching of providing a table which including the client and servers assignments and the communication delay times would have provided Bernat, Dutta, Shraer, Diard, Lowry, Takemoto, Guo and MALDANER’s system with the advantage and capability to allow the system to easily determining and managing the different clients and servers assignment/association for processing the workload which improving the system performance and efficiency.
Bernat, Dutta, Shraer, Diard, Lowry, Takemoto, Guo, MALDANER and Kawai fail to specifically teach the delay time is an average time taken by the set of second processors to compute.
However, Nakamura teaches the delay time is an average time taken by the set of second processors to compute (Nakamura, Fig. 3, processing time management table, distributed node ID, average processing time; [0021] lines 1-10, the client 200 has: a CPU 201, a memory 202; a storage medium 203; and a communication interface 210, and lying on the memory 202 are: an information request program 204 making a request of an information processing program of the distributed node for information processing; a node management table 206 holding network information of all distributed nodes connectable from the client; a processing time management table 207 measuring and holding time required for the information processing in each distributed node).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Shraer, Diard, Lowry, Takemoto, Guo, MALDANER and Kawai with Nakamura because Nakamura’s teaching of average processing time would have provided Bernat, Dutta, Shraer, Diard, Lowry, Takemoto, Guo, MALDANER and Kawai’s system with the advantage and capability to allow the system to easily determining and managing the different average processing times associated with different nodes which improving the system performance and efficiency.
Claim 35 is rejected under 35 U.S.C. 103 as being unpatentable over Bernat, Dutta, Diard, Lowry, Shraer and Guo, as applied to claim 1 above, and further in view of Zadka et al. (US Pub. 2016/0182399 A1).
As per claim 35, Bernat, Dutta, Diard, Lowry, Shraer and Guo teach the invention according to claim 1 above. Guo further teaches receive information from the set of clients related to execution of at least a portion of the one or more video frame transformations (Guo, [0513] one or more application programming interfaces (APIs) in an environment with calling program code interacting with other program code being called through the one or more interfaces; [0514] calling software component interacting with a called software component through an API. A method for operating through an API in this environment includes transferring one or more function calls, messages, other types of invocations or parameters via the API; [0515] Application Programming Interfaces (APIs) may be used in some embodiments. For example, some embodiments of the media exchange module 310 (or 910) provide a set of APIs to other software components for accessing various video processing and encoding functionalities described in FIGS. 3 and 6 such as the functionalities of the TNR module 1500 described in FIG. 15; also see [0091] The video processing module 325 performs image processing on the images and/or the videos captured by the cameras of the device. Examples of such operations include exposure adjustment operations, focus adjustment operations, perspective correction, dynamic range adjustment, image resizing, image compositing, etc. In some embodiments, some image processing operations can also be performed by the media exchange module 310. For instance, as shown in FIG. 3, the media exchange module 310 of some embodiments performs a temporal noise reduction (TNR) operation (e.g., by TNR 315) that reduces noise in video images captured by the cameras of the device. Further examples of such image processing operations of the video processing module 325 and the media exchange module 310 will be provided below; [0517] An API allows a developer of an API-calling component (which may be a third party developer) to leverage specified features provided by an API-implementing component. There may be one API-calling component or there may be more than one such component. An API can be a source code interface that a computer system or program library provides in order to support requests for services from an application; [0523] The API may allow multiple API-calling components written in different programming languages to communicate with the API-implementing component (thus the API may include features for translating calls and returns between the API-implementing component and the API-calling component); however the API may be implemented in terms of a specific programming language. An API-calling component can, in one embodiment, call APIs from different providers such as a set of APIs from an OS provider and another set of APIs from a plug-in provider and another set of APIs from another provider (e.g. the provider of a software library) or creator of the another set of APIs; also see [0284] encodes the image (e.g., using discrete cosine transform, quantization, entropy encoding, etc.) and returns a bitstream with the encoded image to the encoder driver 2200; please note: video frame transformation also taught by Lowry). In addition, Diard teaches use the information to perform load balancing with respect to a plurality of hardware accelerators that comprises the first and second hardware accelerators (Diard, Col 2, line 62- Col 3, line 16, load balancing for graphics processors configured to operate in parallel. A display area is partitioned into at least a first portion to be rendered by a first graphics processor and a second portion to be rendered by a second graphics processor. The graphics processors are instructed to render a number of frames, wherein the first and second graphics processors perform rendering for the first and second portions of the display area, respectively. Feedback data for each of the frames is received from the first and second graphics processors, the feedback data for each frame indicating which of the first and second graphics processors was last to finish rendering the frame. Based on the feedback data, it is determined whether an imbalance exists between respective loads of the first and second graphics processors. In the event that an imbalance exists, based on the feedback data, the one of the first and second graphics processors that is more heavily loaded is identified; the display area is re-partitioned to increase a size of the one of the first and second portions of the display area that is rendered by the more heavily loaded one of the first and second graphics processors and to decrease a size of the other of the first and second portions of the display area).
Bernat, Dutta, Diard, Lowry, Shraer and Guo fail to specifically teach use the information to generate statistical data to use to perform load balancing.
However, Zadka teaches use the information to generate statistical data to use to perform load balancing (Zadka, [0020] The mathematical approximation function uses statistical data, which includes information regarding the response latencies of the pool for various values of the load on the pool, for determining the number of servers to be moved from the pool. For example, the load balancing server selects a set of servers in a pool and increments a current load on the set of servers progressively, e.g., until the response latency exceeds the specified latency threshold, and records values of the response latency of the pool for various load values to generate statistical data. The load balancing server can then use the statistical data to determine an appropriate percentage for increasing the load in a pool while still keeping the pool in the positive state, which can then be used to determine the number of servers to be moved).
It would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to have combined the teaching of Bernat, Dutta, Diard, Lowry, Shraer and Guo with Zadka because Zadka’s teaching of using the statistical data to determine an appropriate percentage for load balancing would have provided Bernat, Dutta, Diard, Lowry, Shraer and Guo’s system with the advantage and capability to allow the system to easily managing and balancing the load among different devices which improving the system efficiency and performance.
Response to Arguments
Applicant’s arguments with respect to independent claims 1-35 under U.S.C. § 103 rejection have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
In the remark applicant’s argue in substance:
(a), Applicant respectfully submits that receiving API call(s) from a set of clients that include request(s) to perform video frame transformation(s), and assigning and reassigning at least some of the set of clients among hardware accelerators, cannot practically be performed in the human mind.
(b), The Specification describes this as a concrete technological solution to a technological problem. In particular, the Specification explains that distributing work to available hardware accelerators may "overcome manual tuning and/or configuration of transformations, reduce bottlenecks, and improve stream density." Specification, at [0054]. The Specification further explains that the disclosed techniques improve the field of load balancing between hardware accelerators and are rooted in computer technology.
(c), As amended, the claims recite a specific combination in which client-submitted requests are received via API call(s), the requests are initially assigned to a pre-selected hardware accelerator, metrics associated with the set of clients are evaluated relative to a threshold, a subset of the set of clients is assigned to second hardware accelerators, and at least one of the set of clients is reassigned back to the first hardware accelerator when appropriate. Applicant respectfully submits that this combination is a specific technical solution for improving operation of a multi-accelerator video-processing system, not merely a generic instruction to apply an abstract idea on a computer.
(d), claims 3-8, 10-12, 14, 16-25, 29-34: the only motivation to combine these references is found in Applicant's own application and not in the cited references, which is impermissible hindsight.
Examiner respectfully disagreed with Applicant’s argument for the following reasons:
As to point (a), in response to applicant’s argument that “receiving API call(s) from a set of clients that include request(s) to perform video frame transformation(s), and assigning and reassigning at least some of the set of clients among hardware accelerators, cannot practically be performed in the human mind”. Examiner would like to point out that “receive one or more calls via an application programming interface (API) from a set of clients, the one or more calls comprising one or more requests to perform one or more video frame transformations” which is insignificant pre-solution data gathering (see MPEP § 2106.05(g)) which are well understood, routine, conventional activity (see MPEP § 2106.05(d)). Courts have identified “receiving and transmitting data, storing and retrieving information”, et cetera as well understood, routine, conventional and mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(f))). These additional elements and combination of the elements does not amount to significant more than the exception itself or provide an inventive concept in Step 2B. That is, the “receive” steps are for the purpose of “communication” and “transmitting the data” and these can be reached on one of court case (Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) see MPEP § 2106.05(d) II). Accordingly, a conclusion that “receive” are well understood, routine, conventional activity is supported under Berkheimer options 2.
Further, “reassigning at least some of the set of clients among hardware accelerators” which can be performed by human mind, such as assigning/scheduling/planning/reassigning at least some of the set of clients among hardware accelerators. Therefore, but for the recitation of generic computing components, these steps may be a Mental Processes that can be performed in the human mind (including an observation, evaluation, judgment, opinion).
Therefore, applicant’s argument has not been found to be persuasive. Please refers to 101 rejection above.
As to point (b)-(c), please refers to point (a) above. In addition, applicant merely recites the specification [0054] to show it is integrated into a practical application and providing the improvement. However, MPEP 2106.05(a) discloses that “It is important to note, the judicial exception alone cannot provide the improvement. The improvement can be provided by one or more additional elements. See the discussion of Diamond v. Diehr, 450 U.S. 175, 187 and 191-92, 209 USPQ 1, 10 (1981)) in subsection II, below. In addition, the improvement can be provided by the additional element(s) in combination with the recited judicial exception”. Here, the claim does NOT providing any additional limitations other than the generic computing components/Functions that perform the abstract idea (i.e., cause” this “video frame transformation,” other than a generic machine such as the “hardware accelerator,” and no details what so ever on how the claimed function (i.e., video frame transformations) will occur, “modify one or more individual frames in the batch of video frames” is not enough for providing details what so ever on how the claimed function will occur. How this “modification” is performed?). Therefore, applicant’s argument has not been found to be persuasive. Please refers to 101 rejection above.
As to points (d), in response to applicant's argument that the examiner's conclusion of obviousness is based upon improper hindsight reasoning, it must be recognized that any judgment on obviousness is in a sense necessarily a reconstruction based upon hindsight reasoning. But so long as it takes into account only knowledge which was within the level of ordinary skill at the time the claimed invention was made, and does not include knowledge gleaned only from the applicant's disclosure, such a reconstruction is proper. See In re McLaughlin, 443 F.2d 1392, 170 USPQ 209 (CCPA 1971).
For the reasons above, Applicant’s argument has not been found to be persuasive, and therefore the rejections are maintained.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZUJIA XU whose telephone number is (571)272-0954. The examiner can normally be reached M-F 9:30-5:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Aimee J Li can be reached at (571) 272-4169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZUJIA XU/Primary Examiner, Art Unit 2195
Prosecution Insights