Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the amendments and remarks filed on 03/03/2026. Claims 1-7, 9-14, 16-22 are pending and have been examined. Claims 8 and 15 have been cancelled. Claims 1-7, 9-14, 16-22 are rejected.
Response to Arguments
Applicant’s argument:
The Rejection of Claims Under 35 U.S.C. 112(b)
Claim 11 and its dependent claim 12 were rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as allegedly being indefinite. Specifically, the Examiner alleged that the term "optimize" renders the claim indefinite. Applicant respectfully disagrees, however, to advance prosecution, these claims have been amended to recite "increase" instead of "optimize."
Applicant respectfully requests that the Examiner withdraws this rejection.
Examiner’s answer:
In response to the amended claims, the examiner withdraws the 35 U.S.C. 112(b) rejection made in the previous office action.
Applicant’s argument:
The Rejections of Claims Under 102 & 103
Applicant respectfully submits that none of the cited references teach or disclose at least the following claim elements recited in the currently amended independent claims:
"monitoring a utilization level of the GPU resources allocated to the
trainer during the model building process; and
in response to determining that the utilization level is below a
threshold, automatically deploying an additional data loader of the one or more
data loaders to execute the data loading task in addition to the data loader."
Examiner’s answer:
In response to the amended claims regarding the 102 (a)(1) rejection for claims 1-2, 16, and 19-20, the examiner acknowledges the amendments overcome the prior art. New grounds of rejection has been made in light of Zhao in view of Jia. Accordingly new grounds of rejection have been made for claims 1-2, 16, and 19-20, under 35 U.S.C. § 103.
In response to the amended claims regarding the 103 rejections for claims 3-7, 9-14, and 17-18. These claims were not amended, however, they depend on the amended claim 1. Therefore, new grounds of rejection have been made in light of Zhao in view of Jia and Johansson for claims 3-7 and 9-14. Similarly, new grounds of rejection have been made in light of Zhao in view Jia and Tang for claims 17-18.
Applicant’s argument:
Regarding new claims 21-22, applicant states “In rejecting the original claim 1, the Examiner alleged that "the term 'trainer' in claim 1 is interpreted broadly to include the 'Execution Flow' subsystem which includes both the 'Client Node' and 'Server Node' acting together as described in the Zhao Reference." Office Action, page 6. In Zhao, however, the Client Node and the Server Node are separate logical and physical entities that do not constitute "a single containerized application executed within a single pod" as recited in the new claim 21. For at least this additional reason, obviousness is not established for dependent claim 21. The new claim 22 recites "a third data loader of the one or more data loaders generates the next training batch concurrently with the trainer executing the training task on the current training batch." This feature is also not disclosed in any of the references cited.”
Examiner’s answer:
Applicant’s arguments are moot in view of new grounds for rejection for claim 21-22. Accordingly, claims 21-22 are rejected in light of Zhao in view of Jia further in view of Graur.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Examiner’s Note: Some rejections will include an Examiner’s Note (labeled ‘EN’) to provide additional context or rationale explaining the basis for the rejection.
Claim 1-2 and 16 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhao et al. (US-20200174840-A1) hereafter (Zhao) in view of non-patent literature Jia et al. (“A Data-Loader Tunable Knob to Shorten GPU Idleness for Distributed Deep Learning”, hereinafter “Jia”).
Claim 1
A method comprising: (Abstract, “Techniques are provided to decouple data pipeline tasks from an execution flow of a high-performance computing task (e.g., distributed deep model training) in a distributed computing system. For example, a method includes…”) accessing, by one or more data loaders in a distributed computer system, input data from a storage component; (figure 2, EN: this
denotes raw data that is being accessed by the preprocessing nodes).
PNG
media_image1.png
661
920
media_image1.png
Greyscale
allocating Central Processing Unit (CPU) resources in the distributed computer system to the one or more data loaders; (Para 46, “Each data pre-processing node 250 comprises a server node having one
or more processors 252 (e.g., central processing units) which perform data-preprocessing operations
on portions of the raw data accessed from the storage nodes 240”)
transmitting, by a trainer in the distributed computer system, a training data request, the trainer being
communicatively coupled to the one or more data loaders, (Para 62, “In one embodiment, the client
node will communicate with the pre-processing/staging node and the server node using a remote
procedure call (RPC) protocol in which the client node sends a request message to the remote nodes
to execute a specified procedure.”) and the trainer and the one or more data loaders being executed
on separate processors; (Para 58, “the logical nodes for the data storage and pre-processing operations can co-exist on a given client, while the logical nodes for the data staging operations can exist on one or more nodes (e.g., GPU server nodes or other compute nodes of a
cluster)”)
allocating Graphics Processing Unit (GPU) resources in the distributed computer system to the trainer; (Para 44, “The sever nodes 220 comprise the logical nodes which provide XaaS to execute HPC jobs using the accelerator devices 226. The accelerator devices 226 include, for example, GPU devices,” Para 53, “FIG. 2 illustrates an exemplary embodiment in which the different stages… are performed by logical nodes that are provisioned on different server nodes.” Para 51, “cluster wide resources, including storage nodes for I/O, CPU nodes for pre-processing, and server nodes (e.g., GPU nodes) for deep learning model training”)
in response to receiving, by a data loader of the one or more data loaders, the training data request,
executing, by the data loader, a data loading task, the data loading task including preprocessing an input
batch read from the input data to generate a training batch; (Para 63, “In response to the RPC request
message received from the client node, the data pre-processing/staging node commences a data pre-
processing service based on the specified parameters (block 610)” Para 65, “the raw data is divided
into mini-batches… and… loaded into memory for pre-processing”)
transmitting, by the data loader, the training batch to the trainer; and (Para 71, “the server node will pre-fetch relevant tensor data structures and associated label information from the staging area into the system memory of the server node.” EN: Zhao discloses transmitting the batch to the server node. The server node is a computing resource operating “under the control of the deep learning application” executing on the client node (paragraph 21, and Figure 1.) Therefore, transmitting the batch to the server node is transmitting the batch to the trainer.)
and executing, by the trainer, a training task, the training task including using the training batch to
execute a machine learning algorithm in a model building process: (Para 44, “The server
daemon 222 executes HPC tasks (e.g., deep learning model training tasks)” Para 71, “The server node
will perform host-to-device memory copy operation to copy a current mini-batch data set to memory
of the accelerator devices for processing (block 624) Para 16, “embodiments of the invention will be
discussed in the context of performing distributed deep learning model training for deep neural
network (DNN) applications.”)
PNG
media_image2.png
664
881
media_image2.png
Greyscale
Examiners Note Regarding the “Trainer”: The term “trainer” in Claim 1 is interpreted broadly to
include the “Execution Flow” subsystem which includes both the “Client Node” and “Server Node”
acting together as described in the Zhao reference. Since the applicant’s specification does not limit
the “trainer” to a single physical device, it is reasonable to map this term to the combined system in
Zhao where the Client Node acts as the manager that requests the training data and the Server Node
acts as the worker that executes the machine learning model. This interpretation matches the claim
limitations because these components work together to run the training process and are distinct from
the “data flow pipeline” (data loaders), thereby satisfying the requirement that the trainer and the
data loader be executed on separate processors.
Zhao does not explicitly disclose:
monitoring a utilization level of the GPU resources allocated to the trainer during the model building process; and
in response to determining that the utilization level is below a threshold, automatically deploying an additional data loader of the one or more data loaders to execute the data loading task in addition to the data loader.
However, Jia teaches:
monitoring a utilization level of the GPU resources allocated to the trainer during the model building process; and (Page 454, “3 The life circle monitor tracks the status of applications, and 4 reports to the worker allocator. […] Worker Allocator: The worker allocator module receives the run-time status of applications from the life-circle monitor. It also obtains the application’s information about hyper- parameters passed by users in submission. Then the worker allocator uses our throughput prediction model (TPM), see Sec. for and TII-B, to estimate the training rate and data-loading rate for each application” Page 450, “We can derive the idle time at GPU kernels by excluding the step time from the iteration duration, see Fig. 2.” – EN: this denotes actively tracking the live status of the applications to estimate training rates and calculating GPU idle time, thus functionally teaching monitoring how efficiently the GPU is being utilized during the training run.)
in response to determining that the utilization level is below a threshold, (Page 455, “If the throughput of specific running applications can benefit and retrieve from those more workers, the scheduler will pause applications with a new worker number. Otherwise, the scheduler will keep those release workers idle until 4) new applications come.” – EN: this denotes the system evaluating if the current raining application is operation below its optimal throughput (i.e. falling below the threshold of maximum utilization) and requires intervention because the GPU is stalling while waiting for data.)
automatically deploying an additional data loader of the one or more data loaders to execute the data loading task in addition to the data loader. (Page 456, “After the scheduler receives the completion buffer, the worker allocation mechanism is triggered to reallocate the just-released workers to the remaining application i.e. Renset18. […] Thus, after pausing and retrieving, Rensetl8 is assigned with eight workers achieve its maximum throughput,” Also see Figure 8 which shows Resnet18 initially started with 2 workers, and was then assigned additional workers to reach a total of 8. – EN: this denotes the system automatically assigning extra data-loading workers to an already-running training process (which previously had fewer workers) so that multiple data loaders can execute the loading tasks together.”)
Before the effective filing date of the claimed invention, it would have been obvious to one skilled in the
art of machine learning to combine the distributed machine learning architecture of Zhao with the dynamic, run-time data-loader allocation of Jia. The motivation for doing so would be to prevent data-loading bottlenecks from causing GPU idleness, thereby maximizing GPU resource utilization. See Page 451 of Jia, “If one can shorten or remove idle intervals, then the DDL training time can be reduced, and the throughput2 of DDL training can be increased. Therefore, the goal of this work is to maximize resource utilization on GPU kernels and obtain high training throughput by avoiding the idle time at each training kernel.”
Claim 2
Zhao further teaches:
wherein the GPU resources are allocated such that each data loader does not utilize any of the
GPU resources. (Para 44, “The sever nodes 220 comprise the logical nodes which provide XaaS to
execute HPC jobs using the accelerator devices 226. The accelerator devices 226 include, for example,
GPU devices,” Para 53, “FIG. 2 illustrates an exemplary embodiment in which the different stages…
are performed by logical nodes that are provisioned on different server nodes.” Para 51, “cluster
wide resources, including storage nodes for I/O, CPU nodes for pre-processing, and server nodes (e.g.,
GPU nodes) for deep learning model training”)
Claim 16:
Zhao further teaches:
implementing, by each data loader, a multi-producer, multi-consumer queue, the implementing comprising performing the preprocessing at least partially in parallel with a reading task, the reading task comprising fetching, by the data loader, one or more input batches from the storage component and adding the one or more input batches to a preprocessing queue of the data loader. (Para 42, “Data processing pipelines are configured to exploit data prefetching, in-parallel and asynchronous pre-processing, to hide latency.” Para 64, “In the pre-processing/staging node(s), the data pre-processing service begins to pre-load the raw data from a target storage node based on the specified data URI (block 612)…. In one embodiment, the data access and pre-loading operations are performed using multiple threads to prefetch the relevant data and temporarily store the pre-fetched data to a local directory.” )
Claim 19:A distributed computing system comprising: (Para 43, “FIG. 2 schematically illustrates a distributed computing system 200 comprising a client node 210, one or more server nodes 220…”) one or more processors; and (Para 94, “The computing node 800 comprises processors 802…”) a non-transitory computer readable storage medium comprising instructions that when executed by the one or processors cause the one or more processors to perform operations comprising: (Para 100, “In one embodiment, the logical nodes 818 comprise program code that is loaded into the system memory 810 (e.g., volatile memory 812), and executed by the processors 802 to perform respective functions as described herein… The term “article of manufacture” as used herein should be understood to exclude transitory, propagating signals.”)
The rest of claim 19 recite the same limitations as claim 1, therefore claim 19 is rejected under the same rationale as claim 1.Claim 20:
A machine-readable non-transitory storage medium having instruction data executable by a machine to cause the machine to perform operations comprising: (Para 100, “In this regard, the system memory 810 resources, and other memory or storage resources as described herein, which have program code and data tangibly embodied thereon, are examples of what is more generally referred to herein as “processor-readable storage media” that store executable program code of one or more software programs. Articles of manufacture comprising such processor-readable storage media are considered embodiments of the invention. An article of manufacture may comprise, for example, a storage device such as a storage disk, a storage array or an integrated circuit containing memory. The term “article of manufacture” as used herein should be understood to exclude transitory, propagating signals.”)
The rest of claim 20 recite the same limitations as claim 1, therefore claim 20 is rejected under the same rationale as claim 1.
Claims 3-7, 9-14 are rejected under 35 U.S.C. 103 as being unpatentable over Zhao et al. (US-20200174840-A1) hereinafter (Zhao) in view of non-patent literature Jia et al. (“A Data-Loader Tunable Knob to Shorten GPU Idleness for Distributed Deep Learning”, hereinafter “Jia”) and non-patent literature Johansson (“Lookaside Load Balancing in a Service Mesh Environment” hereinafter "Johansson")
Claim 3:
Zhao further teaches:
communications between the trainer and the one or more data loaders (Para 24, “the bus/communication network 120 comprises backbone networking infrastructure and communication protocols to implement one or more of various types of intra-node and/or inter-node connection topologies and communication protocols that are utilized to physically connect, and enable communication between, the hardware processor resources which execute the functions of the parameter server nodes 110 and the worker nodes 130.)
Zhao does not distinctly disclose: deploying one or more network proxies such that communications (…) are effected via a service mesh.However, Johansson teaches: deploying one or more network proxies such that communications (…) are effected via a service mesh. (Page 10-11 Section 2.3.4, “A service mesh describes the network of micro services and the interactions between them. Service mesh infrastructure includes a number of components that solve issues such as discoverability, load balancing, monitoring or failure recovery [12]… Two such solutions are the Linkerd and Istio projects [12]. Each of these projects provides load balancing solutions that utilize the concept of sidecar proxies for adding load balancing support to any micro service…”)
Before the effective filing date of the claimed invention it would have been obvious to one skilled in the art of machine learning to combine the work of Zhao and Johansson in order to use a service mesh to run their system. The motivation for doing so would be to allow more efficient load balancing. (Johansson, iii, Abstract, “As more online services are migrated from monolithic systems into decoupled distributed micro services, the need for efficient internal load balancing solutions increases. Today, there exists two main approaches for load balancing internal traffic between micro services. One approach uses either a central or sidecar proxy to load balance queries over all available server endpoints.”)
Claim 4:
Zhao further teaches:
wherein the deploying (…) comprises (…) either the trainer or the one or more data loaders,
(FIG. 2 denotes the execution flow between Client node 210/Server Nodes 220 which correspond to the trainer and the data flow pipeline 230 which includes the data loaders).
Zhao does not distinctly disclose:
“one or more network proxies,” and “defining a data plane of the service mesh by deploying each network proxy in unique association with” and “each network proxy being communicatively coupled to a control plane of the service mesh.”
However, Johansson teaches: deploying one or more network proxies comprises defining a data plane of the service mesh by deploying each network proxy in unique association (Section 2.5.3, page 16, “This is the concept of sidecar proxy load balancing, where a self-managed proxy load balancer is deployed alongside each instance of a micro service… By utilizing containerization and deployment capabilities given by container orchestration we are able to attach a resource thin proxy to the local network interface of each deployed micro service instance.” each network proxy being communicatively coupled to a control plane of the service mesh (Page 29, Figure 3.3 - denotes the control plane interacting with the envoy proxies (data plane).
Before the effective filing date of the claimed invention it would have been obvious to one skilled in the art of machine learning to combine the work of Zhao and Johansson in order to have a service mesh for the trainer and data loaders that includes data plane and control plane.
The motivation for doing so would be to allow more efficient load balancing and to reduce development overhead. (Johansson, Section 2.3.4, page 11, “The purpose of a service mesh is to provide a platform where micro services only require application-specific logic [12]. This means that cost and overhead of development may be reduced while at the same time increasing safety and reliability.” Johansson, iii, Abstract, “As more online services are migrated from monolithic systems into decoupled distributed micro services, the need for efficient internal load balancing solutions increases. Today, there exists two main approaches for load balancing internal traffic between micro services. One approach uses either a central or sidecar proxy to load balance queries over all available server endpoints.” )
Claim 5:Zhao further teaches:
wherein the one or more data loaders is a plurality of data loaders, defining a one-to-many relationship between the trainer and the data loaders, each data loader being uniquely associated with one of the (…). (FIG 2. EN: this depicts the client as a single entity while the data flow pipeline includes a plurality of nodes).
Zhao does not distinctly disclose: “network proxies”However, Johansson teaches: “network proxies” (Section 2.5.3, page 16, “This is the concept of sidecar proxy load balancing, where a self-managed proxy load balancer is deployed alongside each instance of a micro service…”)
Before the effective filing date of the claimed invention, it would have been obvious to one skilled in the art of machine learning to combine the work of Zhao and Johansson in order to include a network proxy to the data loaders.
The motivation for doing so would be to reduce latency. (Johansson, section 2.5.3, page 17, “This form of proxy load balancing has the main benefit of reduced Round Trip Time (RTT) latency since a network hop on the local network interface is negligible compared to the network hop to any centralized proxy (see Figure 2.1). Sidecar proxy-based load balancing is a part of the proxy load balancing approach which has not been widely researched in terms of CPU, memory and latency impact. Because of this, load balancing via a sidecar proxy is an approach that can also be considered within the scope of this study when comparing lookaside load balancing with state of the art proxy and client based solutions.”)
Claim 6:
Zhao further teaches:
transmitting, by the trainer, training data requests to each of the plurality of data loaders. (para 62, “the client node will internally connect to the allocated pre-processing/staging node(s) and server node(s) and provide relevant information for job execution (block 602). For example, the client node will provide relevant information such as the job ID (e.g., a universally unique identifier (UUID)), the directory or path to the raw data to be processed, the mini-batch size, the number of mini-batch iterations, the number of accelerator devices (e.g., GPU devices) to utilize for the deep learning model training job, etc.”)Claim 7:Zhao further teaches:
wherein the training data requests are transmitted using a Remote Procedure Call (RPC) protocol. (Para 62, “In one embodiment, the client node will communicate with the pre-processing/staging node and the server node using a remote procedure call (RPC) protocol in which the client node sends a request message to the remote nodes…”)
Claim 9:
Zhao further teaches:wherein each data loader is a separate instance of a service in the distributed computer system. (Para 52, “…wherein the different stages, e.g., data I/O, data preprocessing, and data staging, can be implemented using multiple instances of logical nodes that run in parallel.”)
Claim 10:
Zhao further teaches:
controlling, (…), traffic between the trainer and the plurality of data loaders. (Para 51, “…the client node 210 orchestrates the data I/O, preprocessing, and staging operations that are performed by the logical nodes of the data flow pipeline 230. The data pipeline composition module 272 dynamically composes data flow pipelines for HPC tasks (e.g., deep learning model training) so that components of the data pipeline can be flexibility decoupled and placed on different nodes, and allow the data flow pipeline and deep learning tasks to be well coordinated by a central point. ’)
Zhao does not disclose “by the service mesh” However Johansson teaches “by the service mesh” (Section 2.3.4, page 10-11, “Service mesh infrastructure includes a number of components that solve issues such as discoverability, load balancing, monitoring or failure recovery [12]…. The purpose of a service mesh is to provide a platform where micro services only require application-specific logic [12].”)
Before the effective filing date of the claimed invention, it would have been obvious to one skilled in the art of machine learning to combine the work of Zhao and Johansson in order to include a service mesh to control the traffic between trainer and data loaders.
The motivation for doing so would be to allow reduction in cost and overhead of development. (Johansson, Section 2.3.4, page 11, “The purpose of a service mesh is to provide a platform where micro services only require application-specific logic [12]. This means that cost and overhead of development may be reduced while at the same time increasing safety and reliability.”)
Claim 11
Zhao further teaches:
routing, (…), the training data requests to the data loaders so as to increase utilization of the trainer.(Para 42, “ A data pipeline is orchestrated and optimized to achieve higher throughput and resource utilization. In one embodiment, a data pipeline is orchestrated by a client for a specific job by, e.g., launching data loading, pre-processing, training mini-batch. Data processing pipelines are configured to exploit data prefetching, in-parallel and asynchronous pre-processing, to hide latency.”)
Zhao does not disclose “by the network proxies”
However Johansson teaches “by the network proxies” (Section 2.5.3, page 16, “This is the concept of sidecar proxy load balancing, where a self-managed proxy load balancer is deployed alongside each instance of a micro service…”)
Before the effective filing date of the claimed invention, it would have been obvious to one skilled in the art of machine learning to combine the work of Zhao and Johansson in order to use network proxies to route the training data requests.
The motivation for doing so would be to reduce latency. (Johansson, section 2.5.3, page 17, “This form of proxy load balancing has the main benefit of reduced Round Trip Time (RTT) latency since a network hop on the local network interface is negligible compared to the network hop to any centralized proxy (see Figure 2.1). Sidecar proxy-based load balancing is a part of the proxy load balancing approach which has not been widely researched in terms of CPU, memory and latency impact. Because of this, load balancing via a sidecar proxy is an approach that can also be considered within the scope of this study when comparing lookaside load balancing with state of the art proxy and client based solutions.”)
Claim 12
Zhao further teaches:
wherein the routing comprises (…) determine to which one of the data loaders to transmit each training data request. (Para 61, a resource manager module 270 (FIG. 3) can be configured to schedule and provision the appropriate GPU server node(s) and pre-processing/staging node(s) and return the login information to the client node.)Zhao does not disclose: “using an exponentially-weighted moving average of response latencies to”However Johansson teaches: “using an exponentially-weighted moving average of response latencies to” (Page 14, “The Linkerd service mesh utilizes an algorithm known as Exponentially Weighted Moving Average (EWMA) [17]. This algorithms functions by inference of endpoint load based on response time. It does this by keeping an exponentially rolling average of the response time to each endpoint and prioritizing endpoints with lower averages when sending requests.”
Before the effective filing date of the claimed invention, it would have been obvious to one skilled in the art of machine learning to combine the work of Zhao and Johansson in order to use an EWMA to make the determination.
The motivation for doing so would be to reduce latency. (Johansson, page 14, “While response time might be affected by other aspects such as underlying network congestion, this algorithm has shown some promise for reducing both client latencies and server load [18].”)
Claim 13Zhao further teaches:
wherein the training data requests are service-to-service calls, (Para 63, “In response to the RPC request message received from the client node, the data pre-processing/staging node commences a data pre-processing service”)
“trainer”, “training data”, and “data loaders” (Fig 2. Depicts execution flow between client node 210 and server nodes 220. It also depicts storage nodes 240 containing raw data 242, and pre-processing nodes 250, and staging nodes 260.)
Zhao does not disclose: “the network proxy associated with the (…) being configured to load balance the (…) requests across the plurality of (…).”However Johansson teaches: “the network proxy associated with the (…) being configured to load balance the (…) requests across the plurality of (…).” (Abstract, “Today, there exists two main approaches for load balancing internal traffic between micro services. One approach uses either a central or sidecar proxy to load balance queries over all available server endpoints.”)
Before the effective filing date of the claimed invention, it would have been obvious to one skilled in the art of machine learning to combine the work of Zhao and Johansson in order to use network proxy to load balance the requests.
The motivation for doing so would be to reduce latency. (Johansson, section 2.5.3, page 17, “This form of proxy load balancing has the main benefit of reduced Round Trip Time (RTT) latency since a network hop on the local network interface is negligible compared to the network hop to any centralized proxy (see Figure 2.1). Sidecar proxy-based load balancing is a part of the proxy load balancing approach which has not been widely researched in terms of CPU, memory and latency impact. Because of this, load balancing via a sidecar proxy is an approach that can also be considered within the scope of this study when comparing lookaside load balancing with state of the art proxy and client based solutions.”)
Claim 14:
Zhao further teaches:
“trainer” and “data loaders” (Fig 2. Depicts execution flow between client node 210 and server nodes 220. It also depicts pre-processing nodes 250, and staging nodes 260.)
Zhao does not disclose: “wherein the … and each … is executed by a respective pod in the distributed computer system, each network proxy being a proxy container added to the pod of the … or the … associated with the network proxy.”
However Johansson teaches:
wherein the … and each … is executed by a respective pod in the distributed computer system, each network proxy being a proxy container added to the pod of the … or the … associated with the network proxy. (Page 16-17 Section 2.5.3 and Page 21-22 Section 3.2.2, this denotes the use of Kubernetes “pods” to execute services and the addition of “sidecar” proxies into those pods sharing the same network interface.)
Before the effective filing date of the claimed invention, it would have been obvious to one skilled in the art of machine learning to combine the work of Zhao and Johansson in order to use network proxy container pods to the trainer and data loaders.
The motivation for doing so would be to reduce latency. (Johansson, Section 2.5.3, Page 17, “This form of proxy load balancing has the main benefit of reduced Round Trip Time (RTT) latency since a network hop on the local network interface is negligible compared to the network hop to any centralized proxy (see Figure 2.1).”)
Claims 17-18 are rejected under 35 U.S.C. 103 as being unpatentable Zhao et al. (US-20200174840-A1) hereafter (Zhao) in view of non-patent literature Jia et al. (“A Data-Loader Tunable Knob to Shorten GPU Idleness for Distributed Deep Learning”, hereinafter “Jia”) and of Tang et al. (US-20200225758-A1) hereinafter (Tang).
Claim 17
Tang teaches:
wherein the input data includes at least one of hand detection data, hand tracking data, gesture detection data, or gesture tracking data. (Para 67, “User specificity may be trained in a calibration phase where the user performs various gestures and this test data is used to train an artificial neural network, for example and/or includes assigning hand joints to a virtual skeleton may be based at least in part on image data of the user performing the first-stage gesture and the second-stage gesture.”)
Before the effective filing date of the claimed invention, it would have been obvious to one skilled in the art of machine learning to combine the work of Zhao and Tang in order use hand detection or hand tracking data to train a model.
The motivation for doing so would be to accurately detect certain hand gestures by computing systems. (Tang, para 59, “In other examples, an artificial neural network may be trained to assess gesture confidence based on one or more frames of feature data input.”)
Claim 18
Zhao further teaches:
wherein the trainer is configured to transmit a plurality of additional training data requests in order to permit the training task to be iterated using a plurality of training batches generated from different input batches by the one or more data loaders, the method comprising generating, by the trainer, an (…) model based on a result of the training tasks in the model building process. (Para 69, “The client orchestrates the deep learning model training iterations. For each iteration, a mini-batch dataset comprising one or more tensors and associated labels are fed to the server node(s). Para 65, the raw data is divided into mini-batches based on the predefined mini-batch size, and a plurality (N) of mini-batches of data can be loaded into memory for pre-processing.” Para 27, “The deep learning model 52 can implement one or more different types of models such as CNN models, recurrent neural network (RNN) models…”)
Zhao does not disclose: “object tracking”
However Tang teaches: “object tracking” (Para 40, “Gesture recognition machine 316 may track different body parts from frame to frame, thereby allowing different gestures to be discerned. For example, the three-dimensional position of fingers may be tracked from frame to frame, thus allowing parameters such as finger position, finger angle, finger velocity, finger acceleration, finger-to-finger proximity, etc. to be discerned.”)
Before the effective filing date of the claimed invention, it would have been obvious to one skilled in the art of machine learning to combine the work of Zhao and Tang in order generate an object tracking model.
The motivation for doing so would be to accurately detect certain hand gestures by computing systems. (Tang, para 59, “In other examples, an artificial neural network may be trained to assess gesture confidence based on one or more frames of feature data input.”)
Claims 21-22 are rejected under 35 U.S.C. 103 as being unpatentable Zhao et al. (US-20200174840-A1) hereafter (Zhao) in view of non-patent literature Jia et al. (“A Data-Loader Tunable Knob to Shorten GPU Idleness for Distributed Deep Learning”, hereinafter “Jia”) and non-patent literature Graur et al. (“Cachew: Machine Learning Input Data Processing as a Service”, hereinafter “Graur”)
Claim 21
Graur teaches:
The method of claim 1, wherein the trainer is a single containerized application executing within a single pod in the distributed computer system, and (Page 693, “Cachew consists of a centralized dispatcher, a dynamic number of input data workers, and a disaggregated storage cluster for data caching, as shown in Figure 3. Users register training nodes (i.e., clients) of ML training jobs with the Cachew dispatcher.” Page 696, “We run the Cachew dispatcher and workers inside Docker containers and use Kubernetes to elastically scale the deployment.” Page 696-697, “Baselines: We compare Cachew’s resource scaling policy with the Kubernetes Horizontal Pod Autoscaler (HPA) policy, which scales input data workers in the service based on a 80% CPU resource utilization target per node [1].” – EN: this denotes the entire distributed training architecture, including the training client, as operating within Kubernetes and Docker ecosystem scaled by the horizontal Pod Autoscaler (HPA), therefore this teaches that the training application is deployed as a containerized pod.)
wherein the training data request is transmitted from the pod of the trainer to a pod of the data loader. (Page 691, “Clients fetch data directly from workers.” Page 693, “Clients fetch data from the workers that are assigned to them by the dispatcher. Clients and workers periodically send heartbeats to the dispatcher (by default every five seconds) to maintain membership in the service and provide metrics.” Page 696, “All communication between clients, workers, and the dispatcher is done over gRPC [24].” – EN: this denotes the client actively fetching dta from the workers and that all of this communication occurs over gRPC, thereby teaching the back and forth pod transmission of training data requests.)
Before the effective filing date of the claimed invention, it would have been obvious to one skilled in the
art of machine learning to combine the distributed machine learning architecture of Zhao and the dynamic, run-time data-loader allocation of Jia with containerized Kubernetes pod deployment and gRPC communication architecture of Graur. The motivation for doing so would be to flexibly scale the disaggregated data processing deployment and improve resource allocation across isolated environments. See page 696 of Graur, “We run the Cachew dispatcher and workers inside Docker containers and use Kubernetes to elastically scale the deployment. All communication between clients, workers, and the dispatcher is done over gRPC [24]. Graur further explains that this containerized disaggregation “is a well-known approach for improving resource allocation flexibility [23,36]. This flexibility can save cost, since users can distribute input data processing across as many or as few CPU worker nodes as needed to avoid data stalls, without provisioning additional expensive GPUs/TPUs.”
Claim 22
Zhao teaches:
The method of claim 1, wherein: the data loader is a first data loader and (Para 46, “Each data pre-processing node 250 comprises a server node having one or more processors 252 (e.g., central processing units) which perform data-preprocessing operations on portions of the raw data accessed from the storage nodes 240”)
Zhao does not explicitly teach:
the additional data loader is a second data loader; and
transmitting the training data request comprises: during execution of the training task on a current training batch using the allocated GPU resources, transmitting, by the trainer, an additional training data request to the one or more data loaders for a next training batch,
wherein, in response to the additional training data request, a third data loader of the one or more data loaders generates the next training batch concurrently with the trainer executing the training task on the current training batch.
However, Jia teaches:
the additional data loader is a second data loader; and (Page 456, “After the scheduler receives the completion buffer, the worker allocation mechanism is triggered to reallocate the just-released workers to the remaining application i.e. Renset18. […] Thus, after pausing and retrieving, Rensetl8 is assigned with eight workers achieve its maximum throughput,”)
a third data loader (Page 456, “At the beginning of the execution, we observe that WAA allocates 2, 8, and 8 workers to Resnet18, smResnet20, and mdResnet12, respectively.” – EN: this denotes that multiple workers exist, thus establishing availability of a third data loader.)
Before the effective filing date of the claimed invention, it would have been obvious to one skilled in the
art of machine learning to combine the distributed machine learning architecture of Zhao with the additional data loaders of Jia. The motivation for doing so would be to prevent data-loading bottlenecks from causing GPU idleness, thereby maximizing GPU resource utilization. See Page 451 of Jia, “If one can shorten or remove idle intervals, then the DDL training time can be reduced, and the throughput2 of DDL training can be increased. Therefore, the goal of this work is to maximize resource utilization on GPU kernels and obtain high training throughput by avoiding the idle time at each training kernel.”
Zhao in view of Jia does not explicitly teach:
transmitting the training data request comprises: during execution of the training task on a current training batch using the allocated GPU resources, transmitting, by the trainer, an additional training data request to the one or more data loaders for a next training batch,
wherein, in response to the additional training data request, (…) of the one or more data loaders generates the next training batch concurrently with the trainer executing the training task on the current training batch.
However, Graur teaches:
transmitting the training data request comprises: during execution of the training task on a current training batch using the allocated GPU resources, transmitting, by the trainer, an additional training data request to the one or more data loaders for a next training batch, (Page 695, “The client includes an incrementing index in its request which the new worker uses as time of failure to fast forward to.” – EN: this quote denotes the clients make requests to workers, Page 693, “Clients fetch data from the workers that are assigned to them by the dispatcher.” – EN: this quote denotes the clients actively fetch data from workers, Page 696 “All communication between clients, workers, and the dispatcher is done over gRPC [24].” Page 694, Table 1 – EN: Table 1 denotes fetching occurs concurrently with training via prefetch buffer
PNG
media_image3.png
264
1218
media_image3.png
Greyscale
Page 694, “To detect if we are on the batch time plateau and can afford to scale down,our intuition is that although batch time will be the same, the result queue will build up if workers are able to provide data faster than the model can ingest it.” – EN: this quote denotes the prefetch operates during ongoing training. Overall, in Graur, the training client actively transmits gRPC requests to data workers to fetch batches of preprocessed data, and the prefetch buffer (Table 1) ensures these requests are transmitted concurrently while the GPU is excuting the training task on the current batch.)
wherein, in response to the additional training data request, (…) of the one or more data loaders generates the next training batch (Page 693, “Input data workers are stateless components responsible for producing batches of preprocessed data for clients. The dispatcher dynamically adjusts the number of input data workers for each job and divides each job’s input dataset (e.g., a list of filenames) into independent partitions, called splits Workers pull new splits (e.g., indexes of the file list) from the dispatcher when they are done processing previous splits. Workers may read splits that correspond to source data which they must transform on-the-fly by executing the job’s input pipeline dataflow graph.”
concurrently with the trainer executing the training task on the current training batch. (Page 450, “the data-loading and data transfer of iteration i can overlap with the step of iteration i — 1 at GPU training kernels because the data-loading is independent of the outputs of step […] Therefore, the goal of this work is to maximize resource utilization on GPU kernels and obtain high training throughput by avoiding the idle time at each training kernel. – EN: this denotes the data loaders prepare the next batch of the training data at the same time the GPU training kernel is executing backpropagation on the current batch.”
Before the effective filing date of the claimed invention, it would have been obvious to one skilled in the
art of machine learning to combine the distributed machine learning architecture of Zhao with the additional data loaders of Jia with the disaggregated, request-driven data processing service of Graur, where training clients actively transmit gRPC requests to remote data processing workers to fetch preprocessed training batches, and in which multiple stateless workers are dynamically assigned per job to produce batches in response to those client requests concurrently with ongoing model training. The motivation for doing so would be to flexibly scale the disaggregated data processing deployment and improve resource allocation across isolated environments. See page 696 of Graur, “We run the Cachew dispatcher and workers inside Docker containers and use Kubernetes to elastically scale the deployment. All communication between clients, workers, and the dispatcher is done over gRPC [24]. Graur further explains that this containerized disaggregation “is a well-known approach for improving resource allocation flexibility [23,36]. This flexibility can save cost, since users can distribute input data processing across as many or as few CPU worker nodes as needed to avoid data stalls, without provisioning additional expensive GPUs/TPUs.”
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NAYMUR RAHMAN ALI whose telephone number is (571)272-0007. The examiner can normally be reached Mon-Fri. 9:30-6:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at (571)270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NAYMUR RAHMAN ALI/Examiner, Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights