DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. CN202010231450.7, filed on 3/27/2020.
Amendments
This Office Action is in response to the amendment filed on 10/24/2025.
Claim(s) 1-4, 8-11, and 15-18 have been amended.
No claims have been cancelled.
No new claims have been added.
The objections and rejections from the prior correspondence that are not restated herein are withdrawn.
Response to Arguments
Applicant's arguments filed on 10/24/2025 have been fully considered.
Applicant's arguments regarding the 35 U.S.C. 103 rejections of the previous office action have been fully considered but are not persuasive. Applicant argues that SHAZEER and LEE do not teach the emphasized claim limitations of claim 1, as indicated on page 12 of the remarks.
Examiner respectfully disagrees. LEE is no longer in the set of prior art used for the rejection. However, SHAZEER teaches output tensor of the first operator is an input tensor of the second operator (SHAZEER [0067] teaches: “Each node of the computation graph represents a respective operation of the plurality of operations.” SHAZEER [0068] teaches: “For any two nodes u and v in the computation graph, an edge (u, v) is a directed edge and represents a data dependency from u to v. A data dependency from u to v means the operation represented by node u (i.e., a first operator) generates an output that is input (i.e., and an output tensor of the first operator is an input tensor of the second operator) to the operation represented by node v (i.e., and a second operator).”). Further, the remaining arguments regarding the 103 rejections of the previous office actions are moot because the new reference of ZHANG in combination with SHAZEER teaches the claim 1 limitations emphasized on page 13 of the remarks. Moreover, the combination of SHAZEER with the new references of ZHANG, RAGHUNATHAN, and GHOLAMI is used to teach the claim limitations, as shown in the rejections below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-5, 7-9, 11-12, 14-16, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over SHAZEER (US 20200042875 A1) in view of ZHANG (US 20220391678 A1), hereafter SHAZEER and ZHANG, respectively.
Regarding Claim 1:
SHAZEER teaches:
A method for processing data, comprising: (SHAZEER [Abstract] teaches: “Methods, systems, and apparatus, including computer programs encoded on computer storage media, for distributing tensor computations across computing devices.” SHAZEER [0087] teaches: “Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus.”)
obtaining a deep neural network model, a tensor layout of an input tensor of at least one operator in the deep neural network model, and a tensor layout of an output tensor of the at least one operator in the deep neural network model, (SHAZEER [0071] teaches: “For example, the computation system 100 can receive a computation graph representing a neural network (i.e., obtaining a deep neural network model) having: an input layer, a hidden layer with nonlinear activation function A, and an output layer.” SHAZEER [0069] teaches: “A laid-out computation graph is a computation graph representing operations (i.e., of at least one operator) that receive input and generate outputs as laid-out tensors (i.e., obtaining […] a tensor layout of an input tensor […] in the deep neural network model […] and a tensor layout of an output tensor […] in the deep neural network model). The computation system 100 can generate a laid-out computation graph from a computation graph by assigning a layout to each input and output tensor in the graph. In this specification, operations for a laid-out computation graph are called laid-out operations, and the layouts for respective input and output tensors for a laid-out operation are called operation layouts.”)
wherein the deep neural network model comprises a first operator and a second operator, and an output tensor of the first operator is an input tensor of the second operator, (SHAZEER [0067] teaches: “Each node of the computation graph (i.e., the deep neural network) represents a respective operation of the plurality of operations.” SHAZEER [0068] teaches: “For any two nodes u and v in the computation graph, an edge (u, v) is a directed edge and represents a data dependency from u to v. A data dependency from u to v means the operation represented by node u (i.e., a first operator) generates an output that is input (i.e., and an output tensor of the first operator is an input tensor of the second operator) to the operation represented by node v (i.e., and a second operator).”)
wherein the sliced computational graph indicates to execute a part of the deep neural network model. (SHAZEER [0085] teaches: “The system assigns, based on the layouts for the input and output tensors, respective device-local operations to each of the plurality of computing devices (step 320). As described above with reference to FIG. 1, the computation system can comprise a mesh execution engine that distributes tensors according to respective layouts for the tensors. Operations can be distributed across the plurality of computing devices by assigning slices of a tensor representing weights (i.e., the sliced computational graph) to each of one or more of the plurality of devices.” SHAZEER [0086] teaches: “The system causes the tensor computations to be executed by the plurality of computing devices by causing each of the plurality of computing devices to execute at least the respective device-local operations assigned to the computing devices (i.e., the sliced computational graph indicates to execute a part of the deep neural network model) (step 330).”)
SHAZEER is not relied upon for teaching, but ZHANG teaches: and wherein a first tensor layout of the output tensor of the first operator is different than a second tensor layout of the input tensor of the second operator; (ZHANG [0173] teaches: “If the operator OP0 is determined to be split in a certain way, as the output of the operator OP0, the Tensor1 is also determined to be split into a series of pieces of sub-tensor data in a certain way (i.e., a first tensor layout of the output tensor of the first operator). Therefore, when the operator OP1 selects the splitting method, it must be ensured that a selected method is compatible with a determined splitting method of the input tensor data Tensor1 (i.e., a second tensor layout of the input tensor of the second operator) […].” ZHANG [0177] teaches: “As shown in FIG. 6B, if a splitting method of current tensor data is not allowed by any splitting method of a subsequent operator, or if the subsequent operator is compatible with the splitting method of the current tensor data, performance improvement brought by the splitting method that is optional is very poor, in this situation, the computer device may insert a glue operator in a calculation graph to adjust the splitting method of current data to another better splitting method (i.e., first tensor layout of the output tensor of the first operator is different than a second tensor layout of the input tensor of the second operator).”)
inserting a redistribution operator in a sliced computational graph of a data processing device, (ZHANG [0171] teaches: “the glue operator may be inserted (i.e., inserting a redistribution operator) between the target operator […].” ZHANG [0194] teaches: “In a step S408, the target operator may be split (i.e., sliced) according to the target splitting path to distribute the target operator to corresponding cores of the multi-core artificial intelligence processor for processing (i.e., of a data processing device).” ZHANG [0221] teaches: “in a step 700′, the split state sets of the tensor data associated with the target operator may be determined according to the target operator in the calculation graph corresponding to the neural network model (i.e., in a sliced computational graph);”)
wherein the redistribution operator is inserted between the first operator and the second operator in the sliced computational graph to convert the first tensor layout to the second tensor layout, (ZHANG [0171] teaches: “In a step S402, the glue operator may be inserted (i.e., the redistribution operator is inserted) between the target operator and the split state set associated with the target operator, and the split states in the split state set may be adjusted, and an adjusted split state set may be obtained, where the glue operator is used to convert the split states obtained by splitting the tensor data according to one splitting method into the split states obtained by splitting the tensor data according to another splitting method (i.e., to convert the first tensor layout to the second tensor layout).” ZHANG [0189] teaches: “inserting the glue operator between the operator Op0 and the Tensor1 input (i.e., between the first operator and the second operator) may convert any one split state of the tensor data into another split state.” ZHANG [0221] teaches: “in a step 700′, the split state sets of the tensor data associated with the target operator may be determined according to the target operator in the calculation graph corresponding to the neural network model (i.e., in the sliced computational graph);” Examiner’s note: ZHANG’s FIG. 6E teaches the glue operator inserted between Op0 (i.e., the first operator) and Op1 (i.e., the second operator).)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of SHAZEER and ZHANG before them, to include ZHANG’s glue operator in SHAZEER’s method for distributing tensor computation across computing devices. One would have been motivated to make such a combination in order to split a neural network calculation task into several calculation tasks to make full use of hardware resources and thereby avoiding extra workloads brought by reimplementation (ZHANG [0003]).
Regarding Claim 2:
SHAZEER in view of ZHANG teaches the elements of claim 1 as outlined above. SHAZEER further teaches:
wherein a device matrix of the first tensor layout is different than a device matrix of the second tensor layout, and/or a tensor map of the first tensor layout is different than a tensor map of the second tensor layout; (SHAZEER [0022] teaches: “Different meshes can be defined for the same set of physical devices (i.e., a device matrix of the first tensor layout is different than a device matrix of the second tensor layout) […].” SHAZEER [0047] teaches: “FIG. 2B shows a tensor laid-out on the mesh 220 of four computing devices A-D with the tensor sliced along tensor-dimension 0 and the slices laid-out over mesh-dimension 0. The tensor is not sliced and laid-out across mesh-dimension 1. The two-tuple representing this layout is <1, Ø> (i.e., a tensor map of the first tensor layout).” SHAZEER [0048] teaches: “FIG. 2C shows a tensor laid-out on the mesh 220 of four computing devices A-D with the tensor sliced along tensor-dimension 0 and the slices laid-out over mesh-dimension 0. The tensor is also sliced along tensor-dimension 1 with the slices laid-out over mesh-dimension 1. The layout shown in FIG. 2C is represented as <0, 1> (i.e., is different than a tensor map of the second tensor layout).”)
ZHANG further teaches: the method further comprises: determining an intermediate tensor layout based on the first tensor layout and the second tensor layout; (ZHANG [0186] teaches: “In FIG. 6E, split states in an updated split state set may include: a state m′1, a state m′2, and a state m′k. Here, the state m′1, the state m′2, and the state m′k are new states generated (i.e., determining an intermediate tensor layout) after states in a first split state set pass through glue operator.” ZHANG [0171] teaches: “In a step S402, the glue operator may be inserted between the target operator and the split state set associated with the target operator, and the split states in the split state set may be adjusted, and an adjusted split state set may be obtained, where the glue operator is used to convert the split states obtained by splitting the tensor data according to one splitting method into the split states obtained by splitting the tensor data according to another splitting method (i.e., based on the first tensor layout and the second tensor layout).”)
determining a first reshape operator based on the first tensor layout and the intermediate tensor layout, a tensor layout of an input tensor of the first reshape operator, and a tensor layout of an output tensor of the first reshape operator, (ZHANG [0281] teaches: “select each inserted glue operator (i.e., determining a first reshape operator) by using the target splitting path of the target operator in the calculation graph including the glue operator, and when a case that in the same glue operator included in the target splitting path (i.e., based on […] the intermediate tensor layout), the split states of the input tensor data (i.e., based on the first tensor layout) are the same as the split states of the output tensor data is satisfied, delete a corresponding inserted glue operator.” ZHANG [0189] teaches: “For any one split state of the input tensor data (i.e., determining […] a tensor layout of an input tensor of the first reshape operator), there is a directed edge directing to all split states of the output tensor data (i.e., determining […] a tensor layout of an output tensor of the first reshape operator).” ZHANG [0185] teaches: “The directed edge between the state in the split state set of the input tensor data of the operator and the state in the split state set of the output tensor represents the splitting method of the operator itself […].”)
wherein the first reshape operator is used to convert the first tensor layout to the intermediate tensor layout; (ZHANG [0189] teaches: “inserting the glue operator (i.e., the first reshape operator) between the operator Op0 and the Tensor1 input may convert any one split state (i.e., is used to convert the first tensor layout) of the tensor data into another split state (i.e., to the intermediate tensor layout).”)
and/or determining a second reshape operator based on the second tensor layout and the intermediate tensor layout, a tensor layout of an input tensor of the second reshape operator, and a tensor layout of an output tensor of the second reshape operator, (ZHANG [0281] teaches: “select each inserted glue operator (i.e., determining a second reshape operator) by using the target splitting path of the target operator in the calculation graph including the glue operator, and when a case that in the same glue operator included in the target splitting path (i.e., based on […] the intermediate tensor layout), the split states of the input tensor data (i.e., based on the second tensor layout) are the same as the split states of the output tensor data is satisfied, delete a corresponding inserted glue operator.” ZHANG [0189] teaches: “For any one split state of the input tensor data (i.e., determining […] a tensor layout of an input tensor of the second reshape operator), there is a directed edge directing to all split states of the output tensor data (i.e., determining […] a tensor layout of an output tensor of the second reshape operator).” ZHANG [0185] teaches: “The directed edge between the state in the split state set of the input tensor data of the operator and the state in the split state set of the output tensor represents the splitting method of the operator itself […].”)
wherein the second reshape operator is located between the first reshape operator and the second operator, and the second reshape operator is used to convert the intermediate tensor layout to the second tensor layout. (ZHANG [0189] teaches: “For any one split state of the input tensor data, there is a directed edge directing to all split states of the output tensor data (i.e., the second reshape operator is used to convert the intermediate tensor layout to the second tensor layout).” ZHANG [0281] teaches: “select each inserted glue operator by using the target splitting path of the target operator in the calculation graph including the glue operator, and when a case that in the same glue operator included in the target splitting path, the split states of the input tensor data are the same as the split states of the output tensor data is satisfied, delete a corresponding inserted glue operator.” ZHANG [0190] teaches: “the glue operator may be inserted between the operator and the corresponding input tensor data, and the glue operator may be inserted between the operator and the corresponding output tensor data, and even the glue operator may be inserted both between the operator and the corresponding input tensor data and between the operator and the corresponding output tensor data (i.e., the second reshape operator is located between the first reshape operator and the second operator).”)
Regarding Claim 4:
SHAZEER in view of ZHANG teaches the elements of claim 1 as outlined above. SHAZEER further teaches:
wherein a device matrix of the first tensor layout is same as a device matrix of the second tensor layout, (SHAZEER [0047] teaches: “FIG. 2B shows a tensor laid-out on the mesh 220 of four computing devices A-D with the tensor sliced along tensor-dimension 0 and the slices laid-out over mesh-dimension 0.” SHAZEER [0048] teaches: “FIG. 2C shows a tensor laid-out on the mesh 220 of four computing devices A-D (i.e., a device matrix of the first tensor layout is same as a device matrix of the second tensor layout) with the tensor sliced along tensor-dimension 0 and the slices laid-out over mesh-dimension 0.”)
a tensor map of the first tensor layout is different than a tensor map of the second tensor layout with a tensor map of the second tensor layout; and (SHAZEER [0047] teaches: “FIG. 2B shows a tensor laid-out on the mesh 220 of four computing devices A-D with the tensor sliced along tensor-dimension 0 and the slices laid-out over mesh-dimension 0. The tensor is not sliced and laid-out across mesh-dimension 1. The two-tuple representing this layout is <1, Ø> (i.e., a tensor map of the first tensor layout).” SHAZEER [0048] teaches: “FIG. 2C shows a tensor laid-out on the mesh 220 of four computing devices A-D with the tensor sliced along tensor-dimension 0 and the slices laid-out over mesh-dimension 0. The tensor is also sliced along tensor-dimension 1 with the slices laid-out over mesh-dimension 1. The layout shown in FIG. 2C is represented as <0,1> (i.e., is different than a tensor map of the second tensor layout with a tensor map of the second tensor layout).”)
ZHANG further teaches: a tensor shape of the first tensor layout is same as a tensor shape of the second tensor layout, and (ZHANG [0189] teaches: “For the glue operator, the input tensor data and the output tensor data have a same shape and a same state space.”)
the method further comprises: determining a map conversion operator, (ZHANG [0281] teaches: “select each inserted glue operator by using the target splitting path of the target operator in the calculation graph including the glue operator (i.e., determining a map conversion operator) […].”)
wherein the tensor map conversion operator comprises a slice operator, a concat operator, or a communication operator, (ZHANG [0178] teaches: “semantics of the glue operator may be obtained through a concat operator and/or a split operator in the neural network model.” ZHANG [0181] teaches: “In the embodiments of the present disclosure, a glue operator may use one of four implementation methods, which are splitting-concatenation, concatenation-splitting, concatenation, and splitting.” ZHANG [0282] teaches: “the glue operator is used to concatenate the split states in the split state set.”)
and the tensor map conversion operator is used to input the output tensor of the first operator and output the input tensor of the second operator. (ZHANG [0173] teaches: “If the operator OP0 is determined to be split in a certain way, as the output of the operator OP0, the Tensor1 is also determined to be split into a series of pieces of sub-tensor data in a certain way. Therefore, when the operator OP1 selects the splitting method, it must be ensured that a selected method is compatible with a determined splitting method of the input tensor data Tensor1 […].” ZHANG [0190] teaches: “It is required to be noted that FIG. 6E or FIG. 6F illustrates that the glue operator may be inserted between the operator and the corresponding input tensor data, and the glue operator may be inserted between the operator and the corresponding output tensor data, and even the glue operator may be inserted both between the operator and the corresponding input tensor data and between the operator and the corresponding output tensor data.” ZHANG [0279] teaches: “the glue operator (i.e., the tensor map conversion operator) is used to convert the split state obtained by splitting the tensor data according to one splitting method into the split state obtained by splitting the tensor data according to another splitting method.” Examiner’s note: ZHANG’s glue operator is inserted between the target operator and the split state associated with the target operator. The output of OP0 (i.e., first operator) is supplied as input to the target operator OP1 (i.e., the second operator) by the glue operator (i.e., is used to input the output tensor of the first operator and output the input tensor of the second operator).)
Regarding Claim 5:
SHAZEER in view of ZHANG teaches the elements of claim 1 as outlined above. SHAZEER further teaches:
wherein the obtaining a deep neural network model, a tensor layout of an input tensor of at least one operator in the deep neural network model, and a tensor layout of an output tensor of the at least one operator in the deep neural network model comprises: (SHAZEER [0071] teaches: “For example, the computation system 100 can receive a computation graph representing a neural network (i.e., obtaining a deep neural network model) having: an input layer, a hidden layer with nonlinear activation function A, and an output layer.” SHAZEER [0069] teaches: “A laid-out computation graph is a computation graph representing operations (i.e., of at least one operator) that receive input and generate outputs as laid-out tensors (i.e., obtaining […] a tensor layout of an input tensor […] in the deep neural network model […] and a tensor layout of an output tensor […] in the deep neural network model). The computation system 100 can generate a laid-out computation graph from a computation graph by assigning a layout to each input and output tensor in the graph. In this specification, operations for a laid-out computation graph are called laid-out operations, and the layouts for respective input and output tensors for a laid-out operation are called operation layouts.”)
obtaining the deep neural network model and […] (SHAZEER [0071] teaches: “For example, the computation system 100 can receive a computation graph representing a neural network (i.e., obtaining a deep neural network model) […].”)
obtaining […] a sharding strategy, (SHAZEER [0084] teaches: “The computation system receives specification data that specifies (i.e., obtaining) a distribution of tensor computations among a plurality of computing devices (i.e., a sharding strategy) (step 310).”)
wherein the sharding strategy comprises a quantity of slices of a tensor of the deep neural network model in at least one dimension; and (SHAZEER [0035] teaches: “The number of slices a tensor is sliced into depends on the number of computing devices in the mesh-dimension the tensor is being laid-out over.”)
determining the tensor layout of the input tensor of at least one operator in the deep neural network model and the tensor layout of the output tensor of the at least one operator in the deep neural network model based on the deep neural network model and the sharding strategy. (SHAZEER [0028] teaches: “As described below, the layout engine 120 can generate a layout according to the layout parameter values 160 provided (i.e., and the sharding strategy) to the computation system 100.” SHAZEER [0069] teaches: “The computation system 100 can generate a laid-out computation graph from a computation graph by assigning a layout to each input and output tensor in the graph (i.e., determining the tensor layout of the input tensor of at least one operator in the deep neural network model and the tensor layout of the output tensor of the at least one operator in the deep neural network model based on the deep neural network model). In this specification, operations for a laid-out computation graph are called laid-out operations, and the layouts for respective input and output tensors for a laid-out operation are called operation layouts.”)
Regarding Claim 7:
SHAZEER in view of ZHANG teaches the elements of claim 5 as outlined above. SHAZEER further teaches:
wherein the sharding strategy is a sharding strategy specified by a user. (SHAZEER [0053] teaches: “The user, through a user interface displayed on the user device, can modify values of the n-tuple, which the computation system 100 can later receive and implement. The computation system 100 can be configured to first identify whether the layout as modified by the user is legal or illegal, and notify the user if the user is attempting to provide the computation system with an illegal layout.”)
Regarding Claim 8:
The claim recites similar limitations as corresponding claim 1 and is rejected for similar reasons as claim 1 using similar teachings and rationale. SHAZEER further teaches:
A data processing device, comprising: a processor; and a memory coupled to the processor to store program instructions, which when executed by the processor, cause the processor to: (SHAZEER [0005] teaches: “Each computing device includes at least one processor and memory for storing tensors. The computing devices can include processor cores, processors, microprocessors, special-purpose logic circuitry, e.g., an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit), or any other appropriate computing devices. In some examples, the computing devices are all the same type of computing device. In other examples, the computing devices can include different types of processing units. For example, one device can include one or more CPUs (central processing units) while other devices can include one or more GPUs (graphics processing units) or TPUs (tensor processing units).” SHAZEER [0093] teaches: “Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.”)
Regarding Claim 9:
SHAZEER in view of ZHANG teaches the elements of claim 8 as outlined above. Additionally, the claim recites similar limitations as corresponding claim 2 and is rejected for similar reasons as claim 2 using similar teachings and rationale.
Regarding Claim 11:
SHAZEER in view of ZHANG teaches the elements of claim 8 as outlined above. Additionally, the claim recites similar limitations as corresponding claim 4 and is rejected for similar reasons as claim 4 using similar teachings and rationale.
Regarding Claim 12:
SHAZEER in view of ZHANG teaches the elements of claim 8 as outlined above. Additionally, the claim recites similar limitations as corresponding claim 5 and is rejected for similar reasons as claim 5 using similar teachings and rationale.
Regarding Claim 14:
SHAZEER in view of ZHANG teaches the elements of claim 12 as outlined above. Additionally, the claim recites similar limitations as corresponding claim 7 and is rejected for similar reasons as claim 7 using similar teachings and rationale.
Regarding Claim 15:
The claim recites similar limitations as corresponding claims 1 and 8 and is rejected for similar reasons as claims 1 and 8 using similar teachings and rationale. SHAZEER further teaches:
A non-transitory computer-readable storage medium, storing one or more instructions that, when executed by at least one processor, cause the at least one processor to: (SHAZEER [0005] teaches: “Each computing device includes at least one processor and memory for storing tensors.” SHAZEER [0094] teaches: “Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.”)
Regarding Claim 16:
SHAZEER in view of ZHANG teaches the elements of claim 15 as outlined above. Additionally, the claim recites similar limitations as corresponding claims 2 and 9 and is rejected for similar reasons as claims 2 and 9 using similar teachings and rationale.
Regarding Claim 18:
SHAZEER in view of ZHANG teaches the elements of claim 15 as outlined above. Additionally, the claim recites similar limitations as corresponding claims 4 and 11 and is rejected for similar reasons as claims 4 and 11 using similar teachings and rationale.
Regarding Claim 19:
SHAZEER in view of ZHANG teaches the elements of claim 15 as outlined above. Additionally, the claim recites similar limitations as corresponding claims 5 and 12 and is rejected for similar reasons as claims 5 and 12 using similar teachings and rationale.
Claims 3, 10, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over SHAZEER in view of ZHANG, as applied respectively above to claims 2, 9, and 16, and further in view of RAGHUNATHAN (US 20130080698 A1), hereafter RAGHUNATHAN.
Regarding Claim 3:
SHAZEER in view of ZHANG teaches the elements of claim 2 as outlined above. ZHANG further teaches:
the determining an intermediate tensor layout based on the first tensor layout and the second tensor layout comprises: (ZHANG [0186] teaches: “In FIG. 6E, split states in an updated split state set may include: a state m′1, a state m′2, and a state m′k. Here, the state m′1, the state m′2, and the state m′k are new states generated (i.e., determining an intermediate tensor layout) after states in a first split state set pass through glue operator.” ZHANG [0171] teaches: “In a step S402, the glue operator may be inserted between the target operator and the split state set associated with the target operator, and the split states in the split state set may be adjusted, and an adjusted split state set may be obtained, where the glue operator is used to convert the split states obtained by splitting the tensor data according to one splitting method into the split states obtained by splitting the tensor data according to another splitting method (i.e., based on the first tensor layout and the second tensor layout).”)
SHAZEER in view of ZHANG is not relied upon for teaching, but RAGHUNATHAN teaches: wherein the tensor shape of the first tensor layout is different than the tensor shape of the second tensor layout, (RAGHUNATHAN [0023] teaches: “The embodiment may include execution of a method with multiple steps: 1) provide two range-partitioned arrays A and B that have the same number of elements globally, but are of different shapes and distributions.”)
determining, based on the first tensor layout and the second tensor layout, a fifth equivalent tensor layout reshaped from the first tensor layout and a sixth equivalent tensor layout reshaped from the second tensor layout, (RAGHUNATHAN [0045] teaches: “The flatten operator for a distributed matrix A creates a unit block-partitioned one-dimensional vector (i.e., a fifth equivalent tensor layout reshaped from the first tensor layout) containing the elements of A stored in the same order as if A were not distributed at all.” RAGHUNATHAN [0046] teaches: “A distributed array B may thus be defined to be a reshaped version of another distributed array A if flat(A)=flat(B) (i.e., based on the first tensor layout and the second tensor layout, […] and a sixth equivalent tensor layout reshaped from the second tensor layout).” Examiner’s note: The flatten operator is applied to array A (i.e., first tensor layout) to produce (i.e., determining) the reshaped vector flat(A) (i.e., fifth equivalent tensor). Similarly, the flatten operator is applied to array B to produce reshaped vector flat(B) (i.e., sixth equivalent tensor).)
wherein a device matrix of the fifth equivalent tensor layout is same as a device matrix of the sixth equivalent tensor layout, and a tensor shape of the fifth equivalent tensor layout is same as a tensor shape of the sixth equivalent tensor layout; and (RAGHUNATHAN [0032] teaches: “A variable D may be used to represent the length of the distributed array A along the distributed dimension, d. The partitioning of the data in A essentially amounts to describing the partitioning of the span [0, D-1] between the P locales.” RAGHUNATHAN [0040] teaches: “One type of range-partitioned arrays are those that are considered "block partitioned". In block-partitioned arrays, the span [0, D-1] is first partitioned into a number of chunks with a minimum block size B. These chunks are in turn assigned in a regular fashion to the P locales such that each locale gets roughly equal number of chunks.” RAGHUNATHAN [0045] teaches: “The flatten operator for a distributed matrix A creates a unit block-partitioned one-dimensional vector containing the elements of A stored in the same order as if A were not distributed at all. […] The flatten operator is therefore independent of the distribution of A and only depends up to its dimensions.” RAGHUNATHAN [0046] teaches: “A distributed array B may thus be defined to be a reshaped version of another distributed array A if flat(A)=flat(B) (i.e., sixth equivalent tensor).” Examiner’s note: RAGHUNATHAN [0045] teaches that flattening creates a unit block-partitioned one-dimensional vector based on only its dimensions, independent of the source distribution. RAGHUNATHAN [0046] teaches flat(A)=flat(B). Thus, flat(A) and flat(B) have the same length and device layout, which under BRI is the same device matrix. Since they have vectors of the same length, they also have the same tensor shape.)
the intermediate tensor layout comprises the fifth equivalent tensor layout and the sixth equivalent tensor layout. (RAGHUNATHAN [0046] teaches: “A distributed array B may thus be defined to be a reshaped version of another distributed array A if flat(A)=flat(B).” Examiner’s note: Under BRI, intermediate tensor layout comprises the fifth equivalent tensor layout and the sixth equivalent tensor layout can be interpreted as flat(A) and flat(B).)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of SHAZEER, ZHANG, and RAGHUNATHAN before them, to include RAGHUNATHAN’s flatten operator as the intermediate step in the redistribution performed by SHAZEER and ZHANG’s method for distributing tensor computation across computing devices. One would have been motivated to make such a combination in order to efficiently send portions of distributed arrays between locales in a deadlock-free manner to avoid computationally expensive scheduling steps (RAGHUNATHAN [0022]).
Regarding Claim 10:
SHAZEER in view of ZHANG teaches the elements of claim 9 as outlined above. Additionally, the claim recites similar limitations as corresponding claim 3 and is rejected for similar reasons as claim 3 using similar teachings and rationale.
Regarding Claim 17:
SHAZEER in view of ZHANG teaches the elements of claim 16 as outlined above. Additionally, the claim recites similar limitations as corresponding claims 3 and 10 and is rejected for similar reasons as claims 3 and 10 using similar teachings and rationale.
Claims 6, 13, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over SHAZEER in view of ZHANG, as applied respectively above to claims 5, 12, and 19, and further in view of GHOLAMI ("Integrated Model, Batch, and Domain Parallelism in Training Neural Networks"), hereafter GHOLAMI.
Regarding Claim 6:
SHAZEER in view of ZHANG teaches the elements of claim 5 as outlined above. SHAZEER in view of ZHANG is not relied upon for teaching, but GHOLAMI teaches:
wherein the sharding strategy comprises a first sharding strategy and a second sharding strategy; (GHOLAMI [page 6, section 3 Simulated Performance in Training ALEXNET] teaches: “At first, the same process grid is used for all layers of the network, which means that if
P
r
>
1
(i.e., a first sharding strategy) then some amount of model parallelism will be used even in convolutional layers. Then we considered the improved case where we force
P
r
=
1
,
P
c
=
P
(i.e., a second sharding strategy) for the convolutional layers and use varying
P
r
×
P
c
grids for the fully connected layers. We compute the communication time for a single iteration with various choices of the mini-batch size B, the number of processes, and the configuration of process grid
P
r
×
P
c
.” GHOLAMI [page 7, section 3 Simulated Performance in Training ALEXNET] teaches: “For example, on P = 512 processes, the best performance is observed with 16 × 32 process grid which results in 2.1× speed up in the overall runtime and 5.0× speedup in communication ( Figure 6-d).”)
the method further comprises: determining a first overall tensor layout corresponding to the first sharding strategy and a second overall tensor layout corresponding to the second sharding strategy, (GHOLAMI [page 5, section 2.3 Integrated Model and Batch Parallelism] teaches: “Conversely, data matrices are replicated
P
r
times and each process holds (
1
/
P
c
)th piece of
X
i
and
Y
i
(i.e., determining a first overall tensor layout corresponding to the first sharding strategy).” GHOLAMI [page 4, section 2.2 Communication Cost Analysis of Pure Batch, Pure Model, and Pure Domain-Parallel Approaches] teaches: “By contrast, in data parallelism, every process starts with the same parameters, which get updated by the same gradient. […] The parallel matrix multiplications in the batch parallel case are illustrated in Figure 2, where the input activations
X
i
and the output activations
Y
i
are distributed 1D columnwise to processes (i.e., and a second overall tensor layout corresponding to the second sharding strategy).”)
wherein the first overall tensor layout comprises a tensor layout of an input tensor of at least one operator in the deep neural network model and a tensor layout of an output tensor of the at least one operator in the deep neural network model that are determined based on the first sharding strategy, (GHOLAMI [page 1, section 1 Introduction and Background] teaches: “The forward phase of DNN training is a sequential combination of affine transformation
Y
i
=
W
i
X
i
(i.e., at least one operator in the deep neural network model), followed by nonlinear transforms
X
i
+
1
=
f
Y
i
. Each column of
X
i
∈
R
d
i
-
1
×
B
holds input activations for one sample (i.e., an input tensor of at least one operator in the deep neural network model) and similarly each column of
Y
i
∈
R
d
i
×
B
holds output activations for one sample (i.e., an output tensor of the at least one operator in the deep neural network model).” GHOLAMI [page 5, section 2.3 Integrated Model and Batch Parallelism] teaches: “We think of our process grid logically partitioned as
P
=
P
r
×
P
c
. Each process holds (
1
/
P
r
)th piece of
W
i
, effectively replicating
W
i
matrix
P
c
times (as opposed to
P
times in batch parallelism). Conversely, data matrices are replicated
P
r
times and each process holds (
1
/
P
c
)th piece of
X
i
and
Y
i
(i.e., the first overall tensor layout comprises a tensor layout of an input tensor […] and a tensor layout of an output tensor […] that are determined based on the first sharding strategy).”)
and the second overall tensor layout comprises a tensor layout of an input tensor of at least one operator in the deep neural network model and a tensor layout of an output tensor of the at least one operator in the deep neural network model that are determined based on the second sharding strategy; and (GHOLAMI [page 4, section 2.2 Communication Cost Analysis of Pure Batch, Pure Model, and Pure Domain-Parallel Approaches] teaches: “The parallel matrix multiplications in the batch parallel case are illustrated in Figure 2, where the input activations
X
i
(i.e., a tensor layout of an input tensor of at least one operator in the deep neural network model) and the output activations
Y
i
(i.e., and a tensor layout of an output tensor of the at least one operator in the deep neural network model) are distributed 1D columnwise to processes (i.e., that are determined based on the second sharding strategy).”)
determining, from the first overall tensor layout and the second overall tensor layout, that the first overall tensor layout comprises the tensor layout of the input tensor of at least one operator in the deep neural network model and the tensor layout of the output tensor of the at least one operator in the deep neural network model, (GHOLAMI [page 5, section 2.3 Integrated Model and Batch Parallelism] teaches: “Furthermore, this algorithm automatically selects the best configuration to distribute the model and batch parallel work given a fixed batch size on
P
processes (i.e., determining, from the first overall tensor layout and the second overall tensor layout, that the first overall tensor layout comprises the tensor layout of the input tensor […] and the tensor layout of the output tensor […]).”)
wherein a sum of communication time and computation time required for training the deep neural network model based on the first overall tensor layout is less than a sum of communication time and computation time required for training the deep neural network model based on the second overall tensor layout. (GHOLAMI [page 6, section 3 Simulated Performance in Training ALEXNET] teaches: “We compute the communication time for a single iteration with various choices of the mini-batch size B, the number of processes, and the configuration of process grid
P
r
×
P
c
. Using this data, we then compute the communication time for a complete epoch by multiplying the communication time form Eq. 4 by N/B. […] Furthermore, we also consider the computational time by empirically measuring the time needed for an SGD iteration for AlexNet on a single KNL using Intel Caffe as shown in Fig. 4. We use this data for cases with the same computational workload to compute the total run time (i.e., a sum of communication time and computation time required for training the deep neural network model).” GHOLAMI [page 7, Figure 6] teaches: “The speedup for the total time compared to pure batch parallel is shown in bold text on top of the best bar chart.” GHOLAMI [page 7, section 3 Simulated Performance in Training ALEXNET] teaches: “For example, on P = 512 processes, the best performance is observed with 16 × 32 process grid which results in 2.1× speed up in the overall runtime (i.e., based on the first overall tensor layout is less than a sum of communication time and computation time required for training the deep neural network model based on the second overall tensor layout) and 5.0× speedup in communication (Figure 6-d).”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of SHAZEER, ZHANG, and GHOLAMI before them, to include GHOLAMI’s step of measuring total run time of process grid configurations and keeping the faster one in SHAZEER and ZHANG’s method for distributing tensor computation across computing devices. One would have been motivated to make such a combination in order to allow optimal selection of per process batch size and model size, resulting in better throughput as compared to pure batch/model parallel algorithms (GHOLAMI [page 10, section 5 Conclusion]).
Regarding Claim 13:
SHAZEER in view of ZHANG teaches the elements of claim 12 as outlined above. Additionally, the claim recites similar limitations as corresponding claim 6 and is rejected for similar reasons as claim 6 using similar teachings and rationale.
Regarding Claim 20:
SHAZEER in view of ZHANG teaches the elements of claim 19 as outlined above. Additionally, the claim recites similar limitations as corresponding claims 6 and 13 and is rejected for similar reasons as claims 6 and 13 using similar teachings and rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
WANG (“Supporting Very Large Models using Automatic Dataflow Graph Partitioning”) relates to inserting split, copy, and concat operators between operators with different partitions.
SHAZEER (“Mesh-TensorFlow: Deep Learning for Supercomputers”) relates to reshaping a tensor when its output layout differs from its input layout.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Alvaro S Laham Bauzo whose telephone number is (571)272-5650. The examiner can normally be reached Mon-Fri 7:30 AM - 11:00 AM | 1:00 PM - 5:30 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached on (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/A.S.L./Examiner, Art Unit 2146
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146
Prosecution Insights