Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 11/19/2025 has been entered.
Response to Arguments
Applicant's arguments filed 10/23/2025 have been fully considered but they are not persuasive.
Applicant argues: Cited references do not teach the claimed invention.
Examiner response: Examiner respectfully disagrees. Regarding the limitation “wherein the output of first portion of the training operation remain in the memory system while transferring the output of the first portion of the training operation from the first memory device to the second memory device”, Tzourfras teaches a memory system as shown in figure 1. The memory system of figure 1 comprises a host (Tzourfras para [0066] “In some implementations, the second set of iterations produces final values of the ANN parameters (e.g., values of weight and biases after the ANN has been fully trained), which are exported to a host computer via host interface 106.”), chips (i.e., memory devices) (para [0066] “Data is cached from RAM chips 100 as needed by SRAM buffer(s)/cache(s) 108 so that it is available to the processor 112.”), and processors which the Examiner interprets as a controller (Tzourfras para [0137] “In addition, in some embodiments, some or all of the above-described functions may be implemented with hardware circuits (e.g., field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), a “system on a chip” that includes processors and memory, or the like). To that end, in some embodiments, CPUs 602 include specialized hardware for performing these and other tasks.”). Data resulting from training is read from a first device to a second device however, the data remains on the memory system of figure 1 (Tzourfras para [0098] “In some implementations, each of the second set of training iterations includes (432) reading values for the set of weights and/or biases (e.g., the intermediate values stored during the first set of training iterations) of the ANN from the second RAM of the type distinct from MRAM, and, using the one or more processors, performing the set of arithmetic operations to update the values for the set of weights and/or biases of the ANN. Each of the second set of training iterations further includes writing the updated set of weights and/or biases of the ANN to the second RAM of the type distinct from the MRAM… In some implementations, the intermediate stored values are stored in MRAM (e.g., on the chip) and the final stored values are stored in non-volatile off-chip memory.”).
Regarding the limitation “receive a select command from the controller to select the first memory device and a second memory device, wherein the select command is generated by the controller on the memory system”, Heaton discloses a runtime engine (i.e., controller) (col. 23 lines 18-21; “At operation 626, runtime engine 314 may control the internal memory device to store the first data at a first location. The first location can be indicated in the first instruction.”) which is coupled to a first memory device 301 and second memory device 301 as shown in figure 3. The runtime engine further generates commands for selecting a memory device to transfer data (col. 17 lines 39-49; col. 21 lines 47-63;).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 8-9 and 11-20 are rejected under 35 U.S.C. 103 as being unpatentable over Tzoufras et al. (US-20200193282-A1) in view of Lai et al. (US-20210011288-A1) and Mappouras et al. (US 20200042859 A1) and Heaton et al. (US-11561833-B1).
Regarding Claim 8,
Tzoufras teaches a system, comprising:
a host (para [0066] “In some implementations, the second set of iterations produces final values of the ANN parameters (e.g., values of weight and biases after the ANN has been fully trained), which are exported to a host computer via host interface 106.”); and
a memory system coupled to the host (fig.1 shows a number of chips (i.e., memory devices) coupled to a host which is a memory system.), wherein the memory system includes:
a controller (para [0137]); and
a number of memory devices each coupled to the controller by a bus (para [0066] “During a second set of iterations of the ANN training process, data used to train the ANN is stored in RAM chips 100. The data is bussed on and off the chip 102 through RAM interface 104.” para [0127] “and one or more communication buses 605 for interconnecting these components.”), wherein each of the number of memory devices are configured as part of a neural network and include a number of memory arrays and wherein the number of memory devices (para [0137]) are configured to:
execute training operation by performing a first portion of the training operation for a first layer within the neural network on a first memory device (para [0065] “FIG. 1 illustrates a schematic diagram of an electronic system that includes a chip 102. The system includes a first type of RAM (e.g., error-prone MRAM) and a second type of RAM (e.g., error-free RAM, which may comprise one or more external (e.g., off-chip) devices, such as RAM chips 100)…For example, during a first set of iterations of an ANN training process, data used to train the ANN is stored in MRAM 110 (e.g., all of the data needed to perform a first set of iterations of an ANN training process is stored in MRAM 110).” Chip 102 with first type of RAM (i.e., first memory device). Chips 100 with second type of RAM (i.e., second memory device).) wherein the first portion of the training operation comprises combining a first input or a first weight, or both, represented as one or more data values stored within the first memory device with another input or another weight, or both (para [0057] “In Equations (1a)-(1c) above, A.sub.i represents the activations, W.sub.i represents the weights, b.sub.i represents the biases. The variable X represents the input data. The parameter n represents the number of layers in the neural network. The last layer of activations, O, is referred to as the outputs. During training, the outputs are compared to known results to determine a loss function. The functions f.sub.i are activation functions, usually a simple non-linear function such as ReLU: f(x)=max(0,x).” The weights and inputs are combined through mathematical functions. The neural network has at least a first layer.), represented as other data stored within the first memory device or received from another memory device (para [0065] “For example, during a first set of iterations of an ANN training process, data used to train the ANN is stored in MRAM 110 (e.g., all of the data needed to perform a first set of iterations of an ANN training process is stored in MRAM 110).”);
receive a first command from the controller at the first memory device and the second memory device and execute the first command to transfer an output of the first portion of the training operation from the first memory device to the second memory device on the bus, wherein the output of first portion of the training operation remain in the memory system while transferring the output of the first portion of the training operation from the first memory device to the second memory device (para [0065] “The chip 102 includes SRAM buffer(s)/cache(s) 108 for buffering data to be used by processors 112 during an ANN training process. For example, SRAM buffer(s)/cache(s) 108 buffers data stored off-chip (e.g., in external RAM 100) and/or data stored in MRAM 110 (e.g., error-prone memory).” Data stored in the MRAM on chip 102 (i.e., first memory device) can be transferred RAM chips 100 (i.e., second memory device) as needed. para [0066] “During a second set of iterations of the ANN training process, data used to train the ANN is stored in RAM chips 100. The data is bussed on and off the chip 102 through RAM interface 104.” Data bussed from chips 102 (i.e. first memory device) to chips 100 (i.e., second memory device). para [0098] “In some implementations, each of the second set of training iterations includes (432) reading values for the set of weights and/or biases (e.g., the intermediate values stored during the first set of training iterations) of the ANN from the second RAM of the type distinct from MRAM, and, using the one or more processors, performing the set of arithmetic operations to update the values for the set of weights and/or biases of the ANN. Each of the second set of training iterations further includes writing the updated set of weights and/or biases of the ANN to the second RAM of the type distinct from the MRAM… In some implementations, the intermediate stored values are stored in MRAM (e.g., on the chip) and the final stored values are stored in non-volatile off-chip memory.” The intermediate values (i.e., output of first portion of the training operation) resulting from the first training iterations (i.e., data that is based at least in part on the inputs or weights) are stored on first RAM (i.e. first memory device. The stored data is bussed/transferred to and from the RAM chips 100 (i.e., second memory device).);
receive and execute a second command to store the output of the first portion of the training operation in the second memory device represented as one or more data values (para [0066] and para [0101] “The instructions include instructions for training an artificial neural network (ANN) using first data by performing a first set of training iterations using the first RAM comprising MRAM, training the ANN using the first data by performing a second set of training iterations using the second RAM comprising a type distinct from MRAM, and storing values for the trained ANN.” The intermediate values (i.e. output of first portion of the training operation) resulting from the first training iteration on the MRAM (i.e., first memory device) is read and stored by the second RAM (i.e. second memory device). para [0098] “In some implementations, each of the second set of training iterations includes (432) reading values for the set of weights and/or biases (e.g., the intermediate values stored during the first set of training iterations) of the ANN from the second RAM of the type distinct from MRAM, and, using the one or more processors, performing the set of arithmetic operations to update the values for the set of weights and/or biases of the ANN.” The processor reads the values stored by the MRAM (i.e., first memory device) cached in the SRAM as discussed in para [0065]. para [0065] “In some implementations, the chip 102 includes a RAM interface 104 (e.g., a DDR3 interface) that facilitates communication between the chip 102 and the external RAM chips 100. The chip 102 includes SRAM buffer(s)/cache(s) 108 for buffering data to be used by processors 112 during an ANN training process.” The chip 102 containing the MRAM (i.e., first memory device) communicates with the RAM chips 100 (i.e., second memory device) by buffering data such as weights and biases in SRAM cache 102 resulting from the first training iteration.); and
execute the training operation by performing a second portion of the training operation for the first layer within the neural network on the second memory device (para [0066] “During a second set of iterations of the ANN training process, data used to train the ANN is stored in RAM chips 100. The data is bussed on and off the chip 102 through RAM interface 104. Data is cached from RAM chips 100 as needed by SRAM buffer(s)/cache(s) 108 so that it is available to the processor 112.” And para [0098]) wherein the second portion of the training operation for the first layer comprises combining the output of the first portion of the training operation as an input of the second portion of the training operation with an additional input or an additional weight, or both (para [0057] inputs are combined with weights through mathematical functions.), represented as additional data stored within the second memory device, (para [0101] “The instructions include instructions for training an artificial neural network (ANN) using first data by performing a first set of training iterations using the first RAM comprising MRAM, training the ANN using the first data by performing a second set of training iterations using the second RAM comprising a type distinct from MRAM, and storing values for the trained ANN.” The first data (i.e. additional input) is received, processed, and stored on the second RAM (i.e. second memory device). para [0102] “In some embodiments, the electronic system includes an off-chip memory (e.g., DRAM, HBM, RAM chips 100) that holds some or all of the first data during the first set of training iterations and/or the second set of training iterations (e.g., the first data is bussed on and off the chip as needed during the first and second sets of iterations).”),
Tzoufras does not explicitly disclose
execute training operation by performing a first portion of the training operation for a first layer within the neural network at a first artificial intelligence (AI) accelerator on a first memory device
receive a select command from the controller to select the first memory device and a second memory device, wherein the select command is generated by the controller on the memory system;
execute the training operation by performing a second portion of the training operation for the first layer within the neural network at a second artificial intelligence (AI) accelerator on the second memory device
wherein together the input and the weight of the first portion of the training operation, the additional input, and the additional weight for the second portion of the training operation for the first layer are too large to be stored on only the first memory device or only the second memory device.
However, Lai (US 20210011288 A1)
execute training operation by performing a first portion of the training operation (para [0045] “The input data 110 can include any type or form of data for configuring, tuning, training and/or activating a neural network 114 of the AI accelerator(s) 108, and/or for processing by the processor(s) 124.”) for a first layer within the neural network at a first artificial intelligence (AI) accelerator on a first memory device (para [0115] “Referring now to FIG. 3A, a system 300 includes a first device 302a, and a second device 302b, according to some embodiments. In some embodiments, the first device 302a is the HMD and/or AR system described in greater detail above with reference to FIGS. 2A-2B.” first device (i.e., first memory device). para [0118] ASIC/FPGA are AI accelerators on the device. para [0119] memory 308 can be a RAM.)
receive and execute a first command to transfer an output of the first portion of the training operation from the first memory device to a second memory device (para [0116] “The first device 302a receives the input data 110 and outputs a reduced data set 310, according to some embodiments. The second device 302b can receive the reduced data set 310 as an input and output or provide the output data 112 to the first device 302a. In some embodiments, the first device 302a outputs the reduced data set 310 to the second device 302b if the first device 302a cannot accurately make a determination based on the input data 110.” Second device (i.e., second memory device).);
receive and execute a second command to store the output of the first portion of the training operation in the second memory device represented as one or more data values (para [0057] “For example, the data can include input data, weight information and/or bias information, activation function information, and/or parameters 128 for one or more neurons (or nodes) and/or layers of the neural network(s) 114, which can be stored in and read or accessed from the storage device 126. The data can include output data from a neuron of the neural network(s) 114, which can be written to and stored at the storage device 126.” Also see para [0121] and para [0125] “In some embodiments, the first device 302a outputs the reduced data set 310 to the second device 302b for further analysis or processing in response to the identified or detected features of the input data 110 being insufficient for performing an action.”); and
execute the training operation by performing a second portion of the training operation for the first layer within the neural network at a second artificial intelligence (AI) accelerator on the second memory device (para [0115] “Referring now to FIG. 3A, a system 300 includes a first device 302a, and a second device 302b, according to some embodiments. In some embodiments, the first device 302a is the HMD and/or AR system described in greater detail above with reference to FIGS. 2A-2B.” second device (i.e., second memory device). para [0118] ASIC/FPGA are AI accelerators on the device. para [0119] memory 308 can be a RAM.)
Tzoufras and Lai are analogous because they are both directed towards transferring data for a neural network between at least two memory devices.
It would have been obvious to modify the neural network device of Tzoufras with the AI accelerators of Lai.
Doing so would allow for performing neural network operations on a first device when the first device can accurately perform the neural network operation or performing the neural network operation on a second device when the first neural network cannot accurately analyze the input data. This would advantageously reduce the amount of data transfers between the two devices (Lai para [0042]).
Mappouras (US 20200042859 A1) teaches
wherein together the input and the weight of the first portion of the training operation, the additional input, and the additional weight for the second portion of the training operation for the first layer are too large to be stored on only the first memory device or only the second memory device (para [0026] “In one implementation, a system includes at least a processing unit for implementing a neural network coupled to a first memory and a second memory. The first memory is a relatively low capacity, high bandwidth memory and the second memory is a relatively high capacity, low bandwidth memory. Typically, when implementing the neural network on the processing unit, the first memory does not have the capacity to store all of the activation buffers utilized by the different layers of the neural network.” Figure 3 further shows a weight buffer on line 3 and an activation buffer which is a combination of weights and input activations on line 5. para [0028] Activations and weights are being transferred between the two memories.)
Mappouras and Tzoufras are analogous because they are both directed towards transferring data for training a neural network between at least two memory devices.
It would have been obvious to modify the neural network of Tzoufras with the memory data transfer of Mappouras.
Doing so would allow for dynamically transferring data between two memories enables for a more efficient data management scheme as some buffers (i.e., activation buffers) allocate significantly more capacity than others (Mappouras para [0047]).
Heaton (US 20200193282 B1) teaches
receive a select command from the controller to select the first memory device and a second memory device, wherein the select command is generated by the controller on the memory system (col. 17 lines 39-49; col. 21 lines 47-63;);
Mappouras and Heaton are analogous because they are both directed towards transferring data for training a neural network.
It would have been obvious to modify the neural network of Tzoufras with the memory device selection of Heaton.
Doing so would allow for selecting memory devices based on memory capacity for processing of a neural network (Heaton col. 21 lines 47-63;).
Regarding Claim 9,
Tzoufras, Lai, Mappouras, and Heaton teach the system of claim 8.
Tzoufras further teaches wherein the memory devices are configured to execute a third portion of the training operation on the first memory device (para [0112] “To that end, in some embodiments, there are more than two sets of iterations, including a third set of iterations. Each of the third set of iterations includes writing values for the set of weights of the ANN to the MRAM using third write parameters corresponding to a third write error rate.”).
Regarding Claim 11,
Tzoufras, Lai, Mappouras, and Heaton teach the system of claim 8. While Tzoufras teaches transferring a result from the first memory device to the second memory device, Tzoufras does not explicitly disclose wherein the memory devices are configured to transfer an output of the second portion of the training operation from the second memory device to the first memory device.
However, Lai teaches
wherein the memory devices are configured to transfer an output of the second portion of the training operation from the second memory device to the first memory device (para [0133] “In some embodiments, if the accuracy a of the identified feature is less than the corresponding threshold value a.sub.threshold (or the error e of the identified feature is greater than the corresponding threshold value e.sub.threshold), the first device 302a provides the reduced data set 310 to the neural network 114b. In some embodiments, the first device 302a can then receive the outputs of the neural network 114b (i.e., the output data 112) and use the outputs of the neural network 114b to perform the action.” The output 310 is transferred from the first memory device to the second memory device. The second memory device then processes it and transfers output 112 it back to the first memory device as shown in figure 3A.).
Tzoufras, Mappouras, and Lai are analogous because they are both directed towards transferring data for a neural network between at least two memory devices.
It would have been obvious to modify the neural network device of Tzoufras with the memory transfer of Lai.
Doing so would allow for performing neural network operations on a first device when the first device can accurately perform the neural network operation or performing the neural network operation on a second device when the first neural network cannot accurately analyze the input data. This would advantageously reduce the amount of data transfers between the two devices (Lai para [0042]).
Regarding Claim 12,
Tzoufras, Lai, Mappouras, and Heaton teach the system of claim 11.
Tzoufras further teaches
wherein the memory devices are configured to execute a third portion of the training operation on the first memory device using the output of the second portion of the training operation as an input of the third portion of the training operation (para [0112] “To that end, in some embodiments, there are more than two sets of iterations, including a third set of iterations. Each of the third set of iterations includes writing values for the set of weights of the ANN to the MRAM using third write parameters corresponding to a third write error rate. The third write error rate is lower than the first write error rate and the second error rate.” As discussed above figure 1 and para [0065]-[0066] discloses how data can be transferred between the first type of RAM (e.g., MRAM) on chip 102 and the second type of RAM distinct from the first type on chip 100. As such, the MRAM (i.e., first memory device) can read updated weights/values (i.e., output of the second portion of the training operation) from RAM chips 100 (i.e. second memory device). para [0066] “During a second set of iterations of the ANN training process, data used to train the ANN is stored in RAM chips 100. The data is bussed on and off the chip 102 through RAM interface 104.”)
Regarding Claim 13,
Tzoufras, Lai, Mappouras, and Heaton teach the system of claim 8. Tzoufras further teaches wherein the memory devices are configured to transfer neural network data from the first memory device to the second memory device (para [0065] “The chip 102 includes SRAM buffer(s)/cache(s) 108 for buffering data to be used by processors 112 during an ANN training process. For example, SRAM buffer(s)/cache(s) 108 buffers data stored off-chip (e.g., in external RAM 100) and/or data stored in MRAM 110 (e.g., error-prone memory).” And para [0066] “During a second set of iterations of the ANN training process, data used to train the ANN is stored in RAM chips 100. The data is bussed on and off the chip 102 through RAM interface 104.” And para [0098] “In some implementations, each of the second set of training iterations includes (432) reading values for the set of weights and/or biases (e.g., the intermediate values stored during the first set of training iterations) of the ANN from the second RAM of the type distinct from MRAM, and, using the one or more processors, performing the set of arithmetic operations to update the values for the set of weights and/or biases of the ANN. Each of the second set of training iterations further includes writing the updated set of weights and/or biases of the ANN to the second RAM of the type distinct from the MRAM… In some implementations, the intermediate stored values are stored in MRAM (e.g., on the chip) and the final stored values are stored in non-volatile off-chip memory.”).
Regarding Claim 14,
Tzoufras, Lai, Mappouras, and Heaton teach the system of claim 8. Tzoufras further teaches wherein the memory devices are configured to transfer activation function data from the first memory device to the second memory device (para [0065] “Data is cached from MRAM 110 as needed by SRAM buffer(s)/cache(s) 108 so that it is available to the processor 112 (e.g., an arithmetic logic unit (ALU)), which performs the calculations necessary to train the ANN. In some implementations, the data includes values, such as weights, activations, and biases, of the ANN. In some implementations the data includes intermediate values (e.g., during the training of the ANN).” And para [0066] “During a second set of iterations of the ANN training process, data used to train the ANN is stored in RAM chips 100. The data is bussed on and off the chip 102 through RAM interface 104. Data is cached from RAM chips 100 as needed by SRAM buffer(s)/cache(s) 108 so that it is available to the processor 112.”).
Regarding Claim 15,
Tzoufras teaches a method, comprising:
executing a first portion of a training operation for a first layer within a neural network on a first memory device that is configured as part of the neural network (para [0065] “FIG. 1 illustrates a schematic diagram of an electronic system that includes a chip 102. The system includes a first type of RAM (e.g., error-prone MRAM) and a second type of RAM (e.g., error-free RAM, which may comprise one or more external (e.g., off-chip) devices, such as RAM chips 100)… For example, during a first set of iterations of an ANN training process, data used to train the ANN is stored in MRAM 110 (e.g., all of the data needed to perform a first set of iterations of an ANN training process is stored in MRAM 110).” Chip 102 with first type of RAM (i.e., first memory device). Chip 100 with second type of RAM (i.e., second memory device).), wherein the first portion of the training operation comprises combining a first input or a first weight, or both, represented as one or more data values stored within the first memory device with another input or another weight, or both (para [0057] In Equations (1a)-(1c) above, A.sub.i represents the activations, W.sub.i represents the weights, b.sub.i represents the biases. The variable X represents the input data. The parameter n represents the number of layers in the neural network. The last layer of activations, O, is referred to as the outputs. During training, the outputs are compared to known results to determine a loss function. The functions f.sub.i are activation functions, usually a simple non-linear function such as ReLU: f(x)=max(0,x). The weights and inputs are combined through mathematical functions. The neural network has at least a first layer.), represented as other data stored within the first memory device or received from another memory device (para [0065] For example, during a first set of iterations of an ANN training process, data used to train the ANN is stored in MRAM 110 (e.g., all of the data needed to perform a first set of iterations of an ANN training process is stored in MRAM 110).);
executing a first command to transfer, from the first memory device to a second memory device, data that is based at least in part on the inputs or weights combined at the first memory device, wherein the first memory device and the second memory device is on a memory system that includes a controller (fig.1;), wherein the memory system is coupled to a host (para [0066] “In some implementations, the second set of iterations produces final values of the ANN parameters (e.g., values of weight and biases after the ANN has been fully trained), which are exported to a host computer via host interface 106.”), wherein the first memory device and the second memory device are coupled to the controller by a bus, and wherein the first command is sent from the controller to the first memory device and the second memory device (para [0066] “The data is bussed on and off the chip 102 through RAM interface 104. Data is cached from RAM chips 100 as needed by SRAM buffer(s)/cache(s) 108 so that it is available to the processor 112.”), and wherein the data that is based at least in part on the inputs or weights combined at the first memory device remain in the memory system while being transferred from the first memory device to the second memory device (para [0065] “FIG. 1 illustrates a schematic diagram of an electronic system that includes a chip 102. The system includes a first type of RAM (e.g., error-prone MRAM) and a second type of RAM (e.g., error-free RAM, which may comprise one or more external (e.g., off-chip) devices, such as RAM chips 100) …The chip 102 includes SRAM buffer(s)/cache(s) 108 for buffering data to be used by processors 112 during an ANN training process. For example, SRAM buffer(s)/cache(s) 108 buffers data stored off-chip (e.g., in external RAM 100) and/or data stored in MRAM 110 (e.g., error-prone memory).” Data stored in the MRAM on chip 102 (i.e., first memory device) can be transferred RAM chips 100 (i.e., second memory device) as needed. para [0066] “During a second set of iterations of the ANN training process, data used to train the ANN is stored in RAM chips 100. The data is bussed on and off the chip 102 through RAM interface 104.”and para [0098] “In some implementations, each of the second set of training iterations includes (432) reading values for the set of weights and/or biases (e.g., the intermediate values stored during the first set of training iterations) of the ANN from the second RAM of the type distinct from MRAM, and, using the one or more processors, performing the set of arithmetic operations to update the values for the set of weights and/or biases of the ANN. Each of the second set of training iterations further includes writing the updated set of weights and/or biases of the ANN to the second RAM of the type distinct from the MRAM… In some implementations, the intermediate stored values are stored in MRAM (e.g., on the chip) and the final stored values are stored in non-volatile off-chip memory.” The intermediate values (i.e., output of first portion of the training operation) resulting from the first training iterations (i.e., data that is based at least in part on the inputs or weights) are stored on first RAM (i.e. first memory device. The stored data is transferred to the second RAM (i.e., second memory device).);
executing a second portion of the training operation for the first layer within the neural network on the second memory device using the data transferred from the first memory device to the second memory device (para [0066] “During a second set of iterations of the ANN training process, data used to train the ANN is stored in RAM chips 100. The data is bussed on and off the chip 102 through RAM interface 104. Data is cached from RAM chips 100 as needed by SRAM buffer(s)/cache(s) 108 so that it is available to the processor 112.” And para [0098]), wherein the second portion of the training operation for the first layer comprises combining a second input or a second weight, or both (para [0057] inputs are combined with weights through mathematical functions. Updated weight (i.e. second weight).), represented as one or more data values stored within the second memory device with an additional input or an additional weight, or both, represented as additional data stored within the second memory device or received from an additional memory device (para [0101] The instructions include instructions for training an artificial neural network (ANN) using first data by performing a first set of training iterations using the first RAM comprising MRAM, training the ANN using the first data by performing a second set of training iterations using the second RAM comprising a type distinct from MRAM, and storing values for the trained ANN. The first data (i.e. additional input) is received, processed, and stored on the second RAM (i.e. second memory device). para [0102] “In some embodiments, the electronic system includes an off-chip memory (e.g., DRAM, HBM, RAM chips 100) that holds some or all of the first data during the first set of training iterations and/or the second set of training iterations (e.g., the first data is bussed on and off the chip as needed during the first and second sets of iterations).”),
Tzoufras does not explicitly disclose
executing a first portion of a training operation for a first layer within a neural network at a first artificial intelligence (AI) accelerator on a first memory device that is configured as part of the neural network
wherein the controller coupled to the first memory device and a second memory device generates and sends a select command to select the first memory device and the second memory device,
executing a second portion of the training operation for the first layer within the neural network at a second artificial intelligence (AI) accelerator on the second memory device using the data transferred from the first memory device to the second memory device
wherein together the first input, the first weight, the second input, and the second weight for the second portion of the training operation for the first layer are too large to be stored on only the first memory device or only the second memory device.
However, Lai (US 20210011288 A1)
executing a first portion of a training operation for a first layer within a neural network operation (para [0045] “The input data 110 can include any type or form of data for configuring, tuning, training and/or activating a neural network 114 of the AI accelerator(s) 108, and/or for processing by the processor(s) 124.”) at a first artificial intelligence (AI) accelerator on a first memory device that is configured as part of the neural network (para [0115] “Referring now to FIG. 3A, a system 300 includes a first device 302a, and a second device 302b, according to some embodiments. In some embodiments, the first device 302a is the HMD and/or AR system described in greater detail above with reference to FIGS. 2A-2B.” first device (i.e., first memory device). para [0118] ASIC/FPGA are AI accelerators. para [0119] memory 308 can be a RAM.)
executing a first command to transfer, from the first memory device to a second memory device, data that is based at least in part on the inputs or weights combined at the first memory device (para [0116] “The first device 302a receives the input data 110 and outputs a reduced data set 310, according to some embodiments. The second device 302b can receive the reduced data set 310 as an input and output or provide the output data 112 to the first device 302a.” Second device (i.e., second memory device).);
receive and execute a second command to store the output of the first portion of the training operation in the second memory device represented as one or more data values (para [0057] “For example, the data can include input data, weight information and/or bias information, activation function information, and/or parameters 128 for one or more neurons (or nodes) and/or layers of the neural network(s) 114, which can be stored in and read or accessed from the storage device 126. The data can include output data from a neuron of the neural network(s) 114, which can be written to and stored at the storage device 126.” Also see para [0121] and para [0125] “In some embodiments, the first device 302a outputs the reduced data set 310 to the second device 302b for further analysis or processing in response to the identified or detected features of the input data 110 being insufficient for performing an action.”); and
executing a second portion of the training operation for the first layer within the neural network (para [0045] “The input data 110 can include any type or form of data for configuring, tuning, training and/or activating a neural network 114 of the AI accelerator(s) 108, and/or for processing by the processor(s) 124.”) at a second artificial intelligence (AI) accelerator on the second memory device using the data transferred from the first memory device to the second memory device (para [0115] “Referring now to FIG. 3A, a system 300 includes a first device 302a, and a second device 302b, according to some embodiments. In some embodiments, the first device 302a is the HMD and/or AR system described in greater detail above with reference to FIGS. 2A-2B.” second device (i.e., second memory device). para [0118] ASIC/FPGA are AI accelerators.)
Tzoufras and Lai are analogous because they are both directed towards transferring data for a neural network between at least two memory devices.
It would have been obvious to modify the neural network device of Tzoufras with the AI accelerators of Lai.
Doing so would allow for performing neural network operations on a first device when the first device can accurately perform the neural network operation or performing the neural network operation on a second device when the first neural network cannot accurately analyze the input data. This would advantageously reduce the amount of data transfers between the two devices (Lai para [0042]).
Mappouras (US 20200042859 A1) teaches
wherein together the first input, the first weight, the second input, and the second weight for the second portion of the training operation for the first layer are too large to be stored on only the first memory device or only the second memory device (para [0026] In one implementation, a system includes at least a processing unit for implementing a neural network coupled to a first memory and a second memory. The first memory is a relatively low capacity, high bandwidth memory and the second memory is a relatively high capacity, low bandwidth memory. Typically, when implementing the neural network on the processing unit, the first memory does not have the capacity to store all of the activation buffers utilized by the different layers of the neural network. Figure 3 further shows a weight buffer on line 3 and an activation buffer which is a combination of weights and input activations on line 5. para [0028] Activations and weights are being transferred between the two memories.)
Mappouras and Tzoufras are analogous because they are both directed towards transferring data for training a neural network between at least two memory devices.
It would have been obvious to modify the neural network of Tzoufras with the memory data transfer of Mappouras.
Doing so would allow for dynamically transferring data between two memories enables for a more efficient data management scheme as some buffers (i.e., activation buffers) allocate significantly more capacity than others (Mappouras para [0047]).
Heaton (US 11561833 B1) teaches
wherein the controller coupled to the first memory device and a second memory device generates and sends a select command to select the first memory device and the second memory device (col. 17 lines 39-49; col. 21 lines 47-63;);
Mappouras and Heaton are analogous because they are both directed towards transferring data for training a neural network.
It would have been obvious to modify the neural network of Tzoufras with the memory device selection of Heaton.
Doing so would allow for selecting memory devices based on memory capacity for processing of a neural network (Heaton col. 21 lines 47-63;).
Regarding Claim 16,
Tzoufras, Lai, Mappouras, and Heaton teach the method of claim 15. Tzoufras further teaches wherein transferring the data from the first memory device to the second memory device includes transferring an output of training operation (para [0065] The chip 102 includes SRAM buffer(s)/cache(s) 108 for buffering data to be used by processors 112 during an ANN training process. For example, SRAM buffer(s)/cache(s) 108 buffers data stored off-chip (e.g., in external RAM 100) and/or data stored in MRAM 110 (e.g., error-prone memory). And para [0066] “During a second set of iterations of the ANN training process, data used to train the ANN is stored in RAM chips 100. The data is bussed on and off the chip 102 through RAM interface 104.”And para [0098] In some implementations, each of the second set of training iterations includes (432) reading values for the set of weights and/or biases (e.g., the intermediate values stored during the first set of training iterations) of the ANN from the second RAM of the type distinct from MRAM, and, using the one or more processors, performing the set of arithmetic operations to update the values for the set of weights and/or biases of the ANN. Each of the second set of training iterations further includes writing the updated set of weights and/or biases of the ANN to the second RAM of the type distinct from the MRAM… In some implementations, the intermediate stored values are stored in MRAM (e.g., on the chip) and the final stored values are stored in non-volatile off-chip memory.).
Regarding Claim 17,
Tzoufras, Lai, Mappouras, and Heaton teach the method of claim 15.
Tzoufras further teaches wherein executing the second portion of the training operation includes using the data transferred from the first memory device to the second memory device as an input for the second portion of the training operation (para [0065] “Data is cached from MRAM 110 as needed by SRAM buffer(s)/cache(s) 108 so that it is available to the processor 112 (e.g., an arithmetic logic unit (ALU)), which performs the calculations necessary to train the ANN. In some implementations, the data includes values, such as weights, activations, and biases, of the ANN. In some implementations the data includes intermediate values (e.g., during the training of the ANN).” Data values including weights, activations, and biases from the MRAM is used as input data for the second iteration of the training on chips 100 (i.e., second memory device). para [0066] “During a second set of iterations of the ANN training process, data used to train the ANN is stored in RAM chips 100. The data is bussed on and off the chip 102 through RAM interface 104.”).
Regarding Claim 18,
Tzoufras, Lai, Mappouras, and Heaton teach the method of claim 15. Tzoufras further including transferring an output of the second portion of the training operation to a controller (para [0066] “During a second set of iterations of the ANN training process, data used to train the ANN is stored in RAM chips 100. The data is bussed on and off the chip 102 through RAM interface 104. Data is cached from RAM chips 100 as needed by SRAM buffer(s)/cache(s) 108 so that it is available to the processor 112.” Processor (i.e., controller).).
Regarding Claim 19,
Tzoufras, Lai, Mappouras, and Heaton teach the method of claim 15. Tzoufras further teaches wherein transferring data from the first memory device to the second memory device includes transferring neural network data for the training (para [0065] The chip 102 includes SRAM buffer(s)/cache(s) 108 for buffering data to be used by processors 112 during an ANN training process. For example, SRAM buffer(s)/cache(s) 108 buffers data stored off-chip (e.g., in external RAM 100) and/or data stored in MRAM 110 (e.g., error-prone memory).) And para [0066] “During a second set of iterations of the ANN training process, data used to train the ANN is stored in RAM chips 100. The data is bussed on and off the chip 102 through RAM interface 104.” And para [0098] In some implementations, each of the second set of training iterations includes (432) reading values for the set of weights and/or biases (e.g., the intermediate values stored during the first set of training iterations) of the ANN from the second RAM of the type distinct from MRAM, and, using the one or more processors, performing the set of arithmetic operations to update the values for the set of weights and/or biases of the ANN. Each of the second set of training iterations further includes writing the updated set of weights and/or biases of the ANN to the second RAM of the type distinct from the MRAM… In some implementations, the intermediate stored values are stored in MRAM (e.g., on the chip) and the final stored values are stored in non-volatile off-chip memory.).
Regarding Claim 20,
Tzoufras, Lai, and Mappouras teach the method claim 15. Tzoufras further teaches wherein transferring data from the first memory device to the second memory device includes transferring activation function data for the training operation (para [0065] “For example, during a first set of iterations of an ANN training process, data used to train the ANN is stored in MRAM 110 (e.g., all of the data needed to perform a first set of iterations of an ANN training process is stored in MRAM 110). Data is cached from MRAM 110 as needed by SRAM buffer(s)/cache(s) 108 so that it is available to the processor 112 (e.g., an arithmetic logic unit (ALU)), which performs the calculations necessary to train the ANN. In some implementations, the data includes values, such as weights, activations, and biases, of the ANN. In some implementations the data includes intermediate values (e.g., during the training of the ANN).” And para [0066] “During a second set of iterations of the ANN training process, data used to train the ANN is stored in RAM chips 100. The data is bussed on and off the chip 102 through RAM interface 104.”).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Tzoufras/Lai/Mappouras/Heaton, as applied above, and further in view of Bleiweiss et al. (US-20190205737-A1).
Regarding Claim 10,
Tzoufras, Lai, and Mappouras teach the system of claim 9.
Tzoufras, Lai, and Mappouras do not explicitly disclose
wherein the third portion of the training operation is executed while the second portion of the training operation is executed.
However, Bleiweiss (US 20190205737 A1) teaches
wherein the third portion of the training operation is executed while the second portion of the training operation is executed (para [0187] “In data parallelism 1904, the different nodes of the distributed network have a complete instance of the model and each node receives a different portion of the data. The results from the different nodes are then combined. While different approaches to data parallelism are possible, data parallel training approaches all require a technique of combining results and synchronizing the model parameters between each node. Exemplary approaches to combining data include parameter averaging and update based data parallelism. Parameter averaging trains each node on a subset of the training data and sets the global parameters (e.g., weights, biases) to the average of the parameters from each node.” There are multiple instances on nodes that perform training. This would include at least a third node (i.e., training portion). para [0188]-[0189]).
Tzoufras, Lai, Mappouras, and Bleiweiss are analogous because they are both directed towards training a neural network.
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Tzoufras, Lai, and Mappouras with the parallel training of Bleiweiss.
Doing so would allow for splitting computations associated with different layers of the neural network to enable the training of very large neural networks. model parallelism includes the ability to scale to large models (Bleiweiss para [0186]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HENRY K NGUYEN whose telephone number is (571)272-0217. The examiner can normally be reached Mon - Fri 7:00am-4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at 5712723768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HENRY NGUYEN/Examiner, Art Unit 2121