DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1-2, 7-17, and 19-20 have been amended. 35 U.S.C. 101 rejection for claims 1-16 and 19 have been withdrawn in light of the amendments. Claims 1-20 are currently pending.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1, 14, and 17 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Interpretation
Claims 1, 14, and 17 state the limitation “a wideband data connection”. Applicant’s Specification filed 09/26/2023, Paragraph [0074] states that a wideband data connection includes any data connection with a bandwidth of a predetermined threshold, such as PCIe. Thus, Examiner will interpret a wideband data connection as any high bandwidth data connection protocol.
Furthermore, claims 1, 14, and 17 state the limitation “persistent memory”. Applicant’s Specification filed 09/26/2023, Paragraph [0066] states that persistent memory includes any non-volatile memory. Thus, Examiner will interpret persistent memory as any non-volatile memory type.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7, 9-12, 14-15, 17-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Shah (US 2020/0167098) in view of Mathuriya (US 11,694,940).
Regarding claim 1, Shah teaches a system (Fig. 5, System; Paragraph 0019, IG. 5 depicts an example inference and storage system), comprising: a persistent storage device configured to perform an inference operation within the persistent storage device (Figs. 4 and 5, Persistent storage device within a single form factor that performs inference shown in same embodiment of both Figures; Paragraph 0018, FIG. 4 depicts an example inference and storage system… Paragraph 0041, processing system 504 can be housed in the same physical casing as that of media device), the persistent storage device comprising: a first persistent memory (Fig. 5, Media 550 includes persistent memory such as SSD; Paragraph 0046, media 550 includes SSDs 1-8); and a control and inference circuit (Fig. 5, Inference engine 510 controls inference models), wherein: the first persistent memory is connected to the control and inference circuit by a wideband data connection (Fig. 5, Media 550 (i.e. first persistent memory) is connected to inference engine 510 via PCIe interface 545 which is a wideband data connection; Paragraph 0041, Media interface 545 can be compatible with one or more of: DDR4, DDR5, PCIe… Paragraph 0094, High speed interconnects can be… Peripheral Component Interconnect express (PCIe)… Paragraph 0097, high-speed link 1018, such as a 40 Gigabit/second (Gb/s)); and the control and inference circuit is configured to perform arithmetic operations based on the inference operation (Fig. 5, Inference engine 510 performs inference operations using arithmetic on inference model weights; Paragraph 0020, use inference engine 510 to perform inference operations. Inference engine 510 can use one or more of: a decode and crop engine 512, compute engine 514… Paragraph 0066, an inference model (e.g., weights) and an address of the decoded data (to be processed using the inference model)… Paragraph 0023, output value of a preceding neuron can be multiplied by the weight of its connection).
Shah does not teach the system comprising: the first persistent memory on a first semiconductor die; and the control and inference circuit on a second semiconductor die, the first semiconductor die stacked on the second semiconductor die, wherein: the first persistent memory is connected to the control and inference circuit by a wideband data connection, configured as a high-bandwidth data connection between the stacked first semiconductor die and the second semiconductor die.
Mathuriya teaches the system comprising: the first persistent memory on a first semiconductor die (Fig. 3A, Memory die 307; Col. 6, Lines 31-35, memory die can be a DRAM, ferroelectric or paraelectric RAM (FeRAM), static random-access memory (SRAM), and other non-volatile memories such as flash, NAND, magnetic RAM (MRAM), Fe-SRAM, Fe-DRAM, and other resistive RAMs (Re-RAMs) etc); and the control and inference circuit on a second semiconductor die (Fig. 3A, Compute chiplet die 304 performs AI processing and inference; Col. 18, Lines 10-11, compute die 304 comprises logic portions of an inference die), the first semiconductor die stacked on the second semiconductor die (Fig. 3A, Memory die 307 is stacked on compute chiplet die 304; Col. 12, Lines 62-63, a memory die 307 is placed over compute chiplet 304), wherein: the first persistent memory is connected to the control and inference circuit by a wideband data connection, configured as a high-bandwidth data connection (Fig. 2, Memory die 1 is connected to compute chiplet die 2 via a wideband connection that is high-bandwidth; Col. 6, Lines 58-60, Ultra-high bandwidth between the memory and compute dies is achieved by tight micro-bump spacing between the two dies… Col. 11, Lines 45-47, memory die is a high bandwidth memory (HBM)) between the stacked first semiconductor die and the second semiconductor die (Fig. 2, Stacked memory die 1 and compute chiplet die 2 are used for AI training using high-bandwidth; Col. 12, Lines 28-30, Architecture 200 provides reduced memory accesses for the compute die (e.g., die 2) by providing data locality for weights, inputs, and outputs).
Shah and Mathuriya are analogous arts because they are in the same field of endeavor of creating integrated systems between random access memory, memory controllers, and persistent memory for inference and AI training.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shah’s system to incorporate the teachings of Mathuriya and include the persistent memory and random access memory as semiconductor chips within a stacked package with the control and inference chip of Shah.
One of ordinary skill in the art would be motivated to make the modifications in order to improve thermal characteristics, increase reliability, and reduce footprint in semiconductor systems (See Mathuriya: Col. 6, Lines 36-41 and Col. 12, Lines 35-39).
Regarding claim 2, Shah in view of Mathuriya teaches the system of claim 1. Shah teaches the system further comprising wherein: the inference operation is related to a neural network (Fig. 5, Inference engine 510 comprises compute engine 510 that uses neural network for inference; Paragraph 0023, Compute engine 514 can use a neural network to perform inferences or analysis); the control and inference circuit is configured to perform the arithmetic operations as a portion of the inference operation (Fig. 5, Inference engine 510 performs inference using arithmetic operations such as matrix multiplication; Paragraph 0026, 1. Matrix-Matrix Multiply), and comprises a first persistent memory controller (Figs. 5 and 8, State machine 516 is a microprocessor that controls storage operations for persistent media 550 and is synonymous with memory controller 822 of Figure 8; Paragraph 0020, State machine 516 can be implemented as a microprocessor… Paragraph 0066, state machine issues a Read command for an inference model (and optionally, weights) and decoded data from the media… Paragraph 0086, memory controller to generate and issue commands to memory 830); the system further comprises: an interface circuit (Fig. 8, Interface 812), a second persistent memory (Fig. 8, Storage 884 is a second persistent memory; Paragraph 0090, storage device(s) 884, which can be or include any conventional medium for storing large amounts of data in a nonvolatile manner, such as one or more magnetic, solid state), and a second persistent memory controller (Fig. 8, Controller 882; Paragraph 0090, storage subsystem 880 includes controller 882 to interface with storage 884); the second persistent memory is connected to the second persistent memory controller (Fig. 8, Controller 882 connected to storage 884); the control and inference circuit is connected to the interface circuit (Fig. 8, Memory subsystem 820 is the control and inference circuit and is coupled to interface 812); and the second persistent memory controller is connected to the interface circuit (Fig. 8, Controller 882 is coupled to interface 814 which further transfers data to interface 812 and thus are connected to each other; Paragraph 0088, system 800 includes interface 814, which can be coupled to interface 812).
Regarding claim 3, Shah in view of Mathuriya teaches the system of claim 1. Shah teaches the system further comprising: an interface circuit (Figs. 4 and 5, Memory board with inference in Figure 4 has interface 503 in Figure 5 which is serial), wherein: the interface circuit comprises a serial interface (Fig. 5, Interface 503 is PCIe or SATA which are both serial; Paragraph 0014, various specifications for M.2 include PCI Express M.2 Specification Revision 3.0, Version 1.2 (2013) and Serial ATA International Organization (SATA-IO) revision 3.2 specification); and the control and inference circuit is connected to the interface circuit (Fig. 5, Inference engine 510 coupled to interface 503).
Regarding claim 4, Shah in view of Mathuriya teaches the system of claim 1. Shah teaches the system further comprising: an interface circuit (Fig. 8, Interface 812); a second persistent memory (Fig. 8, Storage 884); and a second persistent memory controller (Fig. 8, Controller 882), wherein: the control and inference circuit comprises a first persistent memory controller (Fig. 8, Memory controller 822); the interface circuit comprises a serial interface (Fig. 8, Interface 812 is serial; Paragraph 0094, system 800 can be implemented using interconnected compute sleds of processors, memories, storages, network interfaces, and other components. High speed interconnects can be… Peripheral Component Interconnect express (PCIe)); the control and inference circuit is connected to the interface circuit (Fig. 8, Memory subsystem 820 coupled coupled to interface 812 which is the synonymous with Figure 5 where inference engine 510 coupled to interface 503); and the second persistent memory is connected to the interface circuit (Fig. 8, Storage 884 coupled to interface 812 via interface 814).
Regarding claim 5, Shah in view of Mathuriya teaches the system of claim 1. Shah teaches the system further comprising: a second persistent memory (Fig. 8, Storage 884), and a second persistent memory controller (Fig. 8, Controller 882), wherein: the control and inference circuit comprises a first persistent memory controller (Fig. 8, Memory controller 822), the first persistent memory comprises a flash memory (Fig. 8, Memory 830 is flash; Paragraph 0086, Memory subsystem 820 can include one or more memory devices 830 such as read-only memory (ROM), flash memory), the second persistent memory comprises a flash memory (Fig. 8, Storage 884 is flash NAND; Paragraph 0059, nonvolatile memory storage (e.g., flash memory or byte addressable non-volatile memory (e.g., Intel Optane® or Samsung Z-NAND®))… Paragraph 0090, storage device(s) 884, which can be or include any conventional medium for storing large amounts of data in a nonvolatile manner), the first persistent memory controller comprises a first flash memory controller (Fig. 8, Memory controller 822 controls flash memory 830 and thus is a flash memory controller), and the second persistent memory controller comprises a second flash memory controller (Fig. 8, Controller 882 controls flash memory 884 and thus is a flash memory controller).
Regarding claim 6, Shah in view of Mathuriya teaches the system of claim 1. Shah teaches the system comprising wherein the wideband data connection comprises a serial connection (Fig. 5, PCIe interface 545 which is a wideband data connection uses PCIe which is a serial protocol; Paragraph 0041, Media interface 545 can be compatible with one or more of: DDR4, DDR5, PCIe).
Regarding claim 7, Shah in view of Mathuriya teaches the system of claim 1. Shah teaches the system further comprising: a first interface circuit (Fig. 5, Interface 503), and an artificial intelligence accelerator (Fig. 5, Host 502 is an AI accelerator; Paragraph 0019, host 502 can include one or more processors (e.g., CPU, cores, GPU, accelerators… Paragraph 0047, Host 502 can issue a compute or storage command to processing system 504… Paragraph 0048, compute command can include or be associated with an inference model (e.g., model type) and include or reference weights stored in media 550), the artificial intelligence accelerator comprising an artificial intelligence processing circuit and a second interface circuit (Fig. 3, Host 302 is same as host 502 of Figure 5 and includes accelerator circuitry and host fabric interface (i.e. second interface circuit)), wherein: the first interface circuit is connected to the second interface circuit (Fig. 3, Host fabric interface coupled to interface of inference engine 304 with connection 300), the artificial intelligence accelerator is configured to perform additional inference operations of a neural network (Fig. 5, Inference engine 510 can perform more than one operation), with assistance from: the first persistent memory, and the control and inference circuit (Fig. 5, Host 502 transmits compute commands to perform inference of a neural network to inference engine 510 and media 550; Paragraph 0048, compute command can include or be associated with an inference model (e.g., model type) and include or reference weights stored in media 550… Paragraph 0085, Accelerators 842 can provide multiple neural networks, CPUs, processor cores, general purpose graphics processing units, or graphics processing units can be made available for use by artificial intelligence (AI) or machine learning (ML) models).
Regarding claim 9, Shah in view of Mathuriya teaches the system of claim 1. Shah teaches the system comprising wherein: the persistent storage device is configured to conform to a form factor, the first persistent memory is within the form factor; and the control and inference is within the form factor (Figs. 4 and 5, Persistent storage device within a single form factor that contains inference circuit 510 and memory 550 shown in same embodiment of both Figures; Paragraph 0018, FIG. 4 depicts an example inference and storage system… Paragraph 0041, processing system 504 can be housed in the same physical casing as that of media device 550 or a different physical casing than that around media device 550).
Mathuriya teaches the system comprising: the first persistent memory on the first semiconductor die (Fig. 3A, Memory die 307; Col. 6, Lines 31-35, memory die can be a DRAM, ferroelectric or paraelectric RAM (FeRAM), static random-access memory (SRAM), and other non-volatile memories such as flash, NAND, magnetic RAM (MRAM), Fe-SRAM, Fe-DRAM, and other resistive RAMs (Re-RAMs) etc); and the control and inference on the second semiconductor die (Fig. 3A, Compute chiplet die 304 performs AI processing and inference; Col. 18, Lines 10-11, compute die 304 comprises logic portions of an inference die).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shah’s system to incorporate the teachings of Mathuriya and include the persistent memory and random access memory as semiconductor chips within a stacked package form factor with the control and inference chip of Shah.
One of ordinary skill in the art would be motivated to make the modifications in order to improve thermal characteristics, increase reliability, and reduce footprint in semiconductor systems (See Mathuriya: Col. 6, Lines 36-41 and Col. 12, Lines 35-39).
Regarding claim 10, Shah in view of Mathuriya teaches the system of claim 1.
Mathuriya teaches the system comprising: the first semiconductor die and the second semiconductor die are part of a stack of dies (Fig. 3A, Stack of dies).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shah’s system to incorporate the teachings of Mathuriya and include the persistent memory and random access memory as semiconductor chips within a stacked package form factor with the control and inference chip of Shah.
One of ordinary skill in the art would be motivated to make the modifications in order to improve thermal characteristics, increase reliability, and reduce footprint in semiconductor systems (See Mathuriya: Col. 6, Lines 36-41 and Col. 12, Lines 35-39).
Regarding claim 11, Shah in view of Mathuriya teaches the system of claim 1. Shah teaches the system comprising a random-access memory (Fig. 5, Memory 518), wherein: the control and inference circuit is part of a second semiconductor die (Fig. 5, Inference engine 510 is an integrated circuit die; Paragraph 0040, Inference engine 510 can include application specific integrated circuits (ASICs), Intel processor(s), Advanced RISC Machines (ARM) compatible processor(s), CPUs, GPUs).
Mathuriya teaches the system comprising wherein: the random access memory is part of a third semiconductor die (Fig. 1, RAM 104/106 are part of a die; Col. 17, Lines 10-11, Two or more memory dies 707 (e.g., memory 104)); and the first semiconductor die, the second semiconductor die, and the third semiconductor die are part of a stack of dies (Fig. 7A, Compute die 304 and RAM die 701 and dies 701 includes persistent flash die; Col. 19, Lines 39-42, memory dies 703.sub.1-N can be embedded DRAM, SRAM, flash, Fe-RAM, MRAM, Fe-SRAM, Re-RAM, etc. or a combination of them).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shah’s system to incorporate the teachings of Mathuriya and include the persistent memory and random access memory as semiconductor chips within a stacked package form factor with the control and inference chip of Shah.
One of ordinary skill in the art would be motivated to make the modifications in order to improve thermal characteristics, increase reliability, and reduce footprint in semiconductor systems (See Mathuriya: Col. 6, Lines 36-41 and Col. 12, Lines 35-39).
Regarding claim 12, Shah in view of Mathuriya teaches the system of claim 1. Shah teaches the system comprising a random-access memory (Fig. 5, Memory 518).
Mathuriya teaches the system comprising wherein: the random access memory is on a third semiconductor die (Fig. 1, RAM 104/106 are part of a die; Col. 17, Lines 10-11, Two or more memory dies 707 (e.g., memory 104)); and the first semiconductor die is stacked on: the second semiconductor die, and the third semiconductor die (Fig. 7A, Compute die 304 and RAM die 701 and dies 701 includes persistent flash die and stack can be any formation; Col. 19, Lines 39-42, memory dies 703.sub.1-N can be embedded DRAM, SRAM, flash, Fe-RAM, MRAM, Fe-SRAM, Re-RAM, etc. or a combination of them… Col. 17, Lines 59-61, Package configuration 700 has no such limitation, in accordance with some embodiments).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shah’s system to incorporate the teachings of Mathuriya and include the persistent memory and random access memory as semiconductor chips within a stacked package form factor with the control and inference chip of Shah.
One of ordinary skill in the art would be motivated to make the modifications in order to improve thermal characteristics, increase reliability, and reduce footprint in semiconductor systems (See Mathuriya: Col. 6, Lines 36-41 and Col. 12, Lines 35-39).
Regarding claim 14, Shah teaches a method (Fig. 5, Method performed in system 504), comprising: performing an inference operation of a neural network (Fig. 5, Inference engine 510 includes compute engine 514 that performs inference operation of a neural network; Paragraph 0023, Compute engine 514 can use a neural network to perform inferences or analysis) within a persistent storage device (Fig. 4, Persistent storage device), the performing comprising: reading a weight from the persistent storage device using a wideband data connection (Fig. 5, Inference weights are read from persistent memory 550 of persistent storage device of 504/550 of Figure 5; Paragraph 0015, To run an inference operation with a CPU or inference cards being used as an inference accelerator, the data is fetched from storage solid-state drive (SSD)… and then stored in static random access memory (SRAM)… Paragraph 0066, host issues a compute command with an address in the media of an inference model (e.g., weights)… the processing system copies the model and decoded data from media into local memory (e.g., SRAM)); multiplying the weight by an element of an input feature map to form a first product (Fig. 5, Compute engine 514 performs multiplication of weights of a feature map to obtain a product; Paragraph 0063, After the data and model are received by processing system 504, processing system 504 will execute the operation (e.g., Convolution, Relu, MaxPool). For example, Convolution can refer to application of a filter to an input that results in an activation. Repeated application of the same filter to an input results in a map of activations called a feature map… result of the operation can be written to media 550); and calculating an activation based on the first product (Fig. 5, Compute engine 514 can perform further processing such as convolution via multiplication, wherein a multiplication product is obtained and used to determine an activation; Paragraph 0025, Examples of a compute command supported by compute engine 514 include, but are not limited to… Paragraph 0027, 2. Matrix-Filter Convolve… Paragraph 0063, Convolution can refer to application of a filter to an input that results in an activation… host 502 can issue another command such as to retrieve the result or perform further processing on the result), wherein: the persistent storage device comprises a persistent memory and a control and inference circuit (Fig. 5, Media 550 (i.e. first persistent memory) is connected to inference engine 510 via PCIe interface 545 which is a wideband data connection, wherein the inference engine includes memory 518 which is the local SRAM; Paragraph 0041, Media interface 545 can be compatible with one or more of: DDR4, DDR5, PCIe… Paragraph 0094, High speed interconnects can be… Peripheral Component Interconnect express (PCIe)… Paragraph 0097, high-speed link 1018, such as a 40 Gigabit/second (Gb/s)).
Shah does not teach the method comprising: the persistent storage device comprises the persistent memory on a first semiconductor die and the control and inference circuit on a second semiconductor die, the first semiconductor die stacked on the second semiconductor die, and the wideband data is configured as a high-bandwidth connection between the stacked first semiconductor die and second semiconductor die that connects the persistent memory to the control and inference circuit.
Mathuriya teaches the method comprising: the persistent storage device comprises the persistent memory on a first semiconductor die (Fig. 3A, Memory die 307; Col. 6, Lines 31-35, memory die can be a DRAM, ferroelectric or paraelectric RAM (FeRAM), static random-access memory (SRAM), and other non-volatile memories such as flash, NAND, magnetic RAM (MRAM), Fe-SRAM, Fe-DRAM, and other resistive RAMs (Re-RAMs) etc) and the control and inference circuit on a second semiconductor die (Fig. 3A, Compute chiplet die 304 performs AI processing and inference; Col. 18, Lines 10-11, compute die 304 comprises logic portions of an inference die), the first semiconductor die stacked on the second semiconductor die (Fig. 3A, Memory die 307 is stacked on compute chiplet die 304; Col. 12, Lines 62-63, a memory die 307 is placed over compute chiplet 304), and the wideband data is configured as a high-bandwidth connection (Fig. 2, Memory die 1 is connected to compute chiplet die 2 via a wideband connection that is high-bandwidth; Col. 6, Lines 58-60, Ultra-high bandwidth between the memory and compute dies is achieved by tight micro-bump spacing between the two dies… Col. 11, Lines 45-47, memory die is a high bandwidth memory (HBM)) between the stacked first semiconductor die and second semiconductor die that connects the persistent memory to the control and inference circuit (Fig. 2, Stacked memory die 1 and compute chiplet die 2 are used for AI training using high-bandwidth; Col. 12, Lines 28-30, Architecture 200 provides reduced memory accesses for the compute die (e.g., die 2) by providing data locality for weights, inputs, and outputs).
Shah and Mathuriya are analogous arts because they are in the same field of endeavor of creating integrated systems between random access memory, memory controllers, and persistent memory for inference and AI training.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shah’s method to incorporate the teachings of Mathuriya and include the persistent memory and random access memory as semiconductor chips within a stacked package with the control and inference chip of Shah.
One of ordinary skill in the art would be motivated to make the modifications in order to improve thermal characteristics, increase reliability, and reduce footprint in semiconductor systems (See Mathuriya: Col. 6, Lines 36-41 and Col. 12, Lines 35-39).
Regarding claim 15, Shah in view of Mathuriya teaches the method of claim 14. Shah teaches the method further comprising storing the activation in a random-access memory (Figs. 5 and 6, Memory 518 (i.e. random-access memory) stores activation; Paragraph 0066, an inference model (e.g., weights)… processing system copies the model and decoded data from media into local memory (e.g., SRAM)).
Regarding claim 17, Shah teaches a device (Fig. 4, Memory board with inference), comprising: a connector (Fig. 4, Memory board with inference is a memory card with a connector; Paragraph 0018, a board with memory and inference capabilities is available. The board with memory and inference capabilities can be M.2 compatible. A storage or memory device and inference engine can be mounted using an M.2 compatible circuit board. M.2 is a specification for internally mounted computer expansion cards and associated connectors for coupling to other devices in a server, rack, or blade architecture); a first persistent memory (Figs. 4 and 5, Memory board with inference in Figure 4 is the same embodiment as processing system 504 in Figure 5, which shows media 550 which is persistent memory; Paragraph 0018, Reference to storage or memory device can refer to any type of volatile or non-volatile memory); and a control and inference circuit (Fig. 5, Inference engine 510; Paragraph 0020, processing system 504 can use inference engine 510 to perform inference operations), wherein: the first persistent memory is connected to the control and inference circuit by a wideband data connection (Fig. 5, Media interface 545 is PCIe wideband connection coupled to inference engine 510; Paragraph 0041, Media interface 545 can be compatible with one or more of: DDR4, DDR5, PCIe… Paragraph 0094, High speed interconnects can be… Peripheral Component Interconnect express (PCIe)… Paragraph 0097, high-speed link 1018, such as a 40 Gigabit/second (Gb/s)); and the connector is suitable for connecting the device to a mobile computing device (Fig. 4, Memory board with inference is a card capable of connecting to a smartphone, tablets, personal computers which are mobile (i.e. they can be physically moved, such as a laptop); Paragraph 0099, Embodiments herein may be implemented in various types of computing, smart phones, tablets, personal computers), the control and inference circuit is configured to perform arithmetic operations based on an inference operation performed within the device (Fig. 5, Inference engine 510 performs arithmetic operations such as matrix multiplication for inference operations; Paragraph 0025, a compute command supported by compute engine 514 include… Paragraph 0026, 1. Matrix-Matrix Multiply).
Shah does not teach the device comprising: the first persistent memory on a first semiconductor die; and the control and inference circuit on a second semiconductor die, the first semiconductor die stacked on the second semiconductor die, wherein: the first persistent memory is connected to the control and inference circuit by a wideband data connection, configured as a high-bandwidth data connection between the stacked first semiconductor die and the second semiconductor die that connects the persistent memory to the control and inference circuit.
Mathuriya teaches the device comprising: the first persistent memory on a first semiconductor die (Fig. 3A, Memory die 307; Col. 6, Lines 31-35, memory die can be a DRAM, ferroelectric or paraelectric RAM (FeRAM), static random-access memory (SRAM), and other non-volatile memories such as flash, NAND, magnetic RAM (MRAM), Fe-SRAM, Fe-DRAM, and other resistive RAMs (Re-RAMs) etc); and the control and inference circuit on a second semiconductor die (Fig. 3A, Compute chiplet die 304 performs AI processing and inference; Col. 18, Lines 10-11, compute die 304 comprises logic portions of an inference die), the first semiconductor die stacked on the second semiconductor die (Fig. 3A, Memory die 307 is stacked on compute chiplet die 304; Col. 12, Lines 62-63, a memory die 307 is placed over compute chiplet 304), wherein: the first persistent memory is connected to the control and inference circuit by a wideband data connection, configured as a high-bandwidth data connection (Fig. 2, Memory die 1 is connected to compute chiplet die 2 via a wideband connection that is high-bandwidth; Col. 6, Lines 58-60, Ultra-high bandwidth between the memory and compute dies is achieved by tight micro-bump spacing between the two dies… Col. 11, Lines 45-47, memory die is a high bandwidth memory (HBM)) between the stacked first semiconductor die and the second semiconductor die that connects the persistent memory to the control and inference circuit (Fig. 2, Stacked memory die 1 and compute chiplet die 2 are used for AI training using high-bandwidth; Col. 12, Lines 28-30, Architecture 200 provides reduced memory accesses for the compute die (e.g., die 2) by providing data locality for weights, inputs, and outputs).
Shah and Mathuriya are analogous arts because they are in the same field of endeavor of creating integrated systems between random access memory, memory controllers, and persistent memory for inference and AI training.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shah’s device to incorporate the teachings of Mathuriya and include the persistent memory and random access memory as semiconductor chips within a stacked package with the control and inference chip of Shah.
One of ordinary skill in the art would be motivated to make the modifications in order to improve thermal characteristics, increase reliability, and reduce footprint in semiconductor systems (See Mathuriya: Col. 6, Lines 36-41 and Col. 12, Lines 35-39).
Regarding claim 18, Shah in view of Mathuriya teaches the device of claim 17. Shah teaches the device further comprising wherein: the control and inference circuit comprises a first persistent memory controller (Figs. 5 and 8, State machine 516 is a microprocessor that controls storage operations for persistent media 550 and is synonymous with memory controller 822 of Figure 8; Paragraph 0020, State machine 516 can be implemented as a microprocessor… Paragraph 0066, state machine issues a Read command for an inference model (and optionally, weights) and decoded data from the media… Paragraph 0086, memory controller to generate and issue commands to memory 830); the device further comprises: an interface circuit (Fig. 8, Interface 812), a second persistent memory (Fig. 8, Storage 884 is a second persistent memory; Paragraph 0090, storage device(s) 884, which can be or include any conventional medium for storing large amounts of data in a nonvolatile manner, such as one or more magnetic, solid state), and a second persistent memory controller (Fig. 8, Controller 882; Paragraph 0090, storage subsystem 880 includes controller 882 to interface with storage 884); the second persistent memory is connected to the second persistent memory controller (Fig. 8, Controller 882 connected to storage 884); the control and inference circuit is connected to the interface circuit (Fig. 8, Memory subsystem 820 is the control and inference circuit and is coupled to interface 812); and the second persistent memory controller is connected to the interface circuit (Fig. 8, Controller 882 is coupled to interface 814 which further transfers data to interface 812 and thus are connected to each other; Paragraph 0088, system 800 includes interface 814, which can be coupled to interface 812).
Regarding claim 20, Shah in view of Mathuriya teaches the device of claim 17.
Mathuriya teaches the system comprising: the first semiconductor die and the second semiconductor die are part of a stack of dies (Fig. 3A, Stack of dies).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shah’s device to incorporate the teachings of Mathuriya and include the persistent memory and random access memory as semiconductor chips within a stacked package form factor with the control and inference chip of Shah.
One of ordinary skill in the art would be motivated to make the modifications in order to improve thermal characteristics, increase reliability, and reduce footprint in semiconductor systems (See Mathuriya: Col. 6, Lines 36-41 and Col. 12, Lines 35-39).
Claims 8 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Shah (US 2020/0167098) in view of Mathuriya (US 11,694,940) and further in view of Schardt (US 2023/0061268).
Regarding claim 8, Shah in view of Mathuriya teaches the system of claim 1. Shah teaches the system further comprising: a first interface circuit (Fig. 5, Interface 503), and an artificial intelligence accelerator (Fig. 5, Host 502 is an AI accelerator; Paragraph 0019, host 502 can include one or more processors (e.g., CPU, cores, GPU, accelerators… Paragraph 0047, Host 502 can issue a compute or storage command to processing system 504… Paragraph 0048, compute command can include or be associated with an inference model (e.g., model type) and include or reference weights stored in media 550), wherein: the artificial intelligence accelerator is configured to perform additional inference operations of a neural network, with assistance from: the first persistent memory, and the control and inference circuit (Fig. 5, Host 502 transmits compute commands to perform inference of a neural network to inference engine 510 and media 550; Paragraph 0048, compute command can include or be associated with an inference model (e.g., model type) and include or reference weights stored in media 550… Paragraph 0085, Accelerators 842 can provide multiple neural networks, CPUs, processor cores, general purpose graphics processing units, or graphics processing units can be made available for use by artificial intelligence (AI) or machine learning (ML) models); and the first persistent memory and the control and inference circuit are configured to perform an operation selected from the group consisting of compression (Fig. 5, Inference engine 510 and media 550 used for compressing/decompressing; Paragraph 0020, Decode and crop engine 512 can in addition, or alternatively, decompress data stored in media 550. Decompression can include decompressing media (e.g., video, still image, or audio)… Paragraph 0085, an accelerator among accelerators 842 can provide compression (DC) capability).
Neither Shah nor Mathuriya teaches the system comprising to perform an operation selected from the group consisting of pruning, sparsity, compression, quantization, and approximation.
Schardt teaches the system comprising to perform an operation selected from the group consisting of pruning, sparsity, compression, quantization, and approximation (Paragraph 0019, Examples of compression techniques include pruning, quantization, low-rank approximation and sparsity).
Shah, Mathuriya, and Schardt are analogous arts because they are in the same field of endeavor of performing compression via inference using neural networks.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shah/Mathuriya’s system to incorporate the teachings of Schardt and enable the host accelerator of Shah to perform pruning, sparsity, compression, quantization, and approximation via artificial intelligence operations.
One of ordinary skill in the art would be motivated to make the modifications in order to perform model compression to simplify large, complex models to produce lightweight counterpart models (See Schardt: Paragraph 0018).
Regarding claim 19, Shah in view of Mathuriya teaches the device of claim 17. Shah teaches the device comprising wherein: the arithmetic operations performed by the control and inference circuit are a portion of the inference operation selected from the group consisting of compression (Fig. 5, Inference engine 510 and media 550 used for compressing/decompressing; Paragraph 0020, Decode and crop engine 512 can in addition, or alternatively, decompress data stored in media 550. Decompression can include decompressing media (e.g., video, still image, or audio)… Paragraph 0085, an accelerator among accelerators 842 can provide compression (DC) capability).
Neither Shah nor Mathuriya teaches the device comprising to perform an operation selected from the group consisting of pruning, sparsity, compression, quantization, and approximation.
Schardt teaches the device comprising to perform an operation selected from the group consisting of pruning, sparsity, compression, quantization, and approximation (Paragraph 0019, Examples of compression techniques include pruning, quantization, low-rank approximation and sparsity).
Shah, Mathuriya, and Schardt are analogous arts because they are in the same field of endeavor of performing compression via inference using neural networks.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shah/Mathuriya’s device to incorporate the teachings of Schardt and enable the host accelerator of Shah to perform pruning, sparsity, compression, quantization, and approximation via artificial intelligence operations.
One of ordinary skill in the art would be motivated to make the modifications in order to perform model compression to simplify large, complex models to produce lightweight counterpart models (See Schardt: Paragraph 0018).
Claims 13 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Shah (US 2020/0167098) in view of Mathuriya (US 11,694,940) and further in view of Fujiwara (US 2023/0361081).
Regarding claim 13, Shah in view of Mathuriya teaches the system of claim 1. Shah teaches the system comprising a random-access memory (Fig. 5, Memory 518), wherein: the control and inference circuit comprises: a persistent memory controller (Fig. 5, State machine 516 is a microprocessor that controls storage operations for persistent media 550; Paragraph 0020, State machine 516 can control the overall flow of the inference engine 510. State machine 516 can be implemented as a microprocessor… Paragraph 0066, state machine issues a Read command for an inference model (and optionally, weights) and decoded data from the media).
Mathuriya teaches the system comprising wherein: the random access memory is on a fourth semiconductor die (Fig. 1, RAM 104/106 are part of a die; Col. 17, Lines 10-11, Two or more memory dies 707 (e.g., memory 104)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shah’s system to incorporate the teachings of Mathuriya and include the persistent memory and random access memory as semiconductor chips within a stacked package form factor with the control and inference chip of Shah.
One of ordinary skill in the art would be motivated to make the modifications in order to improve thermal characteristics, increase reliability, and reduce footprint in semiconductor systems (See Mathuriya: Col. 6, Lines 36-41 and Col. 12, Lines 35-39).
Neither Shah nor Mathuriya teaches the system comprising a multiply-accumulate circuit; the persistent memory controller is part of a second semiconductor die; the multiply-accumulate circuit is part of a third semiconductor die; and the first semiconductor die is stacked on: the second semiconductor die, the third semiconductor die, and the fourth semiconductor die.
Fujiwara teaches the system (Fig. 1A, Semiconductor stack 100) comprising a multiply-accumulate circuit (Fig. 1A, Memory dies MD includes CIM dies that implement multiply-accumulate functions; Paragraph 0017, An example of applications of CIM is multiply-accumulate (“MAC”) operations… Paragraph 0033, memory dies MD may be or include CIM dies); the first persistent memory is part of a first semiconductor die (Fig. 1A, Memory dies MD further include flash memory die which is persistent memory; Paragraph 0033, memory dies MD may be or include CIM dies that can be implemented with a variety of memory devices, including… Flash); the persistent memory controller is part of a second semiconductor die (Fig. 1A, Core die CD is a flash memory controller; Paragraph 0032, the memory controller is formed on a surface of a semiconductor substrate (e.g., core die)); the multiply-accumulate circuit is part of a third semiconductor die (Fig. 1A, Multiply-accumulate circuit is a CIM die which is part of the MD stack); and the first semiconductor die is stacked on: the second semiconductor die, the third semiconductor die, and the fourth semiconductor die (Fig. 1A, Upper most die UMD can be flash die (i.e. first semiconductor die that is persistent memory) which is stacked on core die CD (i.e. second semiconductor die that is memory controller), CIM die (i.e. third semiconductor die that is multiply-accumulator), and SRAM die (i.e. fourth semiconductor die that is random-access memory); Paragraph 0024, plurality of memory dies MD is stacked layer-by-layer to form a 3D architecture and is coupled to the core die CD… Paragraph 0034, FIG. 1A, the first conductive pillar TSVI also connects the upmost memory die UMD).
Shah, Mathuriya, and Fujiwara are analogous arts because they are in the same field of endeavor of creating integrated systems for machine learning.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shah/Mathuriya’s system to incorporate the teachings of Fujiwara and include a multiply-accumulator die, and have the persistent memory of Shah in a first semiconductor die stacked on top of multiply-accumulator die, memory controller die, and random-access memory die.
One of ordinary skill in the art would be motivated to make the modifications in order to implement complex neural network operations while enabling faster data transfer and power efficiency (See Fujiwara: Paragraphs 0003 and 0016-0021).
Regarding claim 16, Shah in view of Mathuriya teaches the method of claim 14. Shah teaches the method comprising wherein: the performing of the inference operation comprises performing a portion of the inference operation by the control and inference circuit (Fig. 5, Control and inference engine 510 performs at least a portion of the inference operation, See Paragraph 0026), and the control and inference circuit comprises: a persistent memory controller (Fig. 5, State machine 516 is a microprocessor that controls storage operations for persistent media 550; Paragraph 0020, State machine 516 can control the overall flow of the inference engine 510. State machine 516 can be implemented as a microprocessor… Paragraph 0066, state machine issues a Read command for an inference model (and optionally, weights) and decoded data from the media).
Neither Shah nor Mathuriya teaches the method comprising the system a multiply-accumulate circuit.
Fujiwara teaches the method comprising the system (Fig. 1A, Semiconductor stack 100) comprising a multiply-accumulate circuit (Fig. 1A, Memory dies MD includes CIM dies that implement multiply-accumulate functions; Paragraph 0017, An example of applications of CIM is multiply-accumulate (“MAC”) operations… Paragraph 0033, memory dies MD may be or include CIM dies).
Shah and Fujiwara are analogous arts because they are in the same field of endeavor of creating integrated systems for machine learning.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shah/Mathuriya’s method to incorporate the teachings of Fujiwara and include a multiply-accumulator die.
One of ordinary skill in the art would be motivated to make the modifications in order to implement complex neural network operations while enabling faster data transfer and power efficiency (See Fujiwara: Paragraphs 0003 and 0016-0021).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HARRY Z WANG whose telephone number is (571)270-1716. The examiner can normally be reached 9 am - 3 pm (Monday-Friday).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henry Tsai can be reached at 571-272-4176. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/H.Z.W./Examiner, Art Unit 2184
/HENRY TSAI/Supervisory Patent Examiner, Art Unit 2184
