DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/07/2025 is/are compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claim(s) 15 is/are objected to because of the following informalities:
In claim 15, line 1, “The computing device of claim 1, further comprising” should read “The computing device of claim 12, further comprising”.
Appropriate correction is required.
Specification
The disclosure is objected to because of the following informalities:
In Paragraph [0001], line 2, “Application No. 63/658,603, filed on March 22, 2024” should read “Application No. 63/568,603, filed on March 22, 2024”.
Appropriate correction is required.
Office Action Summary
Claim(s) 1 and 11-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lung et al (US 2024/0281142 A1) in view of Rotem et al (US 10,699,190 B1), further in view of Liu et al (US 2022/0318656 A1).
Claim(s) 2 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lung et al (US 2024/0281142 A1) in view of Rotem et al (US 10,699,190 B1) and Liu et al (US 2022/0318656 A1), further in view of Bazarsky et al (US 2021/0304009 A1).
Claim(s) 3 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lung et al (US 2024/0281142 A1) in view of Rotem et al (US 10,699,190 B1) and Liu et al (US 2022/0318656 A1), further in view of Gu et al (US 2019/0214365 A1).
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lung et al (US 2024/0281142 A1) in view of Nag et al (US 2024/0012757 A1) and Alizadeh et al (LLM in a flash: Efficient Large Language Model Inference with Limited Memory), further in view of Franca-Neto et al (US 2020/0310674 A1).
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lung et al (US 2024/0281142 A1) in view of Nag et al (US 2024/0012757 A1), Alizadeh et al (LLM in a flash: Efficient Large Language Model Inference with Limited Memory), and Franca-Neto et al (US 2020/0310674 A1), further in view of Yoon (US 2023/0418695 A1).
Claim(s) 4-10, 15-17, and 20 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1 and 11-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lung et al (US 2024/0281142 A1) in view of Rotem et al (US 10,699,190 B1), further in view of Liu et al (US 2022/0318656 A1).
Regarding claim(s) 1 and 12, Lung teaches a system-on-chip (Figure 2; and Paragraph [0040]) comprising:
a set of chiplets that includes volatile memory (Figure 4; Paragraph [0057]: “The platform includes a processor chip 401 (e.g., a CPU and an AIA cc), a first memory chip 403 (an activation memory storing activation data), and a second memory chip 402 (e.g., a weigh memory storing weight data for a neural network)”; and Paragraph [0058]: “The first memory chip 403 in this example comprises a high capacity, volatile memory 440 such as DRAM or SRAM”);
a non-volatile memory storing a machine learning (ML) model (Figure 1; Paragraph [0029]: “the second memory chip 102, which can be a HB non-volatile memory (NVM) for storing weights (e.g., weight data) used in neural network computations”; and Paragraph [0063]: “The inference model can comprise a computation graph of a deep learning neural network […]”); and
an interconnect to connect the non-volatile memory to the set of chiplets (Figure 1; Paragraph [0031]: “The input/output interface 113 is connected to interconnection wiring 111 on the interposer 110”; Paragraph [0032]: “The first memory chip 103 includes an interface 112 for connection to the interconnection wiring 111 on the interposer 110”; Paragraph [0033]: “The second memory chip 102 includes an interface 114 for connection to the interconnection wiring 111 on the interposer 110”; and Paragraph [0034]: “interconnection wiring 111 provides part of the data path between the first memory chip 103, the second memory chip 102, and the processor chip 101”);
wherein at least one chiplet of the set of chiplet is adapted (Figure 4; and Paragraph [0057]) to:
read a set of values of the ML model (Figure 5; and Paragraph [0067]: “during runtime, an operation retrieves a plurality of neural network weights 602 (e.g., Wb, W1, W2, W3, and Wn) stored in the high bandwidth NVM (operation 501). The weight data is directly transferred into the AI accelerator core 611 to perform calculations”);
apply the set of values of the ML model to input data (Figure 5; and Paragraph [0013]: “Using the accelerator core, the plurality of neural network weights can be applied to input data for each one of a plurality of nodes of a neural network to obtain an intermediate output”).
Lung fails to teach based on applying the set of values to the input data, determine regions of the ML model having fixed values and regions of the ML model having non-fixed values; and migrate or store the regions of the ML model having non-fixed values in the volatile memory of the set of chiplets.
However, Rotem teaches based on applying the set of values to the input data, determine regions of the ML model having fixed values and regions of the ML model having non-fixed values (Figure 4; Col. 5, lines 40-56: “the term "training session," as used herein, may refer to any process during which a neural network is given data in order to improve the performance of the neural network […] a neural network may engage in supervised learning that involves receiving input data that is labelled. Additionally or alternatively, a neural network may engage in unsupervised learning that involves receiving input data that is not labelled […] a neural network may compare output produced from input data with expected output for the input data in order to determine the accuracy of the neural network”; and Col. 7, lines 12-27: “In some embodiments, the values of the weights may be represented as matrices and the difference between the previous values and the new values of the weights may also be represented as a matrix […] before training, a neural network may have weights represented by previous set of values 402. After a training session, the neural network may have weights represented by new set of values 406, which may be produced by adding difference 404 to previous set of values 402 […] difference 404 has only four nonzero values”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify Lung according to the teachings of Rotem such that, based on applying values of the ML model to input data in connection with training the ML model, regions of the ML model having values that remain unchanged and regions having values that change are determined by comparing the model values before and after training. The motivation for this combination of references would have been to make such a modification because Rotem teaches identifying and constraining differences between previous and newly trained neural network values to reduce the size in memory of the difference and thereby reduce the amount of information required to represent and communicate an update to the neural network. This motivation is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Lung and Rotem fail to teach to migrate or store the regions of the ML model having non-fixed values in the volatile memory of the set of chiplets. However, Liu teaches migrate or store the regions of the ML model having non-fixed values in the volatile memory of the set of chiplets (Figure 11; Paragraph [0007]: “illustrative embodiments differentiate model parameters that are pre-trained (and thus are considered immutable) from model parameters that are not pre-trained (and thus are considered mutable). While each of the multiple inference model instances maintain their own local memory for the mutable parameters, the multiple inference model instances access the same shared memory for the immutable parameters”; Paragraph [0031]: “The mutable parameters of the model include the model input, the model output, and the intermediate computing results for each layer of the model”; and Paragraph [0067]: “immutable parameters are pointed to in the GPU shared memory created by shared parameter host manager 1010 while mutable parameters are pointed to in the local memory created by shared parameter guest inference application instance 1030”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to further modify Lung as modified by Rotem according to the teachings of Liu to store the regions of the ML model having non-fixed values in the volatile memory of Lung, by storing mutable model parameters, including intermediate computation results, in the volatile DRAM/SRAM provided by Lung. The motivation for this combination of references would have been to make such a modification because Liu teaches differentiating mutable parameters from immutable pre-trained parameters and separately allocating local memory for mutable parameters so that the respective parameter types can be treated differently and inference performance can be further optimized. Lung already provides a high-capacity volatile DRAM/SRAM memory within the inference system (Paragraph [0058]), thereby providing a predictable memory resource for implementing Liu's separate storage of mutable model parameters. This motivation is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim(s) 11, Lung as modified by Rotem and Liu teaches the system-on-chip of claim 1, where Liu teaches wherein the system-on-chip is included in one of a smartphone, tablet computing device, personal computing device, wearable computing device, or a datacenter server (Paragraph [0025]: “techniques are implementable for training of large models in graphic processing units (GPUs) distributed in multiple computing nodes in a cloud computing environment […] one or more edge servers running one or more application programs that interact with a plurality of heterogeneous computing devices (e.g., X86_64/ARM CPUs (central processing units), FPGAs (field programmable gate arrays), ASICs (application specific integrated circuits), programmable switches, etc.) which are normally computing resource-limited (e.g., limited in terms of processing and/or storage capacities) […] Edge computing moves the computer workload closer to the consumer/data generator to reduce latency, bandwidth and overhead for the centralized data center and intermediate switches, gateways, and servers”).
Claim(s) 2 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lung et al (US 2024/0281142 A1) in view of Rotem et al (US 10,699,190 B1) and Liu et al (US 2022/0318656 A1), further in view of Bazarsky et al (US 2021/0304009 A1).
Regarding claim(s) 2 and 13, Lung as modified by Rotem and Liu teaches the system-on-chip of claim 1, where Lung teaches wherein the non-volatile memory comprises a flash memory chiplet of the system-on-chip (Figure 4; and Paragraph [0059]: “The second memory chip 402, in this example, comprises a high-bandwidth (HB), nonvolatile memory (NVM) configured in one or more banks 430a, 430b, each of which can comprise one or more layers of NVM dies arranged in channels. The NVM can be one of a phase change memory (PCM), a three-dimensional cross point memory (3D Xpoint), and a NAND flash memory”).
Lung, Rotem, and Liu fail to teach wherein the at least one chiplet stores the regions of the ML model having fixed values in the flash memory chiplet. However, Bazarsky teaches wherein the at least one chiplet stores the regions of the ML model having fixed values in the flash memory chiplet (Paragraph [0021]: “The examples herein relate to non-volatile memory (NVM) arrays, and to data storage devices or apparatus for controlling the NVM arrays, such as a controller of a data storage device (DSD), e.g. a solid state device (SSD), and in particular to solid-state memory storage devices such as those that use NAND flash memory, herein “NANDs””; Paragraph [0034]: ““static data memory” refers to memory that has been configured for, or optimized for, storing data that is not updated at all or updated only rarely (e.g. z values). Herein, the term “dynamic data memory” refers to memory that has been configured for, or optimized for, storing data that is updated frequently (e.g. C, δ and ω (or w) values). In some examples, static data is data that does not change at all during the multidimensional computation procedure and dynamic data is data that changes during the multidimensional computation procedure”; and Paragraph [0035]: “the static data memory may be configured to achieve high data retention and write endurance (as compared to the dynamic data memory) via appropriate selection of NVM trim settings or the like, such as by controlling read threshold calibration”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify Lung as modified by Rotem and Liu according to Bazarsky to store regions of the ML model determined to have fixed values in the flash memory of Lung, in view of Bazarsky’s teaching of storing neural-network data that does not change in a static-data portion of non-volatile memory. The motivation for this combination of references would have predictably used the retention characteristics of non-volatile memory for neural-network data that is not updated, consistent with Bazarsky’s differentiated treatment of static and dynamic neural-network data. This motivation is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Claim(s) 3 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lung et al (US 2024/0281142 A1) in view of Rotem et al (US 10,699,190 B1) and Liu et al (US 2022/0318656 A1), further in view of Gu et al (US 2019/0214365 A1).
Regarding claim(s) 3 and 14, Lung as modified by Rotem and Liu teaches the system-on-chip of claim 1, wherein the volatile memory comprises a dynamic random-access-memory (DRAM) (Figure 4; and Paragraph [0058]: “The first memory chip 403 in this example comprises a high capacity, volatile memory 440 such as DRAM or SRAM (or a nonvolatile memory such as 3D NAND or other type of memory implemented using charge trapping storage technology)”).
Lung, Rotem, and Liu fail to teach the system-on-chip of claim 1, wherein the volatile memory comprises a dynamic random-access-memory (DRAM) in a high-bandwidth memory chiplet of the system-on-chip. However, Gu teaches wherein the volatile memory comprises a dynamic random-access-memory (DRAM) in a high-bandwidth memory chiplet of the system-on-chip (Paragraph [0003]: “High Bandwidth Memory (HBM) is a high-performance RAM interface for 3D-stacked dynamic random access memory (DRAM)”; and Paragraph [0047]: “HBM may include stacked dynamic RAM (DRAM) memory that communicates using through-silicon vias (TSV). Generally, High Bandwidth Memory combines through-silicon vias (TSV) and micro-bumps to connect multiple (e.g., 4, 8, etc.) dies of memory cell arrays on top of each other”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify Lung as modified by Rotem and Liu according to Gu such that Lung’s volatile DRAM memory is implemented in a high-bandwidth memory chiplet comprising stacked DRAM dies. The motivation for this combination of references would have been to provides higher bandwidth, reduced power consumption, and a smaller form factor, and further identifies neural-network and artificial-intelligence workloads as memory-intensive applications for HBM (Gu, Paragraphs [0004]-[0005]). This motivation is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lung et al (US 2024/0281142 A1) in view of Nag et al (US 2024/0012757 A1) and Alizadeh et al (LLM in a flash: Efficient Large Language Model Inference with Limited Memory), further in view of Franca-Neto et al (US 2020/0310674 A1).
Regarding claim(s) 18, Lung teaches a system-on-chip (SoC) for vehicle operations, comprising:
a set of chiplets that includes DRAM (Figure 4; Paragraph [0040]: “The AI inference memory device can be packaged as any of a 2.5D through silicon via (TSV) integrated circuit, a three-dimension (3D) through silicon via (TSV) integrated circuit and a three-dimensional (3D) system-on-chip (3D SOC) integrated circuit”; Paragraph [0058]: “The first memory chip 403 in this example comprises a high capacity, volatile memory 440 such as DRAM or SRAM”; and Paragraph [0059]: “The second memory chip 402, in this example, comprises a high-bandwidth (HB), nonvolatile memory (NVM) […] The NVM can be one of a phase change memory (PCM), a three-dimensional cross point memory (3D Xpoint), and a NAND flash memory”)
a flash memory chiplet having flash memory adapted to store a machine learning (ML) model, the ML model including model weights that have been trained (Figure 1; Paragraph [0029]: “the second memory chip 102, which can be a HB non-volatile memory (NVM) for storing weights (e.g., weight data) used in neural network computations”; Paragraph [0063]: “The inference model can comprise a computation graph of a deep learning neural network […]”; and Paragraph [0072]: “During training processing, weights stored in NVM dies (e.g., of the weights memory 402) are adjusted by the processor chip 401 (e.g., a CPU and an AIA cc) based upon training dataset(s). As training progresses, the processor chip 401 will keep updating values for the weights and this value will be stored in the NVM”); and
an interconnect to connect the flash memory to the set of chiplets (Figure 1; Paragraph [0031]: “The input/output interface 113 is connected to interconnection wiring 111 on the interposer 110”; Paragraph [0032]: “The first memory chip 103 includes an interface 112 for connection to the interconnection wiring 111 on the interposer 110”; Paragraph [0033]: “The second memory chip 102 includes an interface 114 for connection to the interconnection wiring 111 on the interposer 110”; and Paragraph [0034]: “interconnection wiring 111 provides part of the data path between the first memory chip 103, the second memory chip 102, and the processor chip 101”);
apply the ML model to the input prompt to perform inference (Paragraph [0068]: “the operations includes executing an inference procedure using the AI accelerator core 611 to apply the plurality of neural network weights to input data 606 […]”) by:
accessing a first set of the model weights of the ML model directly from the flash memory to calculate intermediate values based on the input prompt and the first set of model weights (Figure 5; Paragraph [0013]: “Using the accelerator core, the plurality of neural network weights can be applied to input data for each one of a plurality of nodes of a neural network to obtain an intermediate output”; Paragraph [0067]: “during runtime, an operation retrieves a plurality of neural network weights 602 (e.g., Wb, W1, W2, W3, and Wn) stored in the high bandwidth NVM (operation 501). The weight data is directly transferred into the AI accelerator core 611 to perform calculations”; and Paragraph [0068]: “After loading the weights and configuring the AI accelerator core 611 […] to apply the plurality of neural network weights to input data 606 for each one of a plurality of nodes of a neural network to obtain an intermediate output (operation 503) for one or more levels of nodes of the neural network.”);
storing the intermediate values in the DRAM (Figure 4; Paragraph [0058]: “The first memory chip 403 in this example comprises a high capacity, volatile memory 440 such as DRAM or SRAM […]”; and Paragraph [0069]: “the activation function 604 stored in, for example the first memory chip 103 (e.g., activation memory 403 of FIG. 4) is applied to the intermediate output to obtain an output 610 (e.g., a result) for one or more levels of the neural network (operation 504) […] The result is stored in the high bandwidth NVM, FPGA buffer or any storage device, or activation memory 403 (operation 505) […] restarting operations 501-506 for the next batch of neural network nodes to be processed”);
accessing a second set of the model weights of the ML model directly from the flash memory to calculate an output of the ML model based on the intermediate values and the second set of model weights (Paragraph [0069]: “The procedure includes checking if nodes at further neural network levels are to be processed (operation 506) and if so, restarting operations 501-506 for the next batch of neural network nodes to be processed”; Paragraph [0070]: “the operations loop to operation 501, to process a next level of the neural network. If more neural network nodes are to be processed, the operations 501 to 506 are traversed, and can include changing the activation data to a different function, loading the weights for the nodes of the next neural network level, and executing the different nodes”; and Paragraph [0067]: “during runtime, an operation retrieves a plurality of neural network weights 602 (e.g., Wb, W1, W2, W3, and Wn) stored in the high bandwidth NVM (operation 501). The weight data is directly transferred into the AI accelerator core 611 to perform calculations”).
Lung fails to teach a system-on-chip (SoC) for vehicle operations, comprising: the flash memory having a storage capacity of at least 10 GB; wherein the set of chiplets, when integrated into a vehicle, are adapted to: receive an input prompt generated based on vehicle sensor data generated by the vehicle, passenger input from a passenger of the vehicle, or a combination thereof; storing the output of the ML model in the DRAM; and generate a vehicle action based on the output of the ML model.
However, Nag teaches a system-on-chip (SoC) for vehicle operations, comprising:
wherein the set of chiplets, when integrated into a vehicle, are adapted to: receive an input prompt generated based on vehicle sensor data generated by the vehicle, passenger input from a passenger of the vehicle, or a combination thereof (Paragraph [0067]: “data received from the vehicle 104 may include sensor data collected by the vehicle during its real world services (e.g., when the user is a driver or a passenger)”; Paragraph [0113]: “the historical data includes data received from at least one sensor (e.g., car sensors 216) of the vehicle”; Paragraph [0115]: “the stored data includes sensor data received from at least one sensor of the vehicle”; Paragraph [0116]: “the at least one sensor comprises at least one of a camera, a LIDAR sensor, a GPS sensor, a brake sensor, or a tire sensor”; and Paragraph [0118]: “monitor stream data (e.g., stream data 108 from sensors 140) from the vehicle; and store the stream data in the memory”); and
generate a vehicle action based on the output of the ML model (Paragraph [0111]: “performing the computations comprises generating the instruction as an output from a neural network having the portion of the stored data as an input […] instructions are generated as output from neural network 122. The instructions are read by processor 132 to control vehicle 104”; Paragraph [0121]: at least one processor configured to: perform computations on the historical data to generate the instruction; and write the instruction to the cache for reading by the ADAS, wherein the ADAS is configured to control the vehicle based on the instruction; and Paragraph [0124]: “the instruction causes the ADAS to activate at least one brake of the vehicle”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the AI inference system of Lung according to the teachings of Nag to integrate Lung's neural-network inference architecture into a vehicle, such that vehicle sensor data is provided as input to the neural-network inference processing and the resulting neural-network output is used to generate an instruction for controlling an operation of the vehicle. The motivation for this combination of references would have been to make such a modification because Nag expressly teaches performing neural-network computations on vehicle-related data to generate an instruction for controlling the vehicle, thereby enabling neural-network processing of vehicle sensor data to provide vehicle-control functions (Nag, Paragraph [0111] – Paragraph [0116]; and Paragraph [0121] – Paragraph [0124]). Accordingly, the combination would have predictably applied Lung's hardware-accelerated neural-network inference architecture to Nag's vehicle processing environment to process vehicle sensor data and provide neural network based vehicle control instructions. This motivation is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Lung and Nag fail to teach the flash memory having a storage capacity of at least 10 GB; and storing the output of the ML model in the DRAM. However, Alizadeh teaches the flash memory having a storage capacity of at least 10 GB (Figure 2(a): “100 GB of Flash Memory” and “10 GB of DRAM”; Abstract: “This paper tackles the challenge of efficiently running LLMs that exceed the available DRAM capacity by storing the model parameters on flash memory but bringing them on demand to DRAM”; Page 1, Right Col. 3rd Paragraph: “during inference, we directly and cleverly load the required parameters from the flash memory, avoiding the need to fit the entire model in DRAM”; Page 3, 3 Load From Flash, 1st Paragraph: “[…] devices where the available computational memory is substantially smaller than the size of the model. This necessitates storing the full model weights in flash memory”; and Page 3, 3.1 Reducing Data Transfer, 1st Paragraph: “our approach involves the iterative transfer of only the essential, non-sparse data from flash memory to DRAM for processing during inference”).
Therefore, It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify the AI inference system of Lung as modified by Nag according to the teachings of Alizadeh to provide the flash memory with a storage capacity of at least 10 GB for storing the ML model. The motivation for this combination of references would have been to make such a modification because Alizadeh teaches storing model parameters in flash memory and loading required parameters from the flash memory on demand during inference when the size of the ML model exceeds the available DRAM capacity (Alizadeh, Abstract; and Figure 2(a)). Such a modification would have predictably enabled the combined system of Lung and Nag to perform inference using ML models having storage requirements exceeding the available DRAM capacity while reducing the amount of model weight data required to reside in DRAM at one time. This motivation is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Lung, Nag, and Alizadeh fail to teach storing the output of the ML model in the DRAM. However, Franca-Neto teaches storing the output of the ML model in the DRAM (Paragraph [0065]: “In step 720, the process can save the output of the neural network, for example in the DRAM”; Paragraph [0069]: “The controller 908 can process the data through the layers 902A, 902B, . . . 902C of the neural network implemented in the non-volatile memory and send the output of the neural network to the memory page 906 of the DRAM (or another area of DRAM) at step 4”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify the AI inference system of Lung as modified by Nag and Alizadeh according to the teachings of Franca-Neto to store the output of the ML model in the DRAM. The motivation for this combination of references would have been to make such a modification because Franca-Neto teaches processing data through the layers of a neural network implemented in non-volatile memory and sending the resulting neural-network output to DRAM (Franca-Neto, Paragraph [0069]). Such a modification would have predictably provided the combined system with volatile storage for the inference output, thereby making the resulting output available in DRAM for subsequent access and processing. This motivation is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lung et al (US 2024/0281142 A1) in view of Nag et al (US 2024/0012757 A1), Alizadeh et al (LLM in a flash: Efficient Large Language Model Inference with Limited Memory), and Franca-Neto et al (US 2020/0310674 A1), further in view of Yoon (US 2023/0418695 A1).
Regarding claim(s) 19, Lung as modified by Nag, Alizadeh, and Franca-Neto teaches the SoC of claim 18, where Nag teaches wherein the set of chiplets include (Paragraph [0037]: “This CXL operation maintains a unified, coherent memory space between the CPU (host processor) and any memory on an attached CXL device. This allows both the CPU and device to share resources and operate on the same memory region for higher performance, reduced data-movement, and reduced software stack complexity”; Paragraph [0039]: “The memory may include volatile and non-volatile memory. The CXL device has one or more processing devices configured to perform computations […] using a portion of the stored data”; and Paragraph [0110]: “a coherent memory space is maintained between the processor of the vehicle and the memory”).
Lung, Nag, Alizadeh, and Franca-Neto fail to teach wherein the set of chiplets include a memory controller adapted to map the model weights of the ML model and the intermediate values to a common memory address space. However, Yoon teaches wherein the set of chiplets include a memory controller adapted to map the model weights of the ML model and the intermediate values to a common memory address space (Paragraph [0140] – Paragraph [0142]: “The weight buffer 130 may receive a weight from the L0 memory 120. The weight buffer 130 may transfer the weight to the processing unit 160. The weight buffer 130 may temporarily store the weight before transferring it […] The input activation Act_In and the output activation Act_Out may refer to input values and output values of the layers of a neural network […] The weight may refer to a parameter that is multiplied by the input activation Act_In inputted in each layer. The weight is adjusted and confirmed in the deep learning training phase, and may be used to derive the output activation Act_Out via a fixed value in the inference phase”; Paragraph [0144]: “[…] The activation buffer 150 may temporarily store the input activation Act_In and the output activation Act_Out”; Paragraph [0146]: “The processing unit 160 may receive an input activation Act_In, multiply it by a weight, and then add it to generate an output activation Act_Out”; and Paragraph [0313] – Paragraph [0320]: “the processing unit 160 may issue an instruction including a logical address LA for a task. At this time, the logical address LA may be translated into a physical address PA by a memory management unit MMU […] The memory management unit MMU may include a translation lookaside buffer 114 and a shared page table SPT. The shared page table SPT may include translation information of physical addresses PA for all logical addresses LA […] the logical address LA can be translated into the physical address PA by the shared page table SPT […] the physical address PA of the physical memory PM can be accessed by the instruction […] the neural processing device in accordance with the embodiment can use only one shared page table even when processes are switched […] The PTE may be used to calculate a physical address PA corresponding to a logical address LA”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to further modify the SoC of Lung as modified by Nag, Alizadeh, and Franca-Neto according to Yoon to provide memory management circuitry configured to map memory addresses associated with neural network data, including model weights and intermediate activation values, within the unified memory space taught by Nag. Yoon teaches a neural processing architecture in which neural network weights and inter-layer activation values are used during neural network processing and further teaches a memory management unit using a shared page table to translate logical addresses to physical addresses and provide mapping information for accessing physical memory (Paragraphs [0140] - [0146]; and Paragraph [0313] – Paragraph [0320]). Nag expressly teaches that a unified, coherent memory space allows a processor and an attached memory device to share resources and operate on the same memory region for higher performance, reduced data movement, and reduced software-stack complexity (Paragraph [0037]). The motivation for this combination of references would have been to apply Yoon's shared address-mapping mechanism to the neural network weight and intermediate-value data of the combined system to provide addressable access to such data within Nag's unified memory space, thereby reducing data movement and facilitating access to neural network data across the processing and memory components. This motivation is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Allowable Subject Matter
Claim(s) 4-10, 15-17, and 20 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Relevant Prior Art Directed to State of Art
Kuo et al (US 2024/0281636 A1) are relevant prior art not applied in the rejection(s) above. Kuo discloses a method for storing weight data used to compute node values during artificial intelligence (AI) inferencing operations conducted by a neural network, the method comprising: receiving a neural network definition defining a neural network having a plurality of layers, each layer comprising a plurality of neural network nodes, each neural network node using one or more weights to compute a result for the neural network node by applying a function to (i) the one or more weights and (ii) an input to the neural network node during an inferencing operation; determining, for a layer of the plurality of layers defined in the neural network definition, a set of weights to use to compute a neural network inferencing result for each neural network node of the plurality of neural network nodes in the layer; and storing the set of weights determined for the layer in a page of memory, such that any weights, of the set of weights, used to compute the neural network inferencing result for each neural network node of the plurality of neural network nodes in the layer are stored together in the page of memory for retrieval together.
Golov (US 11,574,659 B2) are relevant prior art not applied in the rejection(s) above. Golov discloses a system, comprising: a processing device; a plurality of memory regions configured to store data used by the processing device; and at least one controller coupled to the plurality of memory regions and configured to: read data from a first memory region of the plurality of memory regions, including reading first data from the first memory region, the first data for use by the processing device in processing associated with machine learning; and write data, received as output data from a computer model, to a second memory region of the plurality of memory regions, including writing second data to the second memory region so that the second data can be accessed sequentially when subsequently read from the second memory region by the processing device for use by the computer model; wherein reading the first data and writing the second data are performed in parallel.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONGBONG NAH whose telephone number is (571) 272-1361. The examiner can normally be reached M - F: 9:00 AM - 5:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ONEAL MISTRY can be reached on 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JONGBONG NAH/Examiner, Art Unit 2674
