DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The present application was filed on 10/12/2021.
This action is in response to arguments and/or filed on 12/30/2025. In the current amendments, claims1, 3-4, 6-8, 12 and 17 have been amended. Claims 1-20 are currently pending and have been examined.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/16/2023, 05/14/2023, 07/15/2023, 11/04/2024, 01/22/2025, 11/10/2025 and 05/12/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. KR10-2020-0144308, filed on11/02/2020.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-7, 11-15 and 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yu et al. (“OPU: An FPGA-Based Overlay Processor for Convolutional Neural Networks”) in view of Zheng et al. (“Optimizing Memory-Access Patterns for Deep Learning Accelerators”) and further in view of Peled et al. (“A Neural Network Prefetcher for Arbitrary Memory Access Patterns”)
Regarding claim 1 (Currently Amended)
Yu teaches a memory system of an artificial neural network (ANN), (abstract “updated. This is not easy for CNN end users. In this article, we propose a domain-specific FPGA overlay processor, named OPU to accelerate CNN networks. It offers software-like programmability for CNN end users, as CNN algorithms are automatically compiled into executable codes, which are loaded and executed by OPU without reconfiguration of FPGA for switch or update of CNN networks.”)
the memory system comprising: a neural network processing unit configured to process an ANN model; (abstract “updated. This is not easy for CNN end users. In this article, we propose a domain-specific FPGA overlay processor, named OPU to accelerate CNN networks. It offers software-like programmability for CNN end users, as CNN algorithms are automatically compiled into executable codes”)
and an ANN memory controller configured to control a rearrangement of data of the ANN model stored in a memory, (pg. 36 left col “FPGA-Based High-Performance Microarchitecture: These architectures are optimized for computation, data communication and reorganization, which are controlled by parameter registers set directly by instructions.” See reorganized corresponds to claim language rearrangement see pg. 37 left col “Memory Read transforms the data from external memory to onboard memory. It operates in two modes to accommodate for different data read patterns. Received data will be reorganized and distributed to three destination buffers corresponding to the feature map (FM), kernel weighs, and instructions, respectively”)
and operate the data of the ANN model stored in the memory in a read-burst mode based on ANN data …of the ANN model. (Section 3 on pg. 40 “Channelwise input concatenation is required for inception module. Taking advantage of our memory storage pattern, we manipulate output memory addresses of preceding layers to place their outputs adjacently. Therefore, input concatenation can be achieved without any extra computational operation or latency cost. The arranged memory can be loaded for the next layer in burst mode for efficient off-chip memory transmission.”)
Yu does not teach locality information.
Zheng teaches locality information (pg. 2 left col “Our approach tries to eliminate unnecessary data movements in the workload (Section 2.1), and for the remainder, maximizing the utilization of the on-chip memory by maintaining data locality in the scratchpad (Section 2.2). Our approach was designed for DL accelerators equipped with powerful compute units and limited on-chip memory.” Also see section 3 “Local mapping which generates mappings within each operator, without propagation, but keeps the output of an operator in on-chip memory if it will be directly used as the input of the next operator”).
Yu and Zheng are analogous art because they are both directed to machine learning.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combined FPGA based overlay processor for CNN of Yu with optimizing memory access patterns for deep learning accelerator of Zheng.
One of ordinary skill in the art would have been motivated to make this modification in order to “leverages the polyhedral model to analyze all operators of a DL model together to minimize the number of memory accesses” as disclosed by (Zheng abstract “This paper proposes a systematic approach which leverages the polyhedral model to analyze all operators of a DL model together to minimize the number of memory accesses. Experiments show that our approach can substantially reduce the impact of memory accesses required by common neural-network models on a homegrown AWS machine-learning inference chip named Inferentia, which is available through Amazon EC2 Inf1 instances.”).
Yu in view of Zheng does not teach wherein the ANN data locality information includes an operation sequence configured in units of memory access requests of the neural network processing unit.
Peled teaches wherein the ANN data locality information includes an operation sequence configured in units of memory access requests of the neural network processing unit. (Section 4 “We start with the basic patterns in Table 1 as a benchmark for real memory access streams. We also add several sequences based on functions that represent patterns of various complexities, as a proxy for more complicated access streams: a (shifted) sine function, a polynomial function, a linear line and a pseudo-random function (LFSR based). Each series is fed into the neural network a single value at a time, and the output is trained to provide the next sequential element.”)
Yu, Zheng and Peled are analogous art because they are all directed to machine learning.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combined FPGA based overlay processor for CNN of Yu in view of Zheng with neural network prefetcher for arbitrary memory access patterns of Peled.
One of ordinary skill in the art would have been motivated to make this modification in order to include “prefetcher correlates program and machine contextual information with memory accesses patterns, using online-training to identify and dynamically adapt to unique access patterns” as disclosed by (Peled abstract “Leveraging recent advances in machine learning, the proposed NN prefetcher correlates program and machine contextual information with memory accesses patterns, using online-training to identify and dynamically adapt to unique access patterns exhibited by the code. By targeting semantic locality in this manner, the prefetcher can discern the useful context attributes and learn to predict previously undetected access patterns, even within noisy memory access streams.”).
Regarding claim 2
Yu in view of Zheng with Peled teaches the memory system of claim 1.
Zheng further teaches wherein the ANN memory controller is further configured to receive pre-generated ANN data locality information (pg. 2 left col “Our approach tries to eliminate unnecessary data movements in the workload (Section 2.1), and for the remainder, maximizing the utilization of the on-chip memory by maintaining data locality in the scratchpad (Section 2.2). Our approach was designed for DL accelerators equipped with powerful compute units and limited on-chip memory.” Also see section 3 “Local mapping which generates mappings within each operator, without propagation, but keeps the output of an operator in on-chip memory if it will be directly used as the input of the next operator”).
Regarding claim 3
Yu in view of Zheng teaches the memory system of claim 1.
Yu further teaches wherein the neural network processing unit is further configured to generate a plurality of data access requests sequentially, (pg. 37 right col “As shown in Fig. 3, after the first instruction read, initial TCI0 is set at t0 for all three operations. At t1, memory access is triggered then executes for t1 − t2. Data fetch is triggered upon finishing memory operation and post process is trigged at t3. Next instruction read that updates TCI1 for Data fetch and post process can be performed at any time point between t3 and t7. Moreover, we store current TCI to avoid setting the same condition repeatedly when modules operate in one pattern consecutively (at time t0 and t5, memory access is triggered by the same TCI). This shortens the instruction sequence over 10×.” also see g. 38 “To address this issue, we explore higher level of parallelism and leave 2-D kernel being computed sequentially. Fig. 5(b) explains how it works: at each clock cycle, a slice of input channel of depth ICi p with width and height as 1 ∗ 1 is read along with corresponding kernel elements. This fits natural data storage pattern and requires much smaller bandwidth.”)
and wherein the ANN memory controller is further configured to generate the ANN data …by monitoring the plurality of data access requests. (Pg. 37 right col “As shown in Fig. 4, the overlay microarchitecture can be decomposed into six main modules following the instruction architecture definition. Each module can be controlled by instruction to accomplish functionalities defined in Section III-A. In addition, four storage buffers (i.e., input FM buffer, kernel buffer, instruction buffer, and output buffer) are placed to cache local data for fast access. With most of the control flow embedded in instruction, overlay only handles the computation of one sub-FM block.”)
Zheng further teaches locality information (pg. 2 left col “Our approach tries to eliminate unnecessary data movements in the workload (Section 2.1), and for the remainder, maximizing the utilization of the on-chip memory by maintaining data locality in the scratchpad (Section 2.2). Our approach was designed for DL accelerators equipped with powerful compute units and limited on-chip memory.” Also see section 3 “Local mapping which generates mappings within each operator, without propagation, but keeps the output of an operator in on-chip memory if it will be directly used as the input of the next operator”).
Regarding claim 4 (Currently Amended)
Yu in view of Zheng with Peled teaches the memory system of claim 1.
Yu further teaches wherein the ANN memory controller is further configured to control communication between the neural network processing unit and the memory in which the data of the ANN model is stored. (Pg. 39 section C “Another key point in memory management is data storage format in both local onboard buffer and external memory. For onboard storage format, data from the same channel slice are stored under the same address so that they can be fetched in one clock cycle. The limit of channel slice depth is set to be the width of on-chip buffer to guarantee such memory arrangement. Benefited from computation pattern, enough data bandwidth is provided and no extra memory latency is caused during the computing stage.”)
Regarding claim 5
Yu in view of Zheng with Peled teaches the memory system of claim 1.
Yu further teaches wherein the ANN memory controller is further configured to rearrange the data of the ANN model stored in the memory in a forward direction based on the ANN data locality information. (Pg. 41 section C “IR contains all the operations included in the current layer groups. Layer index is the sequential number assigned to each conventional layer. Single layer group may have multiple layer indexes for input in the case of inception module, where various previous outputted FMs are concatenated to form the input”)
Regarding claim 6 (Currently Amended)
Yu in view of Zheng with Peled teaches the memory system of claim 1.
Yu further teaches wherein the neural network processing unit is further configured to generate a plurality of data access requests sequentially, each of the plurality of data access requests including a memory address of the memory, (section B “The data fetch module reads FM and kernel data from on-chip buffer, rearranges the data and then sends to the computation unit. As shown in Fig. 7, for input FM read, FM ADDR GEN takes control parameters from instruction and produce FM buffer read address at each clock cycle. The parameters include [Xmin, Xmax, Ymin, Ymax, Xstride, Y stride, Xsize, Y size]. The FM data read from buffer will be selected and copied by the FM REARR to fit the target computation pair of computation unit. For kernel weights, the bandwidth requirement can be as high as 8192 bit/cycle, since all the parallelisms are explored on the kernel side (input channel/output channel).”)
and wherein the ANN memory controller is further configured to rearrange the data of the ANN model by monitoring the memory addresses of the plurality of data access requests. (Section B “The data fetch module reads FM and kernel data from on-chip buffer, rearranges the data and then sends to the computation unit. As shown in Fig. 7, for input FM read, FM ADDR GEN takes control parameters from instruction and produce FM buffer read address at each clock cycle. The parameters include [Xmin, Xmax, Ymin, Ymax, Xstride, Y stride, Xsize, Y size]. The FM data read from buffer will be selected and copied by the FM REARR to fit the target computation pair of computation unit. For kernel weights, the bandwidth requirement can be as high as 8192 bit/cycle, since all the parallelisms are explored on the kernel side (input channel/output channel). We choose a computation pattern that shares the same set of kernel weights during one round of computation. Accordingly, kernel weights are only preloaded once from buffer for each round. We use W PRE-LOAD ADDR GEN to generate weights address for buffer, and each weight takes 32 cycles to load. This loading time is overlapped with the previous round of data fetch process. A pair of local shift registers using ping-pong structure [W SHIFT REG SET 1, W SHIFT REG SET 2] is used to cache the weights.”)
Regarding claim 7(Currently Amended)
Yu teaches a memory system of an artificial neural network (ANN), (abstract “updated. This is not easy for CNN end users. In this article, we propose a domain-specific FPGA overlay processor, named OPU to accelerate CNN networks. It offers software-like programmability for CNN end users, as CNN algorithms are automatically compiled into executable codes, which are loaded and executed by OPU without reconfiguration of FPGA for switch or update of CNN networks.”)
the memory system comprising: a neural network processing unit configured to generate a data access request for processing a neural network model; (abstract “updated. This is not easy for CNN end users. In this article, we propose a domain-specific FPGA overlay processor, named OPU to accelerate CNN networks. It offers software-like programmability for CNN end users, as CNN algorithms are automatically compiled into executable codes”)
an ANN memory controller configured to generate a memory access request corresponding to the data access request based on ANN data …information of the ANN model; (pg. 36 left col “FPGA-Based High-Performance Microarchitecture: These architectures are optimized for computation, data communication and reorganization, which are controlled by parameter registers set directly by instructions.” See reorganized corresponds to claim language rearrangement see pg. 37 left col “Memory Read transforms the data from external memory to onboard memory. It operates in two modes to accommodate for different data read patterns. Received data will be reorganized and distributed to three destination buffers corresponding to the feature map (FM), kernel weighs, and instructions, respectively”)
and a memory configured to provide data corresponding to the memory access request to the ANN controller in a read-burst mode based on the ANN data locality information. (Section 3 on pg. 40 “Channelwise input concatenation is required for inception module. Taking advantage of our memory storage pattern, we manipulate output memory addresses of preceding layers to place their outputs adjacently. Therefore, input concatenation can be achieved without any extra computational operation or latency cost. The arranged memory can be loaded for the next layer in burst mode for efficient off-chip memory transmission.”)
Yu does not teach locality information.
Zheng teaches locality information (pg. 2 left col “Our approach tries to eliminate unnecessary data movements in the workload (Section 2.1), and for the remainder, maximizing the utilization of the on-chip memory by maintaining data locality in the scratchpad (Section 2.2). Our approach was designed for DL accelerators equipped with powerful compute units and limited on-chip memory.” Also see section 3 “Local mapping which generates mappings within each operator, without propagation, but keeps the output of an operator in on-chip memory if it will be directly used as the input of the next operator”).
Yu and Zheng are analogous art because they are both directed to machine learning.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combined FPGA based overlay processor for CNN of Yu with optimizing memory access patterns for deep learning accelerator of Zheng.
One of ordinary skill in the art would have been motivated to make this modification in order to “leverages the polyhedral model to analyze all operators of a DL model together to minimize the number of memory accesses” as disclosed by (Zheng abstract “This paper proposes a systematic approach which leverages the polyhedral model to analyze all operators of a DL model together to minimize the number of memory accesses. Experiments show that our approach can substantially reduce the impact of memory accesses required by common neural-network models on a homegrown AWS machine-learning inference chip named Inferentia, which is available through Amazon EC2 Inf1 instances.”).
Yu in view of Zheng does not teach wherein the ANN data locality information includes an operation sequence configured in units of memory access requests of the neural network processing unit.
Peled teaches wherein the ANN data locality information includes an operation sequence configured in units of memory access requests of the neural network processing unit. (Section 4 “We start with the basic patterns in Table 1 as a benchmark for real memory access streams. We also add several sequences based on functions that represent patterns of various complexities, as a proxy for more complicated access streams: a (shifted) sine function, a polynomial function, a linear line and a pseudo-random function (LFSR based). Each series is fed into the neural network a single value at a time, and the output is trained to provide the next sequential element.”)
Yu, Zheng and Peled are analogous art because they are all directed to machine learning.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combined FPGA based overlay processor for CNN of Yu in view of Zheng with neural network prefetcher for arbitrary memory access patterns of Peled.
One of ordinary skill in the art would have been motivated to make this modification in order to include “prefetcher correlates program and machine contextual information with memory accesses patterns, using online-training to identify and dynamically adapt to unique access patterns” as disclosed by (Peled abstract “Leveraging recent advances in machine learning, the proposed NN prefetcher correlates program and machine contextual information with memory accesses patterns, using online-training to identify and dynamically adapt to unique access patterns exhibited by the code. By targeting semantic locality in this manner, the prefetcher can discern the useful context attributes and learn to predict previously undetected access patterns, even within noisy memory access streams.”).
Regarding claim 11
Yu in view of Zheng with Peled teaches claim 7.
Yu further teaches wherein the ANN memory controller is further configured to set a specific memory area of the memory for the read-burst mode based on the ANN data …information. (Section 3 on pg. 40 “Channelwise input concatenation is required for inception module. Taking advantage of our memory storage pattern, we manipulate output memory addresses of preceding layers to place their outputs adjacently. Therefore, input concatenation can be achieved without any extra computational operation or latency cost. The arranged memory can be loaded for the next layer in burst mode for efficient off-chip memory transmission.”)
Zheng further teaches locality information (pg. 2 left col “Our approach tries to eliminate unnecessary data movements in the workload (Section 2.1), and for the remainder, maximizing the utilization of the on-chip memory by maintaining data locality in the scratchpad (Section 2.2). Our approach was designed for DL accelerators equipped with powerful compute units and limited on-chip memory.” Also see section 3 “Local mapping which generates mappings within each operator, without propagation, but keeps the output of an operator in on-chip memory if it will be directly used as the input of the next operator”).
Regarding claim 12 (Currently Amended)
Claim 12 recites analogous limitations to independent claim 7 and therefore is rejected on the same ground as independent claim 7.
Regarding claim 13
Yu in view of Zheng with Peled teaches claim 12.
Yu further teaches wherein the ANN memory controller includes a cache memory, (pg. 37 right col “In addition, four storage buffers (i.e., input FM buffer, kernel buffer, instruction buffer, and output buffer) are placed to cache local data for fast access. With most of the control flow embedded in instruction, overlay only handles the computation of one sub-FM block”)
and wherein the cache memory is configured to store the data provided by the read-burst mode. (Section D pg. 40 “Channelwise input concatenation is required for inception module. Taking advantage of our memory storage pattern, we manipulate output memory addresses of preceding layers to place their outputs adjacently. Therefore, input concatenation can be achieved without any extra computational operation or latency cost. The arranged memory can be loaded for the next layer in burst mode for efficient off-chip memory transmission.”)
Regarding claim 14
Yu in view of Zheng with Peled teaches the memory system of claim 12.
Yu further teaches wherein the ANN memory controller includes a cache memory, (pg. 37 right col “In addition, four storage buffers (i.e., input FM buffer, kernel buffer, instruction buffer, and output buffer) are placed to cache local data for fast access. With most of the control flow embedded in instruction, overlay only handles the computation of one sub-FM block”)
and wherein the cache memory is configured to store a weight value corresponding to the ANN data locality information of the ANN model. (Section B on pg. 39 “We choose a computation pattern that shares the same set of kernel weights during one round of computation. Accordingly, kernel weights are only preloaded once from buffer for each round. We use W PRE-LOAD ADDR GEN to generate weights address for buffer, and each weight takes 32 cycles to load. This loading time is overlapped with the previous round of data fetch process. A pair of local shift registers using ping-pong structure [W SHIFT REG SET 1, W SHIFT REG SET 2] is used to cache the weights.”)
Regarding claim 15
Yu in view of Zheng with Peled teaches the memory system of claim 12.
Zheng further teaches wherein the at least one memory includes a plurality of memories, and wherein the ANN memory controller is further configured to distribute and store the data of the ANN model in the plurality of memories. (Section 2.2 “In order to maximize the internal memory bandwidth, accelerators typically organize on-chip memories into multiple banks with disjoint address spaces, each of which connects to one portion of the compute units (e.g., a specific row of the systolic array). Data movement between different banks is very slow through the main memory; therefore, tensor data needs to be carefully spread across the banks for computation. For example, in a Conv2D operator, data from different channels of the feature map and weights must be mapped to different memory banks so that the internal compute units can read and process the data in parallel.”)
Regarding claim 17 (Currently Amended)
Yu in view of Zheng with Peled teaches the memory system of claim 12.
Zheng further teaches wherein the ANN memory controller is further configured to obtain mapping data in which memory access requests corresponding to data access requests generated by the processor are mapped to each other based on the ANN data locality information. (Pg. 2 left col “Our approach tries to eliminate unnecessary data movements in the workload (Section 2.1), and for the remainder, maximizing the utilization of the on-chip memory by maintaining data locality in the scratchpad (Section 2.2). Our approach was designed for DL accelerators equipped with powerful compute units and limited on-chip memory.”)
Regarding claim 18
Yu in view of Zheng with Peled teaches the memory system of claim 12.
Yu further teaches wherein the ANN memory controller is further configured to rearrange the data of the ANN model stored in the at least one memory based on the ANN data …information. (pg. 36 left col “FPGA-Based High-Performance Microarchitecture: These architectures are optimized for computation, data communication and reorganization, which are controlled by parameter registers set directly by instructions.” See reorganized corresponds to claim language rearrangement see pg. 37 left col “Memory Read transforms the data from external memory to onboard memory. It operates in two modes to accommodate for different data read patterns. Received data will be reorganized and distributed to three destination buffers corresponding to the feature map (FM), kernel weighs, and instructions, respectively”)
Zheng teaches locality information (pg. 2 left col “Our approach tries to eliminate unnecessary data movements in the workload (Section 2.1), and for the remainder, maximizing the utilization of the on-chip memory by maintaining data locality in the scratchpad (Section 2.2). Our approach was designed for DL accelerators equipped with powerful compute units and limited on-chip memory.” Also see section 3 “Local mapping which generates mappings within each operator, without propagation, but keeps the output of an operator in on-chip memory if it will be directly used as the input of the next operator”).
Regarding claim 19
Yu in view of Zheng with Peled teaches the memory system of claim 12.
Yu further teaches wherein the at least one memory includes a volatile or a non-volatile memory having the read-burst mode. (Section D pg. 40 “Channelwise input concatenation is required for inception module. Taking advantage of our memory storage pattern, we manipulate output memory addresses of preceding layers to place their outputs adjacently. Therefore, input concatenation can be achieved without any extra computational operation or latency cost. The arranged memory can be loaded for the next layer in burst mode for efficient off-chip memory transmission.”)
Regarding claim 20
Yu in view of Zheng with Peled teaches the memory system of claim 12.
Yu further teaches wherein the ANN memory controller is further configured to rearrange the data of the ANN model stored in the at least one memory so as to optimize for the read-burst mode, (pg. 36 left col “FPGA-Based High-Performance Microarchitecture: These architectures are optimized for computation, data communication and reorganization, which are controlled by parameter registers set directly by instructions.” See reorganized corresponds to claim language rearrangement see pg. 37 left col “Memory Read transforms the data from external memory to onboard memory. It operates in two modes to accommodate for different data read patterns. Received data will be reorganized and distributed to three destination buffers corresponding to the feature map (FM), kernel weighs, and instructions, respectively”… section 3 on pg. 40 “Channelwise input concatenation is required for inception module. Taking advantage of our memory storage pattern, we manipulate output memory addresses of preceding layers to place their outputs adjacently. Therefore, input concatenation can be achieved without any extra computational operation or latency cost. The arranged memory can be loaded for the next layer in burst mode for efficient off-chip memory transmission.”)
…
and update the ANN data …information of the ANN model to correspond to the rearranged data. (Pg. 37 right col “As shown in Fig. 3, after the first instruction read, initial TCI0 is set at t0 for all three operations. At t1, memory access is triggered then executes for t1 − t2. Data fetch is triggered upon finishing memory operation and post process is trigged at t3. Next instruction read that updates TCI1 for Data fetch and post process can be performed at any time point between t3 and t7. Moreover, we store current TCI to avoid setting the same condition repeatedly when modules operate in one pattern consecutively (at time t0 and t5, memory access is triggered by the same TCI).” also see pg. 36 left col “FPGA-Based High-Performance Microarchitecture: These architectures are optimized for computation, data communication and reorganization, which are controlled by parameter registers set directly by instructions.” See reorganized corresponds to claim language rearrangement see pg. 37 left col “Memory Read transforms the data from external memory to onboard memory. It operates in two modes to accommodate for different data read patterns. Received data will be reorganized and distributed to three destination buffers corresponding to the feature map (FM), kernel weighs, and instructions, respectively”
Zheng teaches based on the ANN data locality information of the ANN model, (pg. 2 left col “Our approach tries to eliminate unnecessary data movements in the workload (Section 2.1), and for the remainder, maximizing the utilization of the on-chip memory by maintaining data locality in the scratchpad (Section 2.2). Our approach was designed for DL accelerators equipped with powerful compute units and limited on-chip memory.” Also see section 3 “Local mapping which generates mappings within each operator, without propagation, but keeps the output of an operator in on-chip memory if it will be directly used as the input of the next operator”).
Claim(s) 8-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yu et al. (“OPU: An FPGA-Based Overlay Processor for Convolutional Neural Networks”) in view of Zheng et al. (“Optimizing Memory-Access Patterns for Deep Learning Accelerators”) in view of Peled et al. and further in view of Roine et al. (US 2016/0163379 A1).
Regarding claim 8 (Currently Amended)
Yu in view of Zheng with Peled teaches the memory system of claim 7.
Yu in view of Zheng with Peled does not teach wherein the processor is further configured to generate a plurality of data access requests sequentially, and wherein the ANN memory controller is further configured to determine whether the plurality of data access requests are operable in the read-burst mode based on memory addresses of the memory corresponding to the plurality of data access requests.
Roine teaches wherein the processor is further configured to generate a plurality of data access requests sequentially, (para [0044] “FIG. 7 is a timing diagram of an SRAM device incorporating an aspect of the application. In FIG. 7, two sequential SRAM read accesses are illustrated. In the novel methods, as sequential memory cell accesses are performed, the word line firing, and the precharge operation are no longer necessarily performed in every memory access cycle.”)
and wherein the ANN memory controller is further configured to determine whether the plurality of data access requests are operable in the read-burst mode based on memory addresses of the memory corresponding to the plurality of data access requests. (Para [0050] “FIG. 11 depicts, in a table, an example arrangement of signals that can be used to implement the arrangements described above. However, the various methods of this application are not limited to this example implementation or any particular device architecture. In FIG. 11, an input control signal to an SRAM device is labeled “BM for “burst mode enable.” For an SRAM access, this signal indicates that a sequence of SRAM accesses will be performed so that the precharge operations, and word line firing operations, are not to be performed for each SRAM access as in the conventional SRAM accesses”)
Yu, Zheng, Peled and Roine are analogous art because they are all directed to computer system.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combined FPGA based overlay processor for CNN of Yu in view of Zheng and Peled to include a method for performance optimization of SRAM memory of Roine.
One of ordinary skill in the art would have been motivated to make this modification in order to increase the throughput of data transfer as disclosed by (Roine para [0050] “FIG. 11 depicts, in a table, an example arrangement of signals that can be used to implement the arrangements described above. However, the various methods of this application are not limited to this example implementation or any particular device architecture. In FIG. 11, an input control signal to an SRAM device is labeled “BM for “burst mode enable.” For an SRAM access, this signal indicates that a sequence of SRAM accesses will be performed so that the precharge operations, and word line firing operations, are not to be performed for each SRAM access as in the conventional SRAM accesses”).
Regarding claim 9
Yu in view of Zheng with Peled and Roine teaches the memory system of claim 8.
Roine further teaches wherein, if it is determined that the memory cannot operate in the read-burst mode, the ANN memory controller is further configured to store data corresponding to the plurality of data access requests in memory addresses of the memory, the memory addresses enabling the read-burst mode. (Para [0050] “FIG. 11 depicts, in a table, an example arrangement of signals that can be used to implement the arrangements described above. However, the various methods of this application are not limited to this example implementation or any particular device architecture. In FIG. 11, an input control signal to an SRAM device is labeled “BM for “burst mode enable.” For an SRAM access, this signal indicates that a sequence of SRAM accesses will be performed so that the precharge operations, and word line firing operations, are not to be performed for each SRAM access as in the conventional SRAM accesses”).
Regarding claim 10
Yu in view of Zheng with Peled and Roine teaches the memory system of claim 8.
Roine further teaches wherein the memory addresses of the memory include a first memory address corresponding to a data access request of the plurality of data access requests and a second memory address enabling operation of the read-burst mode, (para [0050] “FIG. 11 depicts, in a table, an example arrangement of signals that can be used to implement the arrangements described above. However, the various methods of this application are not limited to this example implementation or any particular device architecture. In FIG. 11, an input control signal to an SRAM device is labeled “BM for “burst mode enable.” For an SRAM access, this signal indicates that a sequence of SRAM accesses will be performed so that the precharge operations, and word line firing operations, are not to be performed for each SRAM access as in the conventional SRAM accesses”).
and wherein the ANN memory controller is further configured to exchange data stored in the first memory address and data stored in the second memory address. (Claim 4 “wherein the memory controller circuit further comprises precharge mode control circuitry operable to output; a burst mode enable signal to the SRAM circuit indicating that a next memory address to be accessed is to a memory cell along the same row as the current row corresponding to the current address input to the SRAM circuit; a precharge first modesignal to the SRAM cir cuit when a sequence of SRAM memory accesses will occur along the same row of SRAM cells”)
Claim(s) 16 is rejected under 35 U.S.C. 103 as being unpatentable over Yu et al. (“OPU: An FPGA-Based Overlay Processor for Convolutional Neural Networks”) in view of Zheng et al. (“Optimizing Memory-Access Patterns for Deep Learning Accelerators”) in view of Peled et al. and further in view of Yahata et al. (US 2003/0031073 A1).
Regarding claim 16
Yu in view of Zheng with Peled teaches the memory system of claim 12.
Yu in view of Zheng with Peled does not teach wherein the ANN memory controller is further configured to control a refresh timing of a specific global bit line of the at least one memory, based on the ANN data locality information of the ANN model and a memory address at which the data of the ANN model is stored.
Yahata teaches wherein the ANN memory controller is further configured to control a refresh timing of a specific global bit line of the at least one memory, based on the ANN data locality information of the ANN model and a memory address at which the data of the ANN model is stored. (Para [0044] “A simplified timing chart for describing one example of the operation of the embodiment shown in FIG. 1 is shown in FIG. 2. The same drawing shows an example in which an internal refresh request is detected earlier than the transition of an address Signal AR. After the refresh operation, i.e., after a bit line pair BL and BLB is changed to a high level and a low level according to information Stored in each memory cell under the Selection of a word line WL by Refresh and the operation of a sense amplifier, and hence Such a refresh operation as described above is executed, the word line WL is temporarily brought to a non-Selected State of a low level”)
Yu, Zheng, Peled and Yahata are analogous art because they are all directed to computer system.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combined FPGA based overlay processor for CNN of Yu in view of Zheng and Peled to include method for control a refresh timing of a specific global bit line of Yahata.
One of ordinary skill in the art would have been motivated to make this modification in order to detect transition of an address early as disclosed by (Para [0044] “A simplified timing chart for describing one example of the operation of the embodiment shown in FIG. 1 is shown in FIG. 2. The same drawing shows an example in which an internal refresh request is detected earlier than the transition of an address Signal AR. After the refresh operation, i.e., after a bit line pair BL and BLB is changed to a high level and a low level according to information Stored in each memory cell under the Selection of a word line WL by Refresh and the operation of a sense amplifier, and hence Such a refresh operation as described above is executed, the word line WL is temporarily brought to a non-Selected State of a low level”)
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VAN C MANG whose telephone number is (571)270-7598. The examiner can normally be reached Mon - Fri 8:00-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at 5712707519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VAN C MANG/Primary Examiner, Art Unit 2126
Prosecution Insights