Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claims 1-19 are presented for examination in this application, 17/705,068 filed 2022-03-25, having an effective filing date of 2021-09-24 via foreign priority of application KR10-2021-0126490.
The Examiner cites particular sections in the references as applied to the claims
below for the convenience of the applicant(s). Although the specified citations are
representative of the teachings in the art and are applied to the specific limitations within
the individual claim, other passages and figures may apply as well. It is respectfully
requested that, in preparing responses, the applicant(s) fully consider the references in
their entirety as potentially teaching all or part of the claimed invention, as well as the
context of the passage as taught by the prior art or disclosed by the Examiner.
Drawings
The drawings submitted on 2022-03-25 have been considered and accepted.
Information Disclosure Statement
Acknowledgement is made of the information disclosure statements filed 2022-03-25 and 2025-04-17. All patents and non-patent literature have been considered.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
The following appears to be the closest portions of the specification corresponding to the 35 U.S.C 112(f) invocations:
controller
[0066]-[0067] The controller 320 may control the overall operation of the neural network processor 300. The controller 320 may set and 20 manage parameters related to the neural network computation, so
that the in-memory computation device 310 may normally perform the neural network computation. The controller 320 may be implemented in the form of a combination of hardware, software, and
software (or firmware) executed on hardware. The controller 320 may be implemented with at least one processor, for example, a central processing unit (CPU), a microprocessor, or the like, and may execute instructions stored in the RAM 330 in order to implement various functions.
computation device
[0065] Referring to FIG. 5, the neural network processor 300 may be a processor or an accelerator specialized for a neural network
computation, and may include an in-memory computation device 310,
updater
[0079] Referring to FIG. 6, the pooling controller 500 may include a
buffer allocator 510, a pooling map configuration circuit 520, and an
updater 530; and [0082] The updater 530 may provide the POOL 319 of FIG. 5 with the pooling elements sequentially outputted from the computation memory 311, and update the pooling value storage unit of the global 20buffer 313 according to a pooling result for each pooling window
data processing system
[0021] The computing system 10 may include a host device 100 and a data processing system 200 that processes the computation of
an application requested by the host device 100; [0030]-[0031] In an embodiment, the data processing system 200 may be an application processor mounted in a mobile device. The data processing system 200 may include at least a neural network processor 300.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-19 are rejected under 35 U.S.C 101 as being unpatentable because the claimed invention in these claims is directed to an abstract idea without significantly more. The analysis of the claims will follow the 2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg 50-57 (January 7, 2019) (“2019 PEG”).
Regarding claim 1:
Step 1 – Is the claim directed to a process, machine, manufacture, or a composition of matter?
Yes, the claim is directed to a machine.
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites abstract ideas:
a computation device including a storage unit allocated to each of a plurality of integration groups, and configured to perform a convolution operation on the input feature map and the weight filter — this limitation amounts to the abstract idea of mathematical calculations (see MPEP 2106.04(a)(2) I. C.).
perform a pooling operation on the plurality of pooling elements, the plurality of pooling elements corresponding to each of the plurality of integration groups — this limitation amounts to the abstract idea of mathematical calculations (see MPEP 2106.04(a)(2) I. C.).
integrating a pooling value read from the storage unit and each of the plurality of pooling elements into a single value — this limitation amounts to the abstract idea of mathematical calculations (see MPEP 2106.04(a)(2) I. C.).
wherein the computation device is configured to perform the pooling operation by: integrating a pooling value read from the storage unit and each of the plurality of pooling elements into a single value — this limitation amounts to the abstract idea of mathematical calculations (see MPEP 2106.04(a)(2) I. C.).
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim recites additional elements that do not integrate the judicial exception into a practical application:
a data processing system comprising: a controller configured to receive a request including an input feature map and a weight filter from a host device, wherein the request is for processing a neural network computation — this limitation is directed to mere data gathering and outputting which has been recognized by the courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) as insignificant extra-solution activity (see MPEP 2106.05(g))).
sequentially output a plurality of pooling elements as a result of the convolution operation — this limitation is directed to mere data gathering and outputting which has been recognized by the courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) as insignificant extra-solution activity (see MPEP 2106.05(g))).
updating the pooling value stored in the storage unit with a result of the integrating, wherein the integrating and the updating are repeated until all of the plurality of pooling elements are integrated — this limitation amounts to mere instructions to apply an exception, as the use of a computer or other machinery in its ordinary capacity amounts to invoking computer components merely as a tool to perform an existing process (see MPEP 2106.05(f)(2)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception. Any additional elements that were determined to be insignificant extra-solution activities in step 2A prong 2 are further evaluated in step 2B on whether they are well-understood, routine, and conventional activities. The “a data processing system comprising: a controller configured to receive a request including an input feature map and a weight filter from a host device, wherein the request is for processing a neural network computation” and “sequentially output a plurality of pooling elements as a result of the convolution operation” limitations were found to be insignificant extra-solution activities in claim 1. These limitations are recited at a high level of generality and amounts to transmitting data over a network, which is a well-understood, routine, and conventional activity (see MPEP 2106.05(d) II.). Thus, the claim is not patent eligible.
Independent claims 10 and 15 are analogous claims, therefore the same rationale and rejection applies to them.
Regarding claim 2:
This claim recites a machine learning process recited at a high level of generality, which amounts to mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Claim 16 is analogous.
Regarding claim 3:
This claim recites a machine learning process recited at a high level of generality, which amounts to mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Claims 12 and 17 are analogous.
Regarding claim 4:
This claim recites a machine learning process recited at a high level of generality, which amounts to mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Claims 8, 13, and 18 are analogous.
Regarding claim 5:
This claim recites a machine learning process recited at a high level of generality, which amounts to mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Claims 9, 14, and 19 are analogous.
Regarding claim 6:
Step 1 – Is the claim directed to a process, machine, manufacture, or a composition of matter?
Yes, the claim is directed to a machine.
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites abstract ideas:
sequentially perform a convolution operation on the weight filter and each of a plurality of division maps included in the input feature map — this limitation amounts to the abstract idea of mathematical calculations (see MPEP 2106.04(a)(2) I. C.).
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim recites additional elements that do not integrate the judicial exception into a practical application:
a data processing system comprising: a computation memory configured to receive a request including an input feature map and a weight filter from a host device — this limitation is directed to mere data gathering and outputting which has been recognized by the courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) as insignificant extra-solution activity (see MPEP 2106.05(g))).
sequentially output each of a plurality of pooling elements as a result of the convolution operation — this limitation is directed to mere data gathering and outputting which has been recognized by the courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) as insignificant extra-solution activity (see MPEP 2106.05(g))).
a global buffer including a storage unit allocated to each of a plurality of integration groups, the plurality of pooling elements corresponding to each of the plurality of integration groups — this limitation amounts to mere instructions to apply an exception, as the use of a computer or other machinery in its ordinary capacity amounts to invoking computer components merely as a tool to perform an existing process (see MPEP 2106.05(f)(2)).
a pooling controller configured to receive each of the plurality of pooling elements from the computation memory, read out a pooling value from the storage unit, and provide said each of the plurality of pooling elements and the pooling value to a pooler — this limitation is directed to mere data gathering and outputting which has been recognized by the courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) as insignificant extra-solution activity (see MPEP 2106.05(g))).
the pooler configured to integrate said each of the plurality of pooling elements and the pooling value into a single value, so that the pooling value stored in the storage unit is updated with a result of the integrating, wherein the integrating and the updating are repeated until all of the plurality of pooling elements are integrated — this limitation amounts to mere instructions to apply an exception, as the use of a computer or other machinery in its ordinary capacity amounts to invoking computer components merely as a tool to perform an existing process (see MPEP 2106.05(f)(2)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception. Any additional elements that were determined to be insignificant extra-solution activities in step 2A prong 2 are further evaluated in step 2B on whether they are well-understood, routine, and conventional activities. The “a data processing system comprising: a computation memory configured to receive a request including an input feature map and a weight filter from a host device”, “sequentially output each of a plurality of pooling elements as a result of the convolution operation”, and “a pooling controller configured to receive each of the plurality of pooling elements from the computation memory, read out a pooling value from the storage unit, and provide said each of the plurality of pooling elements and the pooling value to a pooler” limitations were found to be insignificant extra-solution activities in claim 6. These limitations are recited at a high level of generality and amounts to transmitting data over a network, which is a well-understood, routine, and conventional activity (see MPEP 2106.05(d) II.). Thus, the claim is not patent eligible.
Regarding claim 7:
This claim recites a machine learning process recited at a high level of generality, which amounts to mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)).
Regarding claim 11:
This claim recites a machine learning process recited at a high level of generality, which amounts to mere instructions to apply the judicial exception on a computer (see MPEP 2106.05(f)). Claims 9, 14, and 19 are analogous.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 5, 10-11, 14-15, and 19 are rejected under 35 U.S.C 102 as being unpatentable over Holm et al. (US20210295140A1 hereinafter referred to as Holm).
Regarding claim 1:
Holm teaches a data processing system comprising: a controller configured to receive a request including an input feature map and a weight filter from a host device (see para [0160]: “The data processing system comprises an interconnect 6 which provides an interface between the various processors (ISP, CPU, GPU and NPU) and a memory controller 7. The memory controller 7 is operable to manage memory read and write requests, and to control reading and writing of data to off-chip memory 8.”. Also see para [0063]: “A convolution or pooling operation of the technology described herein should, and in an embodiment does, operate on an input data array (an “input feature map” (IFM)) to produce an output data array (an “output feature map” (OFM)).”.) Also see para [0175]: “A convolution (filter) kernel may be an array of data elements, with each data element of the array representing a “weight”.”. Also see para [0146]: “The neural network processor of the technology described herein is in an embodiment implemented in and as part of an overall data processing system that (further) includes one or more of: a host processor (central processing unit (CPU)) … ).
wherein the request is for processing a neural network computation (see claim 1: “A neural network processor operable to execute a neural network, the neural network processor comprising: a combined convolution and pooling circuit configured to perform convolution and pooling operations for a neural network being executed by the processor; the combined convolution and pooling circuit comprising: a multiply circuit configured to determine a product of data values …”.);
and a computation device including storage unit allocated to each of a plurality of integration groups (see para [0054]: “As will be explained in more detail below, this can enable the combined convolution and pooling circuit to perform “maximum pooling” operations by the maximum circuit determining a maximum value of input feature map data values, and storing the determined maximum value in the storage.”),
and configured to perform a convolution operation on the input feature map and the weight filter (see para [0071]: “It is in an embodiment a discrete convolution operation performed between an input data array (feature map) and a convolution (filter) kernel to generate an output data array (feature map).”),
sequentially output a plurality of pooling elements as a result of the convolution operation (see para [0065]: “The combined convolution and pooling circuit can receive data of an input data array (feature map) for neural network processing in any suitable manner. In an embodiment, the combined convolution and pooling circuit further comprises an input circuit configured to read data values of an input data array from memory (and to provide the read data values to other circuits of the combined convolution and pooling circuit for processing as appropriate).”),
and perform a pooling operation on the plurality of pooling elements (see para [0065]: “The combined convolution and pooling circuit can receive data of an input data array (feature map) for neural network processing in any suitable manner. In an embodiment, the combined convolution and pooling circuit further comprises an input circuit configured to read data values of an input data array from memory (and to provide the read data values to other circuits of the combined convolution and pooling circuit for processing as appropriate).”),
the plurality of pooling elements corresponding to each of the plurality of integration groups (see para [0087]: “For example, and in an embodiment, an input data array (feature map) may be divided into non-overlapping pooling windows that each encompass a respective 2×2 block of data elements of the input data array (feature map). In this case, each value of the output data array (feature map) will be determined from the values for a respective 2×2 block of data elements of the input data array (feature map), and thus the pooling operation will operate to reduce the size of the input data array (feature map) in two dimensions by a factor of two.”[(Examiner’s note: integration groups as defined by the applicant’s spec in [0052] can be understood by a person having ordinary skill in the art using broadest reasonable interpretation in light of the specification as the output of the pooling operation wherein the integration group could be interpreted as the single value for that space that comes from methods such as max pooling and average pooling.)]),
wherein the computation device is configured to perform the pooling operation by: integrating a pooling value read from the storage unit and each of the plurality of pooling elements into a single value (see para [0088]: “A pooling operation may operate on the values for data elements which are encompassed by a respective pooling window in any suitable manner, to e.g., “summarise” the data values that fall within the respective pooling window. For example, in the case of “average pooling”, a pooling operation may operate to determine a mean of data values that fall within a respective pooling window. In the case of “maximum pooling”, a pooling operation may operate to determine a maximum value of data values that fall within a respective pooling window.”);
and updating the pooling value stored in the storage unit with a result of the integrating (see para [0090]: “As discussed above, to facilitate average pooling operations the add and divisor circuits of the combined convolution and pooling circuit can, and in an embodiment do, operate in conjunction with the storage to perform average pooling operations.”),
wherein the integrating and the updating are repeated until all of the plurality of pooling elements are integrated (see para [0176]: “The output feature map may be a suitable array of data elements. Calculating the output feature map may involve a kernel “sliding” (convolving) over an input feature map, such that the kernel is positioned in a different location with respect to the input feature map for each element of the output feature map. At each location that the kernel is positioned with respect to the input feature map, the product of each weight value of the kernel with the value of the corresponding element of the input feature map (i.e. the element wise multiplication) may be calculated, and the sum of the products may be used as the corresponding output value for the output feature map. Where the input feature map comprises multiple channels, the convolution operation may typically be such that output values represent a sum over all of the channels.”).
Regarding claim 5:
Holm teaches the system of claim 1.
Holm teaches wherein the integrating includes determining, as the single value, a maximum value between the pooling value and said each of the plurality of pooling elements (see para [0006]: “A pooling typically operation operates to “downsample” or “summarise” an input array of data. For example, an “average pooling” operation can summarise input array data values by determining their mean, and a “maximum pooling” operation can summarise input array data values by determining their maximum”)
or an average value of the pooling value and said each of the plurality of pooling elements (see abstract: “The circuit can perform an average pooling operation by the add circuit accumulating input feature map data values in the storage, a divisor circuit determining a divisor value, and a division circuit dividing the data value accumulated in the storage by the determined divisor value.”)
, and
wherein the updating includes updating the pooling value stored in the storage unit with the maximum value or the average value (see abstract: “The circuit can perform an average pooling operation by the add circuit accumulating input feature map data values in the storage, a divisor circuit determining a divisor value, and a division circuit dividing the data value accumulated in the storage by the determined divisor value. The circuit can perform a maximum pooling operation by a maximum circuit determining a maximum value of input feature map data values, and storing the determined maximum value in the storage.”).
Regarding claims 14 and 19:
Claims 14 and 19 recite analogous limitations to claim 5 and therefore are rejected on the same grounds.
Regarding claim 10:
Holm teaches an operating method of a data processing system, the operating method comprising: allocating, by a controller, a storage unit to each of a plurality integration groups;
receiving, by the controller, a request including an input feature map and a weight filter from a host device (see para [0160]: “The data processing system comprises an interconnect 6 which provides an interface between the various processors (ISP, CPU, GPU and NPU) and a memory controller 7. The memory controller 7 is operable to manage memory read and write requests, and to control reading and writing of data to off-chip memory 8.”. Also see para [0063]: “A convolution or pooling operation of the technology described herein should, and in an embodiment does, operate on an input data array (an “input feature map” (IFM)) to produce an output data array (an “output feature map” (OFM)).”.) Also see para [0175]: “A convolution (filter) kernel may be an array of data elements, with each data element of the array representing a “weight”.”. Also see para [0146]: “The neural network processor of the technology described herein is in an embodiment implemented in and as part of an overall data processing system that (further) includes one or more of: a host processor (central processing unit (CPU)) … );
performing, by a computation device, a convolution operation on the input feature map and the weight filter (see para [0071]: “It is in an embodiment a discrete convolution operation performed between an input data array (feature map) and a convolution (filter) kernel to generate an output data array (feature map).”);
sequentially outputting, by the computation device, a plurality of pooling elements as a result of performing the convolution operation, the plurality of pooling elements corresponding to each of the plurality of integration groups (see para [0065]: “The combined convolution and pooling circuit can receive data of an input data array (feature map) for neural network processing in any suitable manner. In an embodiment, the combined convolution and pooling circuit further comprises an input circuit configured to read data values of an input data array from memory (and to provide the read data values to other circuits of the combined convolution and pooling circuit for processing as appropriate).”);
integrating, by the computation device, a pooling value read from the storage unit and each of the plurality of pooling elements into a single value (see para [0088]: “A pooling operation may operate on the values for data elements which are encompassed by a respective pooling window in any suitable manner, to e.g., “summarise” the data values that fall within the respective pooling window. For example, in the case of “average pooling”, a pooling operation may operate to determine a mean of data values that fall within a respective pooling window. In the case of “maximum pooling”, a pooling operation may operate to determine a maximum value of data values that fall within a respective pooling window.”);
and updating, by the computation device, the pooling value of the storage unit according to a result of the integrating (see para [0090]: “As discussed above, to facilitate average pooling operations the add and divisor circuits of the combined convolution and pooling circuit can, and in an embodiment do, operate in conjunction with the storage to perform average pooling operations.”),
wherein the integrating and the updating are repeated until all of the plurality of pooling elements are integrated (see para [0176]: “The output feature map may be a suitable array of data elements. Calculating the output feature map may involve a kernel “sliding” (convolving) over an input feature map, such that the kernel is positioned in a different location with respect to the input feature map for each element of the output feature map. At each location that the kernel is positioned with respect to the input feature map, the product of each weight value of the kernel with the value of the corresponding element of the input feature map (i.e. the element wise multiplication) may be calculated, and the sum of the products may be used as the corresponding output value for the output feature map. Where the input feature map comprises multiple channels, the convolution operation may typically be such that output values represent a sum over all of the channels.”).
Regarding claim 11:
Holm teaches the system of claim 10.
Holm further teaches wherein the performing of the convolution operation comprises: sequentially performing the convolution operation on each of a plurality of division maps and the weight filter (see para [0176]: “Calculating the output feature map may involve a kernel “sliding” (convolving) over an input feature map, such that the kernel is positioned in a different location with respect to the input feature map for each element of the output feature map. At each location that the kernel is positioned with respect to the input feature map, the product of each weight value of the kernel with the value of the corresponding element of the input feature map (i.e. the element wise multiplication) may be calculated, and the sum of the products may be used as the corresponding output value for the output feature map”) [(Examiner’s note: A person having ordinary skill in the art using broadest reasonable interpretation in light of the specification, could interpret division maps to be sections of a feature map that will have the weight filter slide over them)],
the input feature map being divided into the plurality of division maps (see para [0177]: “The size of an output feature map may thus depend on the size of the input feature map and convolution kernel. The size of an output feature map may also depend on the convolution “stride”. For example, in the case of a stride value of 1, output data values may be calculated for each possible location in the input feature map, whereas larger stride values will result in smaller sized output feature maps.”).
Regarding claim 15:
A computing system comprising: a host device (see para [0146]: “The neural network processor of the technology described herein is in an embodiment implemented in and as part of an overall data processing system that (further) includes one or more of: a host processor (central processing unit (CPU)) … ).;
and a data processing system configured to: receive a request including an input feature map and a weight filter from the host device (see para [0160]: “The data processing system comprises an interconnect 6 which provides an interface between the various processors (ISP, CPU, GPU and NPU) and a memory controller 7. The memory controller 7 is operable to manage memory read and write requests, and to control reading and writing of data to off-chip memory 8.”. Also see para [0063]: “A convolution or pooling operation of the technology described herein should, and in an embodiment does, operate on an input data array (an “input feature map” (IFM)) to produce an output data array (an “output feature map” (OFM)).”.) Also see para [0175]: “A convolution (filter) kernel may be an array of data elements, with each data element of the array representing a “weight”.”. Also see para [0146]: “The neural network processor of the technology described herein is in an embodiment implemented in and as part of an overall data processing system that (further) includes one or more of: a host processor (central processing unit (CPU)) … ).;
perform a convolution operation on the input feature map and the weight filter (see para [0071]: “It is in an embodiment a discrete convolution operation performed between an input data array (feature map) and a convolution (filter) kernel to generate an output data array (feature map).”);
sequentially output a plurality of pooling elements as a result of performing the convolution operation, the plurality of pooling elements corresponding to each of a plurality of integration groups (see para [0065]: “The combined convolution and pooling circuit can receive data of an input data array (feature map) for neural network processing in any suitable manner. In an embodiment, the combined convolution and pooling circuit further comprises an input circuit configured to read data values of an input data array from memory (and to provide the read data values to other circuits of the combined convolution and pooling circuit for processing as appropriate).”);
allocate a storage unit to each of the plurality of integration groups (see para [0054]: “As will be explained in more detail below, this can enable the combined convolution and pooling circuit to perform “maximum pooling” operations by the maximum circuit determining a maximum value of input feature map data values, and storing the determined maximum value in the storage.”);
integrate a pooling value read from the storage unit and each of the plurality of pooling elements into a single value (see para [0088]: “A pooling operation may operate on the values for data elements which are encompassed by a respective pooling window in any suitable manner, to e.g., “summarise” the data values that fall within the respective pooling window. For example, in the case of “average pooling”, a pooling operation may operate to determine a mean of data values that fall within a respective pooling window. In the case of “maximum pooling”, a pooling operation may operate to determine a maximum value of data values that fall within a respective pooling window.”);
and update the pooling value stored in the storage unit with a result of the integrating (see para [0090]: “As discussed above, to facilitate average pooling operations the add and divisor circuits of the combined convolution and pooling circuit can, and in an embodiment do, operate in conjunction with the storage to perform average pooling operations.”),
wherein the integrating and the updating are repeated until all of the plurality of pooling elements are integrated (see para [0176]: “The output feature map may be a suitable array of data elements. Calculating the output feature map may involve a kernel “sliding” (convolving) over an input feature map, such that the kernel is positioned in a different location with respect to the input feature map for each element of the output feature map. At each location that the kernel is positioned with respect to the input feature map, the product of each weight value of the kernel with the value of the corresponding element of the input feature map (i.e. the element wise multiplication) may be calculated, and the sum of the products may be used as the corresponding output value for the output feature map. Where the input feature map comprises multiple channels, the convolution operation may typically be such that output values represent a sum over all of the channels.”)
.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2, 3, 6, 9, 12, 16, 17 are rejected under 35 U.S.C 103 as being unpatentable over Holm et al. (US20210295140A1 hereinafter referred to as Holm) in view of Peng et al. (“DNN+NeuroSim V2.0: An End-to-End Benchmarking Framework for Compute-in-Memory Accelerators for On-Chip Training” hereinafter referred to as Peng) in further view of Liang et al. (“Design of 16-bit fixed-point CNN coprocessor based on FPGA” hereinafter referred to as Liang).
Regarding claim 2:
Holm teaches the system of claim 1.
Holm further teaches wherein the input feature map is divided into a plurality of division maps (see para [0176]: “Calculating the output feature map may involve a kernel “sliding” (convolving) over an input feature map, such that the kernel is positioned in a different location with respect to the input feature map for each element of the output feature map. ”),
and wherein the computation device comprises: a computation memory configured to sequentially perform the convolution operation on each of the plurality of division maps and the weight filter (see para [0176]: “Calculating the output feature map may involve a kernel “sliding” (convolving) over an input feature map, such that the kernel is positioned in a different location with respect to the input feature map for each element of the output feature map. ”),
and sequentially output, as the plurality of pooling elements, a result of the convolution operation for the plurality of division maps (see para [0176]: “At each location that the kernel is positioned with respect to the input feature map, the product of each weight value of the kernel with the value of the corresponding element of the input feature map (i.e. the element wise multiplication) may be calculated, and the sum of the products may be used as the corresponding output value for the output feature map. Where the input feature map comprises multiple channels, the convolution operation may typically be such that output values represent a sum over all of the channels.”);
Holm does not explicitly teach a global buffer including the storage unit, a pooling controller configured to sequentially provide a pooler with the plurality of pooling elements outputted from the computation memory, update the pooling value stored in the storage unit io according to a result of the pooling operation of the pooler, and the pooler configured to perform the pooling operation when the plurality of pooling elements are sequentially provided to the pooler.
Peng, however, analogously teaches a global buffer including the storage unit (see pg. 7 section ‘F. Weight Update’: “ In other words, during such successive accumulation operations, we only need the global buffer storage to be (2× synaptic-array-size× highest-possible accumulated weight gradients precision).”);
Before the effective filing of the date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Holm and Peng before him or her, to modify the system of claim 2 to include attributes of a global buffer including the storage unit in order to potentially support the use of compute-in-memory accelerators (see pg. 3 section III ‘CIM Architecture for Training’: “As Fig. 2 shows, in chip-level, the key components of the CIM accelerators are tiles, global buffer, neural functional units including pooling, accumulation and activation.”).
Neither Holm nor Peng explicitly teaches a pooling controller configured to sequentially provide a pooler with the plurality of pooling elements outputted from the computation memory, update the pooling value stored in the storage unit according to a result of the pooling operation of the pooler, and the pooler configured to perform the pooling operation when the plurality of pooling elements are sequentially provided to the pooler.
Liang, however, analogously teaches a pooling controller configured to sequentially provide a pooler with the plurality of pooling elements outputted from the computation memory (see pg. 5 section III “How to Implement General CNN Coprocessor”: “When the pooling operation starts, the pooling controller reads the feature data from the global cache into the pooling input cache. The pooling calculation module chooses max-pooling unit or mean-pooling unit to calculate data according to the pooling type.”),
and update the pooling value stored in the storage unit according to a result of the pooling operation of the pooler (see pg. 3 section III “How to Implement General CNN Coprocessor” :“The results will be written back to the global cache when the operation completes.”);
and the pooler configured to perform the pooling operation when the plurality of pooling elements are sequentially provided to the pooler (see pg. 5 section III “How to Implement General CNN Coprocessor”: “When the pooling operation starts, the pooling controller reads the feature data from the global cache into the pooling input cache. The pooling calculation module chooses max-pooling unit or mean-pooling unit to calculate data according to the pooling type. The results will be written back to the global cache when the operation completes.”).
Before the effective filing of the date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Holm, Peng, and Liang before him or her, to modify the system of claim 2 to include attributes of a pooling controller configured to sequentially provide a pooler with the plurality of pooling elements outputted from the computation memory, update the pooling value stored in the storage unit according to a result of the pooling operation of the pooler, and the pooler configured to perform the pooling operation when the plurality of pooling elements are sequentially provided to the pooler in order to process feature data from the global cache into the pooling input cache, complete the pooling computation, and write back the results to a global cache (see pg. 5 section III “How to Implement General CNN Coprocessor”: “When the pooling operation starts, the pooling controller reads the feature data from the global cache into the pooling input cache. The pooling calculation module chooses max-pooling unit or mean-pooling unit to calculate data according to the pooling type. The results will be written back to the global cache when the operation completes.”).
Regarding claim 16:
Claim 16 recites analogous limitations to claim 2 and therefore is rejected on the same grounds as claim 2.
Regarding claim 3:
Holm in view of Peng in further view of Liang teaches the system of claim 2.
Holm further teaches wherein the input feature map is provided in the form of a matrix and the weight filter is provided in the form of a matrix (see para [0064]: “An input or output data array (feature map) can be any suitable array of data elements, and can have any suitable size and number of dimensions. For instance, in the case of two-dimensional RGB image data, an (input) data array may comprise a three dimensional array of data elements, with two dimensions of the array corresponding to the two dimensions of the image, and the third dimension of the array corresponding to the red, green and blue channels of the image.”. Also see para [0175]: “The size and dimensions of a convolution (filter) kernel can be selected as desired, but it should be smaller in size than the input feature map.”. Also see para [0176]: “The output feature map may be a suitable array of data elements. Calculating the output feature map may involve a kernel “sliding” (convolving) over an input feature map, such that the kernel is positioned in a different location with respect to the input feature map for each element of the output feature map”),
each of the plurality of division maps having the same size as the weight filter (see para [0085]: “A pooling operation can operate to downsample or summarise an input data array (feature map) in any suitable manner. In an embodiment, a value for each data element of the output data array (feature map) is determined from the values for plural corresponding data elements of the input data array (feature map), which corresponding data elements in an embodiment fall within a respective, in an embodiment two-dimensional, “window” (region) of the input data array (feature map).”).
Regarding claims 12 and 17:
Claims 12 and 17 recite analogous limitations to claim 3 and therefore are rejected on the same grounds as claim 3.
Regarding claim 6:
Holm teaches a data processing system comprising: a computation memory configured to receive a request including an input feature map and a weight filter from a host device (see para [0160]: “The data processing system comprises an interconnect 6 which provides an interface between the various processors (ISP, CPU, GPU and NPU) and a memory controller 7. The memory controller 7 is operable to manage memory read and write requests, and to control reading and writing of data to off-chip memory 8.”. Also see para [0063]: “A convolution or pooling operation of the technology described herein should, and in an embodiment does, operate on an input data array (an “input feature map” (IFM)) to produce an output data array (an “output feature map” (OFM)).”.) Also see para [0175]: “A convolution (filter) kernel may be an array of data elements, with each data element of the array representing a “weight”.”. Also see para [0146]: “The neural network processor of the technology described herein is in an embodiment implemented in and as part of an overall data processing system that (further) includes one or more of: a host processor (central processing unit (CPU)) … );
sequentially perform a convolution operation on the weight filter and each of a plurality of division maps included in the input feature map (see para [0071]: “It is in an embodiment a discrete convolution operation performed between an input data array (feature map) and a convolution (filter) kernel to generate an output data array (feature map).”),
and sequentially output each of a plurality of pooling elements as a result of the convolution operation (see para [0065]: “The combined convolution and pooling circuit can receive data of an input data array (feature map) for neural network processing in any suitable manner. In an embodiment, the combined convolution and pooling circuit further comprises an input circuit configured to read data values of an input data array from memory (and to provide the read data values to other circuits of the combined convolution and pooling circuit for processing as appropriate).”);
the plurality of pooling elements corresponding to each of the plurality of integration groups (see fig. 3. Also see para [0182]: “As discussed above, a pooling operation may operate to downsample an input data array (e.g. feature map), e.g. in order to summarise featu