DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1 and 19 have been amended. Claims 1-20 are pending.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-9, 17, 19-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
The claims at a high level recite sparse image processing.
Step 1: Does the Claim Fall within a Statutory Category?
Yes. Claims 1-9, 17, 19-20 recite a method and a system and therefore, are directed to the statutory class of machine and a product.
The USPTO Guidance recites:
(1) any judicial exceptions, including certain groupings of abstract ideas (i.e., mathematical concepts, certain methods of organizing human activity such as a fundamental economic practice, or mental processes) (Step 2A, Prong 1); and
(2) additional elements that integrate the judicial exception into a practical application (Step 2A, Prong 2). MPEP §§ 2106.04(a), (d).
Only if the claim (1) recites a judicial exception and (2) does not integrate that exception into a practical application, do we then look in Step 2B to whether the claim:
(3) adds a specific limitation beyond the judicial exception that is not “well-understood, routine, conventional” in the field; or
(4) simply appends well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception. MPEP § 2106.05(d).
Step 2A, Prong One: Is a Judicial Exception Recited?
First, determine whether the claims recite any judicial exceptions, including certain groupings of abstract ideas (i.e., mathematical concepts, certain methods of organizing human activity, or mental processes). MPEP § 2106.04(a).
Claim 1 recites –
▪ the input data comprising a plurality of groups of data elements, each group being associated with a channel of a plurality of channels, the weights comprising a plurality of weight tensors, each weight tensor being associated with a channel of the plurality of channels (Abstract Idea of a mental process, see MPEP § 2106.04(a)(2)(III). Under the broadest reasonable interpretation, this limitation is an abstract idea of “a mental process” because it recites a process that can be performed in the human mind (i.e., observation, determination, evaluation, judgment, and opinion) — a user can group data, associate with a channel and compute tensor weights, which are logical / mathematical computations);
▪ a data sparsity map generation circuit configured to generate, based on the input data, a data sparsity map comprising a channel sparsity map and a spatial sparsity map, the channel sparsity map indicating one or more channels associated with one or more first weights tensors to be selected from the plurality of weight tensors, the spatial sparsity map indicating spatial locations of first data elements to be selected from the plurality of groups of data elements (Abstract Idea of a mental process, see MPEP § 2106.04(a)(2)(III). Under the broadest reasonable interpretation, this limitation is an abstract idea of “a mental process” because it recites a process that can be performed in the human mind (i.e., observation, determination, evaluation, judgment, and opinion) — a user can generate sparsity maps (with a pen and paper) and indicate channels associated with weights);
▪ fetch, based on the channel sparsity map, the one or more first weights tensors (Abstract Idea of a mental process, see MPEP § 2106.04(a)(2)(III). Under the broadest reasonable interpretation, this limitation is an abstract idea of “a mental process” because it recites a process that can be performed in the human mind (i.e., observation, determination, evaluation, judgment, and opinion) — a user can mentally determine and output weights tensors); and
▪ fetch, based on the spatial sparsity map, the first data elements (Abstract Idea of a mental process, see MPEP § 2106.04(a)(2)(III). Under the broadest reasonable interpretation, this limitation is an abstract idea of “a mental process” because it recites a process that can be performed in the human mind (i.e., observation, determination, evaluation, judgment, and opinion) — a user can mentally determine data based on the map); and
▪ computations on the first data elements based on the one or more first weights tensors to generate a processing result of the input data (Mental process: Determining a computations is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. Nothing in this claim element precludes the step from practically being performed in the mind. For example, determining a computations generating results based on computations).
These limitations, based on their broadest reasonable interpretation, recite a mental process, i.e. a judicial exception. For these reasons, the independent claim 1, as well as independent claim 19, which include limitations commensurate in scope with claim 1, recite a judicial exception.
A method, like the claimed method, “a process that employs mathematical algorithms to manipulate existing information to generate additional information is not patent eligible.” See Digitech Image Techs, LLC v. Elecs. for Imaging, Inc., 758 F.3d 1344, 1351 (Fed. Cir. 2014). See Electric Power Group, LLC v. Alstom S.A., 830 F.3d 1350 (Fed. Cir. 2016) where collecting information, analyzing it, and displaying results from certain results of the collection and analysis was held to be an abstract idea. See In re Meyer, 688 F.2d 789, 795—96 (CCPA 1982), which held that “a mental process that a neurologist should follow” when testing a patient for nervous system malfunctions was not patentable.
Accordingly, the claims recite an abstract idea.
Step 2A, Prong Two: Is the Abstract Idea Integrated into a Practical Application?
Next determine whether the claims recite additional elements that integrate the judicial exception into a practical application (see MPEP §§ 2106.05(a)-(c), (e)-(h)). To integrate the exception into a practical application, the additional claim elements must, for example, improve the functioning of a computer or any other technology or technical field (see MPEP § 2106.05(a)), apply the judicial exception with a particular machine (see MPEP § 2106.05(b)), or apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment (see MPEP § 2106.05(e)).
Additional elements:
▪ a memory configured to store input data and weights (Amount to mere instruction to apply the abstract idea using a generic computer component. A mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f).)
▪ a gating circuit configured to: fetch, based on the channel sparsity map, the one or more first weights tensors from the memory (A generic computer functions of receiving and processing that are well-understood, routine, and conventional activities previously known to the industry. Extracting caption data and natural text processing are merely extra-solution activities and does not meaningfully limit the independent claims. Generic computer implementation does not provide significantly more than the abstract idea);
▪ a processing circuit configured to perform, using a neural network, computations (Amount to “Apply it”. Merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea, see MPEP § 2106.05(f). Examiner’s note: high level application of using neural network amount to merely invoking a computer component to apply the exception).
See MPEP 2106.05(d)(Il). Taking the claim elements separately, the function performed by the computer at each step of the process is purely conventional. Using a computer and associated computer network to obtain data, use data to identify other data, and comparing data, are some of the most basic functions of a computer. All of these computer functions are well-understood, routine, conventional activities previously known to the industry. The method claims do not, for example, purport to improve the functioning of the computer itself. Nor do they effect an improvement in any other technology or technical field. Instead, the claims at issue amount to nothing significantly more than an instruction to apply the abstract idea of displaying, processing and storing data using some unspecified, generic computer.
No “inventive concept” sufficient to transform the abstract method of organizing human activity into a patent-eligible application. See MPEP § 2106.05. Rather, the additional elements identified above are merely well-understood, conventional computer components, as confirmed by the Specification. See MPEP § 2106.05(d)(1). For example, the Specification refers to the additional elements in generic terms.
As discussed above with respect to integration of the abstract idea into a practical application, the additional elements relating to computing components amount to no more than applying the exception using a generic computing components. Mere instructions to apply an exception using a generic computing component cannot provide an inventive concept. Furthermore, the broadest reasonable interpretation of the claimed computer components (i.e., additional elements) includes any generic computing components that are capable of being programmed to communicate and process known data.
Therefore, the Office finds no improvements to another technology or field, no improvements to the function of the computer itself, and no meaningful limitations beyond generally linking the use of an abstract idea to a particular technological environment. Therefore, based on the two-part Alice Corp. analysis, there are no limitations in any of the claims that transform the exception (i.e., the abstract idea) into a patent eligible application.
Accordingly, independent claims 1 and 19 are patent ineligible because they are directed to an abstract idea that does not recite an inventive concept that amounts to significantly more than the abstract idea.
Dependent claims 2-18 and 20 do not recite additional limitations that demonstrate integration of the abstract idea into a practical application or an inventive concept that amounts to significantly more than the abstract idea.
With respect to claims 2-6:
Step 2A Prong 1: the claims recite a judicial exception (an abstract idea)
▪ the neural network comprises a first neural network layer and a second neural network layer; the gating circuit comprises a first gating layer and a second gating layer; the first gating layer is configured to perform, based on a first data sparsity map generated based on the plurality of groups of data elements (Amount to “Apply it”. Merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea, see MPEP § 2106.05(f). Examiner’s note: high level application of using neural network amount to merely invoking a computer component to apply the exception)
▪ a first channel gating operation on the plurality of weight tensors to provide first weights of the one or more first weights tensors to the first neural network layer, or a first spatial gating operation on the plurality of groups of data elements to provide first input data including the first data elements to the first neural network layer; the first neural network layer is configured to generate first intermediate outputs based on the first input data and the first weights, the first intermediate outputs having first groups of data elements associated with different channels; the second gating layer is configured to perform, based on a second data sparsity map generated based on the first intermediate outputs, at least one of: a second channel gating operation on the plurality of weight tensors to provide second weights of the one or more first weights tensors to the second neural network layer, or a second spatial gating operation on the first intermediate outputs to provide second input data to the second neural network layer; the second neural network layer is configured to generate second intermediate outputs based on the second input data and the second weights, the second intermediate outputs having second groups of data elements associated with different channels; and the processing result is generated based on the second intermediate outputs (Amount to mere instruction to apply the abstract idea using a generic computer component. A mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f). Reciting the idea of a solution or outcome without detailing how the result is accomplished is equivalent to saying “apply it” and thus, fails to provide significantly more to the judicial exception).
Claims 3-6 further expand on functionality of neural network and directed to an abstract idea that does not recite an inventive concept that amounts to significantly more than the abstract idea.
Step 2A Prong 2: the additional elements that are not sufficient to integrate the judicial exception into a practical application.
Additional elements:
a first neural network layer and a second neural network layer; the gating circuit comprises a first gating layer and a second gating layer (Amount to mere instruction to apply the abstract idea using a generic computer component. A mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f).)
Step 2B: the additional element is not sufficient to amount to significantly more than the judicial exception.
With respect to claims 7-9 and 20:
Dependent claims 7-9, which include limitations commensurate in scope with the dependent claims 2-6, recite a judicial exception.
▪ Claims 7 and 20 - Amount to mere instruction to apply the abstract idea using a generic computer component. A mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f).)
▪ Claims 8 - (Abstract Idea of a mental process. Under the broadest reasonable interpretation, the obtaining/determining probability distribution and divergence, as drafted, is an abstract idea of “a mental process” because it recites a process that can be performed in the human mind (i.e., observation, determination, evaluation, judgment, and opinion) — a user can manually determine array and bitmasks).
▪ Claims 9 - (Abstract idea of “a mathematical concept” — see MPEP § 2106.04(a)(2)(l). Note: under the broadest reasonable interpretation of the claim, the claimed invention encompasses mathematical concept (e.g., Mathematical Formula or Equations)).
Additional elements: the additional element listed above in step 2A Prong 2 is merely instructions to be implemented on a generic computer component. Therefore, the additional element does not amount to an inventive concept, particularly when the activity is well understood or conventional (MPEP 2106.05(d)).
Step 2A Prong 1: The claim does not recite any of the judicial exceptions enumerated in the 2019 PEG. Step 2A Prong 2: The judicial exception is not integrated into a practical application.
Claim 10 - (A generic computer functions of receiving and processing that are well-understood, routine, and conventional activities previously known to the industry. Extracting caption data and natural text processing are merely extra-solution activities and does not meaningfully limit the independent claims. Generic computer implementation does not provide significantly more than the abstract idea. Amount to no more than mere instructions to apply the abstract idea using a generic computer component- see MPEP 2106.05(f))).
Claim 17 –
▪ a programmable pixel cell array and a programming circuit (A generic computer functions of receiving and processing that are well-understood, routine, and conventional activities previously known to the industry. Extracting caption data and natural text processing are merely extra-solution activities and does not meaningfully limit the independent claims. Generic computer implementation does not provide significantly more than the abstract idea. Amount to no more than mere instructions to apply the abstract idea using a generic computer component- see MPEP 2106.05(f)))
▪ wherein the input data is first input data; determine a region of interest based on the processing result from the processing circuit; generate a programming signal indicating the region of interest to select a subset of pixel cells of the programmable pixel cell array to perform light sensing operations to perform a sparse image capture operation (Amount to mere instruction to apply the abstract idea using a generic computer component. A mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f).);
▪ transmit the programming signal to the programmable pixel cell array to perform the sparse image capture operation to capture second input data (A generic computer functions of receiving and processing that are well-understood, routine, and conventional activities previously known to the industry. Extracting caption data and natural text processing are merely extra-solution activities and does not meaningfully limit the independent claims. Generic computer implementation does not provide significantly more than the abstract idea).
Additional elements: a programmable pixel cell array and a programming circuit, perform light sensing operations and transmitting the programming signal (The additional element listed above in step 2A Prong 2 is merely instructions to be implemented on a generic computer component. Therefore, the additional element does not amount to an inventive concept, particularly when the activity is well understood or conventional (MPEP 2106.05(d)).
Step 2A Prong 1: The claim does not recite any of the judicial exceptions enumerated in the 2019 PEG. Step 2A Prong 2: The judicial exception is not integrated into a practical application.
Dependent claims 2-9, 17, 20 are thus, also patent ineligible for the reasons discussed above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7-10, 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yous et al. (US 2019/0340511) in view of Hunter et al. (US 20220108157).
Regarding claim 1, Yous teaches an apparatus comprising:
a memory configured to store input data and weights (F7:222), the input data comprising a plurality of groups of data elements ([0027], F3:252), each group being associated with a channel of a plurality of channels, the weights comprising a plurality of weight tensors, each weight tensor being associated with a channel of the plurality of channels ([0033]-[0036]);
a data sparsity map generation circuit configured to generate, based on the input data, a data sparsity map comprising a channel sparsity map and a spatial sparsity map ([0043]), the channel sparsity map indicating one or more channels associated with one or more first weights tensors to be selected from the plurality of weight tensors ([0034]-[0036], [0041], F4A-B), the spatial sparsity map indicating spatial locations of first data elements to be selected from the plurality of groups of data elements ([0037], [0043])(see NOTE);
a gating circuit ([0072]) configured to: fetch, based on the channel sparsity map, the one or more first weights tensors from the memory ([0041], [0043]); and
fetch, based on the spatial sparsity map, the first data elements from the memory ([0035]); and
a processing circuit configured to perform, using a neural network ([0048]), computations on the first data elements based on the one or more first weights tensors to generate a processing result of the input data ([0033]-[0034] “generate a number of channels … using … weights”; “sparse processing of activations and weights”, [0054], [0098], F3).
NOTE Yous teaches generating a plurality of sparsity maps (see FIGS. 12A and 12B). Yous does not explicitly teach the spatial sparsity map indicating spatial locations of first data elements to be selected from the plurality of groups of data elements. However, it is well-known that in a convolutional neural network (CNN), the sparsity map indicates the location of important features in the input image. Sparsity, which is often induced by a rectified linear unit (ReLU) activation function (ReLU forces all negatively valued activations to be clamped to zero) or through pruning, identifies which neurons or connections are the most critical for a given task. Yous teaches “sparsity maps … allow access to non-zero elements for processing” (aka important features)[0034]. Identifying and accessing “non-zero elements for processing” reasonably implicates that locations of such elements in the sparse map are obviously determined in order for a data to be packed (transformed or compressed). By definition – a map is an indication of a location. (I.e. the primary purpose of a map is to provide information about a position, direction in relation to a location). Therefore, the sparsity map that “allow access to non-zero elements for processing” is obviously and implicitly indicates spatial locations.
However, to merely obviate such reasoning, Hunter discloses the spatial sparsity map indicating spatial locations of first data elements to be selected from the plurality of groups of data elements ([0086], [0116]-[0119], [0121]).
It would have been obvious to one of ordinary skill in the art at the time of invention to modify the teachings of Yous to include programmable pixel cell array as disclosed by Hunter. Doing so would provide more efficient operations related to a sparse neural network (Hunter [0077]).
Regarding claim 2, Yous as modified teaches the apparatus of claim 1, wherein: the neural network comprises a first neural network layer and a second neural network layer (Yous [0028], [0032], [0048]); the gating circuit comprises a first gating layer and a second gating layer (Yous [0043], Hunter [0103]); the first gating layer is configured to perform, based on a first data sparsity map generated based on the plurality of groups of data elements (Yous [0034]-[0035], Hunter [0121]), at least one of:
a first channel gating operation on the plurality of weight tensors to provide first weights of the one or more first weights tensors to the first neural network layer (Yous [0035], [0041], [0043], Hunter [0123]),
or a first spatial gating operation on the plurality of groups of data elements to provide first input data including the first data elements to the first neural network layer (Yous [0028], [0032], [0048]);
the first neural network layer is configured to generate first intermediate outputs based on the first input data and the first weights, the first intermediate outputs having first groups of data elements associated with different channels (Yous [0032], [0034], [0048]);
the second gating layer is configured to perform, based on a second data sparsity map generated based on the first intermediate outputs (Yous [0044]-[0045], [0052]), at least one of:
a second channel gating operation on the plurality of weight tensors to provide second weights of the one or more first weights tensors to the second neural network layer (Yous [0050]), or a second spatial gating operation on the first intermediate outputs to provide second input data to the second neural network layer (Hunter [0071], [0078]); the second neural network layer is configured to generate second intermediate outputs based on the second input data and the second weights, the second intermediate outputs having second groups of data elements associated with different channels (Yous [0047], [0053]); and the processing result is generated based on the second intermediate outputs (Hunter [0090], [0123], [0125]).
Regarding claim 3, Yous as modified teaches the apparatus of claim 2, wherein: the neural network further comprises a third neural network layer (Yous [0049]-[0053], [0053]. Hunter [0062]); the gating circuit further comprises a third gating layer; the third gating layer is configured to perform, based on a third data sparsity map generated based on the second intermediate outputs, at least one of: a third channel gating operation on the plurality of weight tensors to provide third weights of the one or more first weights tensors to the third neural network layer, or a third spatial gating operation on the second intermediate outputs to provide third input data to the third neural network layer (Hunter [0090], [0123], [0125]); and the third neural network layer is configured to generate outputs including the processing result based on the third input data and the third weights (Yous [0112], Hunter [0078], [0131]-[0132], [0135]).
Regarding claim 4, Yous as modified teaches the apparatus of claim 3, wherein the second neural network layer comprises a convolution layer; and wherein the third neural network layer comprises a fully connected layer (Yous [0036], [0104], Hunter [0062]).
Regarding claim 7, Yous as modified teaches the apparatus of claim 1, wherein the neural network is a first neural network; and wherein the data sparsity map generation circuit is configured to use a second neural network to generate the data sparsity map (Yous [0036], [0049], [0104], Hunter [0062]).
Regarding claim 10, Yous teaches the apparatus of claim 1, wherein the data sparsity map generation circuit, the gating circuit, and the processing circuit are parts of a neural network hardware accelerator (Yous F1-2, 7); and wherein the memory is an external memory external to the neural network hardware accelerator (Yous [0067], [0069], Hunter [0059], [0119], [0078]-[0079], [0083] [0131]).
Regarding claim 19, Yous teaches a method comprising: storing, at a memory, input data and weights, the input data comprising a plurality of groups of data elements, each group being associated with a channel of a plurality of channels, the weights comprising a plurality of weight tensors, each weight tensor being associated with a channel of the plurality of channels; generating, based on the input data, a data sparsity map comprising a channel sparsity map and a spatial sparsity map, the channel sparsity map indicating one or more channels associated with one or more first weights tensors to be selected from the plurality of weight tensors, the spatial sparsity map indicating 75spatial locations of first data elements to be selected from the plurality of groups of data elements; fetching, based on the channel sparsity map, the one or more first weights tensors from the memory; fetching, based on the spatial sparsity map, the first data elements from the memory; and performing, using a neural network, computations on the first data elements and the one or more first weights tensors to generate a processing result of the input data.
Claim 19 recites substantially the same limitations as claim 1, and is rejected for substantially the same reasons.
Regarding claim 20, Yous teaches the method of claim 19, wherein the neural network is a first neural work; and wherein the data sparsity map is generated using a second neural network (Yous [0028], [0036], [0056], Hunter [0062]).
Claim 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yous as modified in further view of SON et al. (US 20210365790) and Herrmann et al. “Channel Selection Using Gumbel Softmax”.
Regarding claim 5, Yous as modified does not explicitly teach, however SON and Herrmann disclose the apparatus of claim 3, wherein: the first gating layer is configured to perform the first spatial gating operation but not the first channel gating operation (Herrmann p.243 ¶1.1, SON [0056]-[0057]); the second gating layer is configured to perform the second spatial gating operation but not the second channel gating operation; and the third gating layer is configured to perform the third channel gating operation but not the third spatial gating operation (Herrmann p.245 ¶ 3.1, SON [0059], [0063][0064]).
It would have been obvious to one of ordinary skill in the art at the time of invention to modify the teachings of Yous to include channel gating operation as disclosed by SON and Herrmann. Doing so would provide a key advantage of neural nets and flexibly avoid computing certain filters and their resulting channels (Herrmann p.242]) and increase the processing speed without greatly decreasing the performance of the neural network by increasing the resource utilization (SON [0052]).
Claim 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yous as modified in further view of Gunnam et al. (US 20210303976) and Zeng et al. (US 20190361454).
Regarding claim 6, Yous as modified as modified does not explicitly teach, however Gunnam and Herrmann disclose the apparatus of claim 5, wherein the second data sparsity map is generated based on a spatial tensor (Zeng [0018]), the spatial tensor being generated based on performing a channel-wise pooling operation between the first groups of data elements of the first intermediate outputs associated with different channels (Gunnam [0124], [0131], [0146], [0173], Zeng [0111]-[0113], [0115], [0121]); and
wherein the third data sparsity map is generated based on a channel tensor, the channel tensor being generated based on performing an inter-group pooling operation within each group of the second groups of data elements of the second intermediate outputs, such that the channel tensor is associated with the same channels as the second intermediate outputs (Gunnam [0160], [0162], Zeng [0021], [0111], [0115], [0118]).
It would have been obvious to one of ordinary skill in the art at the time of invention to modify the teachings of Yous to include channel-wise pooling operation n as disclosed by Gunnam and Zeng. Doing so would improve performance and computing efficiency (Zeng [0170]) and reduce storage space and further increase performance (Gunnam [0044]).
Claims 8-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yous as modified in further view of SON et al. (US 20210365790) and Paulus et al. “Gradient Estimation with Stochastic Softmax Tricks”.
Regarding claim 8, Yous as modified teaches the apparatus of claim 7, wherein the data sparsity map comprises an array of binary map
Yous as modified does not explicitly teach, however SON discloses an array of binary masks, each binary mask having one of two values (AA [0058], [0068]); wherein the data sparsity map generation circuit is configured to: generate, using the second neural network, an array of soft masks, each soft mask corresponding to a binary mask of the array of binary masks and having a range of values ([0069]); and generate the data sparsity map based on applying a differentiable function that approximates an arguments of the
Yous as modified by SON does not explicitly teach, however Paulus discloses applying a differentiable function that approximates an arguments of the maxima (argmax) function to the array of soft masks (see page 3 formulas 2-4).
It would have been obvious to one of ordinary skill in the art at the time of invention to modify the teachings of Yous to include array of binary masks as disclosed by SON and Paulus. Doing so would increase the processing speed without greatly decreasing the performance of the neural network by increasing the resource utilization (SON [0052]).
Regarding claim 9, Yous as modified teaches the apparatus of claim 8, wherein the data sparsity map generation circuit is configured to: add random numbers from a Gumbel distribution to the array of soft masks to generate random samples of the array of soft masks (SON [0077]); and apply a soft max function on the random samples to approximate the argmax function (SON [0077]-[0079], Paulus pages 3-4).
Claim 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yous as modified and in view of applicant’s admitted prior art Choudhury et a. (US 20220253716).
Regarding claim 11, Yous as modified teaches the apparatus of claim 10, wherein the neural network hardware accelerator further includes a local memory, a computation engine (Yous F7), an output buffer (Hunter [0082]), and a controller (Yous [0059], F15); wherein the controller is configured to: fetch, based on the channel sparsity map, the one or more first weights tensors from the external memory (Yous [0034]-[0035], Hunter [0059], [0119]); fetch, based on the spatial sparsity map, the first data elements from the external memory (Yous [0043], Hunter [0083]-[0084], [0119]); store the one or more first weights tensors and the first data elements at the local memory (Yous [0045], [0040], Hunter [0059], [0119], [0078]-[0079], [0083] [0131]);
Yous as modified does not explicitly teach, however Choudhury discloses herein the neural network hardware accelerator further includes a local memory, a computation engine, an output buffer, and a controller ([0092]); wherein the controller is configured to: fetch, based on the channel sparsity map, the one or more first weights tensors from the external memory ([0094]-[0095]); fetch, based on the spatial sparsity map ([0033]), the first data elements from the external memory ([0096]); store the one or more first weights tensors and the first data elements at the local memory ([0094], [0101]); control the computation engine to fetch the one or more first weights tensors and the first data elements from the local memory ([0115]-[0116]), and to perform the computations of a first neural network layer of the neural network to generate intermediate outputs ([0097], [0146]); control the output buffer to perform post-processing operations on the intermediate outputs ([0097]); and store the post-processed intermediate outputs at the external memory to provide inputs ([0098], [0101], [0108]) for a second neural network layer of the neural network ([0113]).
It would have been obvious to one of ordinary skill in the art at the time of invention to modify the teachings of Yous to include programmable pixel cell array as disclosed by Choudhury. Doing so would improve overall system performances (e.g., frame rate, accuracy) (Choudhury [0042]).
Claims 12-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yous as modified and in view of applicant’s admitted prior art Yao et a. (US 20220207293).
Regarding claim 12, Yous as modified as modified teaches the apparatus of claim 11, wherein
Yous as modified as modified does not explicitly teach, however Yao discloses the local memory further stores an address table that maps between addresses of the local memory and addresses of the external memory ([0073], [0093]); and wherein the controller is configured to, based on the address table, fetch the one or more first weights tensors and the first data elements from the external memory and store the one or more first weights tensors and the first data elements at the local memory ([0118], [0122]).
It would have been obvious to one of ordinary skill in the art at the time of invention to modify the teachings of Yous to include address table as disclosed by Yao. Doing so enables an efficient execution environment in the face of higher latency memory accesses (Yao [0275]).
Regarding claim 13, Yous as modified as modified teaches the apparatus of claim 12, wherein the address table comprises a translation lookaside buffer (TLB); and wherein the TLB includes multiple entries, each entry being mapped to an address of the local memory, and each entry further storing an address of the external memory (Yao [0073], [0093], [0118], [0122]).
Regarding claim 14, Yous as modified as modified teaches the apparatus of claim 13, wherein the controller is configured to: receive a first instruction to store a data element of the plurality of groups of data elements at a first address of the local memory, the data element having a first spatial location in the plurality of groups of data elements (Hunter [0082]-[0083], [0118]-[0120]); determine, based on the spatial sparsity map, that the data element at the first spatial location is to be fetched (Hunter [0084]-[0086]); and based on determining that the data element at the first spatial location is to be fetched: retrieve a first entry of the address table mapped to the first address; retrieve a second address stored in the first entry (Hunter [0111], [0144], [0146]); fetch the data element from the second address of the external memory; and store the data element at the first address of the local memory (Hunter [0089]-[0090], [0118]-[0119], [0130]).
Regarding claim 15, Yous as modified as modified teaches the apparatus of claim 13, wherein the controller is configured to: receive a second instruction to store a weight tensor of the plurality of weight tensors at a third address of the local memory, the weight tensor being associated with a first channel of the plurality of channels (Hunter [0082]-[0083], [0118]-[0120]); determine, based on the channel sparsity map, that a weight tensor of the first channel is to be fetched (Hunter [0084]-[0086]); and based on determining that the weight tensor of the first channel is to be fetched: retrieve a second entry of the address table mapped to the third address (Hunter [0111], [0144], [0146]); retrieve a fourth address stored in the second entry; fetch the weight tensor from the fourth address of the external memory; and store the weight tensor at the third address of the local memory (Hunter [0089]-[0090], [0118]-[0119], [0130]).
Claims 17-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yous as modified and in view of applicant’s admitted prior art BERKOVICH et al. (US 20200195875).
Regarding claim 17, Yous as modified teaches, the apparatus of claim 1, further comprising
Yous as modified does not explicitly teach, however BERKOVICH discloses the apparatus of claim 1, further comprising a programmable pixel cell array and a programming circuit ([0039]); wherein the input data is first input data ([0043]); and wherein the programming circuit is configured to: determine a region of interest based on the processing result from the processing circuit; generate a programming signal indicating the region of interest to select a subset of pixel cells of the programmable pixel cell array to perform light sensing operations to perform a sparse image capture operation ([0044], [0100]-[0102]); and transmit the programming signal to the programmable pixel cell array to perform the sparse image capture operation to capture second input data ([0055], [0098]).
It would have been obvious to one of ordinary skill in the art at the time of invention to modify the teachings of Yous to include programmable pixel cell array as disclosed by BERKOVICH. Doing so would improve overall system performances (e.g., frame rate, accuracy) (BERKOVICH [0042]).
Regarding claim 18, Yous as modified teaches the apparatus of claim 17, wherein the data sparsity map generation circuit, the gating circuit, the processing circuit, and the programmable pixel cell array are housed within a chip package to form a chip (Yous [0085], BERKOVICH [0039], [0098]).
Allowable Subject Matter
Claim 16 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Response to Arguments
Applicant's arguments filed 09/08/2025 have been fully considered but they are not persuasive.
◊ With respect to the rejection under 35 USC 101, the applicant argues –
“the present claims are not simply performing an abstract concept using generic computer components, but in fact improve the underlying functionality of such computing systems and provide several technical benefits and advancements.” Specifically, the claims provide “the technical benefits of substantial memory data transfer and computational reductions, in addition to battery and power savings — all of which are problems unique to mobile and wearable devices.”
The arguments are not persuasive. The present invention is directed to processing images (data) to detect important objects (analyzing data) using machine learning (such as neural network). The input (image) is received, pixels are analyzed to identify features and locations of such features. To determine the locations a data sparsity is generated (additional data).
Therefore, the image data is compressed (by means of neural network) to reduce processing and storage. The invention uses (at the level of “apply it”) a well-known concept of "sparse convolutional neural networks" (started around 2017) was used to describe networks that skip computations on zero-valued inputs, a technique particularly relevant for 3D data like point clouds. The benefits of spatial sparsity are - Performance acceleration: Reduced computation on redundant data speeds up both training and inference. Memory reduction: Less memory is required to store and process the sparse data and its corresponding feature maps. Hardware friendly: The structured nature of spatial sparsity can be more easily optimized for specialized hardware. Efficiency gains: This selective processing saves significant computational resources and memory, especially for large, high-dimensional inputs with large areas of redundant or uninformative data.
Such benefit are apparent to the use of the “sparse convolutional neural networks." The applicant merely “using a neural network” to implement the computations at the “apply it” level.
The claim simply generates a bit map (sparsity map) by means of basic mathematical, logical tensor computations (i.e. ReLU, SoftMax). Based on the generated sparsity map, the data is fetched. However, none of the compression functionality, computational reductions, battery and power savings or even image analysis is particularly disclosed. There is no improvement in the neural network itself or any specific data training of such network is disclosed by the claims.
With respect to Enfish, LLC v. Microsoft Corp., No.2015-12-14, 11 (Fed. Cir. 2016), it is respectfully noted that present claims do not claim a particular type of table as in Enfish v. Microsoft Corp., that provides synchronization between columns, where columns are entered as rows, to enhance searching. The Appellants claimed invention does not provide any search enhancement by means of the self-referenced table. The present claims merely disclose generating sparsity map and fetching data based on the map, which are basic functions of any generic computer. The identified abstract concept is not similar to Enfish since the focus of the current applications claims is not on a specific asserted improvement in computer capabilities or on an improvement to computer related technology / computer functionality.
The user is interacting with the computer in a conventional manner. The effect of the contribution does not operate at the level of architecture, nor is it independent of the kind of data being processed. Relatedly, the computer, that is to say the hardware upon which the program is operating, is operating in an entirely conventional manner. The proposed technical effect is not causing the computer in of itself to operate different as the method is operating at the level of application and thus is not interacting with the hardware at a level beyond that which any computer program would do so. Similarly, the computer upon which the program is operating is not operating more efficiently or effectively. The computer itself is operating entirely conventionally and the contribution is not having an effect on the efficiency of the computer itself.
The rejection is maintained.
◊ With respect to the 35 USC 103 rejection and the Yous reference, the applicant argues –
“Yous are actually directed to a re-ordering of weights … such sparsity maps are most closely related to re-ordering of weights in the inference model itself, rather than based on any particular form or type of input data. Accordingly, Yous cannot disclose or reasonably suggest, “the spatial sparsity map indicating spatial locations of first data elements to be selected from the plurality of groups of data elements,” as recited in claim 1.”
The arguments are not persuasive and somewhat confusing. Specifically - the applicant argues “sparsity maps are most closely related to re-ordering of weights in the inference model itself, rather than based on any particular form or type of input data.” It is not clear to what “particular form or type of input data” the applicant is referring to. Claim 1 requires – “input data and weights.” Although, It is understood, based on the specification, that the invention is directed to an image processing, there is no requirements of any “particular form or type of input data” in the claim 1.
Yous clearly and explicitly teaches “input data” and “generating output data 254 from input data 252”; “process a portion of the input data 252 tensor … and generate a number of channels … using … weights” [0033], wherein “set of output channels can be grouped in three-dimensional volume to be processed as an input for the next layer” [0037] (aka “plurality of groups of data elements”).
Further, the use of sparsity map in artificial networks is well-known. The concept is most relevant for convolutional neural networks (CNNs) that process spatially organized data like images, video, or 3D point clouds. The map itself is a binary mask (or a heatmap in some cases) that highlights the active spatial regions. During processing, a neural network can generate an "importance map" that measures the significance of each region of the input. Based on the importance map, a threshold is applied to determine which regions are critical. During computation, the network's kernels and operations are only applied to the areas marked as important, while calculations for the "sparse" (unimportant) regions are skipped. Such concept is well-known and is applicable to various concepts (i.e. not just image processing, which is not even claimed.)
Yes, indeed Yous reference provides a different solution, such as – “transforming the weight space to fit a pattern, which is different than merely skipping ineffectual computations (e.g. , weights with a zero value, weights with a near zero value … to balance the compute load … and … reduce the idle time or stalls” (see [0019]). However, different uses of the sparsity maps does not teaches away from the present invention. The claim only requires - “using a neural network, computations on the first data elements based on the one or more first weights tensors to generate a processing result.” The claimed “processing result” is broad enough to encompass a wide range of functionalities. Yous analogously teachers using a neural network, computations on the first data elements based on the one or more first weights to transform (pack) the weight space scarcity map to fit a pattern, which is broadly analogous to the claimed “processing result” as required by the independent claims.
Still with respect to the argued limitations above, Yous teaches taking an input data, transforming such data and providing output (F3), i.e. – taking an input and weights, generating sparsity map (F4A) and transforming the sparsity map into the transformed weight space sparsity map (F5A, F8A). The sparsity maps are clearly based on the input data 252.
Now, it is also well-known that in a convolutional neural network (CNN), the sparsity map indicates the location of important features in the input image (that’s why it’s called a map). The sparsity map is also analogous to an index (locations of nun null values in a matrix). Although such location is not explicitly disclosed, Yous still teaches identifying “sparsity maps … allow access to non-zero elements for processing” (aka important features)[0034]. Identifying and accessing “non-zero elements for processing” reasonably implicates that locations of such elements in the sparse map are obviously determined in order for a data to be packed (transformed or compressed). See specifically Figure 3, which takes input data 252 and outputs packed data 254. The packed data corresponds to the transformed packed weight map Figure 5B, which is based on the