DETAILED ACTION
This Office action is responsive to the Request for Continued Examination (RCE) filed under 37 CFR §1.53(d) for the instant application on July 31, 2026. The Applicants have properly set forth the RCE, which has been entered into the application, and an examination on the merits follows herewith.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitations use a generic placeholder (e.g. “unit”) that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are: the “coefficient storage unit,” the “feature storage unit,” the “storage control unit,” the “selection unit” and the “convolution operation unit” first recited in claim 1 and required by its dependent claims; the “transformation unit” in claim 3; the “first convolution operation unit,” the “second convolution operation unit,” and the “third convolution operation unit” in claim 8 and required by its dependent claims; the “detection unit” recited in claim 9; and the “unit” recited in claim 11.
Because these claim limitations are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 4, 7 and 12-14 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No. 2021/0073558 to Li et al. (“Li”), over U.S. Patent Application Publication No. 2020/0117989 to Huang et al. (“Huang”), and also over the article entitled “Heterogeneous Siamese Tracking System Based on PYNQ Framework” by Cui et al. (“Cui”).
Regarding claim 1, Li describes a method for detecting a target object, which includes inputting each of at least two feature groups output from at least two network layers of a neural network into a detector, and whereby the detector outputs a classification result and a regression result for each group (see e.g. paragraphs 0005-0008). Li particularly teaches that such a method can be implemented via an information processing apparatus (see e.g. paragraphs 0188-0190) which, like claimed, comprises:
a storage control unit configured to set a part of previously obtained feature data as template feature data, and align an order of the template feature data with an order of data referenced by filter coefficients in a convolution computation processing (see e.g. paragraphs 0043-0047 and 0069-0070: Li teaches extracting a feature of a “reference frame” and a feature of a “frame under detection,” wherein the reference frame indicates a target object and the frame under detection, which can occur subsequent to the reference frame in a video sequence, indicates a current frame which is to be subject to target object detection. The extracted feature of the reference frame and the extracted feature of the frame under detection are input as a feature group into a detector, which outputs a corresponding classification result and a regression result, wherein the classification result indicates a probability that a candidate box is a bounding box for the target object and the regression result indicates a position offset of the candidate box – see e.g. paragraphs 0053-0058 and 0071-0072. Li particularly discloses that such detection entails obtaining a classification weight for the detector and a regression weight for the detector based on the extracted feature of the reference frame, wherein the extracted feature of the frame under detection is then processed with the classification and regression weights to obtain the classification and regression results – see e.g. paragraphs 0073-0074, 0078 and 0081. The extracted feature of the reference frame, or the obtained classification and/or regression weight based thereon, is considered “template feature data” like claimed. Accordingly, Li teaches setting a part of previously obtained feature data, e.g. the extracted feature of the reference frame or the obtained classification and/or regression weight, as template feature data. Li further teaches aligning an order of such template feature data with an order of data referenced by filter coefficients in a convolution computation processing. Particularly, the classification weight and regression weight serve as filter coefficients in a convolution operation; the classification and regression weights are each convolved with the extracted feature data of the frame under detection to obtain the classification result and regression result, respectively – see e.g. paragraphs 0085-0087. Since the classification and regression weights serve as filter coefficients in the convolution operation, the order of the classification and regression weights would necessarily be aligned with an order of data referenced by filter coefficients in a convolution computation processing, in this case, with the extracted feature data of the frame under detection. Li teaches that the above-described convolution operations can be performed via a suitably programmed processor – see e.g. paragraphs 0188-0191. Such a processor performing the above-noted operations is considered a “storage control unit” like claimed.); and
a convolution operation unit configured to compute new feature data by a convolution operation between feature data and filter coefficients, and compute, by a convolution operation between feature data and the template feature data, correlation data between the feature data and the template feature data (see e.g. paragraphs 0043-0047 and 0069-0070: Like noted above, Li teaches extracting a feature of a “reference frame” and a feature of a “frame under detection,” wherein the reference frame indicates a target object and the frame under detection, which can occur subsequent to the reference frame in a video sequence, indicates a current frame which is to be subject to target object detection. Li particularly teaches that such feature extraction can be performed via a convolution operation by a convolutional layer in a neural network – see e.g. paragraphs 0049-0051 and 0065. Such a convolutional layer necessarily comprises filter coefficients, and thus Li is considered to teach computing new feature data, e.g. extracting a feature of a frame under detection, by a convolution operation between feature data and filter coefficients. Moreover, like further noted above, Li teaches that the extracted feature of the reference frame and the extracted feature of the frame under detection are input as a feature group into a detector, which outputs a corresponding classification result and a regression result, wherein the classification result indicates a probability that a candidate box is a bounding box for the target object and the regression result indicates a position offset of the candidate box – see e.g. paragraphs 0053-0058 and 0071-0072. As further noted above, Li discloses that such detection entails obtaining a classification weight for the detector and a regression weight for the detector based on the extracted feature of the reference frame, and then processing the extracted feature of the frame under detection with these classification and regression weights to obtain the classification and regression results, respectively – see e.g. paragraphs 0073-0074, 0078 and 0081. In particular, a convolution operation is performed with the classification weight on the extracted feature of the fame under detection so as to obtain the classification result, and a convolution operation is performed with the regression weight on the extracted feature of the frame under detection so as to obtain the regression result – see e.g. paragraph 0085-0087. As noted above, the extracted feature of the reference frame, or the obtained classification and/or regression weight based thereon, is considered “template feature data” like claimed. The classification result and/or regression result can be considered “correlation data” like claimed. Consequently, Li is further considered to teach computing, by a convolution operation between feature data and template feature data, i.e. between the extracted feature of the frame under detection and the classification and/or regression weights, correlation data between the feature data and the template feature data, i.e. the classification and/or regression results. As noted above, the claimed “convolution operation unit” is interpreted under 35 U.S.C. 112(f), and is therefore interpreted to cover the corresponding structure described in the specification as performing the claimed function. The specification of the instant application teaches that the convolution operation unit is implemented via a “computation processing unit” having a multiplier and a cumulative adder – see e.g. paragraphs 0034 and 0040 of the application as filed. Correspondingly, Li teaches that the above-described convolution operations can be performed via a suitably programmed processor, which understandably comprises a multiplier and cumulative adder – see e.g. paragraphs 0188-0191. Accordingly, such a processor programmed to perform the convolution operations described by Li is considered a “convolution operation unit” like claimed.).
Li thus teaches an information processing apparatus similar to that of claim 1, which is operable to perform computation processing in a neural network. However, while Li teaches that the information processing apparatus comprises storage (see e.g. paragraph 0193), Li does not particularly disclose: (i) that the feature data is stored in a “feature storage unit” of the information processing apparatus, wherein the feature storage unit is configured to store feature data; (ii) that the filter coefficients and template feature data are stored in a “coefficient storage unit” of the information processing apparatus, wherein the coefficient storage unit is configured to store filter coefficients of the neural network; and (iii) that the storage control unit is further configured to convert the template feature data into a coefficient storage format in which feature amounts of the template feature data are arranged, in an order in which the filter coefficients are referenced by the convolution operation unit in the convolution computation processing, and stored in units of an address, the coefficient storage format being the same as a storage format of the filter coefficients, and store, in a first memory region different from a second memory region in which the filter coefficients are stored in the coefficient storage unit, the converted template feature data, as is required by claim 1. Li also does not disclose that the information processing apparatus comprises a selection unit configured to designate the second memory region as a coefficient reference region in a case where the convolution operation unit computes new feature data, and designate the first memory region as the coefficient reference region in a case where the convolution operation unit computes correlation data, wherein the convolution operation unit is configured to compute the new feature data by referencing the second memory region designated as the coefficient reference region, and compute the correlation data by referencing the first memory region designated as the coefficient reference region, and wherein the convolution operation unit is configured to perform the convolution operation for computing the new feature data and the convolution operation for computing the correlation data using a same multiplier and cumulative adder, as is further required by claim 1.
Huang generally describes a memory chip capable of performing an artificial intelligence operation (see e.g. paragraph 0005). Regarding the claimed invention, Huang particularly teaches that the memory chip comprises: (i) a coefficient storage unit (i.e. a weight data area or cache) configured to store coefficients (i.e. weights) of a neural network; (ii) a feature storage unit (i.e. a feature map data area or cache) configured to store feature data; and (iii) a storage control unit (e.g. a memory controller or AI engine functionality) configured to store data in the coefficient storage unit in a coefficient storage format in which values of coefficients are arranged, in an order in which the coefficients are referenced by a convolution operation unit (i.e. an AI engine) in convolution computation processing, and stored in units of an address; and (iv) wherein the convolution operation unit is configured to compute new feature data by an operation (e.g. a convolution operation) between feature data stored in the feature storage unit and coefficients (i.e. weights) stored in the coefficient storage unit (see e.g. paragraphs 0010-0012, 0014, 0020-0021, 0036, 0045-0049 and 0052-0054).
It would have been obvious to one of ordinary skill in the art, having the teachings of Li and Huang before the effective filing date of the claimed invention, to modify the information processing apparatus taught by Li so as to include the memory chip taught by Huang to execute the neural network, wherein the memory chip comprises, inter alia: (i) a coefficient storage unit configured to store the coefficients (i.e. the filter coefficients) of the neural network; (ii) a feature storage unit configured to store the feature data; (iii) a storage control unit configured to store data in the coefficient storage unit in a coefficient storage format in which values of filter coefficients are arranged, in an order in which the filter coefficients are referenced by a convolution operation unit in convolution computation processing, and stored in units of an address; and (iv) the convolution operation unit, which is configured to perform the convolution operations required by the neural network (i.e. to compute the new feature data by a convolution operation between feature data stored in the feature storage unit and filter coefficients stored in the coefficient storage unit, and to compute, by a convolution operation between feature data stored in the feature storage unit and the template feature data, correlation data between the feature data stored in the feature storage unit and the template feature data). It would have been advantageous to one of ordinary skill to utilize such a memory chip because it can improve performance of artificial intelligence operations, as is taught by Huang (see e.g. paragraphs 0003-0005).
Like noted above, Huang teaches that convolution filter coefficients (i.e. weights) are stored, and supplied to the AI engine, in a coefficient storage format in which values of the filter coefficients are arranged, in an order in which the filter coefficients are referenced by the convolution operation unit (i.e. AI engine) in convolution computation processing, and stored in units of an address (see e.g. paragraphs 0010-0011, 0045-0049, and 0053-0054). It thus follows that the template feature data described by Li, which are similarly used as weights in convolution operations (see e.g. paragraphs 0073-0074 and 0085 of Li), would necessarily be converted (i.e. by the storage control unit) into the same coefficient storage format in which values (i.e. feature amounts) of the coefficients (i.e. the template feature data) are arranged, in an order in which the filter coefficients are referenced by the convolution operation unit in the convolution computation processing, and stored in units of an address. This would be necessary so that the template feature data can be appropriately accessed by the AI engine (i.e. convolution operation unit) and used for convolution operations.
Cui generally describes “a Siamese network tracking scheme based on PYNQ framework, which is deployed on ZYNQ platform.” (Abstract). Regarding the claimed invention, Cui particularly demonstrates that platform comprises a coefficient storage unit (i.e. input buffers) configured to store filter coefficients (e.g. weights and biases) of a neural network, wherein template feature data is stored in a first memory region (e.g. in a template image buffer or template output buffer) of the coefficient storage unit, and the first memory region is different from a second memory region (e.g. a weight buffer) that the filter coefficients are stored in the coefficient storage unit (see e.g. sections IV.A SNA_IP Core and IV.B RPNA_IP core, and Figures 2 and 3).
It would have been obvious to one of ordinary skill in the art, having the teachings of Li, Huang and Cui before the effective filing date of the claimed invention, to modify the information processing apparatus taught by Li and Huang such that the coefficient storage unit comprises a first memory region (i.e. a template image buffer or template output buffer) to store the converted template feature data, and a second memory region to store the filter coefficients, as is taught by Cui. It would have been advantageous to one of ordinary skill to utilize such a combination, because it would enable the template data to be readily accessed for processing, as is evident from Cui (see e.g. sections IV.A SNA_IP Core and IV.B RPNA_IP core, and Figures 2 and 3).
Like alluded to above, the neural network described by Li comprises a plurality of convolution operations in sequence, including a first convolution operation to compute new feature data (e.g. to compute a feature of a frame under detection) by a convolution operation between feature data and filter coefficients (see e.g. paragraphs 0049-0051 and 0070), and a subsequent convolution operation to compute correlation data by a convolution operation between feature data and the template feature data (i.e. between a feature of a frame under detection and classification weights) (see e.g. paragraphs 0073 and 0085). Thus, as implemented on a memory chip like described by Huang, and which has a first memory region for storing template feature data and a second memory region for storing filter coefficients like taught by Cui, a selection of the filter coefficients (i.e. the second memory region) would be necessary to access those coefficients to compute the new feature data and a selection of the template feature (i.e. the first memory region) data would be necessary to access the template feature data to compute the correlation data. These selections would be necessary to provide the appropriate weights (i.e. filter coefficients or template feature data) to the AI engine (i.e. the convolution operation unit) to perform the respective convolution operations. The functionality necessary for performing such a selection is considered a “selection unit” like claimed, which is configured to designate the second memory region as a coefficient reference region in a case where the convolution operation unit computes new feature data, and designate the first memory region as the coefficient reference region in a case where the convolution operation unit computes correlation data, and wherein the convolution operation unit computes the new feature data by referencing the second memory region designated as the coefficient reference region, and computes the correlation data by referencing the first memory region designated as the coefficient reference region, as is claimed. The AI engine is considered a multiplier and cumulative adder, which are necessary operations for performing the convolution operation. Accordingly, it follows that the convolution operation unit (i.e. the AI engine) would perform the convolution operation for computing the new feature data and the convolution operation for computing the correlation data using the same multiplier and cumulative adder. Consequently, Li, Huang and Cui are considered to teach, to one of ordinary skill in the art, an information processing apparatus like that of claim 1.
As per claim 2, Li further teaches that a part of the feature data (i.e. feature data of the reference frame) computed by the convolution operation unit is applied as template feature data (i.e. as classification weights and/or regression weights) (see e.g. paragraphs 0073, 0078 and 0081). Like described above, it would have been obvious to configure the storage control unit taught by Li and Huang so as to store, in a first memory region different from a second memory region that the filter coefficients are stored in the coefficient storage unit, the converted template feature data, as is taught by Cui. Accordingly, the above-described combination of Li, Huang and Cui is further considered to teach an information processing apparatus like that of claim 2, in which the storage control unit stores in the coefficient storage unit a part of the feature data (i.e. the classification weights and/or regression weights), which is computed by the convolution operation unit, as the template feature data.
As per claim 4, it would have been obvious, as is described above, to modify the information processing apparatus taught by Li so as to include the memory chip taught by Huang to execute the neural network, wherein the memory chip comprises a storage control unit configured to store, in a coefficient storage unit, data in a coefficient storage format in which values of filter coefficients are arranged in an order in which the filter coefficients are referenced by a convolution operation unit in convolution computation processing. Huang teaches that convolution filter coefficients (i.e. weights) are stored in the coefficient storage unit, and supplied to the AI engine (i.e. convolution operation unit), in this coefficient storage format (see e.g. paragraphs 0010-0011, 0045-0049, and 0053-0054). It thus follows that the template feature data described by Li, which are similarly used as weights in convolution operations (see e.g. paragraphs 0073-0074 and 0085 of Li), would necessarily be converted (i.e. by the storage control unit) into the same coefficient storage format in which values (i.e. feature amounts) of the coefficients (i.e. the template feature data) are arranged in an order in which the filter coefficients are referenced by the convolution operation unit. This would be necessary so that the template feature data can be appropriately accessed by the AI engine (i.e. convolution operation unit) and used for convolution operations. That is, the storage control unit is necessarily configured to convert the template feature data into the same format as the filter coefficients, and store the converted template feature data in the coefficient storage unit, so that it can be accessed and used by the AI engine. The above-described combination of Li, Huang and Cui is thus further considered to teachy an information processing apparatus like that of claim 4.
As per claim 7, Li further teaches that the feature data is a feature map (i.e. a feature of a frame under detection), and that the template feature data (i.e. the classification and/or regression weights) comprises feature amounts in a region of a target object (i.e. as indicated in a reference frame) to be a target of tracking in the feature map (see e.g. paragraphs 0006, 0044-0047, 0049, 0065 and 0073). Like described above, it would have been obvious to configure the storage control unit taught by Li and Huang so as to store, in a first memory region different from a second memory region that the filter coefficients are stored in the coefficient storage unit, the template feature data, as is taught by Cui. The storage control unit would thus store in the coefficient storage unit feature amounts in a region of a target object to be a target of tracking in the feature map as the template feature data. Accordingly, the above-described combination of Li, Huang and Cui is further considered to teach an information processing apparatus like that of claim 7.
As per claim 12, Li further teaches periodically updating the template feature data (i.e. by taking the bounding box image of the target object in the frame under detection as a next reference frame) (see e.g. paragraphs 0063-0064). As described above, it would have been obvious to configure the storage control unit taught by Li and Huang so as to store, in a first memory region different from a second memory region that the filter coefficients are stored in the coefficient storage unit, the template feature data, as is taught by Cui. It thus follows that the storage control unit would particularly be configured to periodically update the template feature data stored in the coefficient storage unit, i.e. to determine whether or not to update the template feature data and, in a case where it determines to update the template feature data, to transfer new template feature data to the coefficient storage unit. Accordingly, the above-described combination of Li, Huang and Cui is further considered to teach an information processing apparatus like that of claim 12.
Regarding claims 13 and 14, Li describes a method for detecting a target object, as is noted above, and which includes inputting each of at least two feature groups output from at least two network layers of a neural network into a detector, and whereby the detector outputs a classification result and a regression result for each group (see e.g. paragraphs 0005-0008). Li particularly teaches that such a method comprises
setting a part of previously obtained feature data as template feature data (see e.g. paragraphs 0043-0047 and 0069-0070: Li teaches extracting a feature of a “reference frame” and a feature of a “frame under detection,” wherein the reference frame indicates a target object and the frame under detection, which can occur subsequent to the reference frame in a video sequence, indicates a current frame which is to be subject to target object detection. The extracted feature of the reference frame and the extracted feature of the frame under detection are input as a feature group into a detector, which outputs a corresponding classification result and a regression result, wherein the classification result indicates a probability that a candidate box is a bounding box for the target object and the regression result indicates a position offset of the candidate box – see e.g. paragraphs 0053-0058 and 0071-0072. Li particularly discloses that such detection entails obtaining a classification weight for the detector and a regression weight for the detector based on the extracted feature of the reference frame, wherein the extracted feature of the frame under detection is then processed with the classification and regression weights to obtain the classification and regression results – see e.g. paragraphs 0073-0074, 0078 and 0081. The extracted feature of the reference frame, or the obtained classification and/or regression weight based thereon, is considered “template feature data” like claimed. Accordingly, Li teaches setting a part of previously obtained feature data, e.g. the extracted feature of the reference frame or the obtained classification and/or regression weight, as template feature data.);
aligning an order of the template feature data with an order of data referenced by the filter coefficients in the convolution computation processing (see e.g. paragraphs 0085-0087: Li discloses that the classification weight and regression weight serve as filter coefficients in a convolution operation; the classification and regression weights are each convolved with the extracted feature data of the frame under detection to obtain the classification result and regression result, respectively. Since the classification and regression weights serve as filter coefficients in the convolution operation, the order of the classification and regression weights would necessarily be aligned with an order of data referenced by filter coefficients in a convolution computation processing, in this case, with the extracted feature data of the frame under detection. Accordingly, Li further teaches aligning an order of such template feature data with an order of data referenced by filter coefficients in a convolution computation processing.); and
computing new feature data by a convolution operation between feature data and filter coefficients, and computing, by a convolution operation between feature data and the template feature data, correlation data between the feature data and the template feature data (see e.g. paragraphs 0043-0047 and 0069-0070: Like noted above, Li teaches extracting a feature of a “reference frame” and a feature of a “frame under detection,” wherein the reference frame indicates a target object and the frame under detection, which can occur subsequent to the reference frame in a video sequence, indicates a current frame which is to be subject to target object detection. Li particularly teaches that such feature extraction can be performed via a convolution operation by a convolutional layer in a neural network – see e.g. paragraphs 0049-0051 and 0065. Such a convolutional layer necessarily comprises filter coefficients, and thus Li is considered to teach computing new feature data, e.g. extracting a feature of a frame under detection, by a convolution operation between feature data and filter coefficients. Moreover, like further noted above, Li teaches that the extracted feature of the reference frame and the extracted feature of the frame under detection are input as a feature group into a detector, which outputs a corresponding classification result and a regression result, wherein the classification result indicates a probability that a candidate box is a bounding box for the target object and the regression result indicates a position offset of the candidate box – see e.g. paragraphs 0053-0058 and 0071-0072. As further noted above, Li discloses that such detection entails obtaining a classification weight for the detector and a regression weight for the detector based on the extracted feature of the reference frame, and then processing the extracted feature of the frame under detection with these classification and regression weights to obtain the classification and regression results, respectively – see e.g. paragraphs 0073-0074, 0078 and 0081. In particular, a convolution operation is performed with the classification weight on the extracted feature of the frame under detection so as to obtain the classification result, and a convolution operation is performed with the regression weight on the extracted feature of the frame under detection so as to obtain the regression result – see e.g. paragraph 0085-0087. As noted above, the extracted feature of the reference frame, or the obtained classification and/or regression weight based thereon, is considered “template feature data” like claimed. The classification result and/or regression result can be considered “correlation data” like claimed. Consequently, Li is further considered to teach computing, by a convolution operation between feature data and the template feature data, i.e. between the extracted feature of the frame under detection and the classification and/or regression weights, correlation data between the feature data and the template feature data, i.e. the classification and/or regression results.).
Li thus teaches an information processing method similar to that of claim 13. Li discloses that such teachings can be implemented via a computer software program stored on a non-transitory computer-readable storage medium (see e.g. paragraph 0196). A non-transitory computer-readable storage medium comprising a software program to implement the above-described teachings of Li is considered a non-transitory computer-readable storage medium similar to that of claim 14. However, Li does not particularly teach: (i) storing in a coefficient storage unit filter coefficients of the neural network; (ii) storing in a feature storage unit feature data; (iii) converting the template feature data into a coefficient storage format in which feature amounts of the template feature data are arranged, in an order in which the filter coefficients are referenced by a convolution operation unit in the convolution computation processing, and stored in units of an address, the coefficient storage format being the same as a storage format of the filter coefficients; (iv) storing, in a first memory region different from a second memory region in which the filter coefficients are stored in the coefficient storage unit, the converted template feature data; and (iv) designating the second memory region as a coefficient reference region in a case where the convolution operation unit computes new feature data, and designate the first memory region as the coefficient reference region in a case where the convolution operation unit computes correlation data; (v) wherein the convolution operation unit is configured to compute the new feature data by referencing the second memory region designated as the coefficient reference region, and compute the correlation data by referencing the first memory region designated as the coefficient reference region; and (vii) wherein the convolution operation unit is configured to perform the convolution operation for computing the new feature data and the convolution operation for computing the correlation data using a same multiplier and cumulative adder, as is required by claims 13 and 14.
Huang generally describes a memory chip capable of performing an artificial intelligence operation (see e.g. paragraph 0005). Regarding the claimed invention, Huang particularly teaches that the memory chip comprises: (i) a coefficient storage unit (i.e. a weight data area or cache) configured to store coefficients (i.e. weights) of a neural network; (ii) a feature storage unit (i.e. a feature map data area or cache) configured to store feature data; and (iii) a storage control unit (e.g. a memory controller or AI engine functionality) configured to store data in the coefficient storage unit in a coefficient storage format in which values of coefficients are arranged, in an order in which the coefficients are referenced by a convolution operation unit (i.e. an AI engine) in convolution computation processing, and stored in units of an address; and (iv) wherein the convolution operation unit is configured to compute new feature data by an operation (e.g. a convolution operation) between feature data stored in the feature storage unit and coefficients (i.e. weights) stored in the coefficient storage unit (see e.g. paragraphs 0010-0012, 0014, 0020-0021, 0036, 0045-0049 and 0052-0054).
It would have been obvious to one of ordinary skill in the art, having the teachings of Li and Huang before the effective filing date of the claimed invention, to modify the information processing method and non-transitory computer-readable storage medium taught by Li so as to execute the neural network on a memory chip like taught by Huang, wherein the memory chip comprises, inter alia: (i) a coefficient storage unit configured to store the coefficients (i.e. the filter coefficients) of the neural network; (ii) a feature storage unit configured to store the feature data; (iii) a storage control unit configured to store data in the coefficient storage unit in a coefficient storage format in which values of filter coefficients are arranged, in an order in which the filter coefficients are referenced by a convolution operation unit in convolution computation processing, and stored in units of an address; and (iv) the convolution operation unit, which is configured to perform the convolution operations required by the neural network (i.e. to compute the new feature data by a convolution operation between feature data stored in the feature storage unit and filter coefficients stored in the coefficient storage unit, and to compute, by a convolution operation between feature data stored in the feature storage unit and the template feature data, correlation data between the feature data stored in the feature storage unit and the template feature data). It would have been advantageous to one of ordinary skill to utilize such a memory chip because it can improve performance of artificial intelligence operations, as is taught by Huang (see e.g. paragraphs 0003-0005).
Like noted above, Huang teaches that convolution filter coefficients (i.e. weights) are stored, and supplied to the AI engine, in a coefficient storage format in which values of the filter coefficients are arranged, in an order in which the filter coefficients are referenced by the convolution operation unit (i.e. AI engine) in convolution computation processing, and stored in units of an address (see e.g. paragraphs 0010-0011, 0045-0049, and 0053-0054). It thus follows that the template feature data described by Li, which are similarly used as weights in convolution operations (see e.g. paragraphs 0073-0074 and 0085 of Li), would necessarily be converted (i.e. by the storage control unit) into the same coefficient storage format in which values (i.e. feature amounts) of the coefficients (i.e. the template feature data) are arranged, in an order in which the filter coefficients are referenced by the convolution operation unit in the convolution computation processing, and stored in units of an address. This would be necessary so that the template feature data can be appropriately accessed by the AI engine (i.e. convolution operation unit) and used for convolution operations.
Cui generally describes “a Siamese network tracking scheme based on PYNQ framework, which is deployed on ZYNQ platform.” (Abstract). Regarding the claimed invention, Cui particularly demonstrates that platform comprises a coefficient storage unit (i.e. input buffers) configured to store filter coefficients (e.g. weights and biases) of a neural network, wherein template feature data is stored in a first memory region (e.g. in a template image buffer or template output buffer) of the coefficient storage unit, and the first memory region is different from a second memory region (e.g. a weight buffer) in which the filter coefficients are stored in the coefficient storage unit (see e.g. sections IV.A SNA_IP Core and IV.B RPNA_IP core, and Figures 2 and 3).
It would have been obvious to one of ordinary skill in the art, having the teachings of Li, Huang and Cui before the effective filing date of the claimed invention, to modify the information processing method and non-transitory computer-readable storage medium taught by Li and Huang such that the coefficient storage unit comprises a first memory region (i.e. a template image buffer or template output buffer) to store the converted template feature data, and a second memory region to store the filter coefficients, as is taught by Cui. It would have been advantageous to one of ordinary skill to utilize such a combination, because it would enable the template data to be readily accessed for processing, as is evident from Cui (see e.g. sections IV.A SNA_IP Core and IV.B RPNA_IP core, and Figures 2 and 3). Huang and Cui thus further teach storing, in a first memory region different from a second memory region in which the filter coefficients are stored in the coefficient storage unit, the converted template feature data.
Like alluded to above, the neural network described by Li comprises a plurality of convolution operations in sequence, including a first convolution operation to compute new feature data (e.g. to compute a feature of a frame under detection) by a convolution operation between feature data and filter coefficients (see e.g. paragraphs 0049-0051 and 0070), and a subsequent convolution operation to compute correlation data by a convolution operation between feature data and the template feature data (i.e. between a feature of a frame under detection and classification weights) (see e.g. paragraphs 0073 and 0085). Thus, as implemented on a memory chip like described by Huang, and which has a first memory region for storing template feature data and a second memory region for storing filter coefficients like taught by Cui, a selection of the filter coefficients (i.e. in the second memory region) would be necessary to access those coefficients to compute the new feature data and a selection of the template feature (i.e. in the first memory region) data would be necessary to access the template feature data to compute the correlation data. These selections would be necessary to provide the appropriate weights (i.e. filter coefficients or template feature data) to the AI engine (i.e. the convolution operation unit) to perform the respective convolution operations. The above-described combination of Li, Huang and Cui is thus considered to necessitate designating the second memory region as a coefficient reference region in a case where the convolution operation unit computes new feature data, and designating the first memory region as the coefficient reference region in a case where the convolution operation unit computes correlation data, and wherein the convolution operation unit computes the new feature data by referencing the second memory region designated as the coefficient reference region, and computes the correlation data by referencing the first memory region designated as the coefficient reference region, as is claimed. The AI engine is considered a multiplier and cumulative adder, which are necessary operations for performing the convolution operation. Accordingly, it follows that the convolution operation unit (i.e. the AI engine) would perform the convolution operation for computing the new feature data and the convolution operation for computing the correlation data using the same multiplier and cumulative adder. Consequently, Li, Huang and Cui are considered to teach, to one of ordinary skill in the art, an information processing method like that of claim 13, and a non-transitory computer-readable storage medium like that of claim 14.
Claims 3, 5, 6 and 8-10 are rejected under 35 U.S.C. 103 as being unpatentable over the above-described combination of Li, Huang and Cui, and also over U.S. Patent No. 12,159,214 to Ko et al. (“Ko”).
Regarding claim 3, Li, Huang and Cui teach an information processing apparatus like that of claim 1, as is described above, and which comprises a convolution operation unit configured to compute new feature data. Li particularly teaches that a part of the feature data (i.e. feature data of the reference frame) computed by the convolution operation unit is applied as template feature data (i.e. as classification weights and/or regression weights) (see e.g. paragraphs 0073, 0078 and 0081). Li, Huang and Cui, however, do not explicitly disclose that the information processing apparatus further comprises a transformation unit configured to non-linearly transform feature data computed by the convolution operation unit, and wherein the storage control unit stores in the coefficient storage unit a part of the feature data non-linearly transformed by the transformation unit as the template feature data, as is required by claim 3.
Ko generally describes an integrated circuit for executing a machine-trained network (e.g. a convolutional neural network), wherein the integrated circuit comprises, inter alia, a coefficient storage unit (i.e. a weight memory and/or filter slice buffer) configured to store filter coefficients (i.e. weight values) of the machine-trained network, and an operation unit (e.g. a microprocessor, core controller and/or adder trees) configured to compute new feature data by an operation (e.g. a dot product) between feature data stored in the feature storage unit and filter coefficients stored in the filter coefficient storage unit (see e.g. column 1, lines 21-55; column 16, line 66 – column 17, line 19; column 30, lines 12-58; column 31, lines 4-44; column 32, lines 1-16; and FIG. 17). The integrated circuit described by Ko particularly comprises a transformation unit (i.e. a “post-processor”) configured to non-linearly transform feature data computed by a convolution (i.e. dot product) operation (see e.g. column 1, lines 40-55; column 7, lines 46-59; column 8, lines 36-59; and column 29, lines 16-41).
It would have been obvious to one of ordinary skill in the art, having the teachings of Li, Huang, Cui and Ko before the effective filing date of the claimed invention, to further the information processing apparatus taught by Li, Huang and Cui so as to comprise a transformation unit like taught by Ko, which is configured to non-linearly transform feature data computed by a convolution operation (i.e. the convolution operation unit). It would have been particularly obvious to apply the transformation unit so as to non-linearly transform the part of the feature data computed by the convolution operation unit used as the template feature data (i.e. as classification weights and/or regression weights). It would have been advantageous to one of ordinary skill to utilize such a transformation unit because non-linear transformations are typical in convolutional layers of neural networks, as is taught by Ko (see e.g. column 1, lines 40-55). Like described above, it would have been obvious to configure the storage control unit taught by Li and Huang so as to store, in a first memory region different from a second memory region that the filter coefficients are stored in the coefficient storage unit, the template feature data, as is taught by Cui. Accordingly, it follows that the storage control unit would particularly store, in the coefficient storage unit, a part of the feature data non-linearly transformed by the transformation unit as the template feature data. Li, Huang, Cui and Ko are thus considered to teach an information processing apparatus like that of claim 3.
Regarding claim 5, Li, Huang and Cui teach an information processing apparatus like that of claim 1, as is described above, and which comprises a coefficient storage unit comprising a first memory region for storing template feature data and a second memory region for storing filter coefficients. Li, Huang and Cui, however, do not explicitly disclose that the coefficient storage unit is a single memory address, like required by claim 5.
Like noted above, Ko generally describes an integrated circuit for executing a machine-trained network (e.g. a convolutional neural network), wherein the integrated circuit comprises, inter alia, a coefficient storage unit (i.e. a weight memory and/or filter slice buffer) configured to store filter coefficients (i.e. weight values) of the machine-trained network (see e.g. column 1, lines 21-55; column 16, line 66 – column 17, line 19; column 30, lines 12-58; column 31, lines 4-44; column 32, lines 1-16; and FIG. 17). Ko particularly suggests that the coefficient storage unit can be part of a single unified memory apparatus comprising a predefined memory region for each of the weights (see e.g. column 11, lines 1-16; and column 16, line 66 – column 17, line 19).
It would have been obvious to one of ordinary skill in the art, having the teachings of Li, Huang, Cui and Ko before the effective filing date of the claimed invention, to modify the information processing apparatus taught by Li, Huang and Cui such that the coefficient storage unit can be part of a single unified memory apparatus comprising a predefined memory region for each of the weights, as is taught by Ko. It thus follows that the template feature data would be assigned to and stored in a predefined memory region, while the other filter coefficients would similarly be assigned to and stored in another predefined memory region. It would have been advantageous to one of ordinary skill to utilize such a combination because it would enable the neural network to be efficiently executed, as is suggested by Ko (see e.g. column 11, lines 1-16; and column 16, line 66 – column 17, line 19). Accordingly, Li, Huang, Cui and Ko are further considered to teach an information processing apparatus like that of claim 5.
Regarding claim 6, Li, Huang and Cui teach an information processing apparatus like that of claim 1, as is described above, and which comprises a coefficient storage unit comprising a first memory region for storing template feature data and a second memory region for storing filter coefficients. Li, Huang and Cui, however, do not explicitly disclose that the coefficient storage unit comprises a memory apparatus, serving as the second memory region, configured to store the filter coefficients and a memory apparatus, serving as the first memory region, configured to store the template feature data, as is required by claim 6.
Like noted above, Ko generally describes an integrated circuit for executing a machine-trained network (e.g. a convolutional neural network), wherein the integrated circuit comprises, inter alia, a coefficient storage unit (i.e. a weight memory and/or filter slice buffer) configured to store filter coefficients (i.e. weight values) of the machine-trained network (see e.g. column 1, lines 21-55; column 16, line 66 – column 17, line 19; column 30, lines 12-58; column 31, lines 4-44; column 32, lines 1-16; and FIG. 17). Ko suggests that a coefficient storage unit can be part of a single unified memory comprising a predefined memory region for each of the neural weights (see e.g. column 11, lines 1-16; and column 16, line 66 – column 17, line 19). Ko further discloses that, in some embodiments, the unified memory can be comprised of a plurality of memory apparatuses (e.g. banks of SRAMS), wherein the neural network weights are divided across multiple apparatuses (see e.g. column 2, lines 40-48; column 11, lines 1-16; column 19, lines 38-61; and FIG. 11).
It would have been obvious to one of ordinary skill in the art, having the teachings of Li, Huang, Cui and Ko before the effective filing date of the claimed invention, to further modify the information processing apparatus taught by Li, Huang and Cui such that the coefficient storage unit can be comprised of a plurality of memory apparatuses, wherein the neural network weights are divided across multiple apparatuses, as is taught by Ko. It thus follows that the template feature data can be assigned to and stored in a predefined first memory region (i.e. of a first memory apparatus), while other filter coefficients would be assigned to and stored in a second predefined memory region (i.e. of another memory apparatus). It would have been advantageous to one of ordinary skill to utilize such a combination because it would enable the neural network to be efficiently executed, as is suggested by Ko (see e.g. column 2, lines 40-48; column 11, lines 1-16; column 19, lines 38-61; and FIG. 11). Accordingly, Li, Huang, Cui and Ko are further considered to teach an information processing apparatus like that of claim 6.
Regarding claim 8, Li, Huang and Cui teach an information processing apparatus like that of claim 1, as is described above, and which comprises a convolution operation unit configured to compute new feature data. Li further suggests that the convolution operation unit comprises: (i) a first convolution operation unit configured to perform a convolution operation using filter coefficients (i.e. to perform a convolution operation on a frame under detection, which is necessarily performed using filter coefficients, to extract a feature of the frame under detection) (see e.g. paragraphs 0044-0046, 0049, 0051, 0053 and 0065); (ii) a second convolution operation unit configured to perform a convolution operation between a result of a convolution operation by the first convolution operation unit and the template feature data (i.e. the classification and/or regression weights) (see e.g. paragraphs 0070-0074 and 0085); and (iii) a third convolution operation unit configured to perform a convolution operation between a result of the convolution operation by the second convolution operation unit and the filter coefficients (see e.g. paragraphs 0090-0091). As described above, it would have been obvious to configure the storage control unit taught by Li and Huang so as to store, in a first memory region different from a second memory region that the filter coefficients are stored in the coefficient storage unit, the template feature data, as is taught by Cui. Accordingly, the above-described combination of Li, Ko and Cui is further considered to teach that the convolution operation unit comprises: (i) a first convolution operation unit configured to perform a convolution operation using filter coefficients stored in the coefficient storage unit; (ii) a second convolution operation unit configured to perform a convolution operation between a result of a convolution operation by the first convolution operation unit and the template feature data stored in the coefficient storage unit; and (iii) a third convolution operation unit configured to perform a convolution operation between a result of the convolution operation by the second convolution operation unit and the filter coefficients stored in the coefficient storage unit. Li however does not explicitly disclose that the second convolution operation unit is configured to perform a convolution operation between a result of a nonlinear transformation on a result of the convolution operation by the first convolution operation unit and the template feature data like required by claim 8.
Like noted above, Ko generally describes an integrated circuit for executing a machine-trained network (e.g. a convolutional neural network), wherein the integrated circuit comprises, inter alia, a coefficient storage unit (i.e. a weight memory and/or filter slice buffer) configured to store filter coefficients (i.e. weight values) of the machine-trained network, and an operation unit (e.g. a microprocessor, core controller and/or adder trees) configured to compute new feature data by an operation (e.g. a dot product) between feature data stored in the feature storage unit and filter coefficients stored in the filter coefficient storage unit (see e.g. column 1, lines 21-55; column 16, line 66 – column 17, line 19; column 30, lines 12-58; column 31, lines 4-44; column 32, lines 1-16; and FIG. 17). Ko particularly generally teaches applying a nonlinear transformation on a result of a convolution (i.e. dot product) operation (see e.g. column 1, lines 40-55; column 7, lines 46-59; column 8, lines 36-59; and column 29, lines 16-41).
It would have been obvious to one of ordinary skill in the art, having the teachings of Li, Huang, Cui and Ko before the effective filing date of the claimed invention, to modify the information processing apparatus taught by Li, Huang and Cui so as to apply a nonlinear transformation like taught by Ko on the results of the convolution operations produced by the convolution operation unit. It thus follows that the second convolution operation unit would perform a convolution operation between a result of a nonlinear transformation on a result of the convolution operation by the first convolution operation unit and the template feature data stored in the coefficient storage unit. It would have been advantageous to one of ordinary skill to utilize such a nonlinear transformation because they are typical in convolutional layers of neural networks, as is taught by Ko (see e.g. column 1, lines 40-55). Accordingly, Li, Huang, Cui and Ko are considered to teach, to one of ordinary skill in the art, an information processing apparatus like that of claim 8.
As per claim 9, Li further teaches that the information processing apparatus comprises a detection unit configured to detect an object based, in part, on a result of the convolution operation by the third convolution operation unit (see e.g. paragraphs 0053-0054, 0060-0062, 0085 and 0090-0091). As described above, it would have been obvious to modify the information processing apparatus taught by Li, Huang and Cui so as to apply a nonlinear transformation like taught by Ko on the results of the convolution operations produced thereby. Accordingly, it follows that the detection unit would detect the object based on a result of a nonlinear transformation performed on a result of the convolution operation by the third convolution operation unit. Li, Huang, Cui and Ko are thus further considered to teach an information processing apparatus like that of claim 9.
As per claim 10, it would have been obvious, as is described above, to configure the storage control unit taught by Li and Huang so as to store, in a first memory region different from a second memory region that the filter coefficients are stored in the coefficient storage unit, the template feature data, as is taught by Cui. It thus follows that the coefficient storage unit would hold the filter coefficients that are used by the first convolution operation unit taught by Li and the filter coefficients that are used by the third convolution operation unit taught by Li. Accordingly, the above-described combination of Li, Huang, Cui and Ko is further considered to teach an information processing apparatus like that of claim 10.
Response to Arguments
The Examiner acknowledges the Applicant’s amendments to claims 1, 5, 6, 13 and 14, and cancellation of claims 11 and 15. In response to these amendments, the objections presented in the previous Office Action to claims 1-12 and 15 are respectfully withdrawn.
Regarding the pending claims, the Applicant argues that the claims have been amended to recite sufficient structure, material or acts for performing the claimed functions, and therefore do not invoke 35 U.S.C. § 112(f).
The Examiner, however, respectfully disagrees. It is not clear what structure(s) have been added via the amendments to perform the claimed functions. For example, the “storage control unit” is amended to recite that it “convert[s] the template feature data into a coefficient storage format in which feature amounts of the template feature data are arranged, in an order in which the filter coefficients are referenced by a convolution operation unit in the convolution computation processing, and stored in units of an address, the coefficient storage format being the same as a storage format of the filter coefficients.” The amendments thus add an additional function to be performed by the storage control unit (i.e. converting the template feature data) but do not recite any structure (e.g. a processing unit) for performing the function. Under a similar rationale, the Examiner respectfully maintains that the amendments also fail to add sufficient structure for performing the functions recited by the other limitations noted above that have been interpreted under 35 U.S.C. § 112(f).
The Applicants arguments addressing the 35 U.S.C. § 103 rejections presented in the previous Office Action have been considered, but are moot in view of the new grounds of rejection presented above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BLAINE T BASOM whose telephone number is (571)272-4044. The examiner can normally be reached Monday-Friday, 9:00 am - 5:30 pm, EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Ell can be reached at (571)270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BTB/
9/28/2026
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141
