DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is responsive to the Application filed on 03/29/2023. Claims 1-20 are pending in the case. Claims 1 and 14 are independent claims.
Claim Rejections - 35 U.S.C. § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. §§ 102 and 103 (or as subject to pre-AIA 35 U.S.C. §§ 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant are advised of the obligation under 37 C.F.R. § 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. § 102(b)(2)(C) for any potential 35 U.S.C. § 102(a)(2) prior art against the later invention.
Claims 1-5, 14-17, and 19-20 are rejected under 35 U.S.C. § 103 as being unpatentable over Wu (Wu, Chai Wah. "ProdSumNet: reducing model parameters in deep neural networks via product-of-sums matrix decompositions." arXiv (2019), hereinafter ProdSumNet 2019) in view of Huang et al. (U.S. Pat. App. Pub. No. 2021/0365782, hereinafter Huang).
As to independent claim 1, ProdSumNet 2019 teaches:
A model training method, comprising (Title and abstract):
obtaining a to-be-trained first neural network model, wherein the first neural network model comprises a first operator, and the first operator is used to perform a product operation on input data and a target weight matrix (Page 1, "we consider the following N-layer feed-forward network formulation." Equation 1 reads on the claimed first operator used to perform a product operation on input data. Page 1, "Each Wi is a matrix of length mi x ni and yi is a vector of length ni");
replacing the first operator in the first neural network model with a second operator, to obtain a second neural network model, wherein the second operator is used to perform a product operation on input data and a plurality of sub-weight matrices, and the plurality of sub-weight matrices are obtained by performing matrix factorization on the target weight matrix (Page 2, "reduce the full matrix Wi and vectors bi with a parametrized version that has fewer parameters than the number of entries" Equation 2)….
ProdSumNet 2019 does not appear to expressly teach performing model training on the second neural network model to obtain a target neural network model.
Huang teaches performing model training on the second neural network model to obtain a target neural network model (Paragraph 6, "training the third neural network model").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the matrix decomposition of ProdSumNet 2019 to include the neural network techniques of Huang to reduce development costs and time (see Huang at paragraph 4).
As to dependent claim 2, ProdSumNet 2019 further teaches when a same amount of training data is processed, a time period required by the first neural network model is longer than a time period required by the second neural network model (Page 3, "this can speed up training").
As to dependent claim 3, ProdSumNet 2019 further teaches the plurality of sub-weight matrices comprise a first sub-weight matrix and a second sub-weight matrix, in a process of performing the product operation on the input data and the plurality of sub-weight matrices, the first sub-weight matrix and the second sub-weight matrix are any two matrices that are of the plurality of sub-weight matrices and that are multiplied by each other, and a size of a column in the first sub-weight matrix is the same as a size of a row in the second sub-weight matrix (Page 3, "the column vector [a1,... au]T. In dimensionality reduction via random projection[3], the input row vector x of order n is multiplied (on the right) by a matrix W0 of order n x m where m << n in order to form a much smaller row vector of order m which is then used as input to a neural network with a full dense layer with weight matrix B of order m x k where k is the number of features in the input layer").
As to dependent claim 4, ProdSumNet 2019 further teaches the plurality of sub-weight matrices comprise a matrix 1, a matrix 2, ..., a matrix N-1, and a matrix N, and the second operator is used to perform the following operation: M x matrix 1x matrix 2 x...x matrix N-1x matrix N, wherein M represents the input data, and x represents multiplication (Equation 4); and a size of a row in the target weight matrix is the same as a size of a row in the matrix 1, and a size of a column in the target weight matrix is the same as a size of a column in the matrix N (Page 3, "the column vector [a1,... au]T. In dimensionality reduction via random projection[3], the input row vector x of order n is multiplied (on the right) by a matrix W").
As to dependent claim 5, ProdSumNet 2019 further teaches a size of a row in each of the plurality of sub- weight matrices is less than or equal to the size of the row in the target weight matrix, and a size of a column in each of the plurality of sub-weight matrices is less than or equal to the size of the column in the target weight matrix (Page 3, "a matrix W0 of order n x m where m << n").
As to independent claim 14, ProdSumNet 2019 teaches:
A model training apparatus, comprising (Title and abstract):
an obtaining module, configured to obtain a to-be-trained first neural network model, wherein the first neural network model comprises a first operator, and the first operator is used to perform a product operation on input data and a target weight matrix (Page 1, "we consider the following N-layer feed-forward network formulation." Equation 1 reads on the claimed first operator used to perform a product operation on input data. Page 1, "Each Wi is a matrix of length mi x ni and yi is a vector of length ni");
an operator replacing module, configured to replace the first operator in the first neural network model with a second operator, to obtain a second neural network model, wherein the second operator is used to perform a product operation on input data and a plurality of sub-weight matrices, and the plurality of sub-weight matrices are obtained by performing matrix factorization on the target weight matrix (Page 2, "reduce the full matrix Wi and vectors bi with a parametrized version that has fewer parameters than the number of entries" Equation 2)….
ProdSumNet 2019 does not appear to expressly teach a model training module, configured to perform model training on the second neural network model to obtain a target neural network model.
Huang teaches a model training module, configured to perform model training on the second neural network model to obtain a target neural network model (Paragraph 6, "training the third neural network model").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the matrix decomposition of ProdSumNet 2019 to include the neural network techniques of Huang to reduce development costs and time (see Huang at paragraph 4).
As to dependent claim 15, ProdSumNet 2019 further teaches when a same amount of training data is processed, a time period required by the first neural network model is longer than a time period required by the second neural network model (Page 3, "this can speed up training").
As to dependent claim 16, ProdSumNet 2019 further teaches the plurality of sub-weight matrices comprise a first sub-weight matrix and a second sub-weight matrix, in a process of performing the product operation on the input data and the plurality of sub-weight matrices, the first sub-weight matrix and the second sub-weight matrix are any two matrices that are of the plurality of sub- weight matrices and that are multiplied by each other, and a size of a column in the first sub-weight matrix is the same as a size of a row in the second sub-weight matrix (Page 3, "the column vector [a1,... au]T. In dimensionality reduction via random projection[3], the input row vector x of order n is multiplied (on the right) by a matrix W0 of order n x m where m << n in order to form a much smaller row vector of order m which is then used as input to a neural network with a full dense layer with weight matrix B of order m x k where k is the number of features in the input layer").
As to dependent claim 17, ProdSumNet 2019 further teaches a size of a row in each of the plurality of sub-weight matrices is less than or equal to the size of the row in the target weight matrix, and a size of a column in each of the plurality of sub-weight matrices is less than or equal to the size of the column in the target weight matrix (Page 3, "a matrix W0 of order n x m where m << n").
As to dependent claim 19, Huang further teaches a model training apparatus, wherein the apparatus comprises a memory and a processor, the memory stores code, and the processor is configured to obtain the code, and performs the method according to claim 1 (Figure 12, memory 1104, processor 1102. Paragraph 124 et seq.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the matrix decomposition of ProdSumNet 2019 to include the neural network techniques of Huang to reduce development costs and time (see Huang at paragraph 4).
As to dependent claim 20, Huang further teaches a non-transitory computer storage medium, wherein the computer storage medium stores one or more instructions, and when the instructions are executed by one or more processors, the one or more processors are enabled to perform the method according to claim 1 (Paragraph 124 et seq.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the matrix decomposition of ProdSumNet 2019 to include the neural network techniques of Huang to reduce development costs and time (see Huang at paragraph 4).
Claim 12 is rejected under 35 U.S.C. § 103 as being unpatentable over Wu in view of Huang and Fang Muyuan et al. (Chinese Pat. App. Pub. No. CN-112446462-A, hereinafter Fang Muyuan).
As to dependent claim 12, the rejection of claim 1 is incorporated.
ProdSumNet 2019 does not appear to expressly teach sending the target neural network model to a terminal device.
Fang Muyuan teaches sending the target neural network model to a terminal device (Paragraph 22, "sending neural network model configuration information to the speed measuring device").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the matrix decomposition of ProdSumNet 2019 to include the neural network techniques of Fang Muyuan to reduce computing resources and time (see Fang Muyuan at paragraph 5).
Citation of Pertinent Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Cambricon Technologies Corp Ltd (Chinese Pat. App. Pub. No. CN-111353598-A) teaches a neural network compression method. The first weight matrix is compressed by adopting a compression method until the compressed neural network model achieves a better compression effect. The topological structure of the neural network model can be kept unchanged, so that the topological structure of the neural network model is prevented from being irregular, and the calculation amount of the neural network is reduced.
Allowable Subject Matter
Claims 6-11, 13, and 18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to Applicant's disclosure. Applicant is required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Casey R. Garner whose telephone number is 571-272-2467. The examiner can normally be reached Monday to Friday, 8am to 5pm, Eastern Time.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached on 571-270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center and Private PAIR to authorized users only. Should you have questions about access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/Casey R. Garner/Primary Examiner, Art Unit 2123