DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 11/10/2021 has been entered.
Claims 1, 3-6, 8-11, 13-16 and 18-20 are pending in the case. Claims 1, 6, 11 and 16 are independent claims. Claims 2, 7, 12 and 17 are cancelled and claims 1, 6, 11 and 16 are amended.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1, 3-6, 8-11, 13-16 and 18-20 rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Claims 1, 6, 11 and 16 have amended language that cites the means for solving comprising means for performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute eigenvalue decompositions, however examiner could not find, within the specification, support for the amended language. While Applicant’s remarks state the support can be found in [0067], the closest support examiner found was paragraph [0066] that states that a low-rank matrix is a matric that is rank-1 and gives the example of a rank 1 matrix being a moment matrix of a Dirac delta. However, neither paragraph [0066] or [0067] explicitly use or define what a moment matrix is. Further, support for using a moment matrix with a fast matrix-vector or randomized linear algebra to compute eigenvalue decompositions was not found in either paragraph [0066] or [0067].
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 3-6, 8-11, 13-16 and 18-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term fast in claims 1, 6 , 11 and 16 is a relative term which renders the claim indefinite. The term fast is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The relative terminology renders the matrix-vector products indefinite as the term fast is not defined.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 3-6, 8-11, 13-16 and 18-20 are rejected under 35 U.S.C. 101 as they are directed to an abstract idea and do not integrate into a practical application.
Regarding claim 1:
The claim recites modeling a neural network as a set of polynomials which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))) as the polynomials are optimization problems. Alternatively, under the broadest reasonable interpretation, the limitation covers performance of the limitation in the mind. The limitations encompass a user creating/choosing polynomials that represent a neural network. See 2106.04.(a)(2).III.C.
The claim recites relaxing the global polynomial optimization problem including polynomial constraints with a plurality of semi-definite programs which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))).
The claim recites solving the plurality of semi-definite programs based on a pre-defined structure comprising a block Hankel-plus-Toeplitz structure, the solving comprising performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute eigenvalue decompositions which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))).
Subject Matter Eligibility Analysis Step 2A Prong 2:
processor-implemented(merely recites a generic computer on which to perform the abstract idea, e.g. "apply it on a computer" (see MPEP 2106.05(f)))
receiving as input, a global polynomial optimization problem (which amount to mere extra solution activity of obtaining and/or gathering data over a network, see MPEP §2106.05(g))
that approximates a training problem of the neural network based on the set of polynomials(merely specifies a particular technological environment in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h)))
the semi-definite programs having dimensions corresponding to a total number of weights in the neural network, with weights deeper in the neural network having a higher associated polynomial degree, a last layer in the neural network being a deepest layer and weights in the last layer having a highest associated polynomial degree(merely specifies a particular technological environment in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h)))
outputting a solution indicating a location of a global optimum of the optimization problem(which amount to mere extra solution activity of obtaining and/or gathering data over a network, see MPEP §2106.05(g))
with the neural network(merely recites a generic computer on which to perform the abstract idea, e.g. "apply it on a computer" (see MPEP 2106.05(f)))
training the neural network with weights deeper in the neural network having a higher associated polynomial degree based on the solution(merely recites a generic computer on which to perform the abstract idea, e.g. "apply it on a computer" (see MPEP 2106.05(f)))
Subject Matter Eligibility Analysis Step 2B:
Additional elements (a) (f) and (g) do not integrate the abstract idea into a practical application nor do the additional limitation provide significantly more than the abstract idea because the limitation amount to no more than mere instructions to apply the exception using a generic computer component. Please see MPEP §2106.05(f).
Additional elements (c) and (d) do not integrate the abstract idea into a practical application nor do the additional limitation provide significantly more than the abstract idea because the limitation merely specifies a field of use in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h)).
Additional element (b) and (e) recites receiving and sending inputs/outputs which is a well-understood, routine, and conventional activity of “transmitting or receiving data over a network" (see MPEP 2106.05(d)(II)(i) using the Internet to gather data, buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network))
The additional element(s) (a) (b) (c) (d) (e) (f) and (g) in the claim do/does not include any additional elements, when considered separately and in combination, that amount to an integration of the judicial exception into a practical application, nor significantly more than the judicial exception for the reasons set forth in step 2A prong 2 analysis above. The claim is not patent eligible.
Regarding claim 3:
Subject Matter Eligibility Analysis Step 2A Prong 1:
The claim recites solving further includes dimensionality reduction which is an abstract idea(Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))).
The claim recites and/or using a warm start which is a mathematical concept ( as disclosed in para. [0067] of the specification).
Subject Matter Eligibility Analysis Step 2A Prong 2:
The claim does not contain elements that would warrant a Step 2A Prong 2 analysis.
Subject Matter Eligibility Analysis Step 2B:
The claim does not include any additional element, when considered separately and in combination, that amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception. The claim is not patent eligible.
Regarding claim 4:
Subject Matter Eligibility Analysis Step 2A Prong 1:
The claim recites global polynomial optimization problem is non-convex which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))).
Subject Matter Eligibility Analysis Step 2A Prong 2:
The claim does not contain elements that would warrant a Step 2A Prong 2 analysis.
Subject Matter Eligibility Analysis Step 2B:
The claim does not include any additional element, when considered separately and in combination, that amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception. The claim is not patent eligible.
Regarding claim 5:
Subject Matter Eligibility Analysis Step 2A Prong 1:
The claim recites representing the global polynomial optimization problem with a Chebyshev basis which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))).
Subject Matter Eligibility Analysis Step 2A Prong 2:
The claim does not contain elements that would warrant a Step 2A Prong 2 analysis.
Subject Matter Eligibility Analysis Step 2B:
The claim does not include any additional element, when considered separately and in combination, that amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception. The claim is not patent eligible.
Regarding claim 6:
The claim recites to model a neural network as a set of polynomials which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))) as the polynomials are optimization problems. Alternatively, under the broadest reasonable interpretation, covers performance of the limitation in the mind. The limitations encompass a user creating/choosing polynomials that represent a neural network. See 2106.04.(a)(2).III.C.
The claim recites to relax the global polynomial optimization problem including polynomial constraints with a plurality of semi-definite programs which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))).
The claim recites solving the plurality of semi-definite programs based on a pre-defined structure comprising a block Hankel-plus-Toeplitz structure, the solving comprising performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute eigenvalue decompositions which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))).
Subject Matter Eligibility Analysis Step 2A Prong 2:
a memory; and at least one processor coupled to the memory, the at least one processor(merely recites a generic computer on which to perform the abstract idea, e.g. "apply it on a computer" (see MPEP 2106.05(f)))
to receive as input, a global polynomial optimization problem (which amount to mere extra solution activity of obtaining and/or gathering data over a network, see MPEP §2106.05(g))
that approximates a training problem of the neural network based on the set of polynomials (merely specifies a particular technological environment in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h)))
the semi-definite programs having dimensions corresponding to a total number of weights in the neural network, with weights deeper in the neural network having a higher associated polynomial degree, a last layer in the neural network being a deepest layer and weights in the last layer having a highest associated polynomial degree(merely specifies a particular technological environment in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h)))
and output a solution indicating a location of a global optimum of the optimization problem (which amount to mere extra solution activity of obtaining and/or gathering data over a network, see MPEP §2106.05(g))
with the neural network(merely recites a generic computer on which to perform the abstract idea, e.g. "apply it on a computer" (see MPEP 2106.05(f)))
training of the neural network with weights deeper in the neural network having a higher associated polynomial degree based on the solution (merely recites a generic computer on which to perform the abstract idea, e.g. "apply it on a computer" (see MPEP 2106.05(f)))
Subject Matter Eligibility Analysis Step 2B:
Additional elements (a) (f) and (g) do not integrate the abstract idea into a practical application nor do the additional limitation provide significantly more than the abstract idea because the limitation amount to no more than mere instructions to apply the exception using a generic computer component. Please see MPEP §2106.05(f).
Additional elements (c) and (d) do not integrate the abstract idea into a practical application nor do the additional limitation provide significantly more than the abstract idea because the limitation merely specifies a field of use in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h)).
Additional element (b) and (e) recites receiving and sending inputs/outputs which is a well-understood, routine, and conventional activity of “transmitting or receiving data over a network" (see MPEP 2106.05(d)(II)(i) using the Internet to gather data, buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network))
The additional element(s) (a) (b) (c) (d) (e) (f) and (g) in the claim do/does not include any additional elements, when considered separately and in combination, that amount to an integration of the judicial exception into a practical application, nor significantly more than the judicial exception for the reasons set forth in step 2A prong 2 analysis above. The claim is not patent eligible.
Claims 8-10 are rejected under that same 101 claim analysis due to the substantially similarity of the limitations and additional elements of claims 8-5 found in claims 8-10 respectively.
Regarding claim 11:
The claim recites means for modeling a neural network as a set of polynomials; which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))) as the polynomials are optimization problems. Alternatively, under the broadest reasonable interpretation, covers performance of the limitation in the mind. The limitations encompass a user creating/choosing polynomials that represent a neural network. See 2106.04.(a)(2).III.C.
The claim recites means for relaxing the global polynomial optimization problem including polynomial constraints with a plurality of semi-definite programs which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))).
The claim recites solving the plurality of semi-definite programs based on a pre-defined structure comprising a block Hankel-plus-Toeplitz structure, the solving comprising performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute eigenvalue decompositions which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))).
Subject Matter Eligibility Analysis Step 2A Prong 2:
with the neural network(merely recites a generic computer on which to perform the abstract idea, e.g. "apply it on a computer" (see MPEP 2106.05(f)))
means for receiving as input, a global polynomial optimization problem (which amount to mere extra solution activity of obtaining and/or gathering data over a network, see MPEP §2106.05(g))
that approximates a training problem of the neural network based on the set of polynomials (merely specifies a particular technological environment in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h)))
the semi-definite programs having dimensions corresponding to a total number of weights in the neural network, with weights deeper in the neural network having a higher associated polynomial degree, a last layer in the neural network being a deepest layer and weights in the last layer having a highest associated polynomial degree(merely specifies a particular technological environment in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h)))
means for outputting a solution indicating a location of a global optimum of the optimization problem (which amount to mere extra solution activity of obtaining and/or gathering data over a network, see MPEP §2106.05(g))
training the neural network with weights deeper in the neural network having a higher associated polynomial degree based on the solution (merely recites a generic computer on which to perform the abstract idea, e.g. "apply it on a computer" (see MPEP 2106.05(f)))
Subject Matter Eligibility Analysis Step 2B:
Additional elements (a) and (f) do not integrate the abstract idea into a practical application nor do the additional limitation provide significantly more than the abstract idea because the limitation amount to no more than mere instructions to apply the exception using a generic computer component. Please see MPEP §2106.05(f).
Additional elements (c) and (d) do not integrate the abstract idea into a practical application nor do the additional limitation provide significantly more than the abstract idea because the limitation merely specifies a field of use in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h)).
Additional element (b) and (e) recites receiving and sending inputs/outputs which is a well-understood, routine, and conventional activity of “transmitting or receiving data over a network" (see MPEP 2106.05(d)(II)(i) using the Internet to gather data, buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network))
The additional element(s) (a) (b) (c) (d) (e) and (f) in the claim do/does not include any additional elements, when considered separately and in combination, that amount to an integration of the judicial exception into a practical application, nor significantly more than the judicial exception for the reasons set forth in step 2A prong 2 analysis above. The claim is not patent eligible.
Claims 13-15 are rejected under that same 101 claim analysis due to the substantially similarity of the limitations and additional elements of claims 3-5 found in claims 13-15 respectively.
Regarding claim 16:
The claim recites program code to model a neural network as a set of polynomials which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))) as the polynomials are optimization problems. Alternatively, under the broadest reasonable interpretation, covers performance of the limitation in the mind. The limitations encompass a user creating/choosing polynomials that represent a neural network. See 2106.04.(a)(2).III.C.
The claim recites program code to relax the global polynomial optimization problem including polynomial constraints with a plurality of semi-definite programs which is an abstract idea (Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A)))).
The claim recites solving the plurality of semi-definite programs based on a pre-defined structure comprising a block Hankel-plus-Toeplitz structure, the solving comprising performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute eigenvalue decompositions which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))).
The claim recites program code to train … based on the solution which is an abstract idea (Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C))).
Subject Matter Eligibility Analysis Step 2A Prong 2:
non-transitory computer-readable medium having program code recorded thereon, the program code executed by a processor (merely recites a generic computer on which to perform the abstract idea, e.g. "apply it on a computer" (see MPEP 2106.05(f)))
program code to receive as input, a global polynomial optimization problem(which amount to mere extra solution activity of obtaining and/or gathering data over a network, see MPEP §2106.05(g))
that approximates a training problem of the neural network based on the set of polynomials (merely specifies a particular technological environment in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h)))
the semi-definite programs having dimensions corresponding to a total number of weights in the neural network, with weights deeper in the neural network having a higher associated polynomial degree, a last layer in the neural network being a deepest layer and weights in the last layer having a highest associated polynomial degree(merely specifies a particular technological environment in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h)))
and output a solution indicating a location of a global optimum of the optimization problem (which amount to mere extra solution activity of obtaining and/or gathering data over a network, see MPEP §2106.05(g))
with the neural network(merely recites a generic computer on which to perform the abstract idea, e.g. "apply it on a computer" (see MPEP 2106.05(f)))
train the neural network with weights deeper in the neural network having a higher associated polynomial degree based on the solution (merely recites a generic computer on which to perform the abstract idea, e.g. "apply it on a computer" (see MPEP 2106.05(f)))
Subject Matter Eligibility Analysis Step 2B:
Additional elements (a) (f) and (g) do not integrate the abstract idea into a practical application nor do the additional limitation provide significantly more than the abstract idea because the limitation amount to no more than mere instructions to apply the exception using a generic computer component. Please see MPEP §2106.05(f).
Additional elements (c) and (d) do not integrate the abstract idea into a practical application nor do the additional limitation provide significantly more than the abstract idea because the limitation merely specifies a field of use in which the abstract idea is to take place, i.e. a field of use (see MPEP 2106.05(h)).
Additional element (b) and (e) recites receiving and sending inputs/outputs which is a well-understood, routine, and conventional activity of “transmitting or receiving data over a network" (see MPEP 2106.05(d)(II)(i) using the Internet to gather data, buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network))
The additional element(s) (a) (b) (c) (d) (e) (f) and (g) in the claim do/does not include any additional elements, when considered separately and in combination, that amount to an integration of the judicial exception into a practical application, nor significantly more than the judicial exception for the reasons set forth in step 2A prong 2 analysis above. The claim is not patent eligible.
Claims 18-20 are rejected under that same 101 claim analysis due to the substantially similarity of the limitations and additional elements of claims 3-5 found in claims 18-20 respectively.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Regarding claims 1, 3-4, 6, 8-9, 11, 13-14, 16 and 18-19:
Claim(s) 1, 3-4, 6, 8-9, 11, 13-14, 16 and 18-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bartan et al.(Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Time) in view of Genin et al. (“Optimization Problems Over Positive Pseudopolynomial Matrices” henceforth known as Genin) and further in view of Mai et al. (“A hierarchy of spectral relaxations for polynomial optimization” henceforth known as Mai)
Regarding claim 1:
Bartan discloses processor-implemented(Bartan, ABSTRACT, “Remarkably, we show that semidefinite lifting is always exact and therefore computational complexity for global optimization is polynomial in the input dimension and sample size for all input data” where using computation complexity as a measurement implies the method is completed on a computer/processor)
Bartan discloses modeling a neural network as a set of polynomials(Bartan, Page 7, Paragraph 5 and Equation 10, “Consider the network f(x) =
∑
j
=
1
m
σ
(
x
T
u
j
)
α
j
where the activation function σ is the degree two polynomial σ(u) = au2 + bu + c. First, we note that the neural network output can be written as [Equation 10]” where the conversion of the neural network with degree-2 polynomial activation into polynomial form is considered modeling a neural network as a set of polynomials.
Bartan discloses receiving as input, a global polynomial optimization problem that approximates a training problem of the neural network based on the set of polynomials(Bartan, Page 8, Definition 1, “we will show that the original non-convex neural network problem is solved exactly to global optimality when the optimization is performed over a convex set which we define as the Neural Spectrahedron” where the original non-convex neural network problem to be solved via global optimality using optimization over a convex set is considered receiving a global polynomial optimization problem that approximates a training problem of the neural network based on the set of polynomials.)
Bartan discloses relaxing the global polynomial optimization problem including polynomial constraints with a plurality of semi-definite programs(Bartan, Page 8, Definition 1, “ we will show that the original non-convex neural network problem is solved exactly to global optimality when the optimization is performed over a convex set which we define as the Neural Spectrahedron… Moreover, a Neural Spectrahedron can be described by a simple linear matrix inequality” where the original polynomial NN training problem is reformulated as an semi-definite program by representing the feasible parameter set (Neural Spectrahedron) via LMIs is considered relaxing a polynomial optimization problem including polynomial constraints with a plurality of semi-definite programs(See also: “It is equally important that our results characterize neural networks as constrained linear learning methods <φ(x), ψ> in the lifted feature space φ(x), where the constraints on the lifted parameters ψ are precisely described by a Neural Spectrahedron via linear matrix inequalities. These constraints can be easily tackled with convex semidefinite programming or closed-form projections onto these sets in iterative first-order algorithms”), the semi-definite programs having dimensions corresponding to a total number of weights in the neural network(Bartan, Page 7, Equation 10 and Bartan, Page 14, Theorem 3.1, where the SDP having dimensions corresponding to the total number of weights in the neural network as shown in Equation 10 and Theorem 3.1. In the lifted formulation, the optimization variables of PSD matrices Z, Z` of size (d+1) * (d+1) are used in the SDP that have the weights uj and αj encoded from the neural network parameters which is considered having semi-definite programs having dimensions corresponding to a total number of weights in the neural network(See also: Eq. 14 and 18 for PSD matrix construction)), with weights deeper in the neural network having a higher associated polynomial degree, a last layer in the neural network being a deepest layer and weights in the last layer having a highest associated polynomial degree(Barton, Page 3, Paragraph 1, where the training uses output of hidden layer σ(xTuj) after applying its hidden-layer weights uj and corresponds to one degree less than the weighted output contribution of the output layer σ(xTuj)αj and both the outputs use the weights of the hidden and output layer they both correspond to the having weights deeper in the neural network having a higher associated polynomial degreeI
Bartan discloses solving the plurality of semi-definite based on a pre-defined structure(Bartan, Page 9, Paragraph 2, “It is equally important that our results characterize neural networks as constrained linear learning methods <φ(x), ψ> in the lifted feature space φ(x), where the constraints on the lifted parameters ψ are precisely described by a Neural Spectrahedron via linear matrix inequalities. These constraints can be easily tackled with convex semidefinite programming or closed-form projections onto these sets in iterative first-order algorithms” where solving constraints described by the LMI of the Neural Spectrahedron using semidefinite programs is considered solving semi-definite programs based on a pre-defined structure)…to compute eigenvalue decomposition(Bartan, Page 19, Paragraph 5, “Let us decompose Z* via eigenvalue decomposition as Z*”) and outputting a solution indicating a location of a global optimum of the optimization problem(Bartan, Page 14, Theorem 3.1, Equations 21 and 22, Paragraph 1, “In this section we show that the solution of the convex program (21) provides a lower bound for the solution of the non convex problem (22).” where the SDP is convex and exactly reformulates the original non-convex neural network training problem and the SDP yields Z*, Z`* which represents a globally optimal solution for the original non-convex polynomial optimization) and training the neural network with weights deeper in the neural network having the higher associated polynomial degree(Bartan, Page 11, equation 20, where equation 20 shows the outputs of the weights are used in training as shown in the minimizing the loss based on the prediction) based on the solution(Bartan, Page 18, Paragraph 1, " Decomposing the solution of the convex problem Z* and Z`* onto these cones, i.e., neural decomposition, enables the construction of neural network weights from Z* and Z`*" where after neural network decomposition and solving Z, Z`* the neural network and weights are reconstructed which is considered performing using the solution with the neural network to perform training)
Bartan does not disclose a pre-defined structure comprising a block Hankel- plus-Toeplitz structure or the solving comprising performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute computing eigenvalue decomposition
Genin discloses a pre-defined structure comprising a block Hankel- plus-Toeplitz structure(Genin, Page 59, Paragraph 2, “In this light, it is proposed to substitute a basis of Chebyshev polynomials for the natural basis. Such a Chebyshev basis induces a Toeplitz-plus-Hankel structure to the problem with, in principle, a much better numerical conditioning. It is finally recalled how one can take advantage of the Toeplitz-plus-Hankel structure in fast algorithms based on appropriate displacement rank techniques” and Genin, Page 77, Paragraph 6, “For any ˜Q, let us set the block Toeplitz-plus-Hankel matrix”)
References Bartan and Genin are analogous art because they are from same problem-solving area of semidefinite/linear matrix inequality representation and convex optimization of polynomial structures.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Bartan and Genin before him or her, to modify the predefined structure of Bartan to include the Hankel-plus-Toeplitz structure of Genin because to help with ill-conditioned numerical systems. The suggestion/motivation for doing so would have been Genin, Page 22, Paragraph 4, “From a numerical viewpoint, this reformulation of the optimization problem on the real line exhibits a considerable advantage over its initial formulation in the sense that is not intrinsically ill-conditioned…As a result, the numerical behavior of the computational optimization scheme is expected to be substantially improved” and Genin, Page 22, Paragraph 6, “In particular the dual optimization problems can be solved very efficiently using displacement-based factorizations as well as an appropriate divide-and-conquer strategy. These results are direct consequences of the Hankel or Toeplitz structure in the dual constraints.
Bartan-Genin does not explicitly disclose the solving comprising performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute computing eigenvalue decomposition
Mai discloses the solving comprising performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute eigenvalue decomposition(Mai, Page 25, Paragraph 2, “SketchyCGAL provides approximations of the smallest eigenvalue and eigenvector of each matrix C+AT(y+β(z−b))vi by using the randomized Lanczos method” where the randomized Lanczos method corresponds to randomized linear algebra using matrix-vector products with a symmetric matrix C to approximate eigenvalues corresponding to performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute eigenvalue decomposition)
References Bartan-Genin and Mai are analogous art because they are from same problem-solving area of global optimization of non-convex problems using semidefinite/convex reformulations.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Bartan-Genin and Mai before him or her, to modify the eigenvalue decomposition computation of Bartan-Genin to include the eigenvalue decomposition computation of Mai to avoid costly computational problems . The suggestion/motivation for doing so would have been Mai, Page 25, Paragraph 1, “In SDP relaxations of POPs is more expensive than the simple one related to the first order SDP relaxation of MAXCUT, which is solved very efficiently by SketchyCGAL” and Mai, Page 3, Paragraph 8, “Therefore to solve each semidefinite relaxation of a CTP-POP one may avoid the computationally costly interior-point methods in some cases.”
Regarding claim 3:
The rejection of Bartan-Genin-Mai discloses the processor-implemented method of claim 1 is incorporated and further:
Bartan further discloses in which solving further includes dimensionality reduction, and/or using a warm start(Equation 10, Theorem 3.1, and , “The above identity shows that the nonlinear neural network output is linear over the lifted features
PNG
media_image1.png
35
379
media_image1.png
Greyscale
where the transformation of the original space into a lower-dimensional convex space with fixed size make to make solving in polynomial time feasible is considered dimensionality reduction(i.e. going from original space ℝm(d+1) to two PSD matrixes with fixed dimension (d+1)2). As the maximum neuron count (and thus maximum number of weight vectors) is directly bounded by the SDP matrix dimension from theorem 3.1(“It follows that the optimal number of neurons is upper bounded by m∗ ≤ 2(d + 1)”) the large space of m(d+1) parameters collapses to two PSD matrixes of size (d+1)2 independent of m)
Regarding claim 4:
The rejection of Bartan-Genin-Mai discloses the processor-implemented method of claim 1 is incorporated and further:
Bartan further discloses in which the global polynomial optimization problem is non-convex(“Remarkably, it can be shown that the original non-convex problem in (5) can be solved exactly by solving the convex SDP” where the non-convex problem is considered a non-convex global polynomial optimization)
Regarding claim 6:
Bartan discloses a memory; and at least one processor coupled to the memory, the at least one processor(Bartan, ABSTRACT, “Remarkably, we show that semidefinite lifting is always exact and therefore computational complexity for global optimization is polynomial in the input dimension and sample size for all input data” where using computation complexity as a measurement implies the method is completed on a computer/processor)
Bartan discloses modeling a neural network as a set of polynomials(Bartan, Page 7, Paragraph 5 and Equation 10, “Consider the network f(x) =
∑
j
=
1
m
σ
(
x
T
u
j
)
α
j
where the activation function σ is the degree two polynomial σ(u) = au2 + bu + c. First, we note that the neural network output can be written as [Equation 10]” where the conversion of the neural network with degree-2 polynomial activation into polynomial form is considered modeling a neural network as a set of polynomials.
Bartan discloses receiving as input, a global polynomial optimization problem that approximates a training problem of the neural network based on the set of polynomials(Bartan, Page 8, Definition 1, “we will show that the original non-convex neural network problem is solved exactly to global optimality when the optimization is performed over a convex set which we define as the Neural Spectrahedron” where the original non-convex neural network problem to be solved via global optimality using optimization over a convex set is considered receiving a global polynomial optimization problem that approximates a training problem of the neural network based on the set of polynomials.)
Bartan discloses to relax the global polynomial optimization problem including polynomial constraints with a plurality of semi-definite programs(Bartan, Page 8, Definition 1, “ we will show that the original non-convex neural network problem is solved exactly to global optimality when the optimization is performed over a convex set which we define as the Neural Spectrahedron… Moreover, a Neural Spectrahedron can be described by a simple linear matrix inequality” where the original polynomial NN training problem is reformulated as an semi-definite program by representing the feasible parameter set (Neural Spectrahedron) via LMIs is considered relaxing a polynomial optimization problem including polynomial constraints with a plurality of semi-definite programs(See also: “It is equally important that our results characterize neural networks as constrained linear learning methods <φ(x), ψ> in the lifted feature space φ(x), where the constraints on the lifted parameters ψ are precisely described by a Neural Spectrahedron via linear matrix inequalities. These constraints can be easily tackled with convex semidefinite programming or closed-form projections onto these sets in iterative first-order algorithms”), the semi-definite programs having dimensions corresponding to a total number of weights in the neural network(Bartan, Page 7, Equation 10 and Bartan, Page 14, Theorem 3.1, where the SDP having dimensions corresponding to the total number of weights in the neural network as shown in Equation 10 and Theorem 3.1. In the lifted formulation, the optimization variables of PSD matrices Z, Z` of size (d+1) * (d+1) are used in the SDP that have the weights uj and αj encoded from the neural network parameters which is considered having semi-definite programs having dimensions corresponding to a total number of weights in the neural network(See also: Eq. 14 and 18 for PSD matrix construction)), with weights deeper in the neural network having a higher associated polynomial degree, a last layer in the neural network being a deepest layer and weights in the last layer having a highest associated polynomial degree(Barton, Page 3, Paragraph 1, where the training uses output of hidden layer σ(xTuj) after applying its hidden-layer weights uj and corresponds to one degree less than the weighted output contribution of the output layer σ(xTuj)αj and both the outputs use the weights of the hidden and output layer they both correspond to the having weights deeper in the neural network having a higher associated polynomial degree.)
Bartan discloses solving the plurality of semi-definite based on a pre-defined structure(Bartan, Page 9, Paragraph 2, “It is equally important that our results characterize neural networks as constrained linear learning methods <φ(x), ψ> in the lifted feature space φ(x), where the constraints on the lifted parameters ψ are precisely described by a Neural Spectrahedron via linear matrix inequalities. These constraints can be easily tackled with convex semidefinite programming or closed-form projections onto these sets in iterative first-order algorithms” where solving constraints described by the LMI of the Neural Spectrahedron using semidefinite programs is considered solving semi-definite programs based on a pre-defined structure)…to compute eigenvalue decomposition(Bartan, Page 19, Paragraph 5, “Let us decompose Z* via eigenvalue decomposition as Z*”) and outputting a solution indicating a location of a global optimum of the optimization problem(Bartan, Page 14, Theorem 3.1, Equations 21 and 22, Paragraph 1, “In this section we show that the solution of the convex program (21) provides a lower bound for the solution of the non convex problem (22).” where the SDP is convex and exactly reformulates the original non-convex neural network training problem and the SDP yields Z*, Z`* which represents a globally optimal solution for the original non-convex polynomial optimization) and training the neural network with weights deeper in the neural network having the higher associated polynomial degree(Bartan, Page 11, equation 20, where equation 20 shows the outputs of the weights are used in training as shown in the minimizing the loss based on the prediction) based on the solution(Bartan, Page 18, Paragraph 1, " Decomposing the solution of the convex problem Z* and Z`* onto these cones, i.e., neural decomposition, enables the construction of neural network weights from Z* and Z`*" where after neural network decomposition and solving Z, Z`* the neural network and weights are reconstructed which is considered performing using the solution with the neural network to perform training)
Bartan does not disclose a pre-defined structure comprising a block Hankel- plus-Toeplitz structure or the solving comprising performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute computing eigenvalue decomposition
Genin discloses a pre-defined structure comprising a block Hankel- plus-Toeplitz structure(Genin, Page 59, Paragraph 2, “In this light, it is proposed to substitute a basis of Chebyshev polynomials for the natural basis. Such a Chebyshev basis induces a Toeplitz-plus-Hankel structure to the problem with, in principle, a much better numerical conditioning. It is finally recalled how one can take advantage of the Toeplitz-plus-Hankel structure in fast algorithms based on appropriate displacement rank techniques” and Genin, Page 77, Paragraph 6, “For any ˜Q, let us set the block Toeplitz-plus-Hankel matrix”)
References Bartan and Genin are analogous art because they are from same problem-solving area of semidefinite/linear matrix inequality representation and convex optimization of polynomial structures.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Bartan and Genin before him or her, to modify the predefined structure of Bartan to include the Hankel-plus-Toeplitz structure of Genin because to help with ill-conditioned numerical systems. The suggestion/motivation for doing so would have been Genin, Page 22, Paragraph 4, “From a numerical viewpoint, this reformulation of the optimization problem on the real line exhibits a considerable advantage over its initial formulation in the sense that is not intrinsically ill-conditioned…As a result, the numerical behavior of the computational optimization scheme is expected to be substantially improved” and Genin, Page 22, Paragraph 6, “In particular the dual optimization problems can be solved very efficiently using displacement-based factorizations as well as an appropriate divide-and-conquer strategy. These results are direct consequences of the Hankel or Toeplitz structure in the dual constraints.
Bartan-Genin does not explicitly disclose the solving comprising performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute computing eigenvalue decomposition
Mai discloses the solving comprising performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute eigenvalue decomposition(Mai, Page 25, Paragraph 2, “SketchyCGAL provides approximations of the smallest eigenvalue and eigenvector of each matrix C+AT(y+β(z−b))vi by using the randomized Lanczos method” where the randomized Lanczos method corresponds to randomized linear algebra using matrix-vector products with a symmetric matrix C to approximate eigenvalues corresponding to performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute eigenvalue decomposition)
References Bartan-Genin and Mai are analogous art because they are from same problem-solving area of global optimization of non-convex problems using semidefinite/convex reformulations.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Bartan-Genin and Mai before him or her, to modify the eigenvalue decomposition computation of Bartan-Genin to include the eigenvalue decomposition computation of Mai to avoid costly computational problems . The suggestion/motivation for doing so would have been Mai, Page 25, Paragraph 1, “In SDP relaxations of POPs is more expensive than the simple one related to the first order SDP relaxation of MAXCUT, which is solved very efficiently by SketchyCGAL” and Mai, Page 3, Paragraph 8, “Therefore to solve each semidefinite relaxation of a CTP-POP one may avoid the computationally costly interior-point methods in some cases.”
Regarding claims 8-9:
The rejection of claim 6 is incorporated in claims 8-9. Claims 8-9 are rejected under the same rationale as set forth in the rejection of claim 3-4.
Regarding claim 11:
Bartan discloses means for a neural network as a set of polynomials(Bartan, Page 7, Paragraph 5 and Equation 10, “Consider the network f(x) =
∑
j
=
1
m
σ
(
x
T
u
j
)
α
j
where the activation function σ is the degree two polynomial σ(u) = au2 + bu + c. First, we note that the neural network output can be written as [Equation 10]” where the conversion of the neural network with degree-2 polynomial activation into polynomial form is considered modeling a neural network as a set of polynomials.
Bartan discloses means for receiving as input, a global polynomial optimization problem that approximates a training problem of the neural network based on the set of polynomials(Bartan, Page 8, Definition 1, “we will show that the original non-convex neural network problem is solved exactly to global optimality when the optimization is performed over a convex set which we define as the Neural Spectrahedron” where the original non-convex neural network problem to be solved via global optimality using optimization over a convex set is considered receiving a global polynomial optimization problem that approximates a training problem of the neural network based on the set of polynomials.)
Bartan discloses means for relaxing the global polynomial optimization problem including polynomial constraints with a plurality of semi-definite programs(Bartan, Page 8, Definition 1, “ we will show that the original non-convex neural network problem is solved exactly to global optimality when the optimization is performed over a convex set which we define as the Neural Spectrahedron… Moreover, a Neural Spectrahedron can be described by a simple linear matrix inequality” where the original polynomial NN training problem is reformulated as an semi-definite program by representing the feasible parameter set (Neural Spectrahedron) via LMIs is considered relaxing a polynomial optimization problem including polynomial constraints with a plurality of semi-definite programs(See also: “It is equally important that our results characterize neural networks as constrained linear learning methods <φ(x), ψ> in the lifted feature space φ(x), where the constraints on the lifted parameters ψ are precisely described by a Neural Spectrahedron via linear matrix inequalities. These constraints can be easily tackled with convex semidefinite programming or closed-form projections onto these sets in iterative first-order algorithms”), the semi-definite programs having dimensions corresponding to a total number of weights in the neural network(Bartan, Page 7, Equation 10 and Bartan, Page 14, Theorem 3.1, where the SDP having dimensions corresponding to the total number of weights in the neural network as shown in Equation 10 and Theorem 3.1. In the lifted formulation, the optimization variables of PSD matrices Z, Z` of size (d+1) * (d+1) are used in the SDP that have the weights uj and αj encoded from the neural network parameters which is considered having semi-definite programs having dimensions corresponding to a total number of weights in the neural network(See also: Eq. 14 and 18 for PSD matrix construction)), with weights deeper in the neural network having a higher associated polynomial degree, a last layer in the neural network being a deepest layer and weights in the last layer having a highest associated polynomial degree(Barton, Page 3, Paragraph 1, where the training uses output of hidden layer σ(xTuj) after applying its hidden-layer weights uj and corresponds to one degree less than the weighted output contribution of the output layer σ(xTuj)αj and both the outputs use the weights of the hidden and output layer they both correspond to the having weights deeper in the neural network having a higher associated polynomial degree)
Bartan discloses means for solving the plurality of semi-definite programs based on a pre-defined structure(Bartan, Page 9, Paragraph 2, “It is equally important that our results characterize neural networks as constrained linear learning methods <φ(x), ψ> in the lifted feature space φ(x), where the constraints on the lifted parameters ψ are precisely described by a Neural Spectrahedron via linear matrix inequalities. These constraints can be easily tackled with convex semidefinite programming or closed-form projections onto these sets in iterative first-order algorithms” where solving constraints described by the LMI of the Neural Spectrahedron using semidefinite programs is considered solving semi-definite programs based on a pre-defined structure)…to compute eigenvalue decomposition(Bartan, Page 19, Paragraph 5, “Let us decompose Z* via eigenvalue decomposition as Z*”) and means for outputting a solution indicating a location of a global optimum of the optimization problem(Bartan, Page 14, Theorem 3.1, Equations 21 and 22, Paragraph 1, “In this section we show that the solution of the convex program (21) provides a lower bound for the solution of the non convex problem (22).” where the SDP is convex and exactly reformulates the original non-convex neural network training problem and the SDP yields Z*, Z`* which represents a globally optimal solution for the original non-convex polynomial optimization) and means for training the neural network with weights deeper in the neural network having the higher associated polynomial degree(Bartan, Page 11, equation 20, where equation 20 shows the outputs of the weights are used in training as shown in the minimizing the loss based on the prediction) based on the solution(Bartan, Page 18, Paragraph 1, " Decomposing the solution of the convex problem Z* and Z`* onto these cones, i.e., neural decomposition, enables the construction of neural network weights from Z* and Z`*" where after neural network decomposition and solving Z, Z`* the neural network and weights are reconstructed which is considered performing using the solution with the neural network to perform training)
Bartan does not disclose a pre-defined structure comprising a block Hankel- plus-Toeplitz structure or the solving comprising performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute computing eigenvalue decomposition
Genin discloses a pre-defined structure comprising a block Hankel- plus-Toeplitz structure(Genin, Page 59, Paragraph 2, “In this light, it is proposed to substitute a basis of Chebyshev polynomials for the natural basis. Such a Chebyshev basis induces a Toeplitz-plus-Hankel structure to the problem with, in principle, a much better numerical conditioning. It is finally recalled how one can take advantage of the Toeplitz-plus-Hankel structure in fast algorithms based on appropriate displacement rank techniques” and Genin, Page 77, Paragraph 6, “For any ˜Q, let us set the block Toeplitz-plus-Hankel matrix”)
References Bartan and Genin are analogous art because they are from same problem-solving area of semidefinite/linear matrix inequality representation and convex optimization of polynomial structures.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Bartan and Genin before him or her, to modify the predefined structure of Bartan to include the Hankel-plus-Toeplitz structure of Genin because to help with ill-conditioned numerical systems. The suggestion/motivation for doing so would have been Genin, Page 22, Paragraph 4, “From a numerical viewpoint, this reformulation of the optimization problem on the real line exhibits a considerable advantage over its initial formulation in the sense that is not intrinsically ill-conditioned…As a result, the numerical behavior of the computational optimization scheme is expected to be substantially improved” and Genin, Page 22, Paragraph 6, “In particular the dual optimization problems can be solved very efficiently using displacement-based factorizations as well as an appropriate divide-and-conquer strategy. These results are direct consequences of the Hankel or Toeplitz structure in the dual constraints.
Bartan-Genin does not explicitly disclose the solving comprising performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute computing eigenvalue decomposition
Mai discloses the solving comprising performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute eigenvalue decomposition(Mai, Page 25, Paragraph 2, “SketchyCGAL provides approximations of the smallest eigenvalue and eigenvector of each matrix C+AT(y+β(z−b))vi by using the randomized Lanczos method” where the randomized Lanczos method corresponds to randomized linear algebra using matrix-vector products with a symmetric matrix C to approximate eigenvalues corresponding to performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute eigenvalue decomposition)
References Bartan-Genin and Mai are analogous art because they are from same problem-solving area of global optimization of non-convex problems using semidefinite/convex reformulations.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Bartan-Genin and Mai before him or her, to modify the eigenvalue decomposition computation of Bartan-Genin to include the eigenvalue decomposition computation of Mai to avoid costly computational problems . The suggestion/motivation for doing so would have been Mai, Page 25, Paragraph 1, “In SDP relaxations of POPs is more expensive than the simple one related to the first order SDP relaxation of MAXCUT, which is solved very efficiently by SketchyCGAL” and Mai, Page 3, Paragraph 8, “Therefore to solve each semidefinite relaxation of a CTP-POP one may avoid the computationally costly interior-point methods in some cases.”
Regarding claims 13-14:
The rejection of claim 11 is incorporated in claims 13-14. Claims 13-14 are rejected under the same rationale as set forth in the rejection of claim 3-4.
Regarding claim 16:
Bartan discloses A non-transitory computer-readable medium having program code recorded thereon, the program code executed by a processor(Bartan, ABSTRACT, “Remarkably, we show that semidefinite lifting is always exact and therefore computational complexity for global optimization is polynomial in the input dimension and sample size for all input data” where using computation complexity as a measurement implies the method is completed on a computer/processor)
Bartan discloses program code to model a neural network as a set of polynomials(Bartan, Page 7, Paragraph 5 and Equation 10, “Consider the network f(x) =
∑
j
=
1
m
σ
(
x
T
u
j
)
α
j
where the activation function σ is the degree two polynomial σ(u) = au2 + bu + c. First, we note that the neural network output can be written as [Equation 10]” where the conversion of the neural network with degree-2 polynomial activation into polynomial form is considered modeling a neural network as a set of polynomials.
Bartan discloses program code to receive as input, a global polynomial optimization problem that approximates a training problem of the neural network based on the set of polynomials(Bartan, Page 8, Definition 1, “we will show that the original non-convex neural network problem is solved exactly to global optimality when the optimization is performed over a convex set which we define as the Neural Spectrahedron” where the original non-convex neural network problem to be solved via global optimality using optimization over a convex set is considered receiving a global polynomial optimization problem that approximates a training problem of the neural network based on the set of polynomials.)
Bartan discloses program code to relaxing the global polynomial optimization problem including polynomial constraints with a plurality of semi-definite programs(Bartan, Page 8, Definition 1, “ we will show that the original non-convex neural network problem is solved exactly to global optimality when the optimization is performed over a convex set which we define as the Neural Spectrahedron… Moreover, a Neural Spectrahedron can be described by a simple linear matrix inequality” where the original polynomial NN training problem is reformulated as an semi-definite program by representing the feasible parameter set (Neural Spectrahedron) via LMIs is considered relaxing a polynomial optimization problem including polynomial constraints with a plurality of semi-definite programs(See also: “It is equally important that our results characterize neural networks as constrained linear learning methods <φ(x), ψ> in the lifted feature space φ(x), where the constraints on the lifted parameters ψ are precisely described by a Neural Spectrahedron via linear matrix inequalities. These constraints can be easily tackled with convex semidefinite programming or closed-form projections onto these sets in iterative first-order algorithms”), the semi-definite programs having dimensions corresponding to a total number of weights in the neural network(Bartan, Page 7, Equation 10 and Bartan, Page 14, Theorem 3.1, where the SDP having dimensions corresponding to the total number of weights in the neural network as shown in Equation 10 and Theorem 3.1. In the lifted formulation, the optimization variables of PSD matrices Z, Z` of size (d+1) * (d+1) are used in the SDP that have the weights uj and αj encoded from the neural network parameters which is considered having semi-definite programs having dimensions corresponding to a total number of weights in the neural network(See also: Eq. 14 and 18 for PSD matrix construction)), with weights deeper in the neural network having a higher associated polynomial degree, a last layer in the neural network being a deepest layer and weights in the last layer having a highest associated polynomial degree(Barton, Page 3, Paragraph 1, where the training uses output of hidden layer σ(xTuj) after applying its hidden-layer weights uj and corresponds to one degree less than the weighted output contribution of the output layer σ(xTuj)αj and both the outputs use the weights of the hidden and output layer they both correspond to the having weights deeper in the neural network having a higher associated polynomial degree)
Bartan discloses program code to solve the plurality of semi-definite programs based on a pre-defined structure(Bartan, Page 9, Paragraph 2, “It is equally important that our results characterize neural networks as constrained linear learning methods <φ(x), ψ> in the lifted feature space φ(x), where the constraints on the lifted parameters ψ are precisely described by a Neural Spectrahedron via linear matrix inequalities. These constraints can be easily tackled with convex semidefinite programming or closed-form projections onto these sets in iterative first-order algorithms” where solving constraints described by the LMI of the Neural Spectrahedron using semidefinite programs is considered solving semi-definite programs based on a pre-defined structure) …to compute eigenvalue decomposition(Bartan, Page 19, Paragraph 5, “Let us decompose Z* via eigenvalue decomposition as Z*”) and means for outputting a solution indicating a location of a global optimum of the optimization problem(Bartan, Page 14, Theorem 3.1, Equations 21 and 22, Paragraph 1, “In this section we show that the solution of the convex program (21) provides a lower bound for the solution of the non convex problem (22).” where the SDP is convex and exactly reformulates the original non-convex neural network training problem and the SDP yields Z*, Z`* which represents a globally optimal solution for the original non-convex polynomial optimization) and means for training the neural network with weights deeper in the neural network having the higher associated polynomial degree(Bartan, Page 11, equation 20, where equation 20 shows the outputs of the weights are used in training as shown in the minimizing the loss based on the prediction) based on the solution(Bartan, Page 18, Paragraph 1, " Decomposing the solution of the convex problem Z* and Z`* onto these cones, i.e., neural decomposition, enables the construction of neural network weights from Z* and Z`*" where after neural network decomposition and solving Z, Z`* the neural network and weights are reconstructed which is considered performing using the solution with the neural network to perform training)
Bartan does not disclose a pre-defined structure comprising a block Hankel- plus-Toeplitz structure or the solving comprising performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute computing eigenvalue decomposition
Genin discloses a pre-defined structure comprising a block Hankel- plus-Toeplitz structure(Genin, Page 59, Paragraph 2, “In this light, it is proposed to substitute a basis of Chebyshev polynomials for the natural basis. Such a Chebyshev basis induces a Toeplitz-plus-Hankel structure to the problem with, in principle, a much better numerical conditioning. It is finally recalled how one can take advantage of the Toeplitz-plus-Hankel structure in fast algorithms based on appropriate displacement rank techniques” and Genin, Page 77, Paragraph 6, “For any ˜Q, let us set the block Toeplitz-plus-Hankel matrix”)
References Bartan and Genin are analogous art because they are from same problem-solving area of semidefinite/linear matrix inequality representation and convex optimization of polynomial structures.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Bartan and Genin before him or her, to modify the predefined structure of Bartan to include the Hankel-plus-Toeplitz structure of Genin because to help with ill-conditioned numerical systems. The suggestion/motivation for doing so would have been Genin, Page 22, Paragraph 4, “From a numerical viewpoint, this reformulation of the optimization problem on the real line exhibits a considerable advantage over its initial formulation in the sense that is not intrinsically ill-conditioned…As a result, the numerical behavior of the computational optimization scheme is expected to be substantially improved” and Genin, Page 22, Paragraph 6, “In particular the dual optimization problems can be solved very efficiently using displacement-based factorizations as well as an appropriate divide-and-conquer strategy. These results are direct consequences of the Hankel or Toeplitz structure in the dual constraints.
Bartan-Genin does not explicitly disclose the solving comprising performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute computing eigenvalue decomposition
Mai discloses the solving comprising performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute eigenvalue decomposition(Mai, Page 25, Paragraph 2, “SketchyCGAL provides approximations of the smallest eigenvalue and eigenvector of each matrix C+AT(y+β(z−b))vi by using the randomized Lanczos method” where the randomized Lanczos method corresponds to randomized linear algebra using matrix-vector products with a symmetric matrix C to approximate eigenvalues corresponding to performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute eigenvalue decomposition)
References Bartan-Genin and Mai are analogous art because they are from same problem-solving area of global optimization of non-convex problems using semidefinite/convex reformulations.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Bartan-Genin and Mai before him or her, to modify the eigenvalue decomposition computation of Bartan-Genin to include the eigenvalue decomposition computation of Mai to avoid costly computational problems . The suggestion/motivation for doing so would have been Mai, Page 25, Paragraph 1, “In SDP relaxations of POPs is more expensive than the simple one related to the first order SDP relaxation of MAXCUT, which is solved very efficiently by SketchyCGAL” and Mai, Page 3, Paragraph 8, “Therefore to solve each semidefinite relaxation of a CTP-POP one may avoid the computationally costly interior-point methods in some cases.”
Regarding claims 18-19:
The rejection of claims 16 is incorporated in claims 18-19. Claims 18-19 are rejected under the same rationale as set forth in the rejection of claim 3-4.
Regarding claims 5, 10, 15 and 20:
Claim(s) 5, 10, 15 and 20 is is/are rejected under 35 U.S.C. 103 as being unpatentable over Bartan et al.(Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Time) in view of Genin et al. (“Optimization Problems Over Positive Pseudopolynomial Matrices” henceforth known as Genin) and further in view of Mai et al. (“A hierarchy of spectral relaxations for polynomial optimization” henceforth known as Mai) and even further in view of Henrion et al.(Solving nonconvex optimization problems) henceforth known as Henrion.
Regarding claim 5:
The rejection of Bartan-Genin-Mai discloses the processor-implemented method of claim 1 is incorporated and further:
Bartan-Genin-Mai does not explicitly disclose, however Henrion does disclose representing the global polynomial optimization problem with a Chebyshev basis(Henrion, Page 75, Col. 1, Paragraph 7, “…to construct a sequence of convex semidefinite programming(SDP) or LMI relaxations of P”) where the sequence of convex semidefinite programming(SDP) or LMI relaxations of P is considered a global polynomial optimization problem and Henrion, Page 82, Col. 2, Paragraph 2, “Alternative bases such as Chebyshev polynomials might also prove useful” where the disclosure of Chebyshev bases(plural of basis) as an alternative basis for the SDP and relaxation is considered using a Chebyshev basis with the global polynomial optimization problem)
References Bartan-Genin-Mai and Henrion are analogous art because they are from the using constraints in SDP convex problem optimizations.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Bartan-Genin-Mai and Henrion before him or her, to modify the polynomial basis of Bartan-Genin-Mai to include the Chebyshev basis mentioned in Henrion as the alternative bases could be better conditioned for nonconvex SDP. The Suggestion/motivation for doing so would have been Henrion, Page 82, Col. 2, Paragraph 2, “It is well known that problems involving polynomial bases with monomials of increasing powers are naturally badly conditioned … Alternative bases such as Chebyshev polynomials might also prove useful.”
Regarding claims 10:
The rejection of claim 6 is incorporated in claim 10. Claim 10 is rejected under the same rationale as set forth in the rejection of claim 5.
Regarding claims 15:
The rejection of claim 11 is incorporated in claim 15. Claim 15 is rejected under the same rationale as set forth in the rejection of claim 5.
Regarding claims 20:
The rejection of claims 16 is incorporated in claim 20. Claim 20 is rejected under the same rationale as set forth in the rejection of claim 5.
Relevant Art
The following arts were found to be relevant as found art deal use semi definite problems while handing higher polynomials at deeper layers:
Kileel et al(“On the Expressive Power of Deep Polynomial Neural Networks”)
Zeyuan Allen-Zhu et. al (“Backward Feature Correction: How Deep Learning Performs Deep Learning”)
Lasserre et al. (“Global Optimization with Polynomials and the Problem of Moments”)
Henrion et al. (“Detecting Global Optimality and Extracting Solutions in GloptiPoly”)
Bertsimas et al. (“An Accelerated First-Order Method for Solving Unconstrained SOS Polynomial Optimization Problems”)
Yurtsever et al. (“Scalable Semidefinite Programming”)
Response to Arguments
Applicant's arguments filed 07/24/2026 have been fully considered but they are not persuasive. A breakdown of the arguments can be found below:
101:
Applicant appears to argue on pages 7-8 that the claims and the following limitations cannot practically be performed in the human mind :
modeling a neural network as a set of polynomials
relaxing the global polynomial optimization problem including polynomial constraints with a plurality of semi-definite programs
training a neural network with weights deeper in the neural network having a higher associated polynomial degree based on the solution
Examiner respectfully disagrees as that claims lack specific architecture information or technological instructions that provide/reflect an improvement or provides significantly more and for the following reasons:
The limitation of modeling a neural network as a set of polynomials is an abstract idea of Mathematical Calculations (see MPEP 2106.04(a)(2)(I)(C)) as the polynomials are optimization problems. Alternatively, under the broadest reasonable interpretation, the limitation covers performance of the limitation in the mind. The limitations encompass a user creating/choosing polynomials that represent a neural network. See 2106.04.(a)(2).III.C.
The limitation of relaxing the global polynomial optimization problem including polynomial constraints with a plurality of semi-definite programs is an abstract idea of Mathematical Relationships (see MPEP 2106.04(a)(2)(I)(A))).
The limitation of training a neural network with weights deeper in the neural network having a higher associated polynomial degree based on the solution merely recites a generic computer on which to perform the abstract idea, e.g. "apply it on a computer" (see MPEP 2106.05(f))) as a broad statement of training with weights and is recited at a high level of generality and amounts to using a generic computer component that is implementing a process. While the claims states the weights deeper are higher associated polynomial degree it doesn’t state how the network operates or is changed based on the weights being deeper to provide an improvement. Additionally, Claims that require a computer may still recite a mental process (please see MPEP 2106.04(a)(2).III.C) and training is performed based on the mental process of solving a solution. Without specific architecture or training information related to the weight differences the limitation does not arise to significantly more.
Applicant appears to argue on page 8 that the claims as a whole are patent eligible while stating that training of machine learning models is eligible under 101 and citing Ex parte Desjardins and Example 47, claim 3 as support for eligibility.
Examiner respectfully disagrees as each application must be viewed on their own merits. Ex parte Desjardins the specific continual-learning parameter preservation mechanism of training on a second task using posterior distribution from the first task in the claim as reflecting the improvement and not the broad statement of training. Similarly, the eligibility on Example 47, claim 3 “(d) detecting a source address associated with the one or more malicious network packets,” “(e) dropping the one or more malicious network packets,” and “(f) blocking future traffic from the source address” are the limitations that the example cites as making it eligible. Examiner does not see the parallels between the current claims and Ex parte Desjardins or Example 37.
Further, as the limitation concerning weights deeper in the neural network having the higher associated polynomial degree does not overcome 101 as the weights as recited only limits the mathematical function. Amended language only recites specifics of the mathematical function recited in the claim and only limits the abstract idea further. The weights as recited do not add any specificity to the training process and training is recited generically as claims do not specify how the training is applied to layers or how the training is different from generic training concerning the weight and merely clarifies the neural network has weights with higher associated polynomial degrees based on weight depth but does not state how the neural network handles or uses those layers with higher associated polynomial degrees during training or how it effects the output..
102/103:
Applicant appears to argue on page 9 that Barton does not disclose the amended language that removes the previously mapped low-rank structure or underlying geometry structure” as Bartan does not disclose a Hankel-plus-Toeplitz structure.
Examiner respectfully disagrees as Applicant’s arguments have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Specifically, the new amended language defining the Hankel-plus-Toeplitz structure is mapped to new prior art Genin.
Applicant appears to argue on page 9 that Barton does not disclose the added limitation of performing fast matrix-vector products with a moment matrix and randomized linear algebra to compute eigenvalue decompositions.
Examiner respectfully disagrees as Applicant’s arguments have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Specifically, the amended language is mapped to new prior art Mai.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHARLES JEFFREY JONES JR whose telephone number is (703)756-1414. The examiner can normally be reached Monday - Friday 8:00 - 5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at 571-272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.J.J./Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights