DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in responsive to communication(s): original application filed on 07/22/2024 and preliminary amendment filed on 09/19/2024, said application claims a priority filing date of 08/04/2023. Claims 16-29 remain pending. Claims 16 and 28-29 are independent.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because (1) reference character “9” has been used to designate both "measurement data" in FIG. 1 with Page 11, lines 7 and 17 and "output signal" of "step 190" in FIG. 1 (for a vehicle 50, a driver assistance system 51, a robot 60, a system 70 for quality control, a system 80 for monitoring areas, and/or a system 90 for medical imaging, see also Page 14, lines 23-28); (2) reference character “2a” has been used to designate both "parameters" in FIG. 1 with Page 11, line 23 and Page 12, lines 1-2 and "work results" in Page 17, lines 16 and 19-20; and (3) reference character “2b” has been used to designate both "work results obtained for the training examples 1a*" in FIG. 2 with Page 15, lines 6-8 and "work results obtained for the inputs 1a lied outside the domain D" in FIG. 2 with Page 15, lines 6-8 (NOTE: both "work results" are different as described in Page 15, lines 6-9, however, FIG. 2 with Page 15, lines 6-9 use "2b" to represent both). Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference characters "2a" in Page 17, lines 16 and 19-20 and "2b" in FIG. 2 and from Page 11, line 23 to Page 16, lines 27-28 have both been used to designate "(original) work results". Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: curve "b" in FIG. 3. Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities:
in Page 17, lines 14-20, "… In larger neural networks 1, the law of large numbers causes the distribution of work results 2a to automatically approach the normal distribution. In smaller neural networks 1, i.e. especially in neural networks 1 with fewer than approximately 100 neurons 2, it becomes significantly less probable that the working results 2a are already normally distributed" appears to be "… In larger neural networks 1, the law of large numbers causes the distribution of work results 2b to automatically approach the normal distribution. In smaller neural networks 1, i.e. especially in neural networks 1 with fewer than approximately 100 neurons 2, it becomes significantly less probable that the working results 2b are already normally distributed".
Appropriate correction is required.
Claim Objections
Claims 16-17, 21-22, and 28-29 are objected to because of the following informalities:
in Claim 16, lines 3-4, "… the method comprising the following steps …" appears to be "… the method comprising following steps …";
in Claim 16, line 7, "… selecting a group of neurons is selected …" appears to be "… selecting a group of neurons …" (see also 112 Rejections to Claim 16);
in Claim 17, lines 1-2, "… wherein the respective work results are mapped …" appears to be "… wherein the respective work results of the neurons of the group are mapped …" (see also 112 Rejections to Claim 17);
in Claim 21, lines 1-2, "… wherein the unit function is a differentiable approximation of a discontinuous function" appears to be "… wherein the predetermined unit function is a differentiable approximation of a discontinuous function";
in Claim 22, lines 1-4, "… wherein the differentiable approximation is parameterized with a temperature parameter via which a compromise between an accuracy of the differentiable approximation on the one hand and a speed at which the differentiable approximation saturates, on the other hand can be set" appears to be "… wherein the differentiable approximation is parameterized with a temperature parameter via which a compromise between an accuracy of the differentiable approximation on one hand and a speed at which the differentiable approximation saturates, on other hand can be set";
in Claim 28, lines 1-6, "A non-tranistory machine-readable data carrier on which is stored a computer program including machine-readable instructions for … the instructions, when executed by one or more computers and/or compute instances, causing the one or more computers and/or compute instances to perform the following steps …" appears to be "A non-transitory machine-readable data carrier on which is stored a computer program including machine-readable instructions for … the machine-readable instructions, when executed by one or more computers or compute instances, causing the one or more computers or the compute instances to perform following steps …";
in Claim 29, lines 1-7, "One or more computers and/or compute instances equipped with a non-tranistory machine-readable data carrier on which is stored a computer program including machine- readable instructions for … the instructions, when executed by the one or more computers and/or compute instances, causing the one or more computers and/or compute instances to perform the following steps …" appears to be "One or more computers or the compute instances equipped with a non-transitory machine-readable data carrier on which is stored a computer program including machine-readable instructions for … the machine-readable instructions, when executed by the one or more computers or the compute instances, causing the one or more computers or the compute instances to perform following steps …".
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 16-29 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 16 recites the limitation "… the neural network including a plurality of layers of neurons, wherein outputs of neurons of at least one layer are each fed as inputs into neurons of at least one subsequent layer … processing inputs supplied to each neuron according to parameters associated with the neuron to produce a respective work result of the neuron; selecting a group of neurons …" in lines 1-6 (see also Claim Objections to Claim 16), which rendering the claim indefinite because (1) it is .
Claim 16 recites the limitation "… processing inputs supplied to each neuron according to parameters associated with the neuron to produce a respective work result of the neuron; selecting a group of neurons; … mapping the respective work results of the neurons of the group with a predetermined unit function onto unit values in an interval [0,1], so that the unit value associated with each respective work result occupies a same rank on a list of all unit values as the respective work result occupies on a list of all respective work results; calculating normalized work results from the unit values …" in lines 5-14, which rendering the claim indefinite because (1) it is unclear.
Claims 17-27 are rejected for fully incorporating the deficiency of their respective base claims.
Claim 17 recites the limitation "… wherein the respective work results are mapped as unit values to quantiles that indicate which numerical proportion of the other respective work results is less than the respective work result currently being considered" in lines 1-3 (see also Claim Objections to Claim 17), which rendering the claim indefinite because ".
Claim 18 recites the limitation "… for all other respective work results, a score is ascertained that indicates the extent to which the respective work result is greater than or less than the respective work result currently being considered, and the scores are averaged over the other respective work results" in lines 2-5, which rendering the claim indefinite because ".
Claim 19 recites the limitation "… wherein: a score of 1 is assigned to another respective work result that is greater than the respective work result currently being considered, and a score of 0 is assigned to another respective work result that is less than the respective work result currently being considered" in lines 1-5, which rendering the claim indefinite because ".
Claim 20 recites the limitation "… wherein a difference of another respective work result from the respective work result currently being considered is processed by applying a sigmoid function to a score of the other respective work result" in lines 1-3, which rendering the claim indefinite because ".
Claim 24 recites the limitation "… wherein: the respective work results include activations obtained by summing the inputs of the neuron in a weighted manner based on the parameters associated with the neuron, and the further processing of the normalized work results includes applying a predetermined non-linear activation function to the normalized activations" in lines 1-6, which rendering the claim indefinite because ".
Claim 25 recites the limitation "… wherein the unit values are limited to a predetermined interval …" in lines 1-2, which rendering the claim indefinite because ".
Claims 27-29 recite the limitation "and/or" in lines 3-4, lines 5-6, lines 1 and 6-7, respectively, which rendering these claims indefinite because it is unclear what is included or excluded by the claim language. For examination purpose, "A and/or B" will be considered as "A or B or both" or "at least one of A or B".
Claim 28 recites the limitation "… the neural network including a plurality of layers of neurons, wherein outputs of neurons of at least one layer are each fed as inputs into neurons of at least one subsequent layer … processing inputs supplied to each neuron according to parameters associated with the neuron to produce a respective work result of the neuron; selecting a group of neurons …" in lines (see also Claim Objections to Claim 28), which rendering the claim indefinite because (1) it is unclear whether two instances of ".
Claim 28 recites the limitation "… processing inputs supplied to each neuron according to parameters associated with the neuron to produce a respective work result of the neuron; selecting a group of neurons; … mapping the respective work results of the neurons of the group with a predetermined unit function onto unit values in an interval [0,1], so that the unit value associated with each respective work result occupies a same rank on a list of all unit values as the respective work result occupies on a list of all respective work results; calculating normalized work results from the unit values …" in lines , which rendering the claim indefinite because (1) it is unclear.
Claim 29 recites the limitation "… the neural network including a plurality of layers of neurons, wherein outputs of neurons of at least one layer are each fed as inputs into neurons of at least one subsequent layer … processing inputs supplied to each neuron according to parameters associated with the neuron to produce a respective work result of the neuron; selecting a group of neurons …" in lines (see also Claim Objections to Claim 29), which rendering the claim indefinite because (1) it is unclear whether two instances of ".
Claim 29 recites the limitation "… processing inputs supplied to each neuron according to parameters associated with the neuron to produce a respective work result of the neuron; selecting a group of neurons; … mapping the respective work results of the neurons of the group with a predetermined unit function onto unit values in an interval [0,1], so that the unit value associated with each respective work result occupies a same rank on a list of all unit values as the respective work result occupies on a list of all respective work results; calculating normalized work results from the unit values …" in lines , which rendering the claim indefinite because (1) it is unclear.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 16-18, 21, and 24-29 are rejected under 35 U.S.C. 103 as being unpatentable over Haase-Schuetz et al. (US 2022/0284287 A1, pub. date: 09/08/2022), hereinafter Haase-Schuetz in view of Kwok et al. ("A Novel Target Value Standardization Method Based on Cumulative Distribution Functions for Training Artificial Neural Networks", 2023 IEEE 13th Symposium on Computer Applications & Industrial Electronics (ISCAIE), May 20-21, 2023, pp. 250-255), hereinafter Kwok.
Independent Claims 16 and 28-29
Haase-Schuetz discloses a method for processing measurement data in a neural network (Haase-Schuetz, ¶¶ [0041]-[0043]: an ANN taking the form of a classifier may be used, e.g., to identify objects and/or states of objects sought within the scope of the specific application, in the input quantities of the ANN; a semantic segmentation of physical measurement data may be generated, e.g., by classifying parts of the measurement data as to the type of object, to which they belong; ¶ [0088] with FIG. 4: at least one sensor 6 is provided for recording physical measurement data 6a; measurement data 6a are supplied as input quantities 11 to ANN 1, which may be present, in particular, in its fully trained state 1 *; the output quantities 12' supplied by ANN 1, 1 * are processed in evaluation unit 7 to form a control signal 7a; ¶ [0091] with FIG. 6: in step 230, physical measurement data 6a, which are ascertained by at least one sensor 6, are supplied to trained ANN 1 * as input quantities 11),
the neural network including a plurality of layers of neurons, wherein outputs of neurons of at least one layer are each fed as inputs into neurons of at least one subsequent layer, the method comprising the following steps: processing inputs supplied to each neuron according to parameters associated with the neuron to produce a respective work result of the neuron (Haase-Schuetz, ¶¶ [0004]-[0007]: in deep neural networks having a multitude of layers, it is problematic that there is no control over the orders of magnitude, over which the numerical values of the data processed by the network range; small changes in the input quantities may then produce large changes in the output quantities; Ioffe et al. describes normalizing the numerical values of the data generated in the ANN per processed mini-batch of training data to a uniform order of magnitude; Clevert et al. describes activations of neurons, using a new kind of activation function, which lessens the above-mentioned problem; an artificial neural network includes a plurality of processing layers connected in series (via connected neurons (or processing nodes) in each layer1); the processing layers are each configured to process input quantities in accordance with trainable parameters of the ANN to form output quantities; the output quantities of a layer may each be directed into at least the next layer as input quantities; ¶ [0053]: operating an ANN having a plurality of processing layers connected in series, which are each configured to process input quantities in accordance with trainable parameters of the ANN, to form output quantities; ¶ [0079] with FIG. 1: the ANN 1 includes three processing layers 21-23; each processing layer 21-23 receives input quantities 2la-23a and processes them to form output quantities 21b-23b; input quantities 21a of first processing layer 21 are also input quantities 11 of the ANN 1 as a whole; output quantities 23b of third processing layer 23 are, at the same time, the output quantities 12, 12' of ANN 1 as a whole);
selecting a group of neurons is selected; (Haase-Schuetz, ¶¶ [0008]-[0040]: a new normalizer is inserted into at least one processing layer and/or between at least two processing layers; this normalizer includes a transformation element; this transformation element is configured to transform input quantities directed into the normalizer into one or more input vectors, using a predefined transformation; each of the input quantities enters into exactly one input vector; thus, a single input vector or a collection of input vectors is produced, which has, in total, exactly the same amount of information, that is, e.g., exactly the same amount of numerical values, as were supplied to the normalizer in the input quantities; the normalizer further includes a normalizing element; this normalizing element is configured to normalize the input vector(s) with the aid of a normalizing function, to form one or more output vectors; normalization of a vector is understood to be, in particular, an arithmetic operation, which leaves the number of components of the vector and its direction in the multidimensional space unchanged, but is able to change its norm defined in this multidimensional space; the norm may correspond to, e.g., a length of the vector in the multidimensional space; in particular, the normalization function may be such, that it is able to map vectors, which have markedly different norms, to vectors, which have similar or like norms; the normalization function has at least two different regimes and changes between the regimes as a function of a norm of the input vector at a point and/or in a range, whose position is a function of a predefined parameter ρ; this means that input vectors, whose norm is to the left of the point and/or range (that is, is somewhat smaller), are treated differently by the normalization function from input vectors, whose norm is to the right of the point and/or range (that is, is somewhat larger); in particular, the one regime may include, e.g., during the calculation of the output vector, changing the norm of the input vector absolutely and/or relatively less markedly than provided under the other regime; one of the regimes may also include, e.g., not changing the input vector at all, but taking it on unchanged as an output vector; the normalizer further includes an inverse transformation element; the inverse transformation element is configured to transform the output vectors into output quantities using the inverse of the predefined transformation; these output quantities have the same dimensionality as the input quantities supplied to the normalizer; in this manner, the normalizer may be inserted at an arbitrary position between two processing steps in the ANN (i.e., a group of neurons in layers of the ANN can be selected for applying normalizer); thus, in the further processing by the ANN, the output quantities of the normalizer may take the place of the quantities, which were acquired previously in the ANN and supplied to the normalizer as input quantities; it has been recognized that the numerical stability of the normalization function may be improved, in particular, by changing the regime as a function of the norm of the input vector and specified parameter ρ; if, between two processing steps in the ANN, the numerical values, which represent intermediate results already present, are now combined to form vectors and these vectors are normalized, then the result of this may be that, on one hand, an interval originally present between the useful signal and its processing products, and on the other hand, noise and/or rounding errors, are leveled partially or even completely; using the change between the regimes, it may now be determined, e.g., that all of the input vectors, whose norm does not reach a certain minimum degree, are not changed or only slightly changed in their norm; if, e.g., input vectors having larger norms are simultaneously mapped to output vectors having equal or similar norms, a sufficiently large norm-like interval with regard to the output vectors, which originates from noise and/or rounding errors, still remains; this, in tum, lowers the standards regarding the statistics of the input quantities, which are supplied to the normalizer; it is not necessary to always fall back upon input quantities, which originate from different samples of input quantities supplied to the ANN; instead, the important information contained in the above-mentioned intermediate result of the ANN is preserved, if only numerical values of this intermediate result, which relate to a single sample of input quantities supplied to the ANN, are supplied to the normalizer; thus, the advantages attainable until now with the aid of batch normalization may be attained to the same extent or to a greater extent, without it being necessary for the normalization to apply to mini-batches of training data processed during the training of the ANN; consequently, the effectiveness of the normalization is also, in particular, no longer a function of the size of the mini-batches selected during the training; this, in turn, allows the size of the mini-batches to be selected completely freely, e.g., from the standpoint of the data throughput during the training of the ANN; in contrast to the batch size of the batch normalization, the parameter ρ used by the normalizing element is a continuous and indiscrete parameter; consequently, this parameter ρ is available for optimization in a markedly more effective manner; e.g., it may be trained together with the trainable parameters of the ANN; the ANN may be trained, all in all, in an efficient manner and, at the same, also becomes robust in opposition to manipulation attempts using so-called adversarial examples; in one particularly advantageous refinement, at least one normalization function is configured to leave input vectors, whose norm is less than parameter ρ, unchanged, and to normalize input vectors, whose norm is greater than parameter ρ, to a uniform norm, while maintaining the direction; one example of such a normalization function, which is clarified for vectors in an arbitrary multidimensional space, includes:
π
^
ρ
(
x
→
)
=
x
→
max
1
,
x
→
ρ
; if the norm
x
→
of vector
x
→
is less than ρ, then vector
x
→
remains unchanged; this is the first regime of the normalization function
π
^
ρ
(
x
→
)
; however, if
x
→
is at least equal to ρ, then
π
^
ρ
(
x
→
)
projects vector
x
→
onto a spherical surface having radius ρ; this means that the normalized vector then points in the same direction as before, but ends on the spherical surface; this is the second regime of normalization function
π
^
ρ
(
x
→
)
; when
x
→
= ρ, then a change is made between the two regimes; in a further, particularly advantageous refinement, the change of at least one normalization function between the different regimes is controlled by a softplus function, whose argument has a zero crossing when the norm of the input vector is equal to parameter ρ; an example of such a function is
π
^
ρ
(
x
→
)
=
x
→
1+softplus
x
→
-
ρ
ρ
; in this, the softplus function is given by
s
o
f
t
p
l
u
s
y
=
ln
1
+
e
x
p
y
; the advantage of this function is that it is differentiable in ρ; now, vectors
x
→
having
x
→
less than ρ no longer remain unchanged, but in comparison with vectors
x
→
having a larger norm
x
→
, they are changed markedly less; when
x
→
tends to 0, then norm
x
→
of the vector x in the multidimensional space is reduced by approximately 25% independently of the value of ρ; there is no norm
x
→
, for which
π
^
ρ
(
x
→
)
results in an increase of the norm; thus, not only is the influence of, e.g., rounding errors and noise prevented from being increased, but also this influence is reduced even further, in that norms
x
→
that are overly low are lowered more and are simply not raised to a uniform level; at least one predefined transformation of the input quantities of the normalizer to the input vectors includes transforming a tensor of input quantities into one or more input vectors; for each of the f feature maps, at least one predefined transformation includes combining the feature information items for all locations contained in this feature map to form an input vector assigned to this feature map; in this manner, the greater the expression of certain features in the input values, the greater the norms; for each of the n locations, at least one predefined transformation includes combining the feature information items assigned to this location by all of the feature maps to form an input vector assigned to this location; in this manner, the more feature-rich the input quantities are with regard to the specific location, the larger the norms are; at least one predefined transformation includes combining all feature information items from tensor X in a single input vector; then, the more feature-rich the utilized sample of the input quantities supplied to the ANN is on the whole, the larger is the norm
x
→
of this input vector
x
→
; tensor X may be subjected to further preprocessing prior to use of the normalization function; in particular, in each instance, an arithmetic mean (overall sample mean=mean over all information items regarding the respective sample of the input quantities of the ANN) may be subtracted from all of the feature information items; and/or from the respective feature information items contained in each of the f feature maps, in each instance, an arithmetic mean of the feature information item calculated over this feature map may be subtracted, and/or from the feature information items assigned to each of the n locations by all of the feature maps, in each instance, an arithmetic mean of the feature information items belonging to this location may be subtracted; the normalizer may be "looped in" at any desired position in the ANN, since its output quantities have the same dimensionality as its input quantities and may therefore take the place of these input quantities during the further processing in the ANN; at least one normalizer receives a weighted summation of input quantities of a processing layer as input quantities; the output quantities of this normalizer are directed into a nonlinear activation function for calculating output quantities of the processing layer; if a normalizer is connected to this position in many or even all of the processing layers, then the behavior of the nonlinear activation functions within the ANN may be standardized to a large extent, since these activation functions always operate on values in mainly the same order of magnitude; at least one normalizer receives output quantities of a first processing layer as input quantities, which were calculated, using a nonlinear activation function; the output quantities of this normalizer are directed as input quantities into a further processing layer, which sums these input quantities in a weighted manner in accordance with the trainable parameters; if many or even all transitions between adjacent processing layers in the ANN lead through a normalizer, then the orders of magnitude of the input quantities, which each enter into the weighted summation, may be substantially standardized within the ANN; this ensures that the training converges more effectively; in particular, the accuracy, with which it learns a classification, a regression, and/or a semantic segmentation of real and/or simulated physical measurement data, may be improved markedly; ¶¶ [0052] and [0054]-[0056]: the effect described above and attainable by the normalizer in an ANN is not limited to the normalizer' s constituting a unit encapsulated in some form; it is only important that intermediate products generated during the processing are subjected to the normalization at a suitable location in the ANN, and that the result of the normalization is used in place of the intermediate products during the further processing in the ANN; in at least one processing layer and/or between at least two processing layers, a set of quantities ascertained as input quantities during the process is extracted from the ANN for normalization; the input quantities for the normalization are transformed, using a predefined transformation, into one or more input vectors; each of these input quantities going into exactly one input vector; the input vector(s) are normalized with the aid of a normalization function to form one or more output vectors; this normalization function having at least two different regimes and changing between the regimes as a function of a norm of the input vector at a point and/or in a range, whose position is a function of a predefined parameter ρ; the output vectors are transformed by the inverse of the predefined transformation into output quantities of the normalization, which have the same dimensionality as the input quantities of the normalization; subsequently, the processing in the ANN is continued; the output quantities of the normalization taking the place of the previously extracted input quantities of the normalization; ¶¶ [0080]-[0083] with FIG. 1: two exemplary options of how a normalizer 3 may be introduced into ANN 1, are drawn into FIG. 1; one option is to supply output quantities 21b of first processing layer 21 to normalizer 3 as input quantities 31, and then to supply output quantities 35 of the normalizer to second processing layer 22 as input quantities 22a; the processing proceeding in second processing layer 22, including a second option for integrating normalizer(s) 3, is schematically represented inside of box 22; input quantities 22a are initially summed in accordance with trainable parameters 20 of ANN 1 to form one or more weighted sums, which is indicated by the summation sign; the result is supplied to normalizer 3 as input quantities 31; output quantities 35 of normalizer 3 are converted by a nonlinear activation function (in FIG. 1, indicated as an ReLU function) to output quantities 22b of second processing layer 22; a plurality of different normalizers 3 may be used within one and the same ANN 1; each normalizer 3 may then have, in particular, its own parameters ρ for the transition between the regimes of its normalization function 33; in addition, each normalizer 3 may also be coupled to its own specific preprocessing element; ¶¶ [0084]-[0085] with FIG. 2: normalizer 3 transforms its input quantities 31 into one or more input vectors 32, using a transformation element 3a, which implements a predefined transformation 3a'; these input vectors 32 are supplied to normalization element 3b, and there, they are normalized to form output vectors 34; output vectors 34 are transformed in inverse transformation element 3c in accordance with inverse 32a" of predefined transformation 3a', into output quantities 35 of normalizer 3, which have the same dimensionality as input quantities 31 of normalizer 3; how the normalization of input vectors 32 proceeds to form output vectors 34, is shown in detail inside of box 3b; the normalization function 33 utilized includes two regimes 33a and 33b, in each of which it shows a qualitatively different behavior and acts, in particular, with a different intensity upon input vectors 32; in interaction with at least one predefined parameter ρ, norm 32a of respective input vector 32 decides, which of regimes 33a and 33b is used; for purposes of illustration, this is represented as a binary decision in FIG. 2; in reality, however, it is particularly advantageous for regimes 33a and 33b to merge in a fluid manner, in particular, in a manner that is differentiable in parameter ρ; ¶¶ [0086]-[0087] with FIG. 3: FIG. 3 shows an example of a tensor 31' of input quantities 31 of normalizer 3; tensor 31' is organized as a stack off feature maps 31a; thus, an index i over feature maps 31a runs from 1 to f; each feature map 31a assigns each of n locations 31b feature information item 31c; thus, an index j over locations 31b runs from 1 to n; according to a first option, in each instance, all of the feature information items 31c of a feature map 31a (in this case, the feature map 31a for i=l) are combined in an input vector 32; according to a second option, in each instance, all of the feature information items 31c, which belong to the same location 31b (in this case, the location 31b for j=l), are combined in an input vector 32; a third option, which is not drawn into FIG. 3 for the sake of clarity, is to write all of the feature information items 31c from the entire tensor 31' into a single input vector 32).
Haase-Schuetz further discloses a non-transitory machine-readable data carrier on which is stored a computer program including machine-readable instructions, the instructions, when executed by one or more computers and/or compute instances, causing the one or more computers and/or compute instances to perform the method describe above (Haase-Schuetz, ¶¶ [0070]-[0071] and Claims 42-43: a machine-readable storage medium and/or to a download product including the computer program; a non-transitory machine-readable storage medium on which is stored a computer program for operating an artificial neural network (ANN), including a plurality of processing layers connected in series, which are each configured to process input quantities in accordance with trainable parameters of the ANN to form output quantities, the computer program, when executed by a computer, causing the computer to perform the method describe above).
Haase-Schuetz further discloses one or more computers and/or compute instances equipped with a non-transitory machine-readable data carrier on which is stored a computer program including machine-readable instructions, the instructions, when executed by the one or more computers and/or compute instances, causing the one or more computers and/or compute instances to perform the method describe above (Haase-Schuetz, ¶¶ [0070]-[0071] and Claims 42-43: a machine-readable storage medium and/or to a download product including the computer program; a non-transitory machine-readable storage medium on which is stored a computer program for operating an artificial neural network (ANN), including a plurality of processing layers connected in series, which are each configured to process input quantities in accordance with trainable parameters of the ANN to form output quantities, the computer program, when executed by a computer, causing the computer to perform the method describe above).
Haase-Schuetz fails to explicitly disclose defining a target distribution for the respective work results of the neurons of the group; provide an inverse cumulative density function of the target distribution; mapping the respective work results of the neurons of the group with a predetermined unit function onto unit values in an interval [0,1]; calculating normalized work results from the unit values using the inverse cumulative density function.
Kwok teaches a system and a method relating to artificial neural networks (Kwok, Abstract in Page 250), wherein defining a target distribution for the respective work results of the neurons of the group; provide an inverse cumulative density function of the target distribution; mapping the respective work results of the neurons of the group with a predetermined unit function onto unit values in an interval [0,1]; calculating normalized work results from the unit values using the inverse cumulative density function (Kwok, Abstract in Page 250: function approximation by artificial neural networks (ANNs) are often carried out via a collocation grid approach; however, for certain combinations of grids and functions, the training data for the relevant ANN can be extremely skewed and hence affect training efficiency, thereby necessitating standardization or normalization techniques; the choice of collocation grids often allows uniform training features but not targets; in this paper, we contribute a comparison between methods of target standardization - the relatively common min-max, z-score, normalized log methods and the uncommon empirical cumulative distribution function (CDF) method - in terms of the resulting approximation capabilities of the ANNs using the standardized target values; we demonstrate that the empirical CDF is a general and robust standardization method that allows for efficient training and good approximations in situations of extreme target skewness; novel modifications using a mix of true CDF and empirical CDF in standardizing targets are used to successfully reduce biases arising from the empirical CDF standardization method; Section I in Pages 250-251: function approximation refers to the problem of choosing a function from a specified class to best model a target function, and finds its main applications in describing and forecasting analytically intractable dynamics; ANNs perform most efficiently when training features and training targets are not skewed in distribution or proportions; there is extensive literature on general methods of feature (input) scaling, normalizations and standardizations, but few propose similarly general operations on training targets; common transformations of feature values to reduce skewness are the min-max scaling (alternatively known as 0−1 scaling), z-score standardization and log transformations, but they can be similarly applied to target values; in this paper, we investigate the standardization of target values via their empirical cumulative distribution function (CDF), and their true CDF where feasible, which renders the distribution of target values almost uniform, regardless of their original distribution; we examine the differences in approximation performances of ANNs produced by this standardization technique, relative to ANNs with targets standardized using min-max, z-score and normalized log transformations, under functions that give rise to varying degrees of target skewness; we will only focus on transformations performed on target values before and after training; Section II with FIGS. 1-3 in Pages 251-252: in a collocation grid method used to approximate a function f : X → Y , a sufficiently dense set of points
x
(
i
)
i
=
1
M
is chosen from the domain X, along with their corresponding function values points
f(x
(
i
)
)
i
=
1
M
in Y , to comprise the training dataset
x
(
i
)
,
f
(
x
(
i
)
)
i
=
1
M
; in our investigations, we chose two specific
R
2
→
R
functions f ∈ {f1, f2}:
f
1
≔
exp
-
k
x
2
+
y
2
,
f
1
≔
exp
exp
-
k
x
2
+
y
2
,
where k ∈
R
; both f1 and f2 are surfaces that have distinct bell shapes that peak at the origin (0, 0), with their sharpness controlled by the parameter k; we choose the collocation grid
x
(
i
)
i
=
1
M
,
y
(
j
)
j
=
1
M
via
x
(
i
)
,
y
(
j
)
∼ U(−a, a), a ∈
R
for i, j = 1, …, M, so that X := [−a, a]×[−a, a], where a controls the width of the window over which we wish to approximate f; together, k and a control the skewness of the distribution of the values
f
(
i
,
j
)
≔
f
x
(
i
)
,
y
(
j
)
i
,
j
=
1
M
: due to the collocation grid defined and the shapes of f1 and f2, higher value of a increases the number of smaller values, and a higher value of k increases the sharpness of the peak and decreases the number of larger values; we will examine approximation performances under the settings k = 1.5 and a = 3 to provide considerable skewness; Figure 1 shows the distributions of the points
f
(
i
,
j
)
i
,
j
=
1
M
arising from the collocation grid for f ∈ {f1, f2}; define an arbitrary function F : Y →
R
as the transformation performed on the function value
f
(
i
,
j
)
to produce the corresponding standardized target value
F
(
i
,
j
)
≔
F
f
(
i
,
j
)
=
F
f
x
(
i
)
,
y
(
j
)
in our training dataset; the specific forms of F we will be using are as shown in Eqns. (3)-(6), where each representing the min-max, z-score, normalized log and empirical CDF standardizations respectively; histograms of the target values after these transformations are shown in Figures 2 and 3 for f1 and f2 respectively; Fecdf (i.e., empirical CDF standardizations) eliminates skewness completely, producing a uniform distribution of
F
(
i
,
j
)
’s in both cases; summarily, f1 and f2 define different levels of skewness in our target values; to account for potential skewness (as seen in Figure 2) as well as discontinuities in some of our target values (arising from Fecdf), we explore ANNs with larger depths by trial and error and ultimately chose to use ANNs with K = 8 hidden layers and L = 10 units with tanh activations in each layer; all ANNs were trained using a mini-batched stochastic gradient descent algorithm on a vanilla mean squared error (MSE) loss for a suitable number of epochs; let
ν
f
,
F
x
,
y
be the function that maps an ANN’s inputs to its output, where F is the standardization function performed on values of
f
(
i
,
j
)
; corresponding to the training targets,
ν
f
,
F
x
,
y
approximates
F
(
f
(
x
,
y
)
)
, and hence
f
^
x
,
y
≔
F
-
1
ν
f
,
F
x
,
y
gives the desired approximation to
f
(
x
,
y
)
; the inverse functions of the aforementioned forms of F are given as in Eqns. (7)-(10); to assess the performance of
f
^
on the test set under each type of standardization technique, it is imperative to choose a low-level measure, in which the approximation performance at larger values does not overshadow that of small values; quantities such as R2 (or its equivalent in a non-regression context) and MSE that assess the sum or average performance over all test points are dominated by larger absolute test errors (often arising from larger test values) and obscure potentially large relative inaccuracies of
f
^
at small values; for this reason, we will consider the median of relative squared errors, denoted by η(f, F), for an approximation to f that utilizes the standardization function F - the smaller the value of η, the better the approximation; the relative squared error (RSE) of an approximation
f
^
(
i
,
j
)
≔
f
^
x
(
i
)
,
y
(
j
)
to its corresponding test point
f
(
i
,
j
)
is defined as Eqn. (11); lastly, before interpreting our results, a caveat in our methodology is that there are countless other possible variations to the ANNs’ specifications that may produce better approximations (for example, by using different depths, widths, activations or optimizers); though finding optimal combinations is important, the main focus of this research is to choose adequate models while taking care to avoid obvious issues such as overfitting, and prioritize making the comparisons between models as fair and as straightforward as possible; Section III with FIGS. 4-5 and Tables I-II in Pages 252-253: the performance of the approximation of f1 by
f
^
1
is shown visually in Figure 4; the normalized log transformation gives highly accurate approximations across the entire range of values of f1 and
f
^
1
; the empirical CDF transformation results in decent approximations that seem to have some non-linear bias, and although it performs noticeably poorer than normalized log at all ranges of values, it demonstrates an obvious advantage over min-max and z-score standardizations in approximating small values of orders 10−4 and below; Table I lists the values of η for the approximations of f1 under each standardization technique; in alignment with the visual assessment of performances in Figure 4, normalized log results in the smallest η, followed by empirical CDF, z-score and min-max, with the latter two being several orders of magnitude larger than the former two; similar plots of f2 against
f
^
2
are shown in Figure 5; largely similar observations can be made for the approximation performance of
f
^
2
as that of
f
^
1
under the min-max, z-score and empirical CDF standardization techniques - at large values, min-max and z-score standardizations provide good approximations, better than empirical CDF where a non-linear bias seems to be present; however at small values there is a large number of severely under-trained points under min-max, and z-score to a lesser extent, and this is a pattern not seen under empirical CDF, though it still shows some bias; on the other hand, the approximation performance of
f
^
2
at small values under the normalized log standardization shown in Figure 5 presents a noteworthy difference compared to its counterpart
f
^
1
in Figure 4; specifically, many more points near the bottom of the plot remain under-trained and hence scattered far from the identity line for
f
^
2
than for
f
^
1
; Table II lists the values of η for the approximations of f2 under each standardization technique; in this scenario, standardizing target values using the empirical CDF results in the smallest value of η, followed rather far behind by z-score, normalized log, and min-max; Section IV in Page 254: in approximating both f1 and f2, min-max and z-score standardizations gave rise to extremely skewed target values as seen in Figures 2 and 3, and visually demonstrated poor approximation performances at small values; normalized log offered a clear reduction in skewness for
f
^
1
(
i
,
j
)
’s but not for
f
^
2
(
i
,
j
)
’s, and gave better results - both visually and numerically - in the approximation of f1 than of f2; empirical CDF transformations rendered the target values completely uniform in both cases and produced similarly good approximations, albeit with evident bias at larger values; note that the normalized log standardization is much more effective in the approximation of f1 than of f2; with f1, Flog essentially reduces the target values to be quadratic in (x, y) and hence removes skewness sufficiently for efficient training of the ANN, but with f2, Flog merely renders the target values exponential in (x, y), hence skewness persists and causes inefficient training; in the latter case, the empirical CDF standardization proves to be a much more robust technique to combat skewness and ensure uniform training; the normalized log standardization proves to be more advantageous than the empirical CDF in approximating f1; this shows that complete uniformity, though potentially increases the efficiency in training, is not required to obtain a good function approximation; in fact, the empirical CDF standardization seems to result in some non-linear bias in
f
^
1
, possibly due to discontinuities or "roughness" introduced into the target values by Fecdf, as ANNs with smooth activation functions approximate jump functions less effectively; we also note that
F
ecdf
-
1
(
s
)
becomes increasingly sensitive to small changes in s as s becomes larger due to the sparsity of values in that region, causing inaccuracies or biases from
ν
f
1
,
F
ecdf
x
,
y
to be further magnified; in light of this, we investigate standardization of target values using a mix of the empirical and true CDFs to remove some discontinuities and limit the magnification of such biases in Section V; Section V with FIGS. 6-7 in Pages 254-255: the true CDF of f1(x, y) = exp[−k(x2+y2)] may be derived as [15] shown in Eqn. for G(t); to reduce discontinuities in target values, standardizing the values of
f
1
(
i
,
j
)
by the continuous G is an attractive idea; however, note that there is a clear analytical form for G−1 for s ∈ [1−π/4,1], but not for s ∈ [0,1−π/4); to circumvent this, we simply utilize a mix of the true and empirical CDFs, i.e. the standardization function Fmix defined by:
F
mix
t
=
F
ecdf
(
t
)
for
e
-
2
k
a
2
≤
t
≤
e
-
k
a
2
,
1
-
π
r
(
t
)
4
for
e
-
k
a
2
≤
t
≤
1
,
F
mix
-
1
s
=
inf
t
:
F
ecdf
(
t
)
≥
s
for
0
≤
s
≤
1
-
π
4
,
exp
4
k
a
2
π
s
-
1
for
1
-
π
4
≤
s
≤
1
,
; the histogram of target values
{
F
(
i
,
j
)
}
i
,
j
=
1
M
standardized using Fmix is shown in Figure 6, along with the resulting ANN outputs
ν
f
1
,
F
mix
(
i
,
j
)
corresponding to the test target values
{
F
(
i
,
j
)
}
for
(
i
,
j
)
∈
S
test
; the histogram is uniform below 1 − π/4 due to the use of empirical CDF, and shows fluctuations over the region standardized by the true CDF due to imperfect sampling of
x
(
i
)
’s and
y
(
j
)
’s; the plot of test targets against ANN outputs demonstrates a slight but noticeable jump near the value 1−π/4, as well as some bias for values below 1 − π/4; to end this investigation, we provide plots of test values
f
1
(
i
,
j
)
against the corresponding approximations
f
^
1
(
i
,
j
)
offered by the Fmix standardization - one over the full range of values and one around a neighborhood of
exp
(
-
k
a
2
)
, in Figure 7; compared to purely empirical CDF standardizations (see Figure 4), the bias at large values have been significantly reduced, however this comes with awkward jumps and inaccuracies near the values
exp
(
-
k
a
2
)
≈ 1.37×10−6; the value of η(f1, Fmix) here is 6.68×10−10 - an order of magnitude smaller than under η(f1, Fecdf) (see Table I) - signifying an improvement in approximation performance; alternatively, a separate monotonically increasing function, or ANN, with the mapping Y → [0, 1] may be trained to approximate the true CDF and used to standardize target values).
Haase-Schuetz and Kwok are analogous art because they are from the same field of endeavor, a system and a method relating to artificial neural networks. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Kwok to Haase-Schuetz. Motivation for doing so would provide a much more robust technique to combat skewness and ensure uniform training (Kwok, .
Claim 17
Haase-Schuetz in view of Kwok discloses all the elements as stated in Claim 16 and further discloses wherein the respective work results are mapped as unit values to quantiles that indicate which numerical proportion of the other respective work results is less than the respective work result currently being considered (Haase-Schuetz, ¶¶ [0009]-[0025]: the normalization function has at least two different regimes and changes between the regimes as a function of a norm of the input vector at a point and/or in a range, whose position is a function of a predefined parameter ρ; this means that input vectors, whose norm is to the left of the point and/or range (that is, is somewhat smaller), are treated differently by the normalization function from input vectors, whose norm is to the right of the point and/or range (that is, is somewhat larger); in particular, the one regime may include, e.g., during the calculation of the output vector, changing the norm of the input vector absolutely and/or relatively less markedly than provided under the other regime; one of the regimes may also include, e.g., not changing the input vector at all, but taking it on unchanged as an output vector; it has been recognized that the numerical stability of the normalization function may be improved, in particular, by changing the regime as a function of the norm of the input vector and specified parameter ρ; if, between two processing steps in the ANN, the numerical values, which represent intermediate results already present, are now combined to form vectors and these vectors are normalized, then the result of this may be that, on one hand, an interval originally present between the useful signal and its processing products, and on the other hand, noise and/or rounding errors, are leveled partially or even completely; using the change between the regimes, it may now be determined, e.g., that all of the input vectors, whose norm does not reach a certain minimum degree, are not changed or only slightly changed in their norm; if, e.g., input vectors having larger norms are simultaneously mapped to output vectors having equal or similar norms, a sufficiently large norm-like interval with regard to the output vectors, which originates from noise and/or rounding errors, still remains; this, in tum, lowers the standards regarding the statistics of the input quantities, which are supplied to the normalizer; it is not necessary to always fall back upon input quantities, which originate from different samples of input quantities supplied to the ANN; instead, the important information contained in the above-mentioned intermediate result of the ANN is preserved, if only numerical values of this intermediate result, which relate to a single sample of input quantities supplied to the ANN, are supplied to the normalizer; thus, the advantages attainable until now with the aid of batch normalization may be attained to the same extent or to a greater extent, without it being necessary for the normalization to apply to mini-batches of training data processed during the training of the ANN; consequently, the effectiveness of the normalization is also, in particular, no longer a function of the size of the mini-batches selected during the training; this, in turn, allows the size of the mini-batches to be selected completely freely, e.g., from the standpoint of the data throughput during the training of the ANN; in contrast to the batch size of the batch normalization, the parameter ρ used by the normalizing element is a continuous and indiscrete parameter; consequently, this parameter ρ is available for optimization in a markedly more effective manner; e.g., it may be trained together with the trainable parameters of the ANN; the ANN may be trained, all in all, in an efficient manner and, at the same, also becomes robust in opposition to manipulation attempts using so-called adversarial examples; in one particularly advantageous refinement, at least one normalization function is configured to leave input vectors, whose norm is less than parameter ρ, unchanged, and to normalize input vectors, whose norm is greater than parameter ρ, to a uniform norm, while maintaining the direction; one example of such a normalization function, which is clarified for vectors in an arbitrary multidimensional space, includes:
π
^
ρ
(
x
→
)
=
x
→
max
1
,
x
→
ρ
; if the norm
x
→
of vector
x
→
is less than ρ, then vector
x
→
remains unchanged; this is the first regime of the normalization function
π
^
ρ
(
x
→
)
; however, if
x
→
is at least equal to ρ, then
π
^
ρ
(
x
→
)
projects vector
x
→
onto a spherical surface having radius ρ; this means that the normalized vector then points in the same direction as before, but ends on the spherical surface; this is the second regime of normalization function
π
^
ρ
(
x
→
)
; when
x
→
= ρ, then a change is made between the two regimes; in a further, particularly advantageous refinement, the change of at least one normalization function between the different regimes is controlled by a softplus function, whose argument has a zero crossing when the norm of the input vector is equal to parameter ρ; an example of such a function is
π
^
ρ
(
x
→
)
=
x
→
1+softplus
x
→
-
ρ
ρ
; in this, the softplus function is given by
s
o
f
t
p
l
u
s
y
=
ln
1
+
e
x
p
y
; the advantage of this function is that it is differentiable in ρ; now, vectors
x
→
having
x
→
less than ρ no longer remain unchanged, but in comparison with vectors
x
→
having a larger norm
x
→
, they are changed markedly less; when
x
→
tends to 0, then norm
x
→
of the vector x in the multidimensional space is reduced by approximately 25% independently of the value of ρ; there is no norm
x
→
, for which
π
^
ρ
(
x
→
)
results in an increase of the norm; thus, not only is the influence of, e.g., rounding errors and noise prevented from being increased, but also this influence is reduced even further, in that norms
x
→
that are overly low are lowered more and are simply not raised to a uniform level; ¶¶ [0055], [0061], and [0066]: the input vector(s) are normalized with the aid of a normalization function to form one or more output vectors; this normalization function having at least two different regimes and changing between the regimes as a function of a norm of the input vector at a point and/or in a range, whose position is a function of a predefined parameter ρ; the trainable parameters of the ANN are optimized together with at least one parameter ρ described above, which characterizes the transition between the two regimes of a normalization function; during the further processing of input learning quantities, the objective of this optimization is to obtain output quantities, whose evaluation by the cost function is expected to be more effective; ¶¶ [0080]-[0083] with FIG. 1: two exemplary options of how a normalizer 3 may be introduced into ANN 1, are drawn into FIG. 1; one option is to supply output quantities 21b of first processing layer 21 to normalizer 3 as input quantities 31, and then to supply output quantities 35 of the normalizer to second processing layer 22 as input quantities 22a; the processing proceeding in second processing layer 22, including a second option for integrating normalizer(s) 3, is schematically represented inside of box 22; input quantities 22a are initially summed in accordance with trainable parameters 20 of ANN 1 to form one or more weighted sums, which is indicated by the summation sign; the result is supplied to normalizer 3 as input quantities 31; output quantities 35 of normalizer 3 are converted by a nonlinear activation function (in FIG. 1, indicated as an ReLU function) to output quantities 22b of second processing layer 22; a plurality of different normalizers 3 may be used within one and the same ANN 1; each normalizer 3 may then have, in particular, its own parameters ρ for the transition between the regimes of its normalization function 33; in addition, each normalizer 3 may also be coupled to its own specific preprocessing element; ¶¶ [0084]-[0085] with FIG. 2: normalizer 3 transforms its input quantities 31 into one or more input vectors 32, using a transformation element 3a, which implements a predefined transformation 3a'; these input vectors 32 are supplied to normalization element 3b, and there, they are normalized to form output vectors 34; output vectors 34 are transformed in inverse transformation element 3c in accordance with inverse 32a" of predefined transformation 3a', into output quantities 35 of normalizer 3, which have the same dimensionality as input quantities 31 of normalizer 3; how the normalization of input vectors 32 proceeds to form output vectors 34, is shown in detail inside of box 3b; the normalization function 33 utilized includes two regimes 33a and 33b, in each of which it shows a qualitatively different behavior and acts, in particular, with a different intensity upon input vectors 32; in interaction with at least one predefined parameter ρ, norm 32a of respective input vector 32 decides, which of regimes 33a and 33b is used; for purposes of illustration, this is represented as a binary decision in FIG. 2; in reality, however, it is particularly advantageous for regimes 33a and 33b to merge in a fluid manner, in particular, in a manner that is differentiable in parameter ρ) (Kwok, Section II with FIGS. 1-3 in Pages 251-252: in a collocation grid method used to approximate a function f : X → Y , a sufficiently dense set of points
x
(
i
)
i
=
1
M
is chosen from the domain X, along with their corresponding function values points
f(x
(
i
)
)
i
=
1
M
in Y , to comprise the training dataset
x
(
i
)
,
f
(
x
(
i
)
)
i
=
1
M
; in our investigations, we chose two specific
R
2
→
R
functions f ∈ {f1, f2}:
f
1
≔
exp
-
k
x
2
+
y
2
,
f
1
≔
exp
exp
-
k
x
2
+
y
2
,
where k ∈
R
; both f1 and f2 are surfaces that have distinct bell shapes that peak at the origin (0, 0), with their sharpness controlled by the parameter k; we choose the collocation grid
x
(
i
)
i
=
1
M
,
y
(
j
)
j
=
1
M
via
x
(
i
)
,
y
(
j
)
∼ U(−a, a), a ∈
R
for i, j = 1, …, M, so that X := [−a,a]×[−a,a], where a controls the width of the window over which we wish to approximate f; together, k and a control the skewness of the distribution of the values
f
(
i
,
j
)
≔
f
x
(
i
)
,
y
(
j
)
i
,
j
=
1
M
: due to the collocation grid defined and the shapes of f1 and f2, higher value of a increases the number of smaller values, and a higher value of k increases the sharpness of the peak and decreases the number of larger values; we will examine approximation performances under the settings k = 1.5 and a = 3 to provide considerable skewness; Figure 1 shows the distributions of the points
f
(
i
,
j
)
i
,
j
=
1
M
arising from the collocation grid for f ∈ {f1, f2}; define an arbitrary function F : Y →
R
as the transformation performed on the function value
f
(
i
,
j
)
to produce the corresponding standardized target value
F
(
i
,
j
)
≔
F
f
(
i
,
j
)
=
F
f
x
(
i
)
,
y
(
j
)
in our training dataset; the specific forms of F we will be using are as shown in Eqns. (3)-(6), where each representing the min-max, z-score, normalized log and empirical CDF standardizations respectively; histograms of the target values after these transformations are shown in Figures 2 and 3 for f1 and f2 respectively; Fecdf (i.e., empirical CDF standardizations) eliminates skewness completely, producing a uniform distribution of
F
(
i
,
j
)
’s in both cases; summarily, f1 and f2 define different levels of skewness in our target values; to account for potential skewness (as seen in Figure 2) as well as discontinuities in some of our target values (arising from Fecdf), we explore ANNs with larger depths by trial and error and ultimately chose to use ANNs with K = 8 hidden layers and L = 10 units with tanh activations in each layer; all ANNs were trained using a mini-batched stochastic gradient descent algorithm on a vanilla mean squared error (MSE) loss for a suitable number of epochs; let
ν
f
,
F
x
,
y
be the function that maps an ANN’s inputs to its output, where F is the standardization function performed on values of
f
(
i
,
j
)
; corresponding to the training targets,
ν
f
,
F
x
,
y
approximates
F
(
f
(
x
,
y
)
)
, and hence
f
^
x
,
y
≔
F
-
1
ν
f
,
F
x
,
y
gives the desired approximation to
f
(
x
,
y
)
; the inverse functions of the aforementioned forms of F are given as in Eqns. (7)-(10); to assess the performance of
f
^
on the test set under each type of standardization technique, it is imperative to choose a low-level measure, in which the approximation performance at larger values does not overshadow that of small values; quantities such as R2 (or its equivalent in a non-regression context) and MSE that assess the sum or average performance over all test points are dominated by larger absolute test errors (often arising from larger test values) and obscure potentially large relative inaccuracies of
f
^
at small values; for this reason, we will consider the median of relative squared errors, denoted by η(f, F), for an approximation to f that utilizes the standardization function F - the smaller the value of η, the better the approximation; the relative squared error (RSE) of an approximation
f
^
(
i
,
j
)
≔
f
^
x
(
i
)
,
y
(
j
)
to its corresponding test point
f
(
i
,
j
)
is defined as Eqn. (11); lastly, before interpreting our results, a caveat in our methodology is that there are countless other possible variations to the ANNs’ specifications that may produce better approximations (for example, by using different depths, widths, activations or optimizers); though finding optimal combinations is important, the main focus of this research is to choose adequate models while taking care to avoid obvious issues such as overfitting, and prioritize making the comparisons between models as fair and as straightforward as possible; Section V in Pages 254-255: alternatively, a separate monotonically increasing function, or ANN, with the mapping Y → [0, 1] may be trained to approximate the true CDF and used to standardize target values).
Claim 18
Haase-Schuetz in view of Kwok discloses all the elements as stated in Claim 16 and further discloses wherein: for all other respective work results, a score is ascertained that indicates the extent to which the respective work result is greater than or less than the respective work result currently being considered, and the scores are averaged over the other respective work results (Haase-Schuetz, ¶¶ [0009]-[0025]: the normalization function has at least two different regimes and changes between the regimes as a function of a norm of the input vector at a point and/or in a range, whose position is a function of a predefined parameter ρ; this means that input vectors, whose norm is to the left of the point and/or range (that is, is somewhat smaller), are treated differently by the normalization function from input vectors, whose norm is to the right of the point and/or range (that is, is somewhat larger); in particular, the one regime may include, e.g., during the calculation of the output vector, changing the norm of the input vector absolutely and/or relatively less markedly than provided under the other regime; one of the regimes may also include, e.g., not changing the input vector at all, but taking it on unchanged as an output vector; it has been recognized that the numerical stability of the normalization function may be improved, in particular, by changing the regime as a function of the norm of the input vector and specified parameter ρ; if, between two processing steps in the ANN, the numerical values, which represent intermediate results already present, are now combined to form vectors and these vectors are normalized, then the result of this may be that, on one hand, an interval originally present between the useful signal and its processing products, and on the other hand, noise and/or rounding errors, are leveled partially or even completely; using the change between the regimes, it may now be determined, e.g., that all of the input vectors, whose norm does not reach a certain minimum degree, are not changed or only slightly changed in their norm; if, e.g., input vectors having larger norms are simultaneously mapped to output vectors having equal or similar norms, a sufficiently large norm-like interval with regard to the output vectors, which originates from noise and/or rounding errors, still remains; this, in tum, lowers the standards regarding the statistics of the input quantities, which are supplied to the normalizer; it is not necessary to always fall back upon input quantities, which originate from different samples of input quantities supplied to the ANN; instead, the important information contained in the above-mentioned intermediate result of the ANN is preserved, if only numerical values of this intermediate result, which relate to a single sample of input quantities supplied to the ANN, are supplied to the normalizer; thus, the advantages attainable until now with the aid of batch normalization may be attained to the same extent or to a greater extent, without it being necessary for the normalization to apply to mini-batches of training data processed during the training of the ANN; consequently, the effectiveness of the normalization is also, in particular, no longer a function of the size of the mini-batches selected during the training; this, in turn, allows the size of the mini-batches to be selected completely freely, e.g., from the standpoint of the data throughput during the training of the ANN; in contrast to the batch size of the batch normalization, the parameter ρ used by the normalizing element is a continuous and indiscrete parameter; consequently, this parameter ρ is available for optimization in a markedly more effective manner; e.g., it may be trained together with the trainable parameters of the ANN; the ANN may be trained, all in all, in an efficient manner and, at the same, also becomes robust in opposition to manipulation attempts using so-called adversarial examples; in one particularly advantageous refinement, at least one normalization function is configured to leave input vectors, whose norm is less than parameter ρ, unchanged, and to normalize input vectors, whose norm is greater than parameter ρ, to a uniform norm, while maintaining the direction; one example of such a normalization function, which is clarified for vectors in an arbitrary multidimensional space, includes:
π
^
ρ
(
x
→
)
=
x
→
max
1
,
x
→
ρ
; if the norm
x
→
of vector
x
→
is less than ρ, then vector
x
→
remains unchanged; this is the first regime of the normalization function
π
^
ρ
(
x
→
)
; however, if
x
→
is at least equal to ρ, then
π
^
ρ
(
x
→
)
projects vector
x
→
onto a spherical surface having radius ρ; this means that the normalized vector then points in the same direction as before, but ends on the spherical surface; this is the second regime of normalization function
π
^
ρ
(
x
→
)
; when
x
→
= ρ, then a change is made between the two regimes; in a further, particularly advantageous refinement, the change of at least one normalization function between the different regimes is controlled by a softplus function, whose argument has a zero crossing when the norm of the input vector is equal to parameter ρ; an example of such a function is
π
^
ρ
(
x
→
)
=
x
→
1+softplus
x
→
-
ρ
ρ
; in this, the softplus function is given by
s
o
f
t
p
l
u
s
y
=
ln
1
+
e
x
p
y
; the advantage of this function is that it is differentiable in ρ; now, vectors
x
→
having
x
→
less than ρ no longer remain unchanged, but in comparison with vectors
x
→
having a larger norm
x
→
, they are changed markedly less; when
x
→
tends to 0, then norm
x
→
of the vector x in the multidimensional space is reduced by approximately 25% independently of the value of ρ; there is no norm
x
→
, for which
π
^
ρ
(
x
→
)
results in an increase of the norm; thus, not only is the influence of, e.g., rounding errors and noise prevented from being increased, but also this influence is reduced even further, in that norms
x
→
that are overly low are lowered more and are simply not raised to a uniform level; ¶¶ [0055], [0061], and [0066]: the input vector(s) are normalized with the aid of a normalization function to form one or more output vectors; this normalization function having at least two different regimes and changing between the regimes as a function of a norm of the input vector at a point and/or in a range, whose position is a function of a predefined parameter ρ; the trainable parameters of the ANN are optimized together with at least one parameter ρ described above, which characterizes the transition between the two regimes of a normalization function; during the further processing of input learning quantities, the objective of this optimization is to obtain output quantities, whose evaluation by the cost function is expected to be more effective; ¶¶ [0080]-[0083] with FIG. 1: two exemplary options of how a normalizer 3 may be introduced into ANN 1, are drawn into FIG. 1; one option is to supply output quantities 21b of first processing layer 21 to normalizer 3 as input quantities 31, and then to supply output quantities 35 of the normalizer to second processing layer 22 as input quantities 22a; the processing proceeding in second processing layer 22, including a second option for integrating normalizer(s) 3, is schematically represented inside of box 22; input quantities 22a are initially summed in accordance with trainable parameters 20 of ANN 1 to form one or more weighted sums, which is indicated by the summation sign; the result is supplied to normalizer 3 as input quantities 31; output quantities 35 of normalizer 3 are converted by a nonlinear activation function (in FIG. 1, indicated as an ReLU function) to output quantities 22b of second processing layer 22; a plurality of different normalizers 3 may be used within one and the same ANN 1; each normalizer 3 may then have, in particular, its own parameters ρ for the transition between the regimes of its normalization function 33; in addition, each normalizer 3 may also be coupled to its own specific preprocessing element; ¶¶ [0084]-[0085] with FIG. 2: normalizer 3 transforms its input quantities 31 into one or more input vectors 32, using a transformation element 3a, which implements a predefined transformation 3a'; these input vectors 32 are supplied to normalization element 3b, and there, they are normalized to form output vectors 34; output vectors 34 are transformed in inverse transformation element 3c in accordance with inverse 32a" of predefined transformation 3a', into output quantities 35 of normalizer 3, which have the same dimensionality as input quantities 31 of normalizer 3; how the normalization of input vectors 32 proceeds to form output vectors 34, is shown in detail inside of box 3b; the normalization function 33 utilized includes two regimes 33a and 33b, in each of which it shows a qualitatively different behavior and acts, in particular, with a different intensity upon input vectors 32; in interaction with at least one predefined parameter ρ, norm 32a of respective input vector 32 decides, which of regimes 33a and 33b is used; for purposes of illustration, this is represented as a binary decision in FIG. 2; in reality, however, it is particularly advantageous for regimes 33a and 33b to merge in a fluid manner, in particular, in a manner that is differentiable in parameter ρ) (Kwok, Section II with FIGS. 1-3 in Pages 251-252: in a collocation grid method used to approximate a function f : X → Y , a sufficiently dense set of points
x
(
i
)
i
=
1
M
is chosen from the domain X, along with their corresponding function values points
f(x
(
i
)
)
i
=
1
M
in Y , to comprise the training dataset
x
(
i
)
,
f
(
x
(
i
)
)
i
=
1
M
; in our investigations, we chose two specific
R
2
→
R
functions f ∈ {f1, f2}:
f
1
≔
exp
-
k
x
2
+
y
2
,
f
1
≔
exp
exp
-
k
x
2
+
y
2
,
where k ∈
R
; both f1 and f2 are surfaces that have distinct bell shapes that peak at the origin (0, 0), with their sharpness controlled by the parameter k; we choose the collocation grid
x
(
i
)
i
=
1
M
,
y
(
j
)
j
=
1
M
via
x
(
i
)
,
y
(
j
)
∼ U(−a, a), a ∈
R
for i, j = 1, …, M, so that X := [−a,a]×[−a,a], where a controls the width of the window over which we wish to approximate f; together, k and a control the skewness of the distribution of the values
f
(
i
,
j
)
≔
f
x
(
i
)
,
y
(
j
)
i
,
j
=
1
M
: due to the collocation grid defined and the shapes of f1 and f2, higher value of a increases the number of smaller values, and a higher value of k increases the sharpness of the peak and decreases the number of larger values; we will examine approximation performances under the settings k = 1.5 and a = 3 to provide considerable skewness; Figure 1 shows the distributions of the points
f
(
i
,
j
)
i
,
j
=
1
M
arising from the collocation grid for f ∈ {f1, f2}; define an arbitrary function F : Y →
R
as the transformation performed on the function value
f
(
i
,
j
)
to produce the corresponding standardized target value
F
(
i
,
j
)
≔
F
f
(
i
,
j
)
=
F
f
x
(
i
)
,
y
(
j
)
in our training dataset; the specific forms of F we will be using are as shown in Eqns. (3)-(6), where each representing the min-max, z-score, normalized log and empirical CDF standardizations respectively; histograms of the target values after these transformations are shown in Figures 2 and 3 for f1 and f2 respectively; Fecdf (i.e., empirical CDF standardizations) eliminates skewness completely, producing a uniform distribution of
F
(
i
,
j
)
’s in both cases; summarily, f1 and f2 define different levels of skewness in our target values; to account for potential skewness (as seen in Figure 2) as well as discontinuities in some of our target values (arising from Fecdf), we explore ANNs with larger depths by trial and error and ultimately chose to use ANNs with K = 8 hidden layers and L = 10 units with tanh activations in each layer; all ANNs were trained using a mini-batched stochastic gradient descent algorithm on a vanilla mean squared error (MSE) loss for a suitable number of epochs; let
ν
f
,
F
x
,
y
be the function that maps an ANN’s inputs to its output, where F is the standardization function performed on values of
f
(
i
,
j
)
; corresponding to the training targets,
ν
f
,
F
x
,
y
approximates
F
(
f
(
x
,
y
)
)
, and hence
f
^
x
,
y
≔
F
-
1
ν
f
,
F
x
,
y
gives the desired approximation to
f
(
x
,
y
)
; the inverse functions of the aforementioned forms of F are given as in Eqns. (7)-(10); to assess the performance of
f
^
on the test set under each type of standardization technique, it is imperative to choose a low-level measure, in which the approximation performance at larger values does not overshadow that of small values; quantities such as R2 (or its equivalent in a non-regression context) and MSE that assess the sum or average performance over all test points are dominated by larger absolute test errors (often arising from larger test values) and obscure potentially large relative inaccuracies of
f
^
at small values; for this reason, we will consider the median of relative squared errors, denoted by η(f, F), for an approximation to f that utilizes the standardization function F - the smaller the value of η, the better the approximation; the relative squared error (RSE) of an approximation
f
^
(
i
,
j
)
≔
f
^
x
(
i
)
,
y
(
j
)
to its corresponding test point
f
(
i
,
j
)
is defined as Eqn. (11); lastly, before interpreting our results, a caveat in our methodology is that there are countless other possible variations to the ANNs’ specifications that may produce better approximations (for example, by using different depths, widths, activations or optimizers); though finding optimal combinations is important, the main focus of this research is to choose adequate models while taking care to avoid obvious issues such as overfitting, and prioritize making the comparisons between models as fair and as straightforward as possible; Section V in Pages 254-255: alternatively, a separate monotonically increasing function, or ANN, with the mapping Y → [0, 1] may be trained to approximate the true CDF and used to standardize target values).
Claim 21
Haase-Schuetz in view of Kwok discloses all the elements as stated in Claim 16 and further discloses wherein the unit function is a differentiable approximation of a discontinuous function (Haase-Schuetz, ¶¶ [0009]-[0025]: the normalization function has at least two different regimes and changes between the regimes as a function of a norm of the input vector at a point and/or in a range, whose position is a function of a predefined parameter ρ; this means that input vectors, whose norm is to the left of the point and/or range (that is, is somewhat smaller), are treated differently by the normalization function from input vectors, whose norm is to the right of the point and/or range (that is, is somewhat larger); in particular, the one regime may include, e.g., during the calculation of the output vector, changing the norm of the input vector absolutely and/or relatively less markedly than provided under the other regime; one of the regimes may also include, e.g., not changing the input vector at all, but taking it on unchanged as an output vector; it has been recognized that the numerical stability of the normalization function may be improved, in particular, by changing the regime as a function of the norm of the input vector and specified parameter ρ; if, between two processing steps in the ANN, the numerical values, which represent intermediate results already present, are now combined to form vectors and these vectors are normalized, then the result of this may be that, on one hand, an interval originally present between the useful signal and its processing products, and on the other hand, noise and/or rounding errors, are leveled partially or even completely; using the change between the regimes, it may now be determined, e.g., that all of the input vectors, whose norm does not reach a certain minimum degree, are not changed or only slightly changed in their norm; if, e.g., input vectors having larger norms are simultaneously mapped to output vectors having equal or similar norms, a sufficiently large norm-like interval with regard to the output vectors, which originates from noise and/or rounding errors, still remains; this, in tum, lowers the standards regarding the statistics of the input quantities, which are supplied to the normalizer; it is not necessary to always fall back upon input quantities, which originate from different samples of input quantities supplied to the ANN; instead, the important information contained in the above-mentioned intermediate result of the ANN is preserved, if only numerical values of this intermediate result, which relate to a single sample of input quantities supplied to the ANN, are supplied to the normalizer; thus, the advantages attainable until now with the aid of batch normalization may be attained to the same extent or to a greater extent, without it being necessary for the normalization to apply to mini-batches of training data processed during the training of the ANN; consequently, the effectiveness of the normalization is also, in particular, no longer a function of the size of the mini-batches selected during the training; this, in turn, allows the size of the mini-batches to be selected completely freely, e.g., from the standpoint of the data throughput during the training of the ANN; in contrast to the batch size of the batch normalization, the parameter ρ used by the normalizing element is a continuous and indiscrete parameter; consequently, this parameter ρ is available for optimization in a markedly more effective manner; e.g., it may be trained together with the trainable parameters of the ANN; the ANN may be trained, all in all, in an efficient manner and, at the same, also becomes robust in opposition to manipulation attempts using so-called adversarial examples; in one particularly advantageous refinement, at least one normalization function is configured to leave input vectors, whose norm is less than parameter ρ, unchanged, and to normalize input vectors, whose norm is greater than parameter ρ, to a uniform norm, while maintaining the direction; one example of such a normalization function, which is clarified for vectors in an arbitrary multidimensional space, includes:
π
^
ρ
(
x
→
)
=
x
→
max
1
,
x
→
ρ
; if the norm
x
→
of vector
x
→
is less than ρ, then vector
x
→
remains unchanged; this is the first regime of the normalization function
π
^
ρ
(
x
→
)
; however, if
x
→
is at least equal to ρ, then
π
^
ρ
(
x
→
)
projects vector
x
→
onto a spherical surface having radius ρ; this means that the normalized vector then points in the same direction as before, but ends on the spherical surface; this is the second regime of normalization function
π
^
ρ
(
x
→
)
; when
x
→
= ρ, then a change is made between the two regimes; in a further, particularly advantageous refinement, the change of at least one normalization function between the different regimes is controlled by a softplus function, whose argument has a zero crossing when the norm of the input vector is equal to parameter ρ; an example of such a function is
π
^
ρ
(
x
→
)
=
x
→
1+softplus
x
→
-
ρ
ρ
; in this, the softplus function is given by
s
o
f
t
p
l
u
s
y
=
ln
1
+
e
x
p
y
; the advantage of this function is that it is differentiable in ρ; now, vectors
x
→
having
x
→
less than ρ no longer remain unchanged, but in comparison with vectors
x
→
having a larger norm
x
→
, they are changed markedly less; when
x
→
tends to 0, then norm
x
→
of the vector x in the multidimensional space is reduced by approximately 25% independently of the value of ρ; there is no norm
x
→
, for which
π
^
ρ
(
x
→
)
results in an increase of the norm; thus, not only is the influence of, e.g., rounding errors and noise prevented from being increased, but also this influence is reduced even further, in that norms
x
→
that are overly low are lowered more and are simply not raised to a uniform level; ¶¶ [0055], [0061], and [0066]: the input vector(s) are normalized with the aid of a normalization function to form one or more output vectors; this normalization function having at least two different regimes and changing between the regimes as a function of a norm of the input vector at a point and/or in a range, whose position is a function of a predefined parameter ρ; the trainable parameters of the ANN are optimized together with at least one parameter ρ described above, which characterizes the transition between the two regimes of a normalization function; during the further processing of input learning quantities, the objective of this optimization is to obtain output quantities, whose evaluation by the cost function is expected to be more effective; ¶¶ [0080]-[0083] with FIG. 1: two exemplary options of how a normalizer 3 may be introduced into ANN 1, are drawn into FIG. 1; one option is to supply output quantities 21b of first processing layer 21 to normalizer 3 as input quantities 31, and then to supply output quantities 35 of the normalizer to second processing layer 22 as input quantities 22a; the processing proceeding in second processing layer 22, including a second option for integrating normalizer(s) 3, is schematically represented inside of box 22; input quantities 22a are initially summed in accordance with trainable parameters 20 of ANN 1 to form one or more weighted sums, which is indicated by the summation sign; the result is supplied to normalizer 3 as input quantities 31; output quantities 35 of normalizer 3 are converted by a nonlinear activation function (in FIG. 1, indicated as an ReLU function) to output quantities 22b of second processing layer 22; a plurality of different normalizers 3 may be used within one and the same ANN 1; each normalizer 3 may then have, in particular, its own parameters ρ for the transition between the regimes of its normalization function 33; in addition, each normalizer 3 may also be coupled to its own specific preprocessing element; ¶¶ [0084]-[0085] with FIG. 2: normalizer 3 transforms its input quantities 31 into one or more input vectors 32, using a transformation element 3a, which implements a predefined transformation 3a'; these input vectors 32 are supplied to normalization element 3b, and there, they are normalized to form output vectors 34; output vectors 34 are transformed in inverse transformation element 3c in accordance with inverse 32a" of predefined transformation 3a', into output quantities 35 of normalizer 3, which have the same dimensionality as input quantities 31 of normalizer 3; how the normalization of input vectors 32 proceeds to form output vectors 34, is shown in detail inside of box 3b; the normalization function 33 utilized includes two regimes 33a and 33b, in each of which it shows a qualitatively different behavior and acts, in particular, with a different intensity upon input vectors 32; in interaction with at least one predefined parameter ρ, norm 32a of respective input vector 32 decides, which of regimes 33a and 33b is used; for purposes of illustration, this is represented as a binary decision in FIG. 2; in reality, however, it is particularly advantageous for regimes 33a and 33b to merge in a fluid manner, in particular, in a manner that is differentiable in parameter ρ) (Kwok, Section II with FIGS. 1-3 in Pages 251-252: in a collocation grid method used to approximate a function f : X → Y , a sufficiently dense set of points
x
(
i
)
i
=
1
M
is chosen from the domain X, along with their corresponding function values points
f(x
(
i
)
)
i
=
1
M
in Y , to comprise the training dataset
x
(
i
)
,
f
(
x
(
i
)
)
i
=
1
M
; in our investigations, we chose two specific
R
2
→
R
functions f ∈ {f1, f2}:
f
1
≔
exp
-
k
x
2
+
y
2
,
f
1
≔
exp
exp
-
k
x
2
+
y
2
,
where k ∈
R
; both f1 and f2 are surfaces that have distinct bell shapes that peak at the origin (0, 0), with their sharpness controlled by the parameter k; we choose the collocation grid
x
(
i
)
i
=
1
M
,
y
(
j
)
j
=
1
M
via
x
(
i
)
,
y
(
j
)
∼ U(−a, a), a ∈
R
for i, j = 1, …, M, so that X := [−a,a]×[−a,a], where a controls the width of the window over which we wish to approximate f; together, k and a control the skewness of the distribution of the values
f
(
i
,
j
)
≔
f
x
(
i
)
,
y
(
j
)
i
,
j
=
1
M
: due to the collocation grid defined and the shapes of f1 and f2, higher value of a increases the number of smaller values, and a higher value of k increases the sharpness of the peak and decreases the number of larger values; we will examine approximation performances under the settings k = 1.5 and a = 3 to provide considerable skewness; Figure 1 shows the distributions of the points
f
(
i
,
j
)
i
,
j
=
1
M
arising from the collocation grid for f ∈ {f1, f2}; define an arbitrary function F : Y →
R
as the transformation performed on the function value
f
(
i
,
j
)
to produce the corresponding standardized target value
F
(
i
,
j
)
≔
F
f
(
i
,
j
)
=
F
f
x
(
i
)
,
y
(
j
)
in our training dataset; the specific forms of F we will be using are as shown in Eqns. (3)-(6), where each representing the min-max, z-score, normalized log and empirical CDF standardizations respectively; histograms of the target values after these transformations are shown in Figures 2 and 3 for f1 and f2 respectively; Fecdf (i.e., empirical CDF standardizations) eliminates skewness completely, producing a uniform distribution of
F
(
i
,
j
)
’s in both cases; summarily, f1 and f2 define different levels of skewness in our target values; to account for potential skewness (as seen in Figure 2) as well as discontinuities in some of our target values (arising from Fecdf), we explore ANNs with larger depths by trial and error and ultimately chose to use ANNs with K = 8 hidden layers and L = 10 units with tanh activations in each layer; all ANNs were trained using a mini-batched stochastic gradient descent algorithm on a vanilla mean squared error (MSE) loss for a suitable number of epochs; let
ν
f
,
F
x
,
y
be the function that maps an ANN’s inputs to its output, where F is the standardization function performed on values of
f
(
i
,
j
)
; corresponding to the training targets,
ν
f
,
F
x
,
y
approximates
F
(
f
(
x
,
y
)
)
, and hence
f
^
x
,
y
≔
F
-
1
ν
f
,
F
x
,
y
gives the desired approximation to
f
(
x
,
y
)
; the inverse functions of the aforementioned forms of F are given as in Eqns. (7)-(10); to assess the performance of
f
^
on the test set under each type of standardization technique, it is imperative to choose a low-level measure, in which the approximation performance at larger values does not overshadow that of small values; quantities such as R2 (or its equivalent in a non-regression context) and MSE that assess the sum or average performance over all test points are dominated by larger absolute test errors (often arising from larger test values) and obscure potentially large relative inaccuracies of
f
^
at small values; for this reason, we will consider the median of relative squared errors, denoted by η(f, F), for an approximation to f that utilizes the standardization function F - the smaller the value of η, the better the approximation; the relative squared error (RSE) of an approximation
f
^
(
i
,
j
)
≔
f
^
x
(
i
)
,
y
(
j
)
to its corresponding test point
f
(
i
,
j
)
is defined as Eqn. (11); lastly, before interpreting our results, a caveat in our methodology is that there are countless other possible variations to the ANNs’ specifications that may produce better approximations (for example, by using different depths, widths, activations or optimizers); though finding optimal combinations is important, the main focus of this research is to choose adequate models while taking care to avoid obvious issues such as overfitting, and prioritize making the comparisons between models as fair and as straightforward as possible; Section V in Pages 254-255: alternatively, a separate monotonically increasing function, or ANN, with the mapping Y → [0, 1] may be trained to approximate the true CDF and used to standardize target values).
Claim 24
Haase-Schuetz in view of Kwok discloses all the elements as stated in Claim 16 and further discloses wherein: the respective work results include activations obtained by summing the inputs of the neuron in a weighted manner based on the parameters associated with the neuron, and the further processing of the normalized work results includes applying a predetermined non-linear activation function to the normalized activations (Haase-Schuetz, ¶¶ [0038]-[0039]: at least one normalizer receives a weighted summation of input quantities of a processing layer as input quantities; the output quantities of this normalizer are directed into a nonlinear activation function for calculating output quantities of the processing layer; if a normalizer is connected to this position in many or even all of the processing layers, then the behavior of the nonlinear activation functions within the ANN may be standardized to a large extent, since these activation functions always operate on values in mainly the same order of magnitude; at least one normalizer receives output quantities of a first processing layer as input quantities, which were calculated, using a nonlinear activation function; the output quantities of this normalizer are directed as input quantities into a further processing layer, which sums these input quantities in a weighted manner in accordance with the trainable parameters; if many or even all transitions between adjacent processing layers in the ANN lead through a normalizer, then the orders of magnitude of the input quantities, which each enter into the weighted summation, may be substantially standardized within the ANN; this ensures that the training converges more effectively; ¶¶ [0081]-[0082] with FIG. 1: one option is to supply output quantities 21b of first processing layer 21 to normalizer 3 as input quantities 31, and then to supply output quantities 35 of the normalizer to second processing layer 22 as input quantities 22a; the processing proceeding in second processing layer 22, including a second option for integrating normalizer(s) 3, is schematically represented inside of box 22. Input quantities 22a are initially summed in accordance with trainable parameters 20 of ANN 1 to form one or more weighted sums, which is indicated by the summation sign; the result is supplied to normalizer 3 as input quantities 31; output quantities 35 of normalizer 3 are converted by a nonlinear activation function (in FIG. 1, indicated as an ReLU function) to output quantities 22b of second processing layer 22; Claims 33-34: a normalizer of the at least one normalizer receives a weighted summation of input quantities of a processing layer as input quantities, and output quantities of the normalizer are directed into a nonlinear activation function to calculate output quantities of the processing layer; wherein a normalizer of the at least one normalizer receives, as input quantities, output quantities of a first processing layer, which are calculated, using a nonlinear activation function, and the output quantities of the normalizer are directed as input quantities into a further processing layer, which sums the input quantities in a weighted manner in accordance with the trainable parameters).
Claim 25
Haase-Schuetz in view of Kwok discloses all the elements as stated in Claim 16 and further discloses wherein the unit values are limited to a predetermined interval before being supplied to the inverse cumulative density function (Kwok, Section II with FIGS. 1-3 in Pages 251-252: in a collocation grid method used to approximate a function f : X → Y , a sufficiently dense set of points
x
(
i
)
i
=
1
M
is chosen from the domain X, along with their corresponding function values points
f(x
(
i
)
)
i
=
1
M
in Y , to comprise the training dataset
x
(
i
)
,
f
(
x
(
i
)
)
i
=
1
M
; in our investigations, we chose two specific
R
2
→
R
functions f ∈ {f1, f2}:
f
1
≔
exp
-
k
x
2
+
y
2
,
f
1
≔
exp
exp
-
k
x
2
+
y
2
,
where k ∈
R
; both f1 and f2 are surfaces that have distinct bell shapes that peak at the origin (0, 0), with their sharpness controlled by the parameter k; we choose the collocation grid
x
(
i
)
i
=
1
M
,
y
(
j
)
j
=
1
M
via
x
(
i
)
,
y
(
j
)
∼ U(−a, a), a ∈
R
for i, j = 1, …, M, so that X := [−a,a]×[−a,a], where a controls the width of the window over which we wish to approximate f; together, k and a control the skewness of the distribution of the values
f
(
i
,
j
)
≔
f
x
(
i
)
,
y
(
j
)
i
,
j
=
1
M
: due to the collocation grid defined and the shapes of f1 and f2, higher value of a increases the number of smaller values, and a higher value of k increases the sharpness of the peak and decreases the number of larger values; we will examine approximation performances under the settings k = 1.5 and a = 3 to provide considerable skewness; Figure 1 shows the distributions of the points
f
(
i
,
j
)
i
,
j
=
1
M
arising from the collocation grid for f ∈ {f1, f2}; define an arbitrary function F : Y →
R
as the transformation performed on the function value
f
(
i
,
j
)
to produce the corresponding standardized target value
F
(
i
,
j
)
≔
F
f
(
i
,
j
)
=
F
f
x
(
i
)
,
y
(
j
)
in our training dataset; the specific forms of F we will be using are as shown in Eqns. (3)-(6), where each representing the min-max, z-score, normalized log and empirical CDF standardizations respectively; histograms of the target values after these transformations are shown in Figures 2 and 3 for f1 and f2 respectively; Fecdf (i.e., empirical CDF standardizations) eliminates skewness completely, producing a uniform distribution of
F
(
i
,
j
)
’s in both cases; summarily, f1 and f2 define different levels of skewness in our target values; to account for potential skewness (as seen in Figure 2) as well as discontinuities in some of our target values (arising from Fecdf), we explore ANNs with larger depths by trial and error and ultimately chose to use ANNs with K = 8 hidden layers and L = 10 units with tanh activations in each layer; all ANNs were trained using a mini-batched stochastic gradient descent algorithm on a vanilla mean squared error (MSE) loss for a suitable number of epochs; let
ν
f
,
F
x
,
y
be the function that maps an ANN’s inputs to its output, where F is the standardization function performed on values of
f
(
i
,
j
)
; corresponding to the training targets,
ν
f
,
F
x
,
y
approximates
F
(
f
(
x
,
y
)
)
, and hence
f
^
x
,
y
≔
F
-
1
ν
f
,
F
x
,
y
gives the desired approximation to
f
(
x
,
y
)
; the inverse functions of the aforementioned forms of F are given as in Eqns. (7)-(10); to assess the performance of
f
^
on the test set under each type of standardization technique, it is imperative to choose a low-level measure, in which the approximation performance at larger values does not overshadow that of small values; quantities such as R2 (or its equivalent in a non-regression context) and MSE that assess the sum or average performance over all test points are dominated by larger absolute test errors (often arising from larger test values) and obscure potentially large relative inaccuracies of
f
^
at small values; for this reason, we will consider the median of relative squared errors, denoted by η(f, F), for an approximation to f that utilizes the standardization function F - the smaller the value of η, the better the approximation; the relative squared error (RSE) of an approximation
f
^
(
i
,
j
)
≔
f
^
x
(
i
)
,
y
(
j
)
to its corresponding test point
f
(
i
,
j
)
is defined as Eqn. (11); lastly, before interpreting our results, a caveat in our methodology is that there are countless other possible variations to the ANNs’ specifications that may produce better approximations (for example, by using different depths, widths, activations or optimizers); though finding optimal combinations is important, the main focus of this research is to choose adequate models while taking care to avoid obvious issues such as overfitting, and prioritize making the comparisons between models as fair and as straightforward as possible; Section V in Pages 254-255: alternatively, a separate monotonically increasing function, or ANN, with the mapping Y → [0, 1] may be trained to approximate the true CDF and used to standardize target values).
Claim 26
Haase-Schuetz in view of Kwok discloses all the elements as stated in Claim 16 and further discloses wherein the target distribution is: (i) a normal distribution, or (ii) a member of an exponential family, or (iii) a Student's t-distribution (Kwok, Section II with FIGS. 1-3 in Pages 251-252: in a collocation grid method used to approximate a function f : X → Y , a sufficiently dense set of points
x
(
i
)
i
=
1
M
is chosen from the domain X, along with their corresponding function values points
f(x
(
i
)
)
i
=
1
M
in Y , to comprise the training dataset
x
(
i
)
,
f
(
x
(
i
)
)
i
=
1
M
; in our investigations, we chose two specific
R
2
→
R
functions f ∈ {f1, f2}:
f
1
≔
exp
-
k
x
2
+
y
2
,
f
1
≔
exp
exp
-
k
x
2
+
y
2
,
where k ∈
R
; both f1 and f2 are surfaces that have distinct bell shapes that peak at the origin (0, 0), with their sharpness controlled by the parameter k; we choose the collocation grid
x
(
i
)
i
=
1
M
,
y
(
j
)
j
=
1
M
via
x
(
i
)
,
y
(
j
)
∼ U(−a, a), a ∈
R
for i, j = 1, …, M, so that X := [−a,a]×[−a,a], where a controls the width of the window over which we wish to approximate f; together, k and a control the skewness of the distribution of the values
f
(
i
,
j
)
≔
f
x
(
i
)
,
y
(
j
)
i
,
j
=
1
M
: due to the collocation grid defined and the shapes of f1 and f2, higher value of a increases the number of smaller values, and a higher value of k increases the sharpness of the peak and decreases the number of larger values; we will examine approximation performances under the settings k = 1.5 and a = 3 to provide considerable skewness; Figure 1 shows the distributions of the points
f
(
i
,
j
)
i
,
j
=
1
M
arising from the collocation grid for f ∈ {f1, f2}; define an arbitrary function F : Y →
R
as the transformation performed on the function value
f
(
i
,
j
)
to produce the corresponding standardized target value
F
(
i
,
j
)
≔
F
f
(
i
,
j
)
=
F
f
x
(
i
)
,
y
(
j
)
in our training dataset; the specific forms of F we will be using are as shown in Eqns. (3)-(6), where each representing the min-max, z-score, normalized log and empirical CDF standardizations respectively; histograms of the target values after these transformations are shown in Figures 2 and 3 for f1 and f2 respectively; Fecdf (i.e., empirical CDF standardizations) eliminates skewness completely, producing a uniform distribution of
F
(
i
,
j
)
’s in both cases; summarily, f1 and f2 define different levels of skewness in our target values; to account for potential skewness (as seen in Figure 2) as well as discontinuities in some of our target values (arising from Fecdf), we explore ANNs with larger depths by trial and error and ultimately chose to use ANNs with K = 8 hidden layers and L = 10 units with tanh activations in each layer; all ANNs were trained using a mini-batched stochastic gradient descent algorithm on a vanilla mean squared error (MSE) loss for a suitable number of epochs; let
ν
f
,
F
x
,
y
be the function that maps an ANN’s inputs to its output, where F is the standardization function performed on values of
f
(
i
,
j
)
; corresponding to the training targets,
ν
f
,
F
x
,
y
approximates
F
(
f
(
x
,
y
)
)
, and hence
f
^
x
,
y
≔
F
-
1
ν
f
,
F
x
,
y
gives the desired approximation to
f
(
x
,
y
)
; the inverse functions of the aforementioned forms of F are given as in Eqns. (7)-(10); to assess the performance of
f
^
on the test set under each type of standardization technique, it is imperative to choose a low-level measure, in which the approximation performance at larger values does not overshadow that of small values; quantities such as R2 (or its equivalent in a non-regression context) and MSE that assess the sum or average performance over all test points are dominated by larger absolute test errors (often arising from larger test values) and obscure potentially large relative inaccuracies of
f
^
at small values; for this reason, we will consider the median of relative squared errors, denoted by η(f, F), for an approximation to f that utilizes the standardization function F - the smaller the value of η, the better the approximation; the relative squared error (RSE) of an approximation
f
^
(
i
,
j
)
≔
f
^
x
(
i
)
,
y
(
j
)
to its corresponding test point
f
(
i
,
j
)
is defined as Eqn. (11); lastly, before interpreting our results, a caveat in our methodology is that there are countless other possible variations to the ANNs’ specifications that may produce better approximations (for example, by using different depths, widths, activations or optimizers); though finding optimal combinations is important, the main focus of this research is to choose adequate models while taking care to avoid obvious issues such as overfitting, and prioritize making the comparisons between models as fair and as straightforward as possible).
Claim 27
Haase-Schuetz in view of Kwok discloses all the elements as stated in Claim 16 and further discloses wherein: a control signal is ascertained from the outputs of the neural network, and a vehicle, and/or a driving assistance system, and/or a robot, and/or a system for quality control, and/or a system for monitoring areas, and/or a system for medical imaging, is controlled with the control signal (Haase-Schuetz, ¶ [0058]: a system is configured to control other technical systems on the basis of an evaluation of physical measurement data, using the ANN; the system includes at least one sensor for recording physical measurement data, the ANN described above, as well as a control unit; the control unit is configured to generate a control signal for a vehicle or another autonomous agent (such as a robot), a classification system, a system for the quality control of mass-produced products, and/or a system for medical imaging, from output quantities of the ANN; all of the above-mentioned systems profit from the fact that the ANN learns, in particular, a desired classification, regression and/or semantic segmentation more effectively than ANN' s, which rely on a batch normalization or on an ELU activation function; ¶¶ [0064] and [0067]: the fully trained ANN may be supplied, as input quantities, physical measurement data recorded by at least one sensor; these input quantities may then be processed by the trained ANN to form output quantities; a control signal for a vehicle or another autonomous agent (such as a robot), a classification system, a system for the quality control of mass-produced products, and/or a system for medical imaging, may then be generated from the output quantities; the vehicle, the classification system, the system for the quality control of mass-produced products, and/or the system for medical imaging, may ultimately be controlled by this control signal; ¶ [0088] with FIG. 4: The output quantities 12' supplied by ANN 1, 1* are processed in evaluation unit 7 to form a control signal 7a; this control signal 7a is intended for the control of a vehicle or another autonomous agent (such as a robot) 50, a classification system 60, a system 70 for the quality control of mass-produced products, and/or a system 80 for medical imaging; ¶ [0091] with FIG. 6: in step 250, a control signal 7a is generated from output quantities 12'. In step 260, one or more of systems 50, 60, 70, 80 are controlled, using control signal 7a).
Claims 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Haase-Schuetz in view of Kwok as applied to Claim 18 above, and further in view of Singh et al. ("Investigating the impact of data normalization on classification performance", Applied Soft Computing Journal 97 (2020) 105524, May 23, 2019, pp. 1-23), hereinafter Singh.
Claim 19
Haase-Schuetz in view of Kwok discloses all the elements as stated in Claim 18 and except failing to explicitly disclose wherein: a score of 1 is assigned to another respective work result that is greater than the respective work result currently being considered, and a score of 0 is assigned to another respective work result that is less than the respective work result currently being considered.
Singh teaches a system and a method relating to normalization techniques (Singh, Abstract in Page 1), wherein: a score of 1 is assigned to another respective work result that is greater than the respective work result currently being considered, and a score of 0 is assigned to another respective work result that is less than the respective work result currently being considered (Singh, Section 4.2 in Pages 6-8: the purpose of the optimization algorithms is to deal with complex problems that cannot be solved in polynomial time, probably by searching the best solution from a given set of solutions; Ant Lion Optimization (ALO) is a nature-inspired algorithm which is proposed by Mirjalili for solving optimization problems; the algorithm has good exploration and exploitation capabilities of searching for the best solutions and requires fewer parameters to tune; Antlion optimization can be applied to feature subset selection and feature weighting problems for improving classification performance and reducing learning time of machine learning algorithms; consider P number of antlions (AL) and ants (A), where the position of antlions and ants represents a potential solution in d-dimensional search space; the random walk of an ant is performed around an antlion (ALj) which is modeled as Eqn. (14), where cumsum denotes the cumulative sum, t is current iteration, T is the maximum iterations, and r(t) is a stochastic function which is defined as follows:
r
t
=
1
i
f
r
a
n
d
>
0.5
0
o
t
h
e
r
w
i
s
e
, where rand is a random number which is generated uniformly in the interval of [0, 1]; Elitism is used to sustain the best solutions in the algorithm at any iteration; the newer solutions are generated using two antlions; elite antlion obtained at t iteration and an antlion selected with roulette wheel approach; the elite antlion affects all ants at each iteration whereas roulette wheel selected antlion affects only those ants which are nearby to its positions; in this paper, we use the blend crossover operator (BLX) as a substitute of averaging operation for generating new solutions; this operator has been widely used in many evolutionary algorithms due to its success for convergence towards optimal solutions; it enhances the exploration and exploitation capabilities of the ALO algorithm as compare to an averaging operation which gives deterministic positions; it generates new positions randomly from the expanded search range of elite and selected antlion; then, the BLX operation is used as shown in Eqn. (20), where β is a positive parameter that controls the exploration and exploitation of the search space; then, a uniform random number is chosen to generate a solution as: s = rnd(y1, y2), where rnd is random number generator; in case of feature weighting, the value of s is sufficient to define the new position of the ant as Eqn. (21); but in case of feature subset selection, the conversion of continuous to binary values is necessary; for this purpose, we used S-shaped sigmoid function for obtaining discrete values which is the common choice in many wrapper-based feature selection methods; it is given as shown in
v
=
1
1
+
e
-
s
(22); the new position of the ant is used for evaluating the sigmoid function; the value of Eq. (22) is compared with a random number selected from a uniform distribution in the range of [0, 1] which produces binary value as follows:
A
m
t
=
1
v
>
r
a
n
d
0
o
t
h
e
r
w
i
s
e
; then, the fitness of newer solutions is measured and the global best solution is updated if a solution with better fitness value is obtained; it is equivalent to the consumption of prey by antlion and laying of its new trap for next prey which is defined as follows:
A
L
j
t
+
1
=
A
m
t
if
F
A
m
t
>
F
(
A
L
j
t
)
), where F is the fitness function that measures the classification accuracy using k-NN classifier).
Haase-Schuetz in view of Kwok, and Singh are analogous art because they are from the same field of endeavor, a system and a method relating to normalization techniques. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Singh to Haase-Schuetz in view of Kwok. Motivation for doing so would improve classification performance and reducing learning time of machine learning algorithm (Singh, Section 4.2 in Pages 6-8).
Claim 20
Haase-Schuetz in view of Kwok discloses all the elements as stated in Claim 18 and further discloses wherein a difference of another respective work result from the respective work result currently being considered is processed by applying a (Haase-Schuetz, ¶¶ [0009]-[0025]: the normalization function has at least two different regimes and changes between the regimes as a function of a norm of the input vector at a point and/or in a range, whose position is a function of a predefined parameter ρ; this means that input vectors, whose norm is to the left of the point and/or range (that is, is somewhat smaller), are treated differently by the normalization function from input vectors, whose norm is to the right of the point and/or range (that is, is somewhat larger); in particular, the one regime may include, e.g., during the calculation of the output vector, changing the norm of the input vector absolutely and/or relatively less markedly than provided under the other regime; one of the regimes may also include, e.g., not changing the input vector at all, but taking it on unchanged as an output vector; it has been recognized that the numerical stability of the normalization function may be improved, in particular, by changing the regime as a function of the norm of the input vector and specified parameter ρ; if, between two processing steps in the ANN, the numerical values, which represent intermediate results already present, are now combined to form vectors and these vectors are normalized, then the result of this may be that, on one hand, an interval originally present between the useful signal and its processing products, and on the other hand, noise and/or rounding errors, are leveled partially or even completely; using the change between the regimes, it may now be determined, e.g., that all of the input vectors, whose norm does not reach a certain minimum degree, are not changed or only slightly changed in their norm; if, e.g., input vectors having larger norms are simultaneously mapped to output vectors having equal or similar norms, a sufficiently large norm-like interval with regard to the output vectors, which originates from noise and/or rounding errors, still remains; this, in tum, lowers the standards regarding the statistics of the input quantities, which are supplied to the normalizer; it is not necessary to always fall back upon input quantities, which originate from different samples of input quantities supplied to the ANN; instead, the important information contained in the above-mentioned intermediate result of the ANN is preserved, if only numerical values of this intermediate result, which relate to a single sample of input quantities supplied to the ANN, are supplied to the normalizer; thus, the advantages attainable until now with the aid of batch normalization may be attained to the same extent or to a greater extent, without it being necessary for the normalization to apply to mini-batches of training data processed during the training of the ANN; consequently, the effectiveness of the normalization is also, in particular, no longer a function of the size of the mini-batches selected during the training; this, in turn, allows the size of the mini-batches to be selected completely freely, e.g., from the standpoint of the data throughput during the training of the ANN; in contrast to the batch size of the batch normalization, the parameter ρ used by the normalizing element is a continuous and indiscrete parameter; consequently, this parameter ρ is available for optimization in a markedly more effective manner; e.g., it may be trained together with the trainable parameters of the ANN; the ANN may be trained, all in all, in an efficient manner and, at the same, also becomes robust in opposition to manipulation attempts using so-called adversarial examples; in one particularly advantageous refinement, at least one normalization function is configured to leave input vectors, whose norm is less than parameter ρ, unchanged, and to normalize input vectors, whose norm is greater than parameter ρ, to a uniform norm, while maintaining the direction; one example of such a normalization function, which is clarified for vectors in an arbitrary multidimensional space, includes:
π
^
ρ
(
x
→
)
=
x
→
max
1
,
x
→
ρ
; if the norm
x
→
of vector
x
→
is less than ρ, then vector
x
→
remains unchanged; this is the first regime of the normalization function
π
^
ρ
(
x
→
)
; however, if
x
→
is at least equal to ρ, then
π
^
ρ
(
x
→
)
projects vector
x
→
onto a spherical surface having radius ρ; this means that the normalized vector then points in the same direction as before, but ends on the spherical surface; this is the second regime of normalization function
π
^
ρ
(
x
→
)
; when
x
→
= ρ, then a change is made between the two regimes; in a further, particularly advantageous refinement, the change of at least one normalization function between the different regimes is controlled by a softplus function, whose argument has a zero crossing when the norm of the input vector is equal to parameter ρ; an example of such a function is
π
^
ρ
(
x
→
)
=
x
→
1+softplus
x
→
-
ρ
ρ
; in this, the softplus function is given by
s
o
f
t
p
l
u
s
y
=
ln
1
+
e
x
p
y
; the advantage of this function is that it is differentiable in ρ; now, vectors
x
→
having
x
→
less than ρ no longer remain unchanged, but in comparison with vectors
x
→
having a larger norm
x
→
, they are changed markedly less; when
x
→
tends to 0, then norm
x
→
of the vector x in the multidimensional space is reduced by approximately 25% independently of the value of ρ; there is no norm
x
→
, for which
π
^
ρ
(
x
→
)
results in an increase of the norm; thus, not only is the influence of, e.g., rounding errors and noise prevented from being increased, but also this influence is reduced even further, in that norms
x
→
that are overly low are lowered more and are simply not raised to a uniform level; ¶¶ [0055], [0061], and [0066]: the input vector(s) are normalized with the aid of a normalization function to form one or more output vectors; this normalization function having at least two different regimes and changing between the regimes as a function of a norm of the input vector at a point and/or in a range, whose position is a function of a predefined parameter ρ; the trainable parameters of the ANN are optimized together with at least one parameter ρ described above, which characterizes the transition between the two regimes of a normalization function; during the further processing of input learning quantities, the objective of this optimization is to obtain output quantities, whose evaluation by the cost function is expected to be more effective; ¶¶ [0080]-[0083] with FIG. 1: two exemplary options of how a normalizer 3 may be introduced into ANN 1, are drawn into FIG. 1; one option is to supply output quantities 21b of first processing layer 21 to normalizer 3 as input quantities 31, and then to supply output quantities 35 of the normalizer to second processing layer 22 as input quantities 22a; the processing proceeding in second processing layer 22, including a second option for integrating normalizer(s) 3, is schematically represented inside of box 22; input quantities 22a are initially summed in accordance with trainable parameters 20 of ANN 1 to form one or more weighted sums, which is indicated by the summation sign; the result is supplied to normalizer 3 as input quantities 31; output quantities 35 of normalizer 3 are converted by a nonlinear activation function (in FIG. 1, indicated as an ReLU function) to output quantities 22b of second processing layer 22; a plurality of different normalizers 3 may be used within one and the same ANN 1; each normalizer 3 may then have, in particular, its own parameters ρ for the transition between the regimes of its normalization function 33; in addition, each normalizer 3 may also be coupled to its own specific preprocessing element; ¶¶ [0084]-[0085] with FIG. 2: normalizer 3 transforms its input quantities 31 into one or more input vectors 32, using a transformation element 3a, which implements a predefined transformation 3a'; these input vectors 32 are supplied to normalization element 3b, and there, they are normalized to form output vectors 34; output vectors 34 are transformed in inverse transformation element 3c in accordance with inverse 32a" of predefined transformation 3a', into output quantities 35 of normalizer 3, which have the same dimensionality as input quantities 31 of normalizer 3; how the normalization of input vectors 32 proceeds to form output vectors 34, is shown in detail inside of box 3b; the normalization function 33 utilized includes two regimes 33a and 33b, in each of which it shows a qualitatively different behavior and acts, in particular, with a different intensity upon input vectors 32; in interaction with at least one predefined parameter ρ, norm 32a of respective input vector 32 decides, which of regimes 33a and 33b is used; for purposes of illustration, this is represented as a binary decision in FIG. 2; in reality, however, it is particularly advantageous for regimes 33a and 33b to merge in a fluid manner, in particular, in a manner that is differentiable in parameter ρ) (Kwok, Section II with FIGS. 1-3 in Pages 251-252: in a collocation grid method used to approximate a function f : X → Y , a sufficiently dense set of points
x
(
i
)
i
=
1
M
is chosen from the domain X, along with their corresponding function values points
f(x
(
i
)
)
i
=
1
M
in Y , to comprise the training dataset
x
(
i
)
,
f
(
x
(
i
)
)
i
=
1
M
; in our investigations, we chose two specific
R
2
→
R
functions f ∈ {f1, f2}:
f
1
≔
exp
-
k
x
2
+
y
2
,
f
1
≔
exp
exp
-
k
x
2
+
y
2
,
where k ∈
R
; both f1 and f2 are surfaces that have distinct bell shapes that peak at the origin (0, 0), with their sharpness controlled by the parameter k; we choose the collocation grid
x
(
i
)
i
=
1
M
,
y
(
j
)
j
=
1
M
via
x
(
i
)
,
y
(
j
)
∼ U(−a, a), a ∈
R
for i, j = 1, …, M, so that X := [−a,a]×[−a,a], where a controls the width of the window over which we wish to approximate f; together, k and a control the skewness of the distribution of the values
f
(
i
,
j
)
≔
f
x
(
i
)
,
y
(
j
)
i
,
j
=
1
M
: due to the collocation grid defined and the shapes of f1 and f2, higher value of a increases the number of smaller values, and a higher value of k increases the sharpness of the peak and decreases the number of larger values; we will examine approximation performances under the settings k = 1.5 and a = 3 to provide considerable skewness; Figure 1 shows the distributions of the points
f
(
i
,
j
)
i
,
j
=
1
M
arising from the collocation grid for f ∈ {f1, f2}; define an arbitrary function F : Y →
R
as the transformation performed on the function value
f
(
i
,
j
)
to produce the corresponding standardized target value
F
(
i
,
j
)
≔
F
f
(
i
,
j
)
=
F
f
x
(
i
)
,
y
(
j
)
in our training dataset; the specific forms of F we will be using are as shown in Eqns. (3)-(6), where each representing the min-max, z-score, normalized log and empirical CDF standardizations respectively; histograms of the target values after these transformations are shown in Figures 2 and 3 for f1 and f2 respectively; Fecdf (i.e., empirical CDF standardizations) eliminates skewness completely, producing a uniform distribution of
F
(
i
,
j
)
’s in both cases; summarily, f1 and f2 define different levels of skewness in our target values; to account for potential skewness (as seen in Figure 2) as well as discontinuities in some of our target values (arising from Fecdf), we explore ANNs with larger depths by trial and error and ultimately chose to use ANNs with K = 8 hidden layers and L = 10 units with tanh activations in each layer; all ANNs were trained using a mini-batched stochastic gradient descent algorithm on a vanilla mean squared error (MSE) loss for a suitable number of epochs; let
ν
f
,
F
x
,
y
be the function that maps an ANN’s inputs to its output, where F is the standardization function performed on values of
f
(
i
,
j
)
; corresponding to the training targets,
ν
f
,
F
x
,
y
approximates
F
(
f
(
x
,
y
)
)
, and hence
f
^
x
,
y
≔
F
-
1
ν
f
,
F
x
,
y
gives the desired approximation to
f
(
x
,
y
)
; the inverse functions of the aforementioned forms of F are given as in Eqns. (7)-(10); to assess the performance of
f
^
on the test set under each type of standardization technique, it is imperative to choose a low-level measure, in which the approximation performance at larger values does not overshadow that of small values; quantities such as R2 (or its equivalent in a non-regression context) and MSE that assess the sum or average performance over all test points are dominated by larger absolute test errors (often arising from larger test values) and obscure potentially large relative inaccuracies of
f
^
at small values; for this reason, we will consider the median of relative squared errors, denoted by η(f, F), for an approximation to f that utilizes the standardization function F - the smaller the value of η, the better the approximation; the relative squared error (RSE) of an approximation
f
^
(
i
,
j
)
≔
f
^
x
(
i
)
,
y
(
j
)
to its corresponding test point
f
(
i
,
j
)
is defined as Eqn. (11); lastly, before interpreting our results, a caveat in our methodology is that there are countless other possible variations to the ANNs’ specifications that may produce better approximations (for example, by using different depths, widths, activations or optimizers); though finding optimal combinations is important, the main focus of this research is to choose adequate models while taking care to avoid obvious issues such as overfitting, and prioritize making the comparisons between models as fair and as straightforward as possible; Section V in Pages 254-255: alternatively, a separate monotonically increasing function, or ANN, with the mapping Y → [0, 1] may be trained to approximate the true CDF and used to standardize target values).
Haase-Schuetz in view of Kwok fails to explicitly discloses applying a sigmoid function to a score of the other respective work result.
Singh teaches a system and a method relating to normalization techniques (Singh, Abstract in Page 1), wherein applying a sigmoid function to a score of the other respective work result (Singh, Abstract in Page 1: this study aims to investigate the impact of fourteen data normalization methods on classification performance considering full feature set, feature selection, and feature weighting; also present a modified Ant Lion optimization that search feature subsets and the best feature weights along with the parameter of Nearest Neighbor Classifier; experiments are performed on 21 publicly available real and synthetic datasets, and results are analyzed based on the accuracy, the percentage of feature reduced and runtime; it has been observed from the results that no single method outperforms others; therefore, we have suggested a set of the best and the worst methods combining the normalization procedure and empirical analysis of results; the better performers are z-Score and Pareto Scaling for the full feature set and feature selection, and tanh and its variant for feature weighting; the worst performers are Mean Centered, Variable Stability Scaling and Median and Median Absolute Deviation methods along with un-normalized data; Section 3 in Pages 3-5: 3.6. Sigmoidal Normalization: it is a non-linear transformation for reducing the effects of the outlier in the data; two sigmoidal functions are widely used in literature for normalizing the data which are given as follows: 3.6.1. Logistic Sigmoid (LS) function based normalization: the transformation of data is based on the logistic sigmoid function which maps data within the range of 0 to 1; it squashes larger values present at the tails of the data while rest of the data is mapped linearly; this non-linear normalization method is used when data is not evenly distributed around its mean; the outliers which lie away from the mean of the data are squashed exponentially; 3.6.2. Hyperbolic Tangent (HT) function based normalization: the features are normalized within the range of −1 to 1 using hyperbolic tangent function; this normalization is suitable for scaling the outliers without affecting the rest of the data; it scales the normal data linearly, thus prevents the data being pushed into narrow range due to the presence of outliers; this normalization preserves the significance of the normal data while dealing with the outliers; the values that lie within a standard deviation of the mean of the feature are mapped to almost linear region whereas outliers, as well as extreme values, are mapped along the tails of the range by the given function).
Haase-Schuetz in view of Kwok, and Singh are analogous art because they are from the same field of endeavor, a system and a method relating to normalization techniques. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Singh to Haase-Schuetz in view of Kwok. Motivation for doing so would provide better performance when handling outliers (Singh, Section 3 in Pages 3-5).
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Haase-Schuetz in view of Kwok as applied to Claim 21 above, and further in view of Sather et al. (US 2021/0034955 A1, pub. date: 02/04/2021), hereinafter Sather.
Claim 22
Haase-Schuetz in view of Kwok discloses all the elements as stated in Claim 21 and except failing to explicitly disclose wherein the differentiable approximation is parameterized with a temperature parameter via which a compromise between an accuracy of the differentiable approximation on the one hand and a speed at which the differentiable approximation saturates, on the other hand can be set.
Sather teaches a system and a method relating to neural networks (Sather, Abstract), wherein the differentiable approximation is parameterized with a temperature parameter via which a compromise between an accuracy of the differentiable approximation on the one hand and a speed at which the differentiable approximation saturates, on the other hand can be set (Sather, ¶¶ [0019]-[0032] with FIG. 1: to train the quantized neural network, scale a feature map of a pre-trained floating-point neural network in order to match the range of output values provided by quantized activation functions in a quantized neural network; scale the feature map by determining a distribution of values for each input of a layer of the neural network and selecting a set of scaling and shift values for each layer based on the determined distribution of values; perform an initial scaling that preserves the feature maps computed by the pre-trained neural network using affine transformations that are constrained based on the type of computation performed by a computation node; during training, the learned affine transformations will be able to update the initial scales in order to improve accuracy and adapt to value and weight quantization; a feedforward neural network 100 that has multiple layers of processing nodes, called neurons 102; in all but the first (input) and last (output) layer, each neuron 102 receives two or more outputs of neurons from earlier processing node layers and provides its output to one or more neurons in subsequent layers; the output of the neuron in the last layer represents the output of the network 100; each neuron in neural network 100 has a linear component 110, a nonlinear component 115, and an output transformation component 140; the linear component 110 of each hidden or output neuron in this example computes a dot product of a vector of weight coefficients and a vector of output values of prior nodes, plus an offset; in other words, a hidden or output node's linear operator computes a weighted sum of its inputs (which are outputs of the previous stage neurons that the linear operator receives) plus an offset; the linear component 110 includes a scaling and shifting operation on the inputs (e.g., a fanin component) to apply an affine transformation to each input individually or as a group; each neuron's nonlinear component 115 computes a function based on the output of the neuron's linear component 110; this function is commonly referred to as the activation function; the output transformation component 140 applies a second affine transformation to the result of the nonlinear component 115 (e.g., as a fanout component); the second affine transformation is used to mitigate the effect of the first affine transformation performed by the linear component 110; examples of activation functions include a sigmoid function 120 (f(x)=1/(1+e-x)), a tanh function 125, a ReLU (rectified linear unit) function 130 or a leaky ReLU function 135; once the nonlinear component 115 computes y(l+1), the output transformation component 140 performs an affine transformation (i.e., a scaling and shifting operation) to produce a modified output vector that generally takes the form of Equation (4); where g(yi(l+1)) may be an affine transformation that is applied to all components of the output vector yi(l+1) or a set of affine transformations applied to each component (channel) of the output vector yi(l+1) individually (e.g., different affine transformations for each component); ¶¶ [0038]-[0098] with FIG. 2: the set of scaling operations is designed to set the scale and shift for each node of the network so as to minimize the loss of information due to value quantization; while most nodes can absorb a scale (and shift) of their input, some nodes have a limited ability to do this, and this constrains the ability to optimize the scale/shift of their fanin components (i.e., a component of a network node that accepts inputs from previous network nodes and performs a first scaling and/or shifting operation on each input before further processing by the node); given a distribution of values to quantize, a scale and shift of the optimal quantization can be selected by alternately minimizing the sum of squared quantization errors over the scale, shift, and bin assignment; however, many commonly used activation functions are not homogeneous: sigmoid, tanh, ELU, Swish, etc.; in such situations, possible solutions include: (1) determining the affine transformation that best approximates the change in the output distribution, (2) choosing the look up table (LUT) program that best approximates the portion of the activation that is actually used by the floating-point network, and (3) skip the scaling (and shifting) if the initial quantization is good enough; attempt to minimize the gradient variance of the approximate quantization function while simultaneously minimizing the distortion (measured as [y(x)-Q(x)]2, where y(x) is the approximate quantization function and Q(x) is the quantization function) of the values produced by the approximate quantization function; for y(x) to minimize Lon this interval it must satisfy the Euler-Lagrange differential equation (8); if a new variable,
β
=
λ
, is introduced, the solutions to the Euler-Lagrange Equation (8) satisfy the differential equation β2(y-qi)= (y-qi)" which has the general solution in Equation (9); in Equation (9), A, and B, are selected to satisfy the boundary conditions y(xi)=yi; select the form of each exponent so that its maximum value on the interval [xi, xi+1] is 0 to avoid numerical overflow and minimize underflow; the distortion for each interval [xi, xi+1] is given by Equation (20); in the limit of low temperature T=1/β, Equation (20) is approximated by Equation (21); the average distortion over the entire range of [xmin, xmax] is given by Equation (22); the approximate quantization function satisfies the Helmholtz differential equation (23); the above Equation (23) can be solved for y(x) by finding the Green's function G(x, x0) for the Helmholtz equation which satisfies Equation (24) with boundary conditions given in Equation (25); the required Green's function is:
G
x
,
x
0
=
-
1
2
β
e
x
p
(
-
β
x
-
x
0
)
); the approximate quantization function is then computed in terms of the Green's function as in Equations (27)-(29); the form of the approximate quantization function derived using a Green's function can be interpreted as the expectation of quantized noisy input in Equation (30); the input noise is additive, zero-mean Laplacian noise with diversity equal to the temperature b=T=l/β; this perspective suggests that the variance of the quantized output due to additive input noise should also be taken into account; the variance of the quantized output is defined as Equations (31)-(33); given the analytic form of both the mean y(x) and variance s(x)-y2(x) random values from this distribution are sampled during training and used to learn the temperature of the noise, T; given the analytic expressions for y,(x) and v(x) a reparameterization trick is used to compute derivatives with respect to network parameters including the temperature T; given that the differentiable quantization function produces continuous values, take steps to ensure that training does not exploit this increased expressivity; one solution would be to require T→0 during the optimization; in this limit the gradient becomes a sequence of Dirac delta functions, making optimization difficult; training is not effective for T ≤1/20; an alternative is to add noise to the optimization function that is at least as large as the approximation error so that it cannot be exploited; with this approach the increased expressivity introduced by using a continuous approximation is removed; if T and σ are hyperparameters some embodiments ensure
σ
2
≥
D
-
(
T
)
when they are set; if they are learned parameters per channel (as discussed below) then this constraint is enforced using a Lagrangian; a violation of this constraint can be resolved by increasing σ or by decreasing T (i.e., compromising between error/accuracy and time/speed); some embodiments use Bayesian optimization and hyperband for the hyperparameter optimization; some embodiments use a bilevel optimization approach; in self-tuning networks, the learned parameters, including the learned variance, become linear functions of the hyperparameters; the linear response for the log variance can be parameterized as Equation (48); in Equation (48), the λi are the hyperparameters, of which the VIB coefficients and the temperature T are of particular importance; in some embodiments, Bayesian optimization and hyperband is used in addition to self-tuning networks to learn hyperparameters that are not as easily learned using self-tuning networks).
Haase-Schuetz in view of Kwok, and Sather are analogous art because they are from the same field of endeavor, a system and a method relating to neural networks. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Sather to Haase-Schuetz in view of Kwok. Motivation for doing so would improve accuracy and making optimization easier (Sather, ¶¶ [0020] and [0082]-[0083]).
Claim 23 is rejected under 35 U.S.C. 103 as being unpatentable over Haase-Schuetz in view of Kwok and Sather as applied to Claim 22 above, and further in view of Chai et al. (US 2020/0134461 A1, pub. date: 04/30/2020), hereinafter Chai.
Claim 23
Haase-Schuetz in view of Kwok and Sather discloses all the elements as stated in Claim 22 and further discloses and further discloses in ¶¶ [0006], [0042], and [0046] that the modified (e.g., approximate) quantization function is a differentiable approximation to a stepwise quantization function that is used in a backpropagation operation during training to adjust the weights of the received neural network, wherein relaxed quantization is used to generate a differentiable quantization function.
Haase-Schuetz in view of Kwok and Sather fails to explicitly disclose wherein the temperature parameter is varied during training of the neural network according to a predetermined annealing plan.
Chai teaches a system and a method relating to neural network (Chai, Abstract), wherein the temperature parameter is varied during training of the neural network according to a predetermined annealing plan (Chai, ¶¶ [0045]-[0049]: it may be desirable for computer architects to look into new ways to make DNN processing more amenable for use on mobile platforms with more constraints to speed or energy consumption; ¶¶ [0050]-[0084] with FIG. 1: train a DNN such that the bit precision is part of the parameters in the training procedure; current training methods consist of model selection ( e.g., learning rate, network size, network depth) and a training phase (e.g., using a backpropagation algorithm to set the weights of the network over many iterations); how to include the optimization of bit precision as part of the training phase; note that l(W) is a convex and differentiable relaxation of the negative log likelihood with respect to the quantized parameters; it is clear that l(W) is an upper bound on l(W) which is the Lagrangian corresponding to constraints of small quantization error using a small number of bits; the uniformly spaced quantization of Equation (14) allows this simple functional form for the number of unique values; ¶ [0127]: a flexible tool for training compact DNN s given an indirect specification of the total number of bits available on the target device; a formulation that incorporates such constraints as a regularization on the traditional classification loss function; this formulation is based on controlling the expressive power of the network by dynamically quantizing the range and set of values that the parameters can take; BitNet may have an indirect relationship to the global learning rate; BitNet can be interpreted as having a dynamic learning rate per parameter that depends on the number of bits; in that sense, bit regularization is related to dynamic learning rate schedulers such as AdaGrad; machine learning system 104 may anneal the constraints to leverage fast initial learning combined with high precision fine tuning; in that sense, BitNet can be interpreted as having a joint-optimization to learn a representation of the training data and system architecture; ¶¶ [0137]-[0147] with FIGS.1 and 12: in reference to machine learning system 104, in a step to anneal the constraints, selected weights are selected based on a joint optimization of DNN performance and hardware resources; in reference to the loss function selection the lambda parameters (λ1, λ2 and λ3) in equation (34) based in hardware parameters P, hardware parameters P may be updated during training to affect the selection of the annealing constraints; the selection of hardware parameters P during training allows for a pareto-optimal selection of hardware resources with respect to size, weight, and power, e.g.; such approach in dynamic selection of hardware parameters serves as a design tradeoff method for a designer, using the loss function as a joint-optimization of machine learning performance and hardware resource; this disclosure describes a loss function that uses a correspondence between the bit-precision and hardware parameters, with respect to equation (34) and in reference to r(b, P) function; it is understood that the correspondence can be a similar mapping of machine learning hyper parameters (e.g. bit precision, learning rate, DNN depth, fusion layers, activation, learning algorithm) to hardware parameters, to enable a joint-optimization; it is also understood then that the correspondence can be other similar mapping of machine learning parameters to other physical constraints (e.g. environmental, data biases) that influence the DNN training, wherein a joint-optimization can be performed similarly, as in equation (34)).
Haase-Schuetz in view of Kwok and Sather and Chai are analogous art because they are from the same field of endeavor, a system and a method relating to neural network. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Chai to Haase-Schuetz in view of Kwok and Sather. Motivation for doing so would anneal the constraints to leverage fast initial learning combined with high precision fine tuning (.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HWEI-MIN LU whose telephone number is (313)446-4913. The examiner can normally be reached Mon - Fri: 9:00 AM - 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela D. Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HWEI-MIN LU/Primary Examiner, Art Unit 2142
1 See, for example US 2021/0034955 A1 to Sather et al., pub. date on 04/04/2021, ¶¶ [0021]-[0022] with FIG. 1.
Prosecution Insights