DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. KR10-2021-0158515, filed on 11/17/2021 and No. KR10-2020-0188326, filed on 12/30/2020.
Status of Claims
Claims 1-2, 4-6, 8-9, 11-13 were amended. Claims 3, 10 are canceled. Claims 14-15 are new. Claims 1-2, 4-9, 11-15 are pending and examined herein.
Claims 2, 4, 5, 9, 11, 12, 14 are rejected under 35 U.S.C. 112(b).
Claims 1-2, 4-9, 11-15 are rejected under 35 U.S.C. 101.
Claims 1-2, 4-9, 11-15 are rejected under 35 U.S.C. 103.
Response to Arguments
Applicant’s arguments filed February 18th, 2026 have been fully considered but are not persuasive. The amendments filed with the RCE have been entered and considered. To the extent applicant repeats arguments previously presented after final, those arguments remain unpersuasive for the reasons previously stated in the Advisory Action and for the reasons below.
Regarding the 101 rejection, the amendments do not overcome the rejection. The amended claims further specify the mathematical normalization performed on embedding vectors, including applying a same scale parameter value and a same shift parameter value to elements of the embedding vector. These limitations describe the mathematical operation used in the neural network model, but do not recite a separate technological improvement beyond the mathematical processing itself. The generic electronic device, processor, memory, embedding layer, normalization layer, and neural network layer merely provide the environment for performing the recited mathematical operations.
Regarding the 103 rejection, applicant’s arguments against the combination of Wang and Nguyen are not persuasive. Wang is relied upon for teaching normalization in a CTR neural network environment involving feature embeddings and normalization parameters, while Nguyen is relied upon for teaching use of shared normalization parameterization instead of independent per dimension parameterization. The rejection is based on the combined teachings of the references. It would have been obvious to modify Wang’s normalization parameter values in view of Nguyen’s shared parameter teaching to reduce unnecessary independent parameterization while retaining Wang’s CTR feature embedding normalization framework.
Accordingly, the rejections are maintained.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2, 4, 5, 9, 11, 12, 14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 2, 4, 5, 9, 11, and 12 recites the limitation “the scale parameter”, “the shift parameter.” However, independent claims 1,8 as amended recite “a same scale parameter value” and “a same shift parameter value.” Other than these limitations, independent claims do not introduce any “scale parameter” nor “shift parameter”. There is insufficient antecedent basis for these limitations in the claim. Appropriate correction is required for consistency and antecedent basis of these claims.
Claim 14 recites the equation comprising “Bx” and “γx”. However, the claim does not define variables “Bx” and “γx”. It is unclear whether “Bx” and “γx” correspond to scale and shift parameters, scalar values, vector values, or some other values used in the normalization equation. For the purpose of examination, “Bx” and “γx” are interpreted as corresponding to the scale parameter and shift parameter recited in claim 1.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-2, 4-9, 11-15 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
MPEP § 2109(III) sets out steps for evaluating whether a claim is drawn to patent-eligible subject matter. The analysis of claims 1-2, 4-9, 11-15, in accordance with these steps, follows.
Step 1 Analysis:
Step 1 is to determine whether the claim is directed to a statutory category (process, machine, manufacture, or composition of matter.
Claims 1-2, 4 – 6, 14-15 are directed to a method, meaning that it is directed to the statutory category of process. Claim 7 is directed to a non-transitory, computer-readable recording medium in combination with hardware, which is the statutory category of manufacture. Claims 8-9, 11- 13 are directed to a neural network system, which can be the statutory category of machine.
Step 2A Prong One, Step 2A Prong Two, and Step 2B Analysis:
Step 2A Prong One asks if the claim recites a judicial exception (abstract idea, law of nature, or natural phenomenon). If the claim recites a judicial exception, analysis proceeds to Step 2A Prong Two, which asks if the claim recites additional elements that integrate the abstract idea into a practical application. If the claim does not integrate the judicial exception, analysis proceeds to Step 2B, which asks if the claim amounts to significantly more than the judicial exception. If the claim does not amount to significantly more than the judicial exception, the claim is not eligible subject matter under 35 U.S.C. 101.
Regarding claim 1, the following claim elements are abstract ideas:
mapping a plurality of features included in a feature vector to a plurality of embedding vectors; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components. In addition, mapping features to vector representations fall under a mathematical concept.)
normalizing each embedding vector of the plurality of embedding vectors on the basis of applying a same scale parameter value and a same shift parameter value to all elements of the respective embedding vector; (The normalizing vector on basis of a feature-wise linear transformation is merely mathematical calculation, which is mathematical concept. The amended same scale and same shift language merely defines how the mathematical normalization is parameterized.)
predicting a click-through rate (CTR) of a user, in an electronic device, (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.)
The following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
and inputting the normalized embedding vectors into a neural network layer; and (This is mere data retrieval or data transmission, an insignificant extra solution activity, which is a well-understood, routine conventional activity. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(d). Therefore, this does not amount to significantly more than the judicial exception.)
at least in part based on the normalizing; (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 2, the rejection of claim 1 is incorporated herein. Further, claim 2 recites the following abstract ideas:
calculating a mean of the elements of the respective embedding vector; (Calculating a mean recites a mathematical calculation, which is mathematical concept.)
calculating a variance of the elements of the respective embedding vector; (Calculating a variance recites a mathematical calculation, which is mathematical concept.)
and normalizing the respective embedding vector on the basis of the mean, the variance, the scale parameter and the shift parameter. (Normalizing with basis on the mean, the variance, and the parameter recites a mathematical formula, which is mathematical concept.)
Claim 2 does not recite additional elements.
Regarding claim 4, the rejection of claim 1 is incorporated herein. Further, claim 4 recites the following abstract ideas:
each of the scale parameter and the shift parameter is a vector having the same dimension as the respective embedding vector, and all elements thereof have the same value. (Specifying the scale and shift parameter to be a vector with same dimension and containing same values is merely mathematical relationship, which is mathematical concept. )
Claim 4 does not recite additional elements.
Regarding claim 5, the rejection of claim 1 is incorporated herein. Further, claim 5 recites the following abstract ideas:
wherein each of the scale parameter and the shift parameter has a scalar value. (Specifying the scale and shift parameter to be a scalar value is merely mathematical relationship, which is mathematical concept.)
Claim 5 does not recite additional elements.
Regarding claim 6, the rejection of claim 1 is incorporated herein. Further, claim 6 recites the following abstract ideas:
wherein the normalizing is an operation of performing calculation of Equation 1 below: [Equation 1]
PNG
media_image1.png
114
320
media_image1.png
Greyscale
, wherein, in Equation 1, ex is the respective embedding vector, d is a dimension of the respective embedding vector, μx is the mean of all the elements of the respective embedding vector, σx2 is the variance of all the elements of the respective embedding vector, (ex)k is a kth element of the respective embedding vector ex, γ is the scale parameter, and β is the shift parameter. (Claim is merely reciting mathematical equations for normalization, which is mathematical concept.)
Claim 6 does not recite additional elements.
Regarding claim 7, the rejection of claim 1 is incorporated herein. Further, claim 7 recites the following additional element:
A non-transitory computer-readable recording medium to execute the method of claim 1 in combination with hardware(It is merely directing to a computer program product to train neural network model, which is a well-understood, routine conventional activity. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(d). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
Claims 8-9, 11-13 recite substantially similar subject matter to claims 1-2,4- 6 respectively and are rejected with the same rationale, mutatis mutandis.
Regarding claim 14, the rejection of claim 1 is incorporated herein. Further, claim 14 recites the following abstract idea:
PNG
media_image2.png
77
280
media_image2.png
Greyscale
where ex denotes an embedding vector corresponding to a feature x included in the feature vector, 6x is a normalized embedding corresponding to the feature x, Q refers to an element- wise multiplication operation, px and 6x2 are a mean and a variance of elements of the embedding vector ex, respectively, and s is a scalar value added to the variance to prevent overflow when the normalization is performed. (Claim is merely reciting mathematical equations for normalization, which is mathematical concept.)
Claim 14 does not recite additional elements.
Regarding claim 15, the rejection of claim 1 is incorporated herein. Further, claim 15 recites the following additional elements:
each feature of the feature vector includes an element for a user and an element for an item, (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
the element for the user indicates an age of the user, a gender of the user, an access time of an online platform accessed by the user, a click log in the platform, or a usage history of the platform, and (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
the element for the item includes a type of content in the platform, a feature of the content, or an arrangement region of the content. (This falls under mere instructions to apply an exception. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception.)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 4, 5, 7, 8, 11, 12 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (NPL: “Correct Normalization Matters: Understanding the Effect of Normalization On Deep Neural Network Models For Click-Through Rate Prediction”) in view of Klecki (NPL: “Normalize operator”), further in view of Nguyen et al. (NPL: ”Transformers without Tears: Improving the Normalization of Self-Attention”) .
Regarding Claim 1, Wang teaches
mapping a plurality of features included in a feature vector to a plurality of embedding vectors; (Abstract of Wang states “Normalization has become one of the most fundamental components in many deep neural networks for machine learning tasks while deep neural network has also been widely used in CTR estimation field… In this paper, we conduct a systematic study on the effect of widely used normalization schemas by applying the various normalization approaches to both feature embedding and MLP part in DNN model.” Pg. 3 of Wang states “The result of embedding layer is a wide concatenated vector: Vemb = concat(e1, e2, ..., ei , ..., ef ) (4) where f denotes the number of fields, and ei ∈ Rk denotes the embedding of one field… The first one is to quantize each numerical feature into discrete buckets, and the feature is then represented by the bucket ID. We can map bucket ID to an embedding vector. The second method maps the feature field into an embedding vector as follows”)
normalizing each embedding vector of the embedding vector… (Equation 1 under 3.1 Variance-Only Layer Norm section shows general formulation of LayerNorm, which normalize vector with bias b and gain g parameters like a feature-wise linear transformation.
PNG
media_image3.png
73
259
media_image3.png
Greyscale
3.2.1 Normalization on Feature Embedding section states “We apply normalization on feature embedding based on the feature field as follows:
PNG
media_image4.png
25
300
media_image4.png
Greyscale
where N can be BatchNorm, LayerNorm, GroupNorm, SimpleLayerNorm or variance-only LayerNorm. The bias and gain are shared for features in same feature field if the normalization contains these parameters”)
and inputting the normalized embedding vectors into a neural network layer; (pg. 3 of Wang states “Most DNN ranking models use the feature embedding to represent information and shallow MLP layers to model high-order interactions in an implicit way.” pg. 5 of Wang states “we empirically evaluate the effect of various normalization approaches on deep neural networks on three real-world datasets” and Table2. shows effect of Differently Normalized embedding vectors applied into DNN (deep neural network).)
predicting a click-through rate (CTR) of a user, in an electronic device, at least in part based on the normalizing (Pg. 1 of Wang states “Normalization has become one of the most fundamental components in many deep neural networks for machine learning tasks while deep neural network has also been widely used in CTR estimation field.” And “In this paper, we conduct a systematic study on the effect of widely used normalization schemas by applying the various normalization approaches to both feature embedding and MLP part in DNN model. Extensive experiments are conduct on three real-world datasets and the experiment results demonstrate that the correct normalization significantly enhances model’s performance.” Pg. 5 of Wang states using Criteo or Avazu dataset which are widely used for CTR model evaluation that corresponds to click through rate data of users. )
Wang does not explicitly teach that
applying a same scale parameter value and a same shift parameter value to all elements of the respective embedding vector.
However, Lecki and Nguyen teaches that
applying a same scale parameter value and a same shift parameter value to all elements of the respective embedding vector. (Pg. 4-5 of Klecki states “For this purpose, Normalize offers two scalar arguments: shift and scale . Now the normalization formula becomes: 𝑌𝑖 = (𝑋𝑖 – 𝜇)/𝜎⋅ 𝑠𝑐𝑎𝑙𝑒 + 𝑠ℎ𝑖𝑓𝑡 ” In the formula, Xi and Yi are indexed by I, meaning the operation is applied element by element, but scale and shift are not indexed. Therefore, the same scalar scale value and the same scalar shift value are applied to each element Xi being normalized. Pg. 3 of Nguyen states “SCALENORM replaces the 2d scale and shift parameters of LAYERNORM with a single learned scalar, improving computational and parameter efficiency while potentially regularizing the loss landscape. This bias has an explicit interpretation at the final layer: large inner products sharpen the output distribution, causing frequent words to disproportionately dominate rare words. This led Nguyen and Chiang (2018) to introduce FIXNORM(w) =g*w/||w|| with fixed g at the last linear layer, to maximize the angular difference of output representations and aid rare word translation.” Nguyen teaches that neural network normalization parameters may be simplified from per dimension scale/shift parameters to a single learned scalar for efficiency. Therefore, POSITA would have been motivated to configure Wang’s feature embedding normalization using Klecki’s scalar scale and scalar shift values, so that the same scale parameter value and the same shift parameter value are applied across all elements of each respective embedding vector.)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Wang, Klecki, and Nguyen. Wang teaches CTR prediction using deep neural networks with feature embeddings, and applying normalization to feature embeddings, where they use bias and gain parameters (commonly referred for scale and shift parameters in normalization) in same dimension as input. Lecki teaches a normalization operation using scalar scale and scalar shift arguments in which the same unindexed scale and shift values are applied in the element wise formula. Nguyen teaches that neural network normalization parameters may be simplified from per dimension LayerNorm scale/shift parameters to a single learned scalar to improve computational and parameter efficiency. One with ordinary skill in the art would be motivated to incorporate the teachings of Nguyen and Klecki into that of Wang because using a single learned scalar in place can reduce the number of learned parameters and computational overhead while retraining the benefits of normalization and potentially improving training behavior. The combination would have predictably normalized Wang’s embedding vectors while applying the same scalar scale value and the same scalar shift value to the elements of each respective embedding vector.
Regarding Claim 4, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Wang, Nguyen, and Klecki teaches
each of the scale parameter and the shift parameter is a vector having the same dimension as the respective embedding vector, and all elements thereof have the same value (Equation 1 under 3.1 Variance-Only Layer Norm section of Wang shows the use of bias and gain as parameters with the same dimension H as input x for the normalization. In 3.2.1 Normalization on Feature Embedding section of Wang, same normalization formular is applied for input data of CTR task with embedding vectors. “The result of embedding layer is a wide concatenated vector: Vemb = concat(e1, e2, ..., ei , ..., ef ) (4) where f denotes the number of fields, and ei ∈ Rk denotes the embedding of one field. Although the feature lengths of instances can be various, their embedding are of the same length f ×k, where k is the dimension of field embedding… We can map bucket ID to an embedding vector. The second method maps the feature field into an embedding vector as follows: vi = ei xi (5) where ei is an embedding vector for field i, and xi is a scalar value…. We apply normalization on feature embedding based on the feature field as follows: N(Vemb ) = concat(N(e1), N(e2), ..., N(ei ), ..., N(ef )) (6) where N can be BatchNorm, LayerNorm, GroupNorm, Simple- LayerNorm or variance-only LayerNorm. The bias and gain are shared for features in same feature field if the normalization contains these parameters.” Since wang applies normalization to each embedding vector and gain/bias (scale/shift) parameters are vector parameters “with the same dimension” as the normalized input vector, it reads as these vectors being same dimension as the embedding vector. 2.3 Scaled l2 normalization and FixNorm section of Nguyen states “SCALENORM replaces the 2d scale and shift parameters of LAYERNORM with a single learned scalar, improving computational and parameter efficiency while potentially regularizing the loss landscape. This bias has an explicit interpretation at the final layer: large inner products sharpen the output distribution, causing frequent words to disproportionately dominate rare words. This led Nguyen and Chiang (2018) to introduce FIXNORM(w) =g*w/||w|| with fixed g at the last linear layer, to maximize the angular difference of output representations and aid rare word translation.” Pg. 4-5 of Klecki states “For this purpose, Normalize offers two scalar arguments: shift and scale . Now the normalization formula becomes: 𝑌𝑖 = (𝑋𝑖 – 𝜇)/𝜎⋅ 𝑠𝑐𝑎𝑙𝑒 + 𝑠ℎ𝑖𝑓𝑡 ” Klecki teaches that scalar scale and scalar shift are applied uniformly to all elements. Klecki and Nguyen’s scalar parameterization within Wang’s same dimension parameter format by storing the single scalar value as a tied entry vector or by broadcasting the scalar across the vector elements would be predictable implementation design choice. )
Regarding Claim 5, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Wang, Nguyen, and Klecki teaches
each of the scale parameter and the shift parameter has a scalar value. (Pg. 4-5 of Klecki states “For this purpose, Normalize offers two scalar arguments: shift and scale . Now the normalization formula becomes: 𝑌𝑖 = (𝑋𝑖 – 𝜇)/𝜎⋅ 𝑠𝑐𝑎𝑙𝑒 + 𝑠ℎ𝑖𝑓𝑡 ” Klecki teaches both a scalar scale and a scalar shift argument in a normalization operation. Pg. 3 of Nguyen states “SCALENORM replaces the 2d scale and shift parameters of LAYERNORM with a single learned scalar, improving computational and parameter efficiency while potentially regularizing the loss landscape. This bias has an explicit interpretation at the final layer: large inner products sharpen the output distribution, causing frequent words to disproportionately dominate rare words. This led Nguyen and Chiang (2018) to introduce FIXNORM(w) =g*w/||w|| with fixed g at the last linear layer, to maximize the angular difference of output representations and aid rare word translation.” Nguyen teaches motivation to use scalar parameters in normalization for efficiency and parameter reduction. )
Regarding Claim 7, the rejection of claim 1 Is incorporated herein. Furthermore, the combination of Wang, Nguyen, and Klecki teaches
A non-transitory computer-readable recording medium to execute the method of claim 1 in combination with hardware. (4.1.4 Implementation Details section of Wang states “We implement all the models with TensorFlow in our experiments. For optimization method, we use the Adam with a mini-batch size of 1000 and a learning rate is set to 0.0001. Focusing on normalization approaches in our paper, we make the dimension of field embedding for all models to be a fixed value of 10. For models with DNN part, the depth of hidden layers is set to 3, the number of neurons per layer is 400, all activation function are ReLU. We conduct our experiments with 2 Tesla K40 GPUs.” The use of a software program executed on computer-readable media in combination with hardware is needed to run such experiments with deep neural network.)
Claims 8, 11, 12 recite substantially similar subject matter as claims 1, 4, 5 respectively, and are rejected with the same rationale, mutatis mutandis.
Claims 2, 6, 9, 13, 14 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (NPL: “Correct Normalization Matters: Understanding the Effect of Normalization On Deep Neural Network Models For Click-Through Rate Prediction”) in view of Klecki (NPL: “Normalize operator”), Nguyen et al. (NPL: ”Transformers without Tears: Improving the Normalization of Self-Attention”), further in view of Mao (NPL: ”Layer Normalization Explained”).
Regarding Claim 2, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Wang, Klecki, Nguyen teaches
the normalizing includes: calculating a mean of the elements of the respective embedding vector; normalizing the vector on the basis of the scale parameter and the shift parameter. (3.1 Variance-Only LayerNorm section of Wang states Equation 1
PNG
media_image5.png
101
320
media_image5.png
Greyscale
“where h is the output of a LayerNorm layer. ⊙ is a dot production operation. µ and δ are the mean and standard deviation of input. Bias b and gain g are parameters with the same dimension H.” And 3.2.1 Normalization on Feature Embedding section of Wang shows they apply normalization formula on an embedding vector. “Where ei is an embedding vector for field i” and Equation 6
PNG
media_image6.png
26
305
media_image6.png
Greyscale
”Where N can be BatchNorm, LayerNorm, GroupNorm, SimpleLayerNorm or variance-only LayerNorm.”)
Wang and Nguyen does not explicitly teach that
calculating a variance of the elements of the respective embedding vector;
and normalizing the respective embedding vector on the basis of the mean, the variance…
However, Mao teaches that
calculating a variance of the elements of the vector; (Pg. 1 Mathematical Definition section of Mao states “We have its mean µ and Variance δ2“
PNG
media_image7.png
54
183
media_image7.png
Greyscale
)
and normalizing the vector on the basis of the mean, the variance, the scale parameter, and the shift parameter (Pg. 1 Mathematical Definition section of Mao states “Then we normalize each sample such that the elements in the sample have zero mean and unit variance. E is for numerical stability in case the denominator becomes zero by chance.” And “Finally, there is a scaling and shifting step.”
PNG
media_image8.png
73
142
media_image8.png
Greyscale
PNG
media_image9.png
37
183
media_image9.png
Greyscale
)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Wang, Klecki, Mao, and Nguyen. Wang teaches CTR prediction using deep neural networks with feature embeddings, and applying normalization to feature embeddings, where they use bias and gain parameters (commonly referred for scale and shift parameters in normalization) in same dimension as input. Lecki teaches a normalization operation using scalar scale and scalar shift arguments in which the same unindexed scale and shift values are applied in the element wise formula. Nguyen teaches that neural network normalization parameters may be simplified from per dimension LayerNorm scale/shift parameters to a single learned scalar to improve computational and parameter efficiency. Mao teaches the LayerNorm equation with the e stability term in the denominator and final scaling and shifting using gamma and beta. One with ordinary skill in the art would be motivated to incorporate the teachings of Mao into that of Wang, Klecki, Nguyen because e was a known numerical stability term in normalization calculations and Mao explicitly shows the known formular used for layer normalization. The combination would have predictably performed Wang’s feature embedding normalization according to the claimed mean variance equation while using scalar scale and shift values.
Regarding Claim 6, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Wang, Nguyen, Klecki, and Mao teaches
The method of claim 1, wherein the normalizing is an operation of performing calculation of Equation 1 below: [Equation 1]
PNG
media_image10.png
120
294
media_image10.png
Greyscale
, wherein, in Equation 1, ex is the respective embedding vector, d is a dimension of the respective embedding vector, μx is the mean of all the elements of the respective embedding vector, σx2 is the variance of all the elements of the respective embedding vector, (ex)k is a kth element of the respective embedding vector ex, gamma is the scale parameter, and beta is the shift parameter (Mathematical Definition section of Mao shows formulars that matches corresponding parts of the Equation 1.
PNG
media_image11.png
111
187
media_image11.png
Greyscale
PNG
media_image12.png
74
176
media_image12.png
Greyscale
PNG
media_image13.png
38
190
media_image13.png
Greyscale
. Incorporating to 3.2.1. Normalization on Feature Embedding section of Wang, xi vector can be exchanged with ei when normalization is applied to embedding vector. )
Claims 9, 13 recite substantially similar subject matter as claims 2, 6 respectively, and are rejected with the same rationale, mutatis mutandis.
Regarding Claim 14, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Wang, Nguyen, Klecki, and Mao teaches
PNG
media_image14.png
61
284
media_image14.png
Greyscale
where ex denotes an embedding vector corresponding to a feature x included in the feature vector, 6x is a normalized embedding corresponding to the feature x, Q refers to an element- wise multiplication operation, px and 6x2 are a mean and a variance of elements of the embedding vector ex, respectively, and s is a scalar value added to the variance to prevent overflow when the normalization is performed. (Pg. 2 – 3 of Wang states “First, we briefly review the formulation of LayerNorm. Let x = (x1, x2, ..., xH ) denotes the vector representation of an input of size H to normalization layers. LayerNorm re-centers and re-scales input x as
PNG
media_image15.png
96
248
media_image15.png
Greyscale
(1) where h is the output of a LayerNorm layer. ⊙ is a dot production operation. μ and δ are the mean and standard deviation of input. Bias b and gain g are parameters with the same dimension H.” Pg. 1 of Mao states “Then we normalize each sample such that the elements in the sample have zero mean and unit variance. E is for numerical stability in case the denominator becomes zero by chance.” And “Finally, there is a scaling and shifting step.”
PNG
media_image8.png
73
142
media_image8.png
Greyscale
PNG
media_image9.png
37
183
media_image9.png
Greyscale
)
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (NPL: “Correct Normalization Matters: Understanding the Effect of Normalization On Deep Neural Network Models For Click-Through Rate Prediction”) in view of Klecki (NPL: “Normalize operator”), Nguyen et al. (NPL: ”Transformers without Tears: Improving the Normalization of Self-Attention”), further in view of Chen et al. (NPL: “FLEN: Leveraging Field for Scalable CTR Prediction”).
Regarding Claim 15, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Wang, Nguyen, Klecki, and Mao does not teaches
each feature of the feature vector includes an element for a user and an element for an item,
the element for the user indicates an age of the user, a gender of the user, an access time of an online platform accessed by the user, a click log in the platform, or a usage history of the platform,
and the element for the item includes a type of content in the platform, a feature of the content, or an arrangement region of the content.
However, Chen teaches
each feature of the feature vector includes an element for a user and an element for an item, the element for the user indicates an age of the user, a gender of the user, an access time of an online platform accessed by the user, a click log in the platform, or a usage history of the platform, and the element for the item includes a type of content in the platform, a feature of the content, or an arrangement region of the content. (Pg. 1 of Chen states “The data in CTR prediction task is multi-field categorical data, i.e., every feature is categorical and belongs to one and only one field. For example, feature “gender=Female" belongs to field “gender", feature “age=24" belongs to field “age" and feature “item category=cosmetics" belongs to field “item category". The value of feature “gender" is either “male" or “female. Feature “age" is discretized to several age groups: “0-18", “18-25", “25-30", and so on. It is well regarded that, feature conjunctions are essential for accurate CTR prediction [3, 4, 7, 8, 11].” Chen identifies user side fields and item side fields as components of the feature vector used in a CTR prediction. )
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings of Wang, Klecki, Chen, and Nguyen. Wang teaches CTR prediction using deep neural networks with feature embeddings, and applying normalization to feature embeddings, where they use bias and gain parameters (commonly referred for scale and shift parameters in normalization) in same dimension as input. Lecki teaches a normalization operation using scalar scale and scalar shift arguments in which the same unindexed scale and shift values are applied in the element wise formula. Nguyen teaches that neural network normalization parameters may be simplified from per dimension LayerNorm scale/shift parameters to a single learned scalar to improve computational and parameter efficiency. Chen teaches CTR prediction using multi field feature inputs, including user side fields such as gender, age, clicked feed IDs, and liked feed IDs, and item side fields such as item category, feed ID, and tags. One with ordinary skill in the art would be motivated to incorporate the teachings of Chen into that of Wang, Klecki, Nguyen because both references concern CTR prediction using multi-field feature inputs and embedding based neural network models. The combination would have predictably used user side and item side CTR features in Wang’s feature vector while applying the scalar normalization parameterization.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BYUNGKWON HAN whose telephone number is (571)272-5294. The examiner can normally be reached M-F: 9:00AM-6PM PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at (571)272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BYUNGKWON HAN/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights