Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
Claims 23-30 recite “computer-storage medium…” The specification paragraph 100 says, “The terms … ‘computer-storage media,’… specifically exclude carrier waves, modulated data signals, and other such media….” However the specification paragraph 104 says, “The terms … “computer-readable medium,” … include both storage devices/media and carrier waves/modulated data signals.” The claims 23-30 use the term defined in paragraph 100 and will be interpreted to exclude data per se and signals per se, because the similar claim definition in paragraph 104 is for a different unclaimed element.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 1-11, 16 and 27 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation "the transformers" in the last clause. There is insufficient antecedent basis for this limitation in the claim. There is only a singular “transformer” in the claim.
Claims 5, 16 and 27 refer to “the select prediction value” but it is unclear whether that refers to the categorical prediction value inherited form claims 1/12/23 or whether applicant intended “the given prediction value” recited earlier in the claim. Because those values correspond to different fields and different prediction objectives, the metes and bounds cannot be determined with reasonable certainty.
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claims 13 and 24 rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends. Claims 13 and 24 are repeats of the predictions generated in 12 and 23 respectively. Therefore, claims 13 and 24 fail to further limit the subject matter of the claim upon which they depend. Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6, 12-17 and 23-28 are rejected under 35 U.S.C. 103 as being unpatentable over TabTransformer: Tabular Data Modeling Using Contextual Embeddings by Huang et al and US20240312193A1 to Cheng et al.
Claims 7-9, 18-20 and 29-30 are rejected under 35 U.S.C. 103 as being unpatentable over TabTransformer: Tabular Data Modeling Using Contextual Embeddings by Huang et al, US20240312193A1 to Cheng et al and US20230252114A1 to Lozoya et al.
Claims 10-11 and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over TabTransformer: Tabular Data Modeling Using Contextual Embeddings by Huang et al, US20240312193A1 to Cheng et al, US20230252114A1 to Lozoya et al and US20230053182A1 and Bertiger et al.
Huang teaches claim 1. A system comprising:
(Huang fig. 1.)
accessing training data that comprises a data table, the data table comprising a set of rows that store data, each row of the set of rows comprises a plurality of fields; and (Huang sec. 2 “Let (x,y) denote a feature-target pair, where x ≡ {xcat, xcont}. The xcat denotes all the categorical features and xcont ∈ Rc denotes all of the c continuous features…. We minimize the following loss function L(x,y) to learn all the TabTransformer parameters in an end-to-end learning by the first-order gradient methods.” Each x is a row in a table.)
for a select row of the set of rows:
generating, by a set of initial processing operations, a select plurality of intermediate field values based on a select plurality of original field values of the select row, the set of initial processing operations being configured to generate the select plurality of intermediate field values by replacing a set of original categorical values of a set of randomly-selected categorical fields of the select row in the select plurality of intermediate field values with a set of replacement categorical values; (Huang sec. 2 “RTD replaces the original feature by a random value of that feature…. The RTD procedure as proposed in (Clark et al. 2020) uses auxiliary genera tor for sampling a subset of features that a feature should be replaced with.”)
generating, by a feature tokenizer, a select plurality of field token(Huang sec 2 “Let eφi (xi) ∈ Rd for i ∈ {1,··· ,m} be the embedding of the xi feature, and Eφ(xcat) = {eφ1 (x1),··· ,eφm (xm)} be the set of embeddings for all the categorical features.” eφi and Eφ are the tokens.)
generating, by a transformer, a select plurality of transformer output values based on the select plurality of field token vectors; (Huang sec. 2 “these parametric embeddings Eφ(xcat) are in putted to the first Transformer layer. The output of the first Transformer layer is inputted to the second layer Transformer, and so forth. Each parametric embedding is transformed into contextual embedding when outputted from the top layer Transformer…”)
generating, by a set of prediction layers, a set of prediction values based on the select plurality of transformer output values; and (Huang sec. 2 “RTD replaces the original feature by a random value of that feature. Here, the loss is minimized for a binary classifier that tries to predict whether or not the feature has been replaced.”)
training the transformer or the set of prediction layers based on the set of prediction values by adjusting one or more parameters of at least one of the transformers or the set of prediction layers based on the set of prediction values. (Huang sec. 2 “We minimize the following loss function L(x,y) to learn all the TabTransformer parameters in an end-to-end learning by the first-order gradient methods…. RTD replaces the original feature by a random value of that feature. Here, the loss is minimized for a binary classifier that tries to predict whether or not the feature has been replaced…. We name the two pre-training methods as TabTransformer-MLM and TabTransformer-RTD.”)
Huang doesn’t teach a memory and processor.
However, Cheng teaches at least one hardware processor; and
at least one memory storing instructions that cause the at least one hardware processor to perform operations. (Cheng fig. 9)
Cheng, Huang and the claims are all directed to machine learning on tabular data. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to execute Huang a processor and memory because of the numerous mentions of “compute” in Huang.
Huang teaches claim 2. The system of claim 1, wherein a select prediction value in the set of prediction values corresponds to a randomly-selected categorical field of the set of randomly-selected categorical fields, the select prediction value indicating whether an original categorical value of the randomly-selected categorical field was replaced. (Huang sec. 2 “RTD replaces the original feature by a random value of that feature. Here, the loss is minimized for a binary classifier that tries to predict whether or not the feature has been replaced.”)
Huang teaches claim 3. The system of claim 2, wherein the adjusting of the one or more parameters based on the set of prediction values comprises:
comparing the select prediction value to an expected value for the randomly-selected categorical field, the expected value indicating that an original categorical value of the randomly-selected categorical field was replaced; and (Huang sec. 2 calculates loss, eq 1 below, that calculation is a comparison between the expected value and the prediction value.
PNG
media_image1.png
31
282
media_image1.png
Greyscale
Huang sec. 2 “RTD replaces the original feature by a random value of that feature. Here, the loss is minimized for a binary classifier that tries to predict whether or not the feature has been replaced.”)
adjusting the one or more parameters based on the comparing of the select prediction value to the expected value. (Huang sec. 2 “We minimize the following loss function L(x,y) to learn all the TabTransformer parameters in an end-to-end learning by the first-order gradient methods….”)
Cheng teaches claim 4. The system of claim 1, wherein the set of initial processing operations is configured to generate the select plurality of intermediate field values by replacing, in the select plurality of intermediate field values, a set of original numerical values of a set of randomly-selected numerical fields of the select row with a set of masked values. (Cheng para 48 “Feature-Tokenizer Transformer technique is described in Gorishniy, Yury, et al. “Revisiting deep learning models for tabular data.” Advances in Neural Information Processing Systems 34 (2021): 18932-18943, the contents of which are incorporated herein by reference in its entirety. Accordingly, randomly corrupt entries of table samples with values draw according to its marginal probability, and use multilevel perception (MLP) decoders to learn by comparison and denoising. Thus, as illustrated in FIG. 4 , an encoder 408 such as the Feature-Tokenizer Transformer takes an original tabular input 402 that is randomly corrupted 404, and the output from the encoder 408 is decoded by a decoder 410 in an effort to reconstruct the original tabular data 412.” The randomly corrupt values are the set of masked values.)
Cheng teaches claim 5. The system of claim 4, wherein the set of prediction values comprises a given prediction value corresponding to a randomly-selected numerical field of the set of randomly-selected numerical fields, and wherein the adjusting of the one or more parameters based on the set of prediction values comprises:
comparing the given prediction value to an original numerical value of the randomly-selected numerical field; and (Cheng para 48 “Feature-Tokenizer Transformer technique is described in Gorishniy, Yury, et al. “Revisiting deep learning models for tabular data.” Advances in Neural Information Processing Systems 34 (2021): 18932-18943, the contents of which are incorporated herein by reference in its entirety. Accordingly, randomly corrupt entries of table samples with values draw according to its marginal probability, and use multilevel perception (MLP) decoders to learn by comparison and denoising. Thus, as illustrated in FIG. 4 , an encoder 408 such as the Feature-Tokenizer Transformer takes an original tabular input 402 that is randomly corrupted 404, and the output from the encoder 408 is decoded by a decoder 410 in an effort to reconstruct the original tabular data 412.”)
adjusting the one or more parameters based on the comparing of the select prediction value to the original numerical value. (Cheng para 48 “Feature-Tokenizer Transformer technique is described in Gorishniy, Yury, et al. “Revisiting deep learning models for tabular data.” Advances in Neural Information Processing Systems 34 (2021): 18932-18943, the contents of which are incorporated herein by reference in its entirety. Accordingly, randomly corrupt entries of table samples with values draw according to its marginal probability, and use multilevel perception (MLP) decoders to learn by comparison and denoising.”)
Cheng teaches claim 6. The system of claim 1, wherein the feature tokenizer and the transformer are part of a feature-tokenizer transformer (FTT). (Cheng para 15 “specific encoder for tabular data modality by performing Feature-Tokenizer Transformer.”)
Huang teaches claim 7. The system of claim 1, wherein the data table is a first data table, and wherein the operations comprise:
after the training of the transformer based on the set of prediction values, performing an inference operation using the transformer, the inference operation comprising: (Huang sec. 2 “The contextual embed dings explained above are learned in end-to-end supervised training using labeled examples. For a scenario, when there are a few labeled examples and a large number of unlabeled examples, we introduce a pre-training procedure to train the Transformer layers using unlabeled data. This is followed by fine-tuning of the pre-trained Transformer layers along with the top MLP layer using the labeled data.”)
accessing input data that comprises a second data table, the second data table comprising a current row, the current row comprising the plurality of fields; (Huang table 1 p. 4 shows that tabtransformer is run on several tables of different datasets.)
generating, by the feature tokenizer, a current plurality of field token vectors that corresponds to a current plurality of original field values of the current row; (Huang sec 2 “Let eφi (xi) ∈ Rd for i ∈ {1,··· ,m} be the embedding of the xi feature, and Eφ(xcat) = {eφ1 (x1),··· ,eφm (xm)} be the set of embeddings for all the categorical features.” eφi and Eφ are the tokens.)
generating, by the transformer, a current plurality of transformer output values that corresponds to the current plurality of field token vectors; and (Huang sec. 2 “these parametric embeddings Eφ(xcat) are in putted to the first Transformer layer. The output of the first Transformer layer is inputted to the second layer Transformer, and so forth. Each parametric embedding is transformed into contextual embedding when outputted from the top layer Transformer…”)
Huang doesn’t have a mean pool layer.
However, Lozoya teaches generating, by a mean pool operation, a current row embedding value for the current row, the current row embedding value being generated based on the current plurality of transformer output values. (Lozoya para 33 “embedding zsent is defined as the mean-pooling result of the last transformer layer's output.”)
Huang, Lozoya and the claims are all transformer architectures. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to add Lozoya’s pooling layer because it’s a standard way to collapse a set of transformer embeddings into one fixed-dimension embedding – which transforms the output back into the single vector representing the row.
Huang teaches claim 8. The system of claim 7, wherein the second data table comprises a reference row, wherein the reference row comprises the plurality of fields, and wherein the operations comprise: (Huang table 1 p. 4 shows that tabtransformer is run on several tables of different datasets.)
after the performing of the inference operation: (Huang sec. 2 “RTD replaces the original feature by a random value of that feature. Here, the loss is minimized for a binary classifier that tries to predict whether or not the feature has been replaced.”)
Huang doesn’t teach a pooling layer.
However, Lozoya teaches generating a reference row embedding value for the reference row; and (Lozoya para 33 “embedding zsent is defined as the mean-pooling result of the last transformer layer's output.” Mean pooling result is the reference row generated after the prediction output of the transformer layer.)
Huang doesn’t teach a cosine similarity.
However, Cheng teaches determining a similarity between the reference row and the current row based on the reference row embedding value and the current row embedding value. (Cheng para 50 “K priors may selected as the sample with embeddings that are closest to the embeddings of the observed modality (i.e., the image modality), which may be measured by cosine similarity between embeddings of the observed modality eobserved.”)
Huang teaches claim 9. The system of claim 8, wherein the generating of the reference row embedding value for the reference row comprises:
generating, by the set of initial processing operations, a reference plurality of intermediate field values based on a reference plurality of original field values of the reference row; (Huang sec. 2 “RTD replaces the original feature by a random value of that feature…. The RTD procedure as proposed in (Clark et al. 2020) uses auxiliary genera tor for sampling a subset of features that a feature should be replaced with.” The reference row is functionally the same as any of the other rows until something is done with the reference row.)
generating, by the feature tokenizer, a reference plurality of field token vectors that corresponds to the reference plurality of intermediate field values; (Huang sec 2 “Let eφi (xi) ∈ Rd for i ∈ {1,··· ,m} be the embedding of the xi feature, and Eφ(xcat) = {eφ1 (x1),··· ,eφm (xm)} be the set of embeddings for all the categorical features.” eφi and Eφ are the tokens. The reference row is functionally the same as any of the other rows until something is done with the reference row.)
generating, by the transformer, a reference plurality of transformer output values that corresponds to the reference plurality of field token vectors; and (Huang sec. 2 “these parametric embeddings Eφ(xcat) are in putted to the first Transformer layer. The output of the first Transformer layer is inputted to the second layer Transformer, and so forth. Each parametric embedding is transformed into contextual embedding when outputted from the top layer Transformer…” The reference row is functionally the same as any of the other rows until something is done with the reference row.)
Huang doesn’t teach a pooling layer.
However, Lozoya teaches generating, by a mean pool operation, the reference row embedding value for the reference row, the reference row embedding value being generated based on the reference plurality of transformer output values. (Lozoya para 33 “embedding zsent is defined as the mean-pooling result of the last transformer layer's output.” Mean pooling result is the reference row generated after the prediction output of the transformer layer.)
Huang teaches claim 10. The system of claim 7, wherein the operations comprise:
after the performing of the inference operation: (Huang sec. 2 “RTD replaces the original feature by a random value of that feature. Here, the loss is minimized for a binary classifier that tries to predict whether or not the feature has been replaced.”)
Huang doesn’t do a nearest neighbor calculation.
However, Bertiger teaches determining a distance between the current row embedding value and a nearest neighbor row in the first data table of the training data; (Bertiger para 35 “ the anomaly score is taken to be the minimum of these distances, that is, the distance between the embeddings of user u and its nearest neighbor in
PNG
media_image2.png
38
29
media_image2.png
Greyscale
.”)
determining whether the distance satisfies a distance threshold constraint; and (Bertiger para 35 “ the anomaly score is taken to be the minimum of these distances, that is, the distance between the embeddings of user u and its nearest neighbor in
PNG
media_image2.png
38
29
media_image2.png
Greyscale
.”)
determining that the current row is anomalous in response to determining that the distance satisfies the distance threshold constraint. (Bertiger para 37 “anomaly score is compared against a pre-defined anomaly threshold (at 508), and access events with an anomaly score greater than the threshold are flagged as anomalous (in 510)…”)
Huang, Bertiger and the claims are all directed to embeddings and downstream use of embeddings. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to use Huang’s embeddings in anomaly detection because “Anomaly-based intrusion detection provides a complementary approach that has the potential to discover unknown threats and so-called zero-day attacks… [and] An advantage of anomaly detection via graph embeddings compared with a pure organizational approach is that, if users regularly behave as if they belong to a different organization (e.g., because they work on an inter-organizational project), the former approach can learn this behavior, while the latter approach, which is based on fixed metadata, cannot.” Bertiger para 2 and 52.
Huang teaches claim 11. The system of claim 7, wherein the operations comprise:
after the performing of the inference operation: (Huang sec. 2 “RTD replaces the original feature by a random value of that feature. Here, the loss is minimized for a binary classifier that tries to predict whether or not the feature has been replaced.”)
Huang doesn’t do a nearest neighbor calculation.
However, Bertiger teaches determining a distance between the current row embedding value and a nearest neighbor row in the first data table of the training data; (Bertiger para 35 “ the anomaly score is taken to be the minimum of these distances, that is, the distance between the embeddings of user u and its nearest neighbor in
PNG
media_image2.png
38
29
media_image2.png
Greyscale
.”)
determining whether the distance satisfies a distance threshold constraint; and (Bertiger para 35 “ the anomaly score is taken to be the minimum of these distances, that is, the distance between the embeddings of user u and its nearest neighbor in
PNG
media_image2.png
38
29
media_image2.png
Greyscale
.”)
determining that the current row is not anomalous in response to determining that the distance does not satisfy the distance threshold constraint. (Bertiger para 37 “anomaly score is compared against a pre-defined anomaly threshold (at 508), and access events with an anomaly score greater than the threshold are flagged as anomalous (in 510)…”)
Huang, Bertiger and the claims are all directed to embeddings and downstream use of embeddings. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to use Huang’s embeddings in anomaly detection because “Anomaly-based intrusion detection provides a complementary approach that has the potential to discover unknown threats and so-called zero-day attacks… [and] An advantage of anomaly detection via graph embeddings compared with a pure organizational approach is that, if users regularly behave as if they belong to a different organization (e.g., because they work on an inter-organizational project), the former approach can learn this behavior, while the latter approach, which is based on fixed metadata, cannot.” Bertiger para 2 and 52.
Huang teaches claim 12. A method comprising:
accessing, (Huang sec. 2 “Let (x,y) denote a feature-target pair, where x ≡ {xcat, xcont}. The xcat denotes all the categorical features and xcont ∈ Rc denotes all of the c continuous features…. We minimize the following loss function L(x,y) to learn all the TabTransformer parameters in an end-to-end learning by the first-order gradient methods.” Each x is a row in a table.)
for a select row of the set of rows:
generating, by a set of initial processing operations, a select plurality of intermediate field values based on a select plurality of original field values of the select row, the set of initial processing operations being configured to generate the select plurality of intermediate field values by replacing a set of original categorical values of a set of randomly-selected categorical fields of the select row in the select plurality of intermediate field values with a set of replacement categorical values; (Huang sec. 2 “RTD replaces the original feature by a random value of that feature…. The RTD procedure as proposed in (Clark et al. 2020) uses auxiliary genera tor for sampling a subset of features that a feature should be replaced with.”)
generating, by a feature tokenizer, a select plurality of field token vectors based on a select plurality of intermediate field values; (Huang sec 2 “Let eφi (xi) ∈ Rd for i ∈ {1,··· ,m} be the embedding of the xi feature, and Eφ(xcat) = {eφ1 (x1),··· ,eφm (xm)} be the set of embeddings for all the categorical features.” eφi and Eφ are the tokens.)
generating, by a transformer, a select plurality of transformer output values based on the select plurality of field token vectors; (Huang sec. 2 “these parametric embeddings Eφ(xcat) are in putted to the first Transformer layer. The output of the first Transformer layer is inputted to the second layer Transformer, and so forth. Each parametric embedding is transformed into contextual embedding when outputted from the top layer Transformer…”)
generating, by a set of prediction layers, a set of prediction values based on the select plurality of transformer output values, a select prediction value in the set of prediction values corresponding to a randomly-selected categorical field of the set of randomly-selected categorical fields, the select prediction value indicating whether an original categorical value of the randomly-selected categorical field was replaced; and (Huang sec. 2 “RTD replaces the original feature by a random value of that feature. Here, the loss is minimized for a binary classifier that tries to predict whether or not the feature has been replaced.”)
training, by the one or more hardware processors, the transformer or the set of prediction layers based on the set of prediction values by adjusting one or more parameters of at least one of the transformer or the set of prediction layers based on the set of prediction values. (Huang sec. 2 “We minimize the following loss function L(x,y) to learn all the TabTransformer parameters in an end-to-end learning by the first-order gradient methods…. RTD replaces the original feature by a random value of that feature. Here, the loss is minimized for a binary classifier that tries to predict whether or not the feature has been replaced…. We name the two pre-training methods as TabTransformer-MLM and TabTransformer-RTD.”)
Huang doesn’t teach a processor.
However, Cheng teaches one or more hardware processor. (Cheng fig. 9)
Cheng, Huang and the claims are all directed to machine learning on tabular data. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to execute Huang a processor and memory because of the numerous mentions of “compute” in Huang.
Huang teaches claim 13. The method of claim 12, wherein a select prediction value in the set of prediction values corresponds to a randomly-selected categorical field of the set of randomly-selected categorical fields, the select prediction value indicating whether an original categorical value of the randomly-selected categorical field was replaced. (Huang sec. 2 “RTD replaces the original feature by a random value of that feature. Here, the loss is minimized for a binary classifier that tries to predict whether or not the feature has been replaced.”)
Huang teaches claim 14. The method of claim 13, wherein the adjusting of the one or more parameters based on the set of prediction values comprises:
comparing the select prediction value to an expected value for the randomly-selected categorical field, the expected value indicating that an original categorical value of the randomly-selected categorical field was replaced; and (Huang sec. 2 calculates loss, eq 1 below, that calculation is a comparison between the expected value and the prediction value.
PNG
media_image1.png
31
282
media_image1.png
Greyscale
Huang sec. 2 “RTD replaces the original feature by a random value of that feature. Here, the loss is minimized for a binary classifier that tries to predict whether or not the feature has been replaced.”)
adjusting the one or more parameters based on the comparing of the select prediction value to the expected value. (Huang sec. 2 “We minimize the following loss function L(x,y) to learn all the TabTransformer parameters in an end-to-end learning by the first-order gradient methods….”)
Cheng teaches claim 15. The method of claim 12, wherein the set of initial processing operations is configured to generate the select plurality of intermediate field values by replacing, in the select plurality of intermediate field values, a set of original numerical values of a set of randomly-selected numerical fields of the select row with a set of masked values. (Cheng para 48 “Feature-Tokenizer Transformer technique is described in Gorishniy, Yury, et al. “Revisiting deep learning models for tabular data.” Advances in Neural Information Processing Systems 34 (2021): 18932-18943, the contents of which are incorporated herein by reference in its entirety. Accordingly, randomly corrupt entries of table samples with values draw according to its marginal probability, and use multilevel perception (MLP) decoders to learn by comparison and denoising. Thus, as illustrated in FIG. 4 , an encoder 408 such as the Feature-Tokenizer Transformer takes an original tabular input 402 that is randomly corrupted 404, and the output from the encoder 408 is decoded by a decoder 410 in an effort to reconstruct the original tabular data 412.” The randomly corrupt values are the set of masked values.)
Cheng teaches claim 16. The method of claim 15, wherein the set of prediction values comprises a given prediction value corresponding to a randomly-selected numerical field of the set of randomly-selected numerical fields, and wherein the adjusting of the one or more parameters based on the set of prediction values comprises:
comparing the given prediction value to an original numerical value of the randomly-selected numerical field; and (Cheng para 48 “Feature-Tokenizer Transformer technique is described in Gorishniy, Yury, et al. “Revisiting deep learning models for tabular data.” Advances in Neural Information Processing Systems 34 (2021): 18932-18943, the contents of which are incorporated herein by reference in its entirety. Accordingly, randomly corrupt entries of table samples with values draw according to its marginal probability, and use multilevel perception (MLP) decoders to learn by comparison and denoising. Thus, as illustrated in FIG. 4 , an encoder 408 such as the Feature-Tokenizer Transformer takes an original tabular input 402 that is randomly corrupted 404, and the output from the encoder 408 is decoded by a decoder 410 in an effort to reconstruct the original tabular data 412.”)
adjusting the one or more parameters based on the comparing of the select prediction value to the original numerical value. (Cheng para 48 “Feature-Tokenizer Transformer technique is described in Gorishniy, Yury, et al. “Revisiting deep learning models for tabular data.” Advances in Neural Information Processing Systems 34 (2021): 18932-18943, the contents of which are incorporated herein by reference in its entirety. Accordingly, randomly corrupt entries of table samples with values draw according to its marginal probability, and use multilevel perception (MLP) decoders to learn by comparison and denoising.”)
Cheng teaches claim 17. The method of claim 12, wherein the feature tokenizer and the transformer are part of a feature-tokenizer transformer (FTT). (Cheng para 15 “specific encoder for tabular data modality by performing Feature-Tokenizer Transformer.”)
Huang teaches claim 18. The method of claim 12, wherein the data table is a first data table, and wherein the method comprises:
after the training of the transformer based on the set of prediction values, performing an inference operation using the transformer, the inference operation comprising: (Huang sec. 2 “The contextual embed dings explained above are learned in end-to-end supervised training using labeled examples. For a scenario, when there are a few labeled examples and a large number of unlabeled examples, we introduce a pre-training procedure to train the Transformer layers using unlabeled data. This is followed by fine-tuning of the pre-trained Transformer layers along with the top MLP layer using the labeled data.”)
accessing input data that comprises a second data table, the second data table comprising a current row, the current row comprising the plurality of fields; (Huang table 1 p. 4 shows that tabtransformer is run on several tables of different datasets.)
generating, by the feature tokenizer, a current plurality of field token vectors that corresponds to a current plurality of original field values of the current row; (Huang sec 2 “Let eφi (xi) ∈ Rd for i ∈ {1,··· ,m} be the embedding of the xi feature, and Eφ(xcat) = {eφ1 (x1),··· ,eφm (xm)} be the set of embeddings for all the categorical features.” eφi and Eφ are the tokens.)
generating, by the transformer, a current plurality of transformer output values that corresponds to the current plurality of field token vectors; and (Huang sec. 2 “these parametric embeddings Eφ(xcat) are in putted to the first Transformer layer. The output of the first Transformer layer is inputted to the second layer Transformer, and so forth. Each parametric embedding is transformed into contextual embedding when outputted from the top layer Transformer…”)
Huang doesn’t have a mean pool layer.
However, Lozoya teaches generating, by a mean pool operation, a current row embedding value for the current row, the current row embedding value being generated based on the current plurality of transformer output values. (Lozoya para 33 “embedding zsent is defined as the mean-pooling result of the last transformer layer's output.”)
Huang, Lozoya and the claims are all transformer architectures. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to add Lozoya’s pooling layer because it’s a standard way to collapse a set of transformer embeddings into one fixed-dimension embedding – which transforms the output back into the single vector representing the row.
Huang teaches claim 19. The method of claim 18, wherein the second data table comprises a reference row, wherein the reference row comprises the plurality of fields, and wherein the method comprises: (Huang table 1 p. 4 shows that tabtransformer is run on several tables of different datasets.)
after the performing of the inference operation: (Huang sec. 2 “RTD replaces the original feature by a random value of that feature. Here, the loss is minimized for a binary classifier that tries to predict whether or not the feature has been replaced.”)
Huang doesn’t teach a pooling layer.
However, Lozoya generating, by the one or more hardware processors, a reference row embedding value for the reference row; and(Lozoya para 33 “embedding zsent is defined as the mean-pooling result of the last transformer layer's output.” Mean pooling result is the reference row generated after the prediction output of the transformer layer.)
Huang doesn’t teach a cosine similarity.
However, Cheng teaches determining, by the one or more hardware processors, a similarity between the reference row and the current row based on the reference row embedding value and the current row embedding value. (Cheng para 50 “K priors may selected as the sample with embeddings that are closest to the embeddings of the observed modality (i.e., the image modality), which may be measured by cosine similarity between embeddings of the observed modality eobserved.”)
Huang teaches claim 20. The method of claim 19, wherein the generating of the reference row embedding value for the reference row comprises:
generating, by the set of initial processing operations, a reference plurality of intermediate field values based on a reference plurality of original field values of the reference row; (Huang sec. 2 “RTD replaces the original feature by a random value of that feature…. The RTD procedure as proposed in (Clark et al. 2020) uses auxiliary genera tor for sampling a subset of features that a feature should be replaced with.” The reference row is functionally the same as any of the other rows until something is done with the reference row.)
generating, by the feature tokenizer, a reference plurality of field token vectors that corresponds to the reference plurality of intermediate field values; (Huang sec 2 “Let eφi (xi) ∈ Rd for i ∈ {1,··· ,m} be the embedding of the xi feature, and Eφ(xcat) = {eφ1 (x1),··· ,eφm (xm)} be the set of embeddings for all the categorical features.” eφi and Eφ are the tokens. The reference row is functionally the same as any of the other rows until something is done with the reference row.)
generating, by the transformer, a reference plurality of transformer output values that corresponds to the reference plurality of field token vectors; and (Huang sec. 2 “these parametric embeddings Eφ(xcat) are in putted to the first Transformer layer. The output of the first Transformer layer is inputted to the second layer Transformer, and so forth. Each parametric embedding is transformed into contextual embedding when outputted from the top layer Transformer…” The reference row is functionally the same as any of the other rows until something is done with the reference row.)
Huang doesn’t teach a pooling layer.
However, Lozoya teaches generating, by a mean pool operation, the reference row embedding value for the reference row, the reference row embedding value being generated based on the reference plurality of transformer output values. (Lozoya para 33 “embedding zsent is defined as the mean-pooling result of the last transformer layer's output.” Mean pooling result is the reference row generated after the prediction output of the transformer layer.)
Huang teaches claim 21. The method of claim 18, comprising:
after the performing of the inference operation: (Huang sec. 2 “RTD replaces the original feature by a random value of that feature. Here, the loss is minimized for a binary classifier that tries to predict whether or not the feature has been replaced.”)
Huang doesn’t do a nearest neighbor calculation.
However, Bertiger teaches determining, by the one or more hardware processors, a distance between the current row embedding value and a nearest neighbor row in the first data table of the training data; (Bertiger para 35 “ the anomaly score is taken to be the minimum of these distances, that is, the distance between the embeddings of user u and its nearest neighbor in
PNG
media_image2.png
38
29
media_image2.png
Greyscale
.”)
determining, by the one or more hardware processors, whether the distance satisfies a distance threshold constraint; and (Bertiger para 35 “ the anomaly score is taken to be the minimum of these distances, that is, the distance between the embeddings of user u and its nearest neighbor in
PNG
media_image2.png
38
29
media_image2.png
Greyscale
.”)
determining, by the one or more hardware processors, that the current row is anomalous in response to determining that the distance satisfies the distance threshold constraint. (Bertiger para 37 “anomaly score is compared against a pre-defined anomaly threshold (at 508), and access events with an anomaly score greater than the threshold are flagged as anomalous (in 510)…”)
Huang, Bertiger and the claims are all directed to embeddings and downstream use of embeddings. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to use Huang’s embeddings in anomaly detection because “Anomaly-based intrusion detection provides a complementary approach that has the potential to discover unknown threats and so-called zero-day attacks… [and] An advantage of anomaly detection via graph embeddings compared with a pure organizational approach is that, if users regularly behave as if they belong to a different organization (e.g., because they work on an inter-organizational project), the former approach can learn this behavior, while the latter approach, which is based on fixed metadata, cannot.” Bertiger para 2 and 52.
Huang teaches claim 22. The method of claim 18, comprising:
after the performing of the inference operation: (Huang sec. 2 “RTD replaces the original feature by a random value of that feature. Here, the loss is minimized for a binary classifier that tries to predict whether or not the feature has been replaced.”)
Huang doesn’t do a nearest neighbor calculation.
However, Bertiger teaches determining, by the one or more hardware processors, a distance between the current row embedding value and a nearest neighbor row in the first data table of the training data; (Bertiger para 35 “ the anomaly score is taken to be the minimum of these distances, that is, the distance between the embeddings of user u and its nearest neighbor in
PNG
media_image2.png
38
29
media_image2.png
Greyscale
.”)
determining, by the one or more hardware processors, whether the distance satisfies a distance threshold constraint; and (Bertiger para 35 “ the anomaly score is taken to be the minimum of these distances, that is, the distance between the embeddings of user u and its nearest neighbor in
PNG
media_image2.png
38
29
media_image2.png
Greyscale
.”)
determining, by the one or more hardware processors, that the current row is not anomalous in response to determining that the distance does not satisfy the distance threshold constraint. (Bertiger para 37 “anomaly score is compared against a pre-defined anomaly threshold (at 508), and access events with an anomaly score greater than the threshold are flagged as anomalous (in 510)…”)
Huang, Bertiger and the claims are all directed to embeddings and downstream use of embeddings. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to use Huang’s embeddings in anomaly detection because “Anomaly-based intrusion detection provides a complementary approach that has the potential to discover unknown threats and so-called zero-day attacks… [and] An advantage of anomaly detection via graph embeddings compared with a pure organizational approach is that, if users regularly behave as if they belong to a different organization (e.g., because they work on an inter-organizational project), the former approach can learn this behavior, while the latter approach, which is based on fixed metadata, cannot.” Bertiger para 2 and 52.
Huang teaches claim 23. A (Huang fig. 1.)
accessing training data that comprises a data table, the data table comprising a set of rows that store data, each row of the set of rows comprises a plurality of fields; and (Huang sec. 2 “Let (x,y) denote a feature-target pair, where x ≡ {xcat, xcont}. The xcat denotes all the categorical features and xcont ∈ Rc denotes all of the c continuous features…. We minimize the following loss function L(x,y) to learn all the TabTransformer parameters in an end-to-end learning by the first-order gradient methods.” Each x is a row in a table.)
for a select row of the set of rows:
generating, by a set of initial processing operations, a select plurality of intermediate field values based on a select plurality of original field values of the select row, the set of initial processing operations being configured to generate the select plurality of intermediate field values by replacing a set of original categorical values of a set of randomly-selected categorical fields of the select row in the select plurality of intermediate field values with a set of replacement categorical values; (Huang sec. 2 “RTD replaces the original feature by a random value of that feature…. The RTD procedure as proposed in (Clark et al. 2020) uses auxiliary genera tor for sampling a subset of features that a feature should be replaced with.”)
generating, by a feature tokenizer, a select plurality of field token vectors based on a select plurality of intermediate field values; (Huang sec 2 “Let eφi (xi) ∈ Rd for i ∈ {1,··· ,m} be the embedding of the xi feature, and Eφ(xcat) = {eφ1 (x1),··· ,eφm (xm)} be the set of embeddings for all the categorical features.” eφi and Eφ are the tokens.)
generating, by a transformer, a select plurality of transformer output values based on the select plurality of field token vectors; (Huang sec. 2 “these parametric embeddings Eφ(xcat) are in putted to the first Transformer layer. The output of the first Transformer layer is inputted to the second layer Transformer, and so forth. Each parametric embedding is transformed into contextual embedding when outputted from the top layer Transformer…”)
generating, by a set of prediction layers, a set of prediction values based on the select plurality of transformer output values, a select prediction value in the set of prediction values corresponding to a randomly-selected categorical field of the set of randomly-selected categorical fields, the select prediction value indicating whether an original categorical value of the randomly-selected categorical field was replaced; and (Huang sec. 2 “RTD replaces the original feature by a random value of that feature. Here, the loss is minimized for a binary classifier that tries to predict whether or not the feature has been replaced.”)
training the transformer or the set of prediction layers based on the set of prediction values by adjusting one or more parameters of at least one of the transformer or the set of prediction layers based on the set of prediction values. (Huang sec. 2 “We minimize the following loss function L(x,y) to learn all the TabTransformer parameters in an end-to-end learning by the first-order gradient methods…. RTD replaces the original feature by a random value of that feature. Here, the loss is minimized for a binary classifier that tries to predict whether or not the feature has been replaced…. We name the two pre-training methods as TabTransformer-MLM and TabTransformer-RTD.”)
Huang doesn’t teach a memory and processor.
However, Cheng teaches computer-storage medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to… (Cheng fig. 9)
Cheng, Huang and the claims are all directed to machine learning on tabular data. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to execute Huang a processor and memory because of the numerous mentions of “compute” in Huang.
Huang teaches claim 24. The computer-storage medium of claim 23, wherein a select prediction value in the set of prediction values corresponds to a randomly-selected categorical field of the set of randomly-selected categorical fields, the select prediction value indicating whether an original categorical value of the randomly-selected categorical field was replaced. (Huang sec. 2 “RTD replaces the original feature by a random value of that feature. Here, the loss is minimized for a binary classifier that tries to predict whether or not the feature has been replaced.”)
Huang teaches claim 25. The computer-storage medium of claim 24, wherein the adjusting of the one or more parameters based on the set of prediction values comprises:
comparing the select prediction value to an expected value for the randomly-selected categorical field, the expected value indicating that an original categorical value of the randomly-selected categorical field was replaced; and (Huang sec. 2 calculates loss, eq 1 below, that calculation is a comparison between the expected value and the prediction value.
PNG
media_image1.png
31
282
media_image1.png
Greyscale
Huang sec. 2 “RTD replaces the original feature by a random value of that feature. Here, the loss is minimized for a binary classifier that tries to predict whether or not the feature has been replaced.”)
adjusting the one or more parameters based on the comparing of the select prediction value to the expected value. (Huang sec. 2 “We minimize the following loss function L(x,y) to learn all the TabTransformer parameters in an end-to-end learning by the first-order gradient methods….”)
Cheng teaches claim 26. The computer-storage medium of claim 23, wherein the set of initial processing operations is configured to generate the select plurality of intermediate field values by replacing, in the select plurality of intermediate field values, a set of original numerical values of a set of randomly-selected numerical fields of the select row with a set of masked values. (Cheng para 48 “Feature-Tokenizer Transformer technique is described in Gorishniy, Yury, et al. “Revisiting deep learning models for tabular data.” Advances in Neural Information Processing Systems 34 (2021): 18932-18943, the contents of which are incorporated herein by reference in its entirety. Accordingly, randomly corrupt entries of table samples with values draw according to its marginal probability, and use multilevel perception (MLP) decoders to learn by comparison and denoising. Thus, as illustrated in FIG. 4 , an encoder 408 such as the Feature-Tokenizer Transformer takes an original tabular input 402 that is randomly corrupted 404, and the output from the encoder 408 is decoded by a decoder 410 in an effort to reconstruct the original tabular data 412.” The randomly corrupt values are the set of masked values.)
Cheng teaches claim 27. The computer-storage medium of claim 26, wherein the set of prediction values comprises a given prediction value corresponding to a randomly-selected numerical field of the set of randomly-selected numerical fields, and wherein the adjusting of the one or more parameters based on the set of prediction values comprises:
comparing the given prediction value to an original numerical value of the randomly-selected numerical field; and (Cheng para 48 “Feature-Tokenizer Transformer technique is described in Gorishniy, Yury, et al. “Revisiting deep learning models for tabular data.” Advances in Neural Information Processing Systems 34 (2021): 18932-18943, the contents of which are incorporated herein by reference in its entirety. Accordingly, randomly corrupt entries of table samples with values draw according to its marginal probability, and use multilevel perception (MLP) decoders to learn by comparison and denoising. Thus, as illustrated in FIG. 4 , an encoder 408 such as the Feature-Tokenizer Transformer takes an original tabular input 402 that is randomly corrupted 404, and the output from the encoder 408 is decoded by a decoder 410 in an effort to reconstruct the original tabular data 412.”)v
adjusting the one or more parameters based on the comparing of the select prediction value to the original numerical value. (Cheng para 48 “Feature-Tokenizer Transformer technique is described in Gorishniy, Yury, et al. “Revisiting deep learning models for tabular data.” Advances in Neural Information Processing Systems 34 (2021): 18932-18943, the contents of which are incorporated herein by reference in its entirety. Accordingly, randomly corrupt entries of table samples with values draw according to its marginal probability, and use multilevel perception (MLP) decoders to learn by comparison and denoising.”)
Cheng teaches claim 28. The computer-storage medium of claim 23, wherein the feature tokenizer and the transformer are part of a feature-tokenizer transformer (FTT). (Cheng para 15 “specific encoder for tabular data modality by performing Feature-Tokenizer Transformer.”)
Huang teaches claim 29. The computer-storage medium of claim 23, wherein the data table is a first data table, and wherein the operations comprise:
after the training of the transformer based on the set of prediction values, performing an inference operation using the transformer, the inference operation comprising: (Huang sec. 2 “The contextual embed dings explained above are learned in end-to-end supervised training using labeled examples. For a scenario, when there are a few labeled examples and a large number of unlabeled examples, we introduce a pre-training procedure to train the Transformer layers using unlabeled data. This is followed by fine-tuning of the pre-trained Transformer layers along with the top MLP layer using the labeled data.”)
accessing input data that comprises a second data table, the second data table comprising a current row, the current row comprising the plurality of fields; (Huang table 1 p. 4 shows that tabtransformer is run on several tables of different datasets.)
generating, by the feature tokenizer, a current plurality of field token vectors that corresponds to a current plurality of original field values of the current row; (Huang sec 2 “Let eφi (xi) ∈ Rd for i ∈ {1,··· ,m} be the embedding of the xi feature, and Eφ(xcat) = {eφ1 (x1),··· ,eφm (xm)} be the set of embeddings for all the categorical features.” eφi and Eφ are the tokens.)
generating, by the transformer, a current plurality of transformer output values that corresponds to the current plurality of field token vectors; and (Huang sec. 2 “these parametric embeddings Eφ(xcat) are in putted to the first Transformer layer. The output of the first Transformer layer is inputted to the second layer Transformer, and so forth. Each parametric embedding is transformed into contextual embedding when outputted from the top layer Transformer…”)
Huang doesn’t have a mean pool layer.
However, Lozoya teaches generating, by a mean pool operation, a current row embedding value for the current row, the current row embedding value being generated based on the current plurality of transformer output values. (Lozoya para 33 “embedding zsent is defined as the mean-pooling result of the last transformer layer's output.”)
Huang, Lozoya and the claims are all transformer architectures. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to add Lozoya’s pooling layer because it’s a standard way to collapse a set of transformer embeddings into one fixed-dimension embedding – which transforms the output back into the single vector representing the row.
Huang teaches claim 30. The computer-storage medium of claim 29, wherein the second data table comprises a reference row, wherein the reference row comprises the plurality of fields, and wherein the operations comprise: (Huang table 1 p. 4 shows that tabtransformer is run on several tables of different datasets.)
after the performing of the inference operation: (Huang sec. 2 “RTD replaces the original feature by a random value of that feature. Here, the loss is minimized for a binary classifier that tries to predict whether or not the feature has been replaced.”)
Huang doesn’t teach a pooling layer.
However, Lozoya teaches generating a reference row embedding value for the reference row; and (Lozoya para 33 “embedding zsent is defined as the mean-pooling result of the last transformer layer's output.” Mean pooling result is the reference row generated after the prediction output of the transformer layer.)
Huang doesn’t teach a cosine similarity.
However, Cheng teaches determining a similarity between the reference row and the current row based on the reference row embedding value and the current row embedding value. (Cheng para 50 “K priors may selected as the sample with embeddings that are closest to the embeddings of the observed modality (i.e., the image modality), which may be measured by cosine similarity between embeddings of the observed modality eobserved.”)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Austin Hicks whose telephone number is (571)270-3377. The examiner can normally be reached Monday - Thursday 8-4 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AUSTIN HICKS/Primary Examiner, Art Unit 2142
Prosecution Insights