DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Remarks
2. Claims 1-20 have been examined and rejected. This is the first Office action on the merits.
Specification
3. The disclosure is objected to because of the following informalities: In [paragraph 35] of Applicant’s specification, it appears reference number ‘712’ was incorrectly referred to as “Positional embedding 712.” Examiner suggests changing “Positional embedding 712” to --Feature name embedding 712--.
Appropriate correction is required.
Claim Objections
4. Claims 1, 2, and 7 are objected to because of the following informalities:
a. On [line 3] of claim 1, Examiner suggests changing “to provided encoded” to --to provide encoded--.
b. Claim 2 recites the limitation “the transformer” in [line 1] of the claim. There is insufficient antecedent basis for this limitation.
c. Claim 7 recites the limitation “the one or more of the plurality of linear layers” in [line 1] of the claim. There is insufficient antecedent basis for this limitation.
d. On [line 2] of claim 7, Examiner suggests changing “an Lasso” to --a Lasso--.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
5. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
6. Claims 1-5 and 7-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1
6-1. Regarding claims 1-5 and 7-9, the claims are directed to a method (i.e. a process) Thus, each of the claims fall within one of the four statutory categories. Nonetheless, the claims fall within the judicial exception of an abstract idea.
Step 2A – Prong 1
6-2. Independent claim 1 recites steps that, under their broadest reasonable interpretations, cover a mathematical concept. The claim recites:
receiving tabular input data including categorical and continuous data types, at an encoder to vectorize, embed, and apply a reduction tensor to the tabular input data to provided encoded data sequences; and
passing the encoded data sequences to a plurality of self-attention layers and at least one cross-attention layer to provide an output, the plurality self-attention layers and the at least one cross-attention layer each including a plurality of linear layers, multi-head attention, a plurality of low dropout functions, and a non-linear activation, at least one of the plurality of linear layers arranged to receive the encoded data sequences and pass the data to the multi-head attention, the multi-head attention applying a masking function, a softmax function, and a high dropout at each head, the high dropout being greater than 0.1 and greater than dropout of the plurality of low dropout functions.
The bolded portions of limitations above recite mathematical concepts involving conversion of one form of data into a numerical array, multiplication, exponentiation, summation, and division. Thus, the claim recites mathematical concepts, which fall within the “mathematical concept” group of abstract ideas.
Step 2A - Prong 2
6-3. This judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of “receiving tabular input data including categorical and continuous data types, at an encoder”. This additional element amounts to insignificant extra-solution activity, i.e., the mere gathering of information. See MPEP 2106.05(g). Accordingly, the additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim does not recite any particular technological environment, any improvement to computer functionality, any transformation of an article, or any specific technical application of the calculated output.
Step 2B
6-4. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of receiving tabular input data amounts to insignificant extra-solution activity, i.e., the mere gathering of information. See MPEP 2106.05(g). Therefore, the claim is not patent eligible.
6-5. Dependent claims 2-5, 7, and 9 recite the same abstract idea as independent claim 1. No further limitations in the dependent claims integrate the judicial exception into a practical application. The dependent claims also do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
6-6. As per dependent claim 8, the claim recites the additional limitation, “wherein a penalty is applied to an output of the softmax function.” This limitation also constitutes a mathematical concept because it represents a mathematical adjustment of computed results. Dependent claim 8 does not include additional elements that integrate the judicial exception into a practical application or that amount to significantly more than the judicial exception.
Step 1
6-7. Regarding claims 10-17, the claims are directed to a method (i.e. a process). Thus, each of the claims fall within one of the four statutory categories. Nonetheless, the claims fall within the judicial exception of an abstract idea.
6-8. Independent claim 10 recites steps that, under their broadest reasonable interpretations, cover a mathematical concept. The claim recites:
feeding tabular input data to a data encoder, the data encoder vectorizing the tabular input data to provide encoded input data X0;
feeding the encoded input data X0 to a stack of self-attention layers, each self-attention layer having multi-head attention with heads corresponding to linear layers Q, K, V, each head applying a modified softmax function and a dropout of greater than 0.1, the stack of self-attention layers providing a self-attention output XN;
feeding the encoded input data X0 through an input masking module to provide an output Z0;
feeding Z0 and the self-attention output XN to a cross-attention layer, the cross-attention layer having multi-head attention with heads corresponding to linear layers Q, K, V, each head applying a modified softmax function and a dropout of greater than 0.1, where Xo is passed through Q of the cross-attention layer, and XN is passed through K and V of the cross-attention layer to provide a final output; and
making a regression-based prediction derived from the self-attention output XN and/or the final output.
The bolded portions of limitations above recite mathematical concepts involving conversion of one form of data into a numerical array, multiplication, exponentiation, summation, division, and statistical analysis. Thus, the claim recites mathematical concepts, which fall within the “mathematical concept” group of abstract ideas.
Step 2A - Prong 2
6-9. This judicial exception is not integrated into a practical application. No further limitations in the dependent claims integrate the judicial exception into a practical application. The claim does not recite any particular technological environment, any improvement to computer functionality, any transformation of an article, or any specific technical application of the calculated output.
Step 2B
6-10. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Therefore, the claim is not patent eligible.
6-11. Dependent claims 11 and 13-14 recite the same abstract idea as independent claim 10. No further limitations in the dependent claims integrate the judicial exception into a practical application. The dependent claims also do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
6-12. As per dependent claim 12, the claim recites the additional limitation, “wherein the data encoder applies a reduction tensor to remove undefined values from the tabular input data.” This limitation also constitutes a mathematical concept because it requires computing values from multiple elements into a summary result. Dependent claim 12 does not include additional elements that integrate the judicial exception into a practical application or that amount to significantly more than the judicial exception.
6-13. As per dependent claim 15, the claim recites the additional limitation, “wherein a mask M is applied to represent all activity of current manufacturing stations and all activity of previous manufacturing stations.” This limitation also constitutes a mathematical concept. Dependent claim 15 does not include additional elements that integrate the judicial exception into a practical application or that amount to significantly more than the judicial exception.
6-14. As per dependent claim 16, the claim recites the additional limitation, “wherein first masks are applied via the stack of self-attention layers and a second mask is applied via the cross-attention layer, the first masks being different from the second mask.” This limitation also constitutes a mathematical concept. Dependent claim 16 does not include additional elements that integrate the judicial exception into a practical application or that amount to significantly more than the judicial exception.
6-15. As per dependent claim 17, the claim recites the additional limitation, “wherein a L1-L2 penalty is applied to an output of the modified softmax function.” This limitation also constitutes a mathematical concept because it represents a mathematical adjustment of computed results. Dependent claim 17 does not include additional elements that integrate the judicial exception into a practical application or that amount to significantly more than the judicial exception.
Step 1
6-16. Regarding claims 18-20, the claims are directed to a non-transitory computer-readable medium (i.e. a machine). Thus, each of the claims fall within one of the four statutory categories. Nonetheless, the claims fall within the judicial exception of an abstract idea.
Step 2A – Prong 1
6-17. Independent claim 18 recites steps that, under their broadest reasonable interpretations, cover a mathematical concept. The claim recites:
18. A non-transitory computer-readable medium having computer-readable instructions stored thereon, the computer-readable instructions operable by a processor to normalize a dataset, the instructions operable to perform functions of:
apply multi-head attention with heads corresponding to linear layers Q, K, V, each head applying a softmax function to provide a softmax output, applying a penalty to the softmax output, and applying further regularization after applying the softmax function.
The bolded portions of limitations above recite mathematical concepts involving adjusting values measured on different scales to a notionally common scale, multiplication, exponentiation, summation, division, and a mathematical adjustment of computed results. Thus, the claim recites mathematical concepts, which fall within the “mathematical concept” group of abstract ideas.
Step 2A - Prong 2
6-18. This judicial exception is not integrated into a practical application. In particular, the claim recites, “A non-transitory computer-readable medium having computer-readable instructions stored thereon, the computer-readable instructions operable by a processor… the instructions operable to perform functions of:…” The non-transitory computer-readable medium and processor are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using a generic computer component. Accordingly, the additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim does not recite any particular technological environment, any improvement to computer functionality, any transformation of an article, or any specific technical application of the calculated output.
Step 2B
6-19. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of the non-transitory computer-readable medium and processor amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Therefore, the claim is not patent eligible.
6-20. Dependent claims 19-20 recite the same abstract idea as independent claim 18. No further limitations in the dependent claims integrate the judicial exception into a practical application. The dependent claims also do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Claim Rejections - 35 USC § 103
7. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
8. Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Choudhury et al (Pub. No. US 2024/0127044) in view of Cai et al (U.S. Patent No. 12,629,839).
8-1. Regarding claim 18, Choudhury teaches a non-transitory computer-readable medium having computer-readable instructions stored thereon, the computer-readable instructions operable by a processor to normalize a dataset, by disclosing a data processing system for implementing an attention-based neural network in a hardware accelerator 200 (NNA) [paragraph 159; figure 4] comprising a local response normalize (LRN) unit [paragraph 142; figure 2].
Choudhury teaches the instructions operable to perform functions of: apply multi-head attention with heads corresponding to linear layers Q, K, V, each head applying a softmax function to provide a softmax output, by disclosing feeding inputs into the first layer of an encoder stack that comprises a multi-head attention block 110 executing self-attention on the inputs using scaled dot-product attention (SDPA) blocks 112-1 to 112-n [paragraph 113; figure 1A]. Inputs to the SDPA calculation are labeled Q, K, and V [paragraph 126; figure 1C] and a Softmax function is applied within each head [paragraph 133].
Choudhury does not expressly teach applying a penalty to the softmax output, and applying further regularization after applying the softmax function. Cai discloses making predictions using a transformer neural network model [column 11, lines 30-35] that uses Softmax regression [column 11, lines 50-59] and applies a classification entropy loss function [column 12, lines 4-10] and hierarchical dependence loss [column 12, lines 26-44]. This would increase computational efficiency, thus enabling the model to be trained faster. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply loss functions, as taught by Cai. This would increase computational efficiency, thus enabling the model to be trained faster.
9. Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Choudhury et al (Pub. No. US 2024/0127044), in view of Cai et al (U.S. Patent No. 12,629,839), and further in view of Miller (“Attention Is Off By One,” July 24, 2023).
9-1. Regarding claim 19, Choudhury-Cai teach all the limitations of claim 18. Choudhury-Cai do not expressly teach wherein the softmax function is represented by formula (5):
S
o
f
t
m
a
x
1
x
=
e
x
1
+
∑
e
x
(5).
Miller discloses using the softmax function represented by formula (5) [Pages 7-9, “Softmax One and Quiet Attention”]. This would reduce outlier attention weights and help make transformer models easier to compress and deploy. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the softmax function, as taught by Miller. This would reduce outlier attention weights and help make transformer models easier to compress and deploy.
10. Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Choudhury et al (Pub. No. US 2024/0127044) in view of Cai et al (U.S. Patent No. 12,629,839).
10-1. Regarding claim 20, Choudhury-Cai teach all the limitations of claim 18. Choudhury-Cai do not expressly teach wherein the further regularization is a dropout of greater than 0.3. Fedus discloses an attention neural network having a Transformer-based architecture [column 1, lines 42-47] that is trained to perform an initial task, and then fine-tuned on training data for the machine learning task [column 14, lines 57-59]. During fine-tuning, the system can regularize the training of the neural network using dropout by applying a first higher dropout rate within expert neural networks in one or more switch layers while applying a second lower dropout weight to nodes in layers of the neural network other than the switch layers [column 14, line 67 to column 15, line 11]. The lower dropout rate can be less than or equal to 0.1 while the higher dropout rate is greater than 0.1, e.g., 0.4 [column 15, lines 11-13]. This would help prevent over-fitting of the model [column 14, line 59 to column 15, line 2]. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply further regularization having a dropout greater than 0.3, as taught by Fedus. This would help prevent over-fitting of the model.
Allowable Subject Matter
11. Claims 1-17 would be allowable if rewritten or amended to overcome the rejections under 35 U.S.C. 101 set forth in this Office action.
Conclusion
12. Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALVIN H TAN whose telephone number is (571)272-8595. The examiner can normally be reached M-F 10AM-6PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Scott Baderman can be reached at 571-272-3644. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALVIN H TAN/Primary Examiner, Art Unit 2118
Prosecution Insights