DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged. Applicant has not complied with one or more conditions for receiving the benefit of an earlier filing date under 35 U.S.C. 120 as follows:
The later-filed application must be an application for a patent for an invention which is also disclosed in the prior application (the parent or original nonprovisional application or provisional application). The disclosure of the invention in the parent application and in the later-filed application must be sufficient to comply with the requirements of 35 U.S.C. 112(a) or the first paragraph of pre-AIA 35 U.S.C. 112, except for the best mode requirement. See Transco Products, Inc. v. Performance Contracting, Inc., 38 F.3d 551, 32 USPQ2d 1077 (Fed. Cir. 1994).
The disclosure of the prior-filed application, Provisional Application No. 63/326,764 dated 4/1/2022, fails to provide adequate support or enablement in the manner provided by 35 U.S.C. 112(a) or pre-AIA 35 U.S.C. 112, first paragraph for one or more claims of this application. Specifically, exemplary claim 22 recites a Gaussian error linear unit (GeLU) activation function, which is not found in the disclosure of the provisional application or parent application.
Accordingly, claims 22, 29, and 36 are not entitled to the benefit of the prior date of the provisional application or the parent application. Therefore, the priority date of these claims is 7/29/2024, which is the filing date of this application.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 21, 25-28, 32-35, 39-41 is/are rejected under 35 U.S.C. 103 as being unpatentable over Vaswani et al. (hereinafter Vaswani), Attention Is All You Need in view of Merrill et al. (hereinafter Merrill), Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent, further in view of Dey et al. (hereinafter Dey), U.S. Patent Application Publication 2022/0292266.
Regarding Claim 21, Vaswani discloses a system for performing a machine learning task on a network input to generate a network output, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement:
an attention neural network configured to perform the machine learning task, the attention neural network comprising a stack of a plurality of attention layers, each attention layer comprising an attention sub-layer and a feed-forward sub-layer [“The Transformer follows this overall architecture using stacked self-attention and point-wise, fully connected layers” §3 ¶2; Fig. 1], and each attention layer configured to:
receive an input sequence for the attention layer [“an input sequence” §3 ¶1; “Inputs” Fig. 1];
provide the input sequence to the attention sub-layer and to the feed-forward sub- layer [Fig. 1],
wherein the attention sub-layer is configured to generate an attended input sequence based at least in part on applying an attention mechanism to the input sequence by using one or more query linear transformation matrices, one or more key linear transformation matrices, and one or more value linear transformation matrices included in the attention sub-layer [“An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors.” §3.2 ¶1; Fig. 2; “we compute the attention function on a set of queries simultaneously, packed together into a matrix Q. The keys and values are also packed together into matrices K and V.” §3.2.1 ¶2], wherein the one or more query linear transformation matrices, the one or more key linear transformation matrices, and the one or more value linear transformation matrices each include respective weights [“parameter matrices WiQ є Rdmodelxdk ,WiK є Rdmodelxdk ,WiV є Rdmodelxdv and WO є Rhdvxdmodel” §3.2.1 ¶3] but no biases, and
wherein the feed-forward sub-layer configured to generate a transformed input sequence based on applying a sequence of transformations on the input sequence by using two or more weight matrices included in the feed-forward sub-layer [“In addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully connected feed-forward network” §3.3 ¶1; Fig. 1], wherein the two or more weight matrices each include respective weights [§3.3 equation 2] but no biases; and
generate the output sequence for the attention layer by determining a combination of the attended input sequence and the transformed input sequence [“We also use the usual learned linear transformation and softmax function to convert the decoder output to predicted next-token probabilities.” §3.4 ¶1; Fig. 1].
However, while a computer and memory are implicit in the execution of transformers, Vaswani fails to explicitly disclose the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers;
but no biases;
but no biases.
Merrill discloses the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers [“on NVIDIA A100 GPUs” §A ¶3];
but no biases [“the self attention and feed-forward sublayers have no bias terms” §D ¶1];
but no biases [“the self attention and feed-forward sublayers have no bias terms” §D ¶1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Vaswani and Merrill before him before the effective filing date of the claimed invention, to modify the system of Vaswani to incorporate using no biases of Merrill.
Given the advantage of removing a bias calculation to simplify the process since the sublayers are normalized, one having ordinary skill in the art would have been motivated to make this obvious modification.
However, Vaswani fails to explicitly disclose wherein the attention sub-layer and the feed-forward sub-layer are configured to receive a same input sequence in parallel.
Dey discloses wherein the attention sub-layer and the feed-forward sub-layer are configured to receive a same input sequence in parallel [“In contrast with a conventional encoder with multiple layers of a 2 sub-layer stack in sequential processing, the enhanced encoder of this disclosure operates with a single layer of a multi-head attention component in parallel with a feed forward network” §10; “Similar to enhanced encoder, enhanced decoder is enhanced by parallel configuration of multi-head attention components and feed-forward network” §14; Fig. 2].
It would have been obvious to one having ordinary skill in the art, having the teachings of Vaswani, Merrill, and Dey before him before the effective filing date of the claimed invention, to modify the combination to incorporate the attention sub-layer and the feed-forward sub-layer configured to receive a same input sequence in parallel of Dey.
Given the advantage of superior performance of the enhanced encoder with parallel integration demonstrating significantly reduced number of parameters related to neural network operations, which results in computation resources being greatly conserved and model learning accelerated, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 25, Vaswani, Merrill, and Dey disclose the system of claim 21. Vaswani further discloses wherein generating the transformed input sequence based on applying the sequence of transformations on the input sequence by using the two or more fully-connected weight matrices included in the feed-forward sub-layer comprises:
multiplying the input sequence by a weight matrix that represents weights of a first fully-connected layer to generate a product [“we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation, similar to [24]. In the embedding layers, we multiply those weights by p dmodel” §3.4 ¶1].
However, Vaswani fails to explicitly disclose adding no bias to the product.
Merrill discloses adding no bias to the product [“the self attention and feed-forward sublayers have no bias terms” §D ¶1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Vaswani, Merrill, and Dey before him before the effective filing date of the claimed invention, to modify the combination to incorporate using no biases of Merrill.
Given the advantage of removing a bias calculation to simplify the process since the sublayers are normalized, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 26, Vaswani, Merrill, and Dey disclose the system of claim 21. Vaswani further discloses wherein determining the combination of the attended input sequence and the transformed input sequence comprises:
computing a sum of the attended input sequence and the transformed input sequence [“the sums of the embeddings” §5.4 ¶2].
Regarding Claim 27, Vaswani, Merrill, and Dey disclose the system of claim 21. Vaswani further discloses wherein the attention mechanism comprises a multi-head self-attention mechanism [“Multi-Head Attention” §3.2.2; Fig. 1, 2].
Claim 28 is rejected on the same grounds as claim 21.
Claims 32-34 are rejected on the same grounds as claims 25-27 respectively.
Claim 35 is rejected on the same grounds as claim 21.
Claims 39-41 are rejected on the same grounds as claims 25-27 respectively.
Claim(s) 22, 29, 36 is/are rejected under 35 U.S.C. 103 as being unpatentable over Vaswani, Merrill, and Dey in view of So et al. (hereinafter So), Primer: Searching for Efficient Transformers for Language Modeling.
Regarding Claim 22, Vaswani, Merrill, and Dey disclose the system of claim 21. Vaswani further discloses wherein the sequence of transformations comprises a first linear transformation followed by a Gaussian error linear unit (GeLU) activation function followed by a second linear transformation [“This consists of two linear transformations with a ReLU activation in between.” §3.3 ¶1].
However, Vaswani fails to explicitly disclose wherein the sequence of transformations comprises a first linear transformation followed by a Gaussian error linear unit (GeLU) activation function followed by a second linear transformation.
So discloses wherein the sequence of transformations comprises a first linear transformation followed by a Gaussian error linear unit (GeLU) activation function [“the most commonly used activation functions: ReLU, GELU and Swish” §3 ¶3] followed by a second linear transformation.
It would have been obvious to one having ordinary skill in the art, having the teachings of Vaswani, Merrill, Dey, and So before him before the effective filing date of the claimed invention, to modify the combination to incorporate the GELU of So.
Given the advantage of substituting one known activation function for another, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claims 29 and 36 are rejected on the same grounds as claim 22.
Claim(s) 23-24, 30-31, 37-38 is/are rejected under 35 U.S.C. 103 as being unpatentable over Vaswani, Merrill, and Dey in view of Narang et al. (hereinafter Narang), Do Transformer Modifications Transfer Across Implementations and Applications?
Regarding Claim 23, Vaswani, Merrill, and Dey disclose the system of claim 21. Vaswain further discloses wherein the sequence of transformations comprises a first linear transformation followed by a Swish activation function followed by a second linear transformation [“This consists of two linear transformations with a ReLU activation in between.” §3.3 ¶1].
However, Vaswani fails to explicitly disclose wherein the sequence of transformations comprises a first linear transformation followed by a Swish activation function followed by a second linear transformation.
Narang discloses wherein the sequence of transformations comprises a first linear transformation followed by a Swish activation function [“The activation functions that we explored are: (1) GeLU (Hendrycks & Gimpel, 2016), (2) Swish (Ramachandran et al., 2017)” §3.1 ¶1] followed by a second linear transformation.
It would have been obvious to one having ordinary skill in the art, having the teachings of Vaswani, Merrill, Dey, and Narang before him before the effective filing date of the claimed invention, to modify the combination to incorporate the Swish activation function of Narang.
Given the advantage of a simple substitution of one activation function for another, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 24, Vaswani, Merrill, and Dey disclose the system of claim 21. Vaswani further discloses wherein the sequence of transformations comprises a first linear transformation followed by a SwiGLU activation function followed by a second linear transformation [“This consists of two linear transformations with a ReLU activation in between.” §3.3 ¶1].
However, Vaswani fails to explicitly disclose wherein the sequence of transformations comprises a first linear transformation followed by a SwiGLU activation function followed by a second linear transformation.
Narang discloses wherein the sequence of transformations comprises a first linear transformation followed by a SwiGLU activation function [“We also explore Gated Linear Unit" (GLU) variants” §3.1 ¶1; “We find that several activation functions improve performance over the ReLU activation used in the standard Transformer architecture. Specifically, SwiGLU and GeGLU improve performance” §4.1 ¶2] followed by a second linear transformation.
It would have been obvious to one having ordinary skill in the art, having the teachings of Vaswani, Merrill, Dey, and Narang before him before the effective filing date of the claimed invention, to modify the combination to incorporate the SwiGLU activation function of Narang.
Given the advantage of a simple substitution of one activation function for another and improved performance, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claims 30-31 are rejected on the same grounds as claims 23-24 respectively.
Claims 37-38 are rejected on the same grounds as claims 23-24 respectively.
Examiner’s Note
The Examiner respectfully requests of the Applicant in preparing responses, to fully consider the entirety of the reference(s) as potentially teaching all or part of the claimed invention. It is noted, REFERENCES ARE RELEVANT AS PRIOR ART FOR ALL THEY CONTAIN. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain.” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)). A reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art, including non-preferred embodiments (see MPEP 2123). The Examiner has cited particular locations in the reference(s) as applied to the claim(s) above for the convenience of the Applicant. Although the specified citations are representative of the teachings of the art and are applied to the specific limitations within the individual claim(s), typically other passages and figures will apply as well.
Additionally, any claim amendments for any reason should include remarks indicating clear support in the originally filed specification.
Response to Arguments
Regarding the priority of claims 22, 29, and 36, Applicant asserts that amended claims are supported in the provisional application but does not provide any citation to confirm such support. Examiner was unable to find support in both the provision and parent application. Accordingly, these claims are not entitled to the priority date of either earlier filed application.
Regarding the prior art rejections, Applicant's arguments with respect to the claims have been considered but are moot because the arguments do not apply to the references being used in the current rejection of the limitations.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT H BEJCEK II whose telephone number is (571)270-3610. The examiner can normally be reached Monday - Friday: 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T. Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/R.B./ Examiner, Art Unit 2148
/MICHELLE T BECHTOLD/ Supervisory Patent Examiner, Art Unit 2148