DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
2. The information disclosure statement (IDS) submitted on 03/14/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Priority Acknowledgment
3. Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in Application KR10-2024-0056423 filed on 04/29/2024 and KR10-2024-0118290 filed on 09/02/2024 in the Korea Intellectual Property Office.
Claim Rejections - 35 USC § 101
4. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
5. Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
All of the claims are directed towards the statutory category of a machine/apparatus or process.
Claim 1 recites
“1. A method for extending a context window, the method being performed by a computing device, the method comprising:
performing a rotary position embedding computation on a first embedding vector and a second embedding vector input to an artificial intelligence model to calculate a first position embedding and a second position embedding respectively corresponding to the first embedding vector and the second embedding vector;
calculating a self-attention score between the first embedding vector and the second embedding vector, based on the first position embedding and the second position embedding;
performing position interpolation on the self-attention score in order to extend a context window of the artificial intelligence model;
calculating a decay weight related to a relative position difference between the first embedding vector and the second embedding vector; and
applying the calculated decay weight to the self-attention score subjected to the position interpolation, thereby updating the self-attention score.” as recited in Claim 1.
The independent claims 1, 9 and 16 recite substantially the same concept but do so in the context of a method, a device and a non-transitory computer-readable recording medium.
The limitations recited in the independent claims as drafted covers mathematical concepts. More specifically, it relates to calculating a first position embedding and a second position embedding of a first embedding vector and a second embedding vector, calculating a self-attention score between the first embedding vector and the second embedding vector based on the first position embedding and the second position embedding, performing position interpolation on the self-attention score, calculating a decay weight and finally updating the self-attention score.
The judicial exception is not integrated into a practical application. In particular, the independent claims 9 and 16 recite additional elements of “a processor”, “a memory for storing therein instructions”, “a non-transitory computer-readable recording medium…executed by the computing device”. The additional element(s) or combination of elements such as a processor, a memory and a non-transitory computer-readable recording medium in the claim(s) other than the abstract idea per se amount(s) to no more than (i) mere instructions to implement the idea on a computer, and/or (ii) recitation of generic computer structure that serves to perform generic computer functions that are well-understood, routine, and conventional activities previously known to the pertinent industry. Viewed as a whole, these additional claim element(s) do not provide meaningful limitation(s) to transform the abstract idea into a patent eligible application of the abstract idea such that the claim(s) amounts to significantly more than the abstract idea itself. Therefore, the claim(s) are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter. There is no further improvement to the computing device other than updating the self-attention score. The mere recitation of a processor, a memory and non-transitory computer-readable recording medium and/or the like is akin of adding the word “apply it” and/or “use it” with a computer in conjunction with the abstract idea. The paragraphs [0087 and 0090] of the specification disclose “[0087] The processor 510 may control an operation of each of the components of the computing device 500. The processor 510 may include at least one of a central processing unit (CPU), a micro processor unit (MPU), a micro controller unit (MCU), a graphic processing unit (GPU), or any type of processor well known in the technical field of the present disclosure. In addition, the processor 510 may perform a computation on at least one application or program for executing an operation/method according to embodiments of the present disclosure. The computing device 500 may include one or more processors, [0088] Next, the memory 520 may store various data, commands and/or information therein. The memory 520 may load therein the computer program 560 from the storage 550 to execute an operation/method according to embodiments of the present disclosure. The memory 520 may be embodied as a volatile memory such as RAM. However, the present disclosure is not limited thereto, [0090] Next, the communication interface 540 may support wired/wireless Internet communication of the computing device 500. Further, the communication interface 540 may support various communication schemes other than Internet communication. To this end, the communication interface 540 may be configured to include a communication module well known in the technical field of the present disclosure. [0091] Next, the storage 550 may non-temporarily store therein one or more computer programs 560. The storage 550 may include a non-volatile memory, such as a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, or any type of computer-readable recording medium well known in the art to which the present disclosure pertains.”
As filed in the specification, the computer is listed as a general-purpose computer and are mainly used as an application thereof. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of using a computer is noted as a general computer. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claims are not patent eligible.
The dependent claims do not remedy the issues noted above. More specifically, claims 2, 10 and 17 relate to determining a scaling factor as a ratio between lengths and scaling the position difference based on the scaling factor. No additional limitations are presented. Claims 3, 11 and 18 relate to obtaining a rotation matrix and scaling a rotation angle of the rotation matrix. No additional limitations are presented. Claims 4, 12 and 19 define the decay weight. No additional limitations are presented. Claims 5, 13 and 20 relate to determining a hyperparameter of the decay weight. No additional limitations are presented. Claims 6 and 14 relate to determining the preset target ratio. No additional limitations are presented. Claims 7 and 15 relate to using the updated self-attention score. No additional limitations are presented. Claim 8 relates to defining the first embedding vector and the second embedding vector.
For at least the supra provided reasons, claims 1-20 are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter.
Claim Rejections - 35 USC § 102
6. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
7. Claims 1-3, 7-11, 15-18 are rejected under 35 U.S.C. 102(a) (1) as being anticipated by Peng et al. (‘YaRN: Efficient Context Window Extension of Large Language Models.’)
With respect to Claim 1, Peng et al. disclose
A method for extending a context window, the method being performed by a computing device, the method comprising:
performing a rotary position embedding computation on a first embedding vector and a second embedding vector input to an artificial intelligence model to calculate a first position embedding and a second position embedding respectively corresponding to the first embedding vector and the second embedding vector (Peng et al. Section 2.1 disclose performing a rotary position embedding computation to calculate first position embedding and second position embedding);
calculating a self-attention score between the first embedding vector and the second embedding vector, based on the first position embedding and the second position embedding (Peng et al. using equation (2) to calculate a self-attention score between the first embedding vector (e.g. query vector) and the second embedding vector (e.g., key vector));
performing position interpolation on the self-attention score in order to extend a context window of the artificial intelligence model (Peng et al. Section 2.2 disclose position interpolation to extend context window);
calculating a decay weight related to a relative position difference between the first embedding vector and the second embedding vector (Peng In equation (9), the relative distance (m-n) indicates difference between the first embedding vector and the second embedding vector); and
applying the calculated decay weight to the self-attention score subjected to the position interpolation, thereby updating the self-attention score (Peng Equations (10) and (12) shows the self-attention score is updated by applying the relative distance (m-n) in equation (9) based on the position interpolation in equation (10))).
With respect to Claim 2, Peng et al. disclose
wherein the performing of the position interpolation includes:
determining, as a scaling factor, a ratio of a second length of a context window as an extension target length to a first length of a context window initially preset on the artificial intelligence model (Peng et al. Section 2.3 Additional Notation, this section discloses a ratio between the extended context length and the original context length); and
scaling the relative position difference by the scaling factor (Peng et al. equation (11)).
With respect to Claim 3, Peng et al. disclose
wherein the performing of the position interpolation includes:
obtaining a rotation matrix corresponding to the self-attention score (Peng et al. Section 2.1 discloses the RoPE is written in a rotation matrix); and
scaling a rotation angle of the rotation matrix by a preset scaling factor (Peng et al. Equation 15 disclose scaling a rotation angle of the rotation matrix by a preset scaling factor).
With respect to Claim 7, Peng et al. disclose
further comprising fine-tuning the artificial intelligence model using the updated self-attention score (Peng et al. equation (12) updating
PNG
media_image1.png
30
100
media_image1.png
Greyscale
therefore updating self-attention score in order to fine-tune the large language model. See Section Introduction.)
With respect to Claim 8, Peng et al. disclose
wherein the first embedding vector is a query vector, and the second embedding vector is a key vector (Peng et al. Section 2.1 disclose the query vector and the key vector).
With respect to Claim 9, Claim 9 recites similar features as Claim 1, thus Claim 9 is rejected as the same ground as Claim 1.
With respect to Claim 10, Claim 10 recites similar features as Claim 2, thus Claim 10 is rejected as the same ground as Claim 2.
With respect to Claim 11, Claim 11 recites similar features as Claim 3, thus Claim 11 is rejected as the same ground as Claim 3.
With respect to Claim 15, Claim 15 recites similar features as Claim 7, thus Claim 15 is rejected as the same ground as Claim 7.
With respect to Claim 16, Claim 16 recites similar features as Claim 1, thus Claim 16 is rejected as the same ground as Claim 1.
With respect to Claim 17, Claim 17 recites similar features as Claim 2, thus Claim 17 is rejected as the same ground as Claim 2.
With respect to Claim 18, Claim 18 recites similar features as Claim 3, thus Claim 18 is rejected as the same ground as Claim 3.
Allowable Subject Matter
8. Claims 4-6, 12-14 and 19-20 are allowed in view of the prior art of record. The claims stand rejected under 101 Abstract idea, and for the application to pass to allowance this rejection need to be overcome. Any amendments to overcome the 101 rejection that results in any change in scope require further search and/or consideration in order to determine it allowability.
The following is a statement of reasons for the indication of allowable subject matter: the prior art(s) taken alone or in combination fail(s) to teach the following element(s) in combination with the other recited elements in the claim(s).
“wherein the decay weight is exponentially decreased as the relative position difference increases.” as recited in Claim 4.
“wherein a hyperparameter of the decay weight is determined such that a first amplitude of the self-attention score based on the relative position difference after the decay weight has been applied is reduced relative to a second amplitude of the self- attention score based on the relative position difference before the decay weight is applied by a preset target ratio.” As recited in Claim 5.
Claim 6 depends on Claim 5, thus Claim 6 is objected as the same ground by virtue of dependency.
Claims 12 and 19 recite similar features as Claim 4.
Claims 13 and 20 recite similar features as Claim 5.
Claim 14 recites similar features as Claim 6.
Conclusion
9. The prior art made of record and not relied upon is considered pertinent to application’s disclosure. See PTO-892.
a. Sengupta et al. (US 2024/0311559 A1.) In this reference, Sengupta et al. disclose a method and a system for extending the context window.
b. Liu et al. (US 2026/0170257 A1.) In this reference, Liu et al. disclose a method and a system for extending a context window of a text length.
c. Vasylyev (US 2024/0412720 A1.) In this reference, Vasylyev et al. disclose a method and a system for expanding the context window.
10. Any inquiry concerning this communication or earlier communications from the examiner should be directed to THUYKHANH LE whose telephone number is (571)272-6429. The examiner can normally be reached Mon-Fri: 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew C. Flanders can be reached on 571-272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THUYKHANH LE/Primary Examiner, Art Unit 2655
Prosecution Insights