DETAILED ACTION
The present application, filed on 5/20/2024 is being examined under the AIA first inventor to file provisions.
The following is a non-final First Office Action on the Merits. Claims 1-20 are pending and have been considered below.
Priority
This appln claims benefit of 63/527,294 07/17/2023. The priority is acknowledged.
Information Disclosure Statement (IDS)
The information disclosure statement (IDS) submitted on 5/20/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, such IDS is being considered by Examiner.
Claim Rejections - 35 USC § 101
35 USC 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 USC 101 because the claimed invention is not directed to patent eligible subject matter. The claimed matter is directed to a judicial exception, i.e. an abstract idea, not integrated into a practical application, and without significantly more.
Per Step 1 of the multi-step eligibility analysis, claims 1-7 are directed to a computer implemented method, claims 8-14 are directed to a system, and claims 15-20 are directed to computer executable instructions stored on a non-transitory storage medium.
Thus, on its face, each independent claim and the associated dependent claims are directed to a statutory category of invention.
[INDEPENDENT CLAIMS]
Per Step 2A.1. Independent claim 1, (which is representative of independent claims 8, 15) is rejected under 35 USC 101 because the independent claim is directed to an abstract idea, a judicial exception, without reciting additional elements that integrate the judicial exception into a practical application.
The limitations of the independent claim 1 (which is representative of independent claims 8, 15) recite an abstract idea, shown in bold below:
[A] A method comprising:
[B] accessing a plurality of weight matrices of a machine learning model;
and for each weight matrix:
[C] decomposing the weight matrix into a U matrix, an S matrix, and a V matrix using singular value decomposition,
[D] wherein the S matrix is a diagonal matrix and a singular group corresponds to each element in the S matrix;
[E] determining an importance score of each singular group,
[F] wherein the importance score of the singular group represents a change in loss if the singular group is removed from the machine learning model;
[G] ranking the singular groups across the plurality of weight matrices based on the importance scores; and
[H] identifying one or more of the singular groups to prune based on the ranking of the singular groups.
Independent claim 1 (which is representative of independent claims 8, 15) recites: decomposing a weight matrix and determining the importance of one of the matrix groups ([C], [E]); ranking the singular groups by importance ([G]); and pruning some of the singular groups ([H]), which, based on the claim language and in view of the application disclosure, represents a process aimed at: creating singular groups (sub-matrices) for transformer-based language models, ranking and pruning them.
This is a combination of operations that, under its broadest reasonable interpretation, covers performance of limitations expressing mathematical concepts like mathematical relationships, mathematical calculations. These fall under the Mathematical Concepts. i.e., mathematical relationships, mathematical formulas or equations, or mathematical calculations grouping of abstract ideas (see MPEP 2106.04(a)(2) I).
Accordingly, it is concluded that independent claim 1 (which is representative of independent claims 8, 15) recites an abstract idea that corresponds to a judicial exception.
[INDEPENDENT CLAIMS – Additional Elements]
Per Step 2A.2. The identified abstract idea is not integrated into a practical application because the additional elements in the independent claims only amount to instructions to apply the judicial exception to a computer, or are a general link to a technological environment (see MPEP 2106.05(f); MPEP 2106.05(h)).
For example, the added elements “a processing device” recite computing elements at a high level of generality, generally linking the use of a judicial exception to a particular technological environment (see MPEP 2106.05(h)), or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). Further, the additional elements “wherein the S matrix is a diagonal matrix and a singular group corresponds to each element in the S matrix”; “wherein the importance score of the singular group represents a change in loss if the singular group is removed from the machine learning model” as applied to the S matrix, and importance score, are nothing more than (a) descriptive limitations of claim elements, such as describing the nature, structure and/or content of other claim elements, or (b) general links to the computing environment, which amount to instructions to “apply it,” or equivalent (MPEP 2106.05(f)).
These additional elements of the independent claims do not preclude from carrying out the identified abstract idea creating singular groups (sub-matrices) for transformer-based language models, ranking and pruning them, and do not serve to integrate the identified abstract idea into a practical application.
The additional elements in the independent claims, shown not bolded above, recite: accessing weight matrices of a machine learning model ([B]). When considered individually, they amount to nothing more than receiving data, processing data, storing results or transmitting data that serves merely to implement the abstract idea using computing components for performing computer functions (corresponding to the words “apply it” or an equivalent), or merely uses a computer as a tool to perform the identified abstract idea. Thus, it is concluded that these claim elements do not integrate the identified abstract idea (creating singular groups (sub-matrices) for transformer-based language models, ranking and pruning them) into a practical application (see MPEP 2106.05(f)(2)).
Therefore, the additional claim elements of independent claim 1, (which is representative of independent claims 8, 15), evaluated individually, as well as a whole, as an ordered combination, do not integrate the identified abstract idea into a practical application and the claims are directed to the recited judicial exception.
Per Step 2B. Independent claim 1 (which is representative of claims independent 8, 15) does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when the independent claim is reevaluated as a whole, as an ordered combination under the considerations of Step 2B, the outcome is the same like under Step 2A.2.
Overall, it is concluded that independent claims 1, 8, 15 are deemed ineligible.
[DEPENDENT CLAIMS]
Dependent claim 3, which is representative of dependent claims 10, 16, recites:
updating the U matrix, the S matrix, and the V matrix to prune the identified one or more of the singular groups.
The elements in these dependent claims are comparable to receiving/transmitting data, processing data, storing results or transmitting data that serves merely to implement the abstract idea using computing components for performing computer functions (corresponding to the words “apply it” or an equivalent), or merely uses a computer as a tool to perform the identified abstract idea. Thus, it is concluded that these claim elements do not integrate the identified abstract idea (creating singular groups (sub-matrices) for transformer-based language models, ranking and pruning them) into a practical application (see MPEP 2106.05(f)(2)). When considered individually, these added claim elements further elaborate on the abstract idea identified in the independent claims, because the dependent claims continue to recite the identified abstract idea.
The dependent claims elements have the same relationship to the underlying abstract idea as outlined in the independent claims analysis above. It is readily clear that the dependent claim elements are not directed to any specific improvements of the independent claims and do not practically or significantly alter how the identified abstract idea would be performed. When considered as a whole, as an ordered combination, the dependent claims further elaborate on the previously identified abstract idea (creating singular groups (sub-matrices) for transformer-based language models, ranking and pruning them).
Therefore, dependent claim 1 (which is representative of dependent claims 8, 15) is deemed ineligible. As a result, it is concluded that the dependent claim elements do not integrate the identified abstract idea into a practical application (see MPEP 2106.05(f)(2)).
Dependent claims 2, 4-7, which are representative of dependent claims 9, 11-14, 17-20, respectively, recite:
wherein the machine learning model is a transformer model.
wherein each singular group includes: a column or row of the U matrix; an element of the S matrix; and a column or row of the V matrix.
wherein the importance score of the singular group is a sum of: a first importance score for the column or row of the U matrix; a second importance score for the element of the S matrix; and a third importance score for the column or row of the V matrix.
wherein the first importance score represents a change in loss if the respective column or row of the U matrix is removed; the second importance score represents a change in loss if the respective element of the S matrix is removed; and the third importance score represents a change in loss if the respective column or row of the V matrix is removed.
wherein the plurality of weight matrices includes one or more of: a query weight matrix; a key weight matrix; a value weight matrix; and a feedforward network weight matrix.
These further elements in the dependent claims do not perform any claimed method steps. They describe the nature, structure and/or content of other claim elements (in this instance – the machine learning model, the singular groups, the importance score, the weight matrices) and as such, cannot change the nature of the identified abstract idea (see MPEP 2106.07). The nature, form or structure of the other claim elements themselves do not practically or significantly alter how the identified abstract idea would be performed and do not provide more than a general link to a technological environment.
Therefore, dependent claims 2, 4-7, which are representative of dependent claims 9, 11-14, 17-20, respectively, are deemed ineligible.
When the dependent claims are considered as a whole, as an ordered combination, the claim elements noted above appear to merely apply the abstract concept to a technical environment in a very general sense. The most significant elements, which form the abstract concept, are set forth in the independent claims. The fact that the computing devices and the dependent claims are facilitating the abstract concept is not enough to confer statutory subject matter eligibility, since their individual and combined significance do not transform the identified abstract concept at the core of the claimed invention into eligible subject matter. Therefore, it is concluded that the dependent claims of the instant application, considered individually, or as a as a whole, as an ordered combination, do not amount to significantly more (see MPEP 2106.07(a)II).
In sum, claims 1-20 are rejected under 35 USC 101 as being directed to non-statutory subject matter.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the difference between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103(a) are summarized as follows:
i. Determining the scope and contents of the prior art.
ii. Ascertaining the differences between the prior art and the claims at issue.
iii. Resolving the level of ordinary skill in the pertinent art.
iv. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-3, 7-10, 14-16, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Hsu et al (US 2023/0106213).
Regarding Claims 1, 8, 15: Hsu first embodiment discloses: A method comprising:
accessing a plurality of weight matrices of a machine learning model;
and for each weight matrix: {see at least fig2A, fig2B, [0049] U matrix, V matrix, S matrix, weighted machine learning}
decomposing the weight matrix into a U matrix, an S matrix, and a V matrix using singular value decomposition, {see at least fig2A, fig2B, rc202, [0049] decomposed in three matrices; singular values decomposition (SVD)}
wherein the S matrix is a diagonal matrix and a singular group corresponds to each element in the S matrix; {see at least fig2, rc206, [0050] diagonal matrix; fig2B, [0062] S is a diagonal matrix)}
identifying one or more of the singular groups to prune based on the ranking of the singular groups. {see at least fig2, rc254-rc258, [0060] reducing the size (reads on pruning)}
Hsu first embodiment does not disclose, however, Hsu second embodiment discloses:
determining an importance score of each singular group, {see at least [abstract] importance values (reads on importance score); [0005]-[0007] [0033]; [0052]-[0053]}
wherein the importance score of the singular group represents a change in loss if the singular group is removed from the machine learning model; {see at least [0033] importance score (reads on weight value); smaller machine learning models (reads on reads on change in loss)}
ranking the singular groups across the plurality of weight matrices based on the importance scores; and {see at least [0033] low-ranked factorization; importance values}
It would have been obvious to one of ordinary skill in the art, at the time of filing, to modify Hsu first embodiment to include the elements of Hsu second embodiment. One would have been motivated to do so, in order to reduce the size of the transformer-based language model. Furthermore, the Supreme Court has supported that combining well known prior art elements, in a well-known manner, to obtain predictable results is sufficient to determine an invention obvious over such combination (see KSR International Co. v. Teleflex Inc. (KSR), 550 U.S.,82 USPQ2d 1385 (2007) & MPEP 2143). In the instant case, Hsu first embodiment evidently discloses reducing (pruning) sub-matrices (singular groups) of a transformer-based language model. Hsu second embodiment is merely relied upon to illustrate the functionality of determining sub-matrices (singular groups) and ranking them in the same or similar context. Since both reducing (pruning) sub-matrices (singular groups) of a transformer-based language model, as well as determining sub-matrices (singular groups) and ranking them in the same or similar context are implemented through well-known computer technologies in the same or similar context, combining their features as outlined above using such well-known computer technologies (i.e., conventional software/hardware configurations), would be reasonable, according to one of ordinary skill in the art. Moreover, since the elements disclosed by Hsu first embodiment, as well as Hsu second embodiment would function in the same manner in combination as they do in their separate embodiments, it is concluded that their resulting combination would be predictable. Accordingly, the claimed subject matter is obvious over Hsu. **Examiner notes that the reference is being used here as a one-reference combination in this 103 rejection because the reference teaches two clearly different embodiments within the same cited reference.**
Regarding Claims 2, 9: Hsu discloses the limitations of Claims 1, 8. Hsu further discloses:
wherein the machine learning model is a transformer model. {see at least [0064]-[0065] transformer model}
Regarding Claims 3, 10, 16: Hsu discloses the limitations of Claims 1, 8, 15. Hsu further discloses:
updating the U matrix, the S matrix, and the V matrix to prune the identified one or more of the singular groups. {see at least [0061] updating the matrices. The claim element “to prune the identified one or more of the singular groups” consists entirely of language disclosing at most a reason to have performed earlier method steps (intended use or field of use), but does not affect the functions in a manipulative sense (see MPEP 2103 I C) and imparts neither structure nor functionality to the claimed method (see MPEP 2111.05, MPEP 2114 and authorities cited therein), so it is considered but given no patentable weight. The reference is provided for the purpose of compact prosecution.}
Regarding Claims 7, 14, 20: Hsu discloses the limitations of Claims 1, 8, 15. Hsu further discloses:
wherein the plurality of weight matrices includes one or more of: a query weight matrix; a key weight matrix; a value weight matrix; and a feedforward network weight matrix. {see at least [0033] weighted low-ranked factorization (reads on query weight matrix); [0037], [0047], [0049], [0057]}
Examiner Remarks
Claims 4-6, 11-13, 17-19 are objected to as being dependent upon rejected base claims, but would be allowable if rewritten in independent form including all of the limitations of the base claims and any intervening claims.
The prior art made of record and not relied upon which, however, is considered pertinent to applicant's disclosure:
US 20240372754 A1 Åhlander; Mats et al. ROBUST PORT SELECTION ystems and methods for port selection in a wireless communication system are disclosed. In embodiment, a method performed by a radio access network (RAN) node for mapping Sounding Reference Signal (SRS) ports to transmission layers comprises obtaining a channel matrix, H, for one subcarrier or a group of subcarriers for a particular User Equipment (UE) and transforming the channel matrix, H, using a Singular Value Decomposition (SVD) of the channel matrix to thereby provide a transformed channel matrix. The method further comprises computing beamforming weights using the transformed channel matrix. Embodiments of a RAN node are also disclosed.
US 20140358565 A1 Peters; Nils Gunther et al. COMPRESSION OF DECOMPOSED REPRESENTATIONS OF A SOUND FIELD In general, techniques are described for obtaining decomposed versions of spherical harmonic coefficients. A device comprising one or more processors may be configured to perform the techniques, whereby the processors may be configured to obtain, from a bitstream, at least one of one or more vectors decomposed from spherical harmonic coefficients that were recombined with background spherical harmonic coefficients, wherein the spherical harmonic coefficients describe a sound field, and wherein the background spherical harmonic coefficients described one or more background components of the same sound field.
US 9495968 B2 Sen; Dipanjan et al. Identifying sources from which higher order ambisonic audio data is generated In general, techniques are described for obtaining an indication of whether spherical harmonic coefficients are representative of a synthetic audio object. In accordance with the techniques, a device comprising one or more processors may be configured to obtain an indication of whether spherical harmonic coefficients representative of a sound field are generated from a synthetic audio object.
US 20200293864 A1 NAGEL; Markus et al. DATA-AWARE LAYER DECOMPOSITION FOR NEURAL NETWORK COMPRESSION Certain aspects of the present disclosure are directed to methods and apparatus for operating an artificial neural network using data-aware layer decomposition. One exemplary method generally includes receiving a first input signal at a first layer of the artificial neural network; generating a first output signal of the first layer based, at least in part, on a weight matrix of the first layer and the first input signal; decomposing the weight matrix; generating an approximate output signal of the first layer based, at least in part, on the decomposed weight matrix and the first input signal; generating an updated decomposed weight matrix by minimizing a difference between the generated first output signal of the first layer and the approximate output signal of the first layer; and operating the first layer of the artificial neural network using the updated decomposed weight matrix.
US 20230057387 A1 KAMALAKARA; Siddhartha Rao et al. System and Method for Low Rank Training of Neural Networks A method of training a neural network model and related systems are disclosed. The method includes training the neural network model by factorising, based on a singular value decomposition scheme, a first plurality of nodes of the neural network model into a low rank neural network model comprising a second plurality of nodes. Each node of the second plurality of nodes is defined at least in part by at least one weight matrix, and the factorisation is based on a matrix decomposition scheme constrained by one or more directionality criteria.
US 20170161814 A1 Islam; Atiq et al. DISCOVERING PRODUCTS IN ITEM INVENTORY In various example embodiments, a system and method for discovering products in an item inventory are presented. The system receives a corpus of item information listings respectively describing items that are categorized in the same category and including titles but no product identifiers. The system generates a plurality of candidate phrases based on the plurality of titles. The system prunes insignificant phrases from the plurality of candidate phrases to identify a plurality of pruned candidate phrases. The system matches each of the titles to a pruned candidate phrase based on the significance information to identify matched pruned candidate phrases. The matching includes identifying a longest pruned candidate phrase that matches each of the titles. The system stores matched pruned candidate phrases as qualified product titles in the listings to generate a productized corpus of item information and communicates the productized corpus of item information to the sender.
US 20150127354 A1 Peters; Nils Gunther et al. NEAR FIELD COMPENSATION FOR DECOMPOSED REPRESENTATIONS OF A SOUND FIELD In general, techniques are described for compressing higher order ambisonics (HOA) audio data. A device comprising one or more processors may be configured to perform the techniques. The one or more processors may be configured to obtain a plurality of spherical harmonic coefficients from a plurality of near field compensated spherical harmonic coefficients by, at least in part, counterbalancing application of a near field compensation filter to the plurality of spherical harmonic coefficients.
US 20210005182 A1 Han; Kyu Jeong et al. MULTISTREAM ACOUSTIC MODELS WITH DILATIONS Audio signals of speech may be processed using an acoustic model. An acoustic model may be implemented with multiple streams of processing where different streams perform processing using different dilation rates. For example, a first stream may process features of the audio signal with one or more convolutional neural network layers having a first dilation rate, and a second stream may process features of the audio signal with one or more convolutional neural network layers having a second dilation rate. Each stream may compute a stream vector, and the stream vectors may be combined to a vector of speech unit scores, where the vector of speech unit scores provides information about the acoustic content of the audio signal. The vector of speech unit scores may be used for any appropriate application of speech, such as automatic speech recognition.
Inquiries
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Radu Andrei whose telephone number is 313.446.4948. The examiner can normally be reached on Monday – Friday 8:30am – 5pm EST. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Hayes can be reached at 571.272.6708. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http:/www.uspto.gov/interviewpractice.
As disclosed in MPEP 502.03, communications via Internet e-mail are at the discretion of the applicant. Without a written authorization by applicant in place, the USPTO will not respond via Internet e-mail to any Internet correspondence which contains information subject to the confidentiality requirement as set forth in 35 U.S.C. 122. A paper copy of such correspondence will be placed in the appropriate patent application. The following is a sample authorization form which may be used by applicant:
“Recognizing that Internet communications are not secure, I hereby authorize the USPTO to communicate with me concerning any subject matter of this application by electronic mail. I understand that a copy of these communications will be made of record in the application file.”
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center information webpage. Status information for unpublished applications is available to registered users through Patent Center information webpage only.
To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov.
Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (in USA or CANADA) or 571-272-1000.
Any response to this action should be mailed to:
Commissioner of Patents and Trademarks
P.O. Box 1450
Alexandria, VA 22313-1450
or faxed to 571-273-8300
/Radu Andrei/
Primary Examiner, AU 3697
Prosecution Insights