DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the original application filed on July 10th, 2024.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation, “generating, by the one or more processors, a balanced training partition by appending a portion of the domain-specific data partition to the initial training partition; and” (Emphasis added). The claim recites the generation of a training partition, however the term “balanced” is undefined and the combination of data partitions does not further disclose how a “balanced” data portion is achieved. Therefore, since the claim fails to further define the term “balanced”, the claim is rejected under 35 U.S.C. 112(b) for being indefinite. For examination purposes the claim element “balanced training partition” will be interpreted to mean, “a training partition containing a combination of domain specific data and enterprise data”. Claims 2-12 depend on rejected claim 1 and are also rejected under 35 U.S.C. 112(b) by virtue of this dependency. Appropriate correction is required.
Claim 13 recites the limitation, “generate a balanced training partition by appending a portion of the domain-specific data partition to the initial training partition; and” (Emphasis added). The claim recites the generation of a training partition, however the term “balanced” is undefined and the combination of data partitions does not further disclose how a “balanced” data portion is achieved. Therefore, since the claim fails to further define the term “balanced”, the claim is rejected under 35 U.S.C. 112(b) for being indefinite. For examination purposes the claim element “balanced training partition” will be interpreted to mean, “a training partition containing a combination of domain specific data and enterprise data”. Claims 14-19 depend on rejected claim 13 and are also rejected under 35 U.S.C. 112(b) by virtue of this dependency. Appropriate correction is required.
Claim 20 recites the limitation, “generate a balanced training partition by appending a portion of the domain-specific data partition to the initial training partition; and” The claim recites the generation of a training partition, however the term “balanced” is undefined and the combination of data partitions does not further disclose how a “balanced” data portion is achieved. Therefore, since the claim fails to further define the term “balanced”, the claim is rejected under 35 U.S.C. 112(b) for being indefinite. For examination purposes the claim element “balanced training partition” will be interpreted to mean, “a training partition containing a combination of domain specific data and enterprise data”. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis of the claims will follow the 2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50 (“2019 PEG”).
Claim 1
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Claim 1 recites, "A computer-implemented method comprising:" therefore it is directed to the statutory category of a process.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites, inter alia:
“generating, by the one or more processors, a balanced training partition by appending a portion of the domain-specific data partition to the initial training partition; and” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate two sets of data and apply mental functions to them to combine or append them. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “A computer-implemented method comprising:” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“receiving, by one or more processors, an enterprise data partition from a plurality of enterprise data partitions associated with an enterprise data source;” is an insignificant extra-solution activity required for any uses of the mental processes (see MPEP § 2106.05(g)) As such, the claim is ineligible.
“receiving, by the one or more processors, a domain-specific data partition from a plurality of domain-specific data partitions associated with one or more domain data sources that are different than the enterprise data source;” is an insignificant extra-solution activity required for any uses of the mental processes (see MPEP § 2106.05(g)) As such, the claim is ineligible.
“storing, by the one or more processors, the enterprise data partition as an initial training partition of a plurality of balanced training partitions within a balanced training dataset;” is an insignificant extra-solution activity required for any uses of the mental processes (see MPEP § 2106.05(g)) As such, the claim is ineligible.
“training, by the one or more processors, a domain-specific language model based on the balanced training dataset.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “A computer-implemented method comprising:” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“receiving, by one or more processors, an enterprise data partition from a plurality of enterprise data partitions associated with an enterprise data source;” is an insignificant extra-solution activity required for any uses of abstract ideas (see MPEP § 2106.05(g)), and is a well-understood, routine, conventional activity (see MPEP § 2106.05(d)(II)(i); “Receiving or transmitting data over a network, e.g., using the Internet to gather data”.
“receiving, by the one or more processors, a domain-specific data partition from a plurality of domain-specific data partitions associated with one or more domain data sources that are different than the enterprise data source;” is an insignificant extra-solution activity required for any uses of abstract ideas (see MPEP § 2106.05(g)), and is a well-understood, routine, conventional activity (see MPEP § 2106.05(d)(II)(i); “Receiving or transmitting data over a network, e.g., using the Internet to gather data”.
“storing, by the one or more processors, the enterprise data partition as an initial training partition of a plurality of balanced training partitions within a balanced training dataset;” is an insignificant extra-solution activity required for any uses of abstract ideas (see MPEP § 2106.05(g)), and is a well-understood, routine, conventional activity (see MPEP § 2106.05(d)(iv); “Storing and retrieving information in memory”.
“training, by the one or more processors, a domain-specific language model based on the balanced training dataset.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 2
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “wherein the plurality of balanced training partitions of the balanced training dataset respectively corresponds to the plurality of enterprise data partitions.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “wherein the plurality of balanced training partitions of the balanced training dataset respectively corresponds to the plurality of enterprise data partitions.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 3
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “wherein each of the plurality of balanced training partitions comprises a respective enterprise data partition and an equal portion of a respective domain-specific data partition.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “wherein each of the plurality of balanced training partitions comprises a respective enterprise data partition and an equal portion of a respective domain-specific data partition.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 4
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “wherein the enterprise data source comprises a plurality of private documents accessible to an enterprise within a prediction domain and the one or more domain data sources comprise a plurality of public documents that are publicly accessible to a plurality of enterprises within the prediction domain.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “wherein the enterprise data source comprises a plurality of private documents accessible to an enterprise within a prediction domain and the one or more domain data sources comprise a plurality of public documents that are publicly accessible to a plurality of enterprises within the prediction domain.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 5
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “wherein a size of the portion of the domain-specific data partition is based on a number of the plurality of public documents or a number of the plurality of enterprise data partitions.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “wherein a size of the portion of the domain-specific data partition is based on a number of the plurality of public documents or a number of the plurality of enterprise data partitions.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 6
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “wherein the plurality of enterprise data partitions comprises a plurality of first non-overlapping text sequences extracted from the plurality of private documents and the plurality of domain-specific data partitions comprises a plurality of second non-overlapping text sequences extracted from the plurality of public documents.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “wherein the plurality of enterprise data partitions comprises a plurality of first non-overlapping text sequences extracted from the plurality of private documents and the plurality of domain-specific data partitions comprises a plurality of second non-overlapping text sequences extracted from the plurality of public documents.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 7
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “wherein a size of the plurality of first non-overlapping text sequences and the plurality of second non-overlapping text sequences is defined by predefined sequence length.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “wherein a size of the plurality of first non-overlapping text sequences and the plurality of second non-overlapping text sequences is defined by predefined sequence length.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 8
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites, inter alia:
“the computer-implemented method further comprises modifying the balanced training dataset by rearranging a plurality of indexed positions of the plurality of balanced training partitions within the balanced training dataset.” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate a data structure such as indexed array and rearrange the indexes. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “wherein each of the plurality of balanced training partitions is stored at an indexed position within the balanced training dataset, and” is an insignificant extra-solution activity required for any uses of the mental processes (see MPEP § 2106.05(g)) As such, the claim is ineligible.
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “wherein each of the plurality of balanced training partitions is stored at an indexed position within the balanced training dataset, and” is an insignificant extra-solution activity required for any uses of the mental processes (see MPEP § 2106.05(g)) As such, the claim is ineligible. Further, storing training data in an indexed set is well-understood, routine and conventional as it is described in MPEP 2106.05(d) per Genix, (Genix, “Mastering Two-Dimensional Arrays: A Comprehensive Guide”, Jul. 2nd, 2024, Medium.com; “A two-dimensional array, on the other hand, is an array of arrays. Each element in a two-dimensional array is itself an array that can be accessed using two indices: one for the row and one for the column.”. This claim discloses the use of an indexed 2D array which is well understood in the art. This is an example of a website where someone new to programming would learn about different arrays and array manipulation.)
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 9
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “wherein a partition size of the balanced training partition is defined by a predefined hardware constraint.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “wherein a partition size of the balanced training partition is defined by a predefined hardware constraint.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 10
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “wherein the domain-specific language model comprises a bidirectional encoder representation from transformers model.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “wherein the domain-specific language model comprises a bidirectional encoder representation from transformers model.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 11
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “wherein the domain-specific language model is trained using continued masked language modelling based on the balanced training dataset.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “wherein the domain-specific language model is trained using continued masked language modelling based on the balanced training dataset.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 12
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A process, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites, inter alia:
“generating a byte-pair encoding subword for the balanced training dataset; and” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to identify and evaluate data to generate encoding pairs to train machine learning models. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “training the domain-specific language model based on the byte-pair encoding subword.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “training the domain-specific language model based on the byte-pair encoding subword.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 13
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Claim 13 recites, "A system comprising memory and one or more processors communicatively coupled to the memory, the one or more processors configured to:" therefore it is directed to the statutory category of a machine.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites, inter alia:
“generate a balanced training partition by appending a portion of the domain-specific data partition to the initial training partition; and” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate two sets of data and apply mental functions to them to combine or append them. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “A system comprising memory and one or more processors communicatively coupled to the memory, the one or more processors configured to:” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“receive an enterprise data partition from a plurality of enterprise data partitions associated with an enterprise data source;” is an insignificant extra-solution activity required for any uses of the mental processes (see MPEP § 2106.05(g)) As such, the claim is ineligible.
“receive a domain-specific data partition from a plurality of domain-specific data partitions associated with one or more domain data sources that are different than the enterprise data source;” is an insignificant extra-solution activity required for any uses of the mental processes (see MPEP § 2106.05(g)) As such, the claim is ineligible.
“store the enterprise data partition as an initial training partition of a plurality of balanced training partitions within a balanced training dataset;” is an insignificant extra-solution activity required for any uses of the mental processes (see MPEP § 2106.05(g)) As such, the claim is ineligible.
“train a domain-specific language model based on the balanced training dataset.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “A system comprising memory and one or more processors communicatively coupled to the memory, the one or more processors configured to:” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“receive an enterprise data partition from a plurality of enterprise data partitions associated with an enterprise data source;” is an insignificant extra-solution activity required for any uses of abstract ideas (see MPEP § 2106.05(g)), and is a well-understood, routine, conventional activity (see MPEP § 2106.05(d)(II)(i); “Receiving or transmitting data over a network, e.g., using the Internet to gather data”.
“receive a domain-specific data partition from a plurality of domain-specific data partitions associated with one or more domain data sources that are different than the enterprise data source;” is an insignificant extra-solution activity required for any uses of abstract ideas (see MPEP § 2106.05(g)), and is a well-understood, routine, conventional activity (see MPEP § 2106.05(d)(II)(i); “Receiving or transmitting data over a network, e.g., using the Internet to gather data”.
“store the enterprise data partition as an initial training partition of a plurality of balanced training partitions within a balanced training dataset;” is an insignificant extra-solution activity required for any uses of abstract ideas (see MPEP § 2106.05(g)), and is a well-understood, routine, conventional activity (see MPEP § 2106.05(d)(iv); “Storing and retrieving information in memory”.
“train a domain-specific language model based on the balanced training dataset.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 14
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A machine, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “wherein the plurality of balanced training partitions of the balanced training dataset respectively corresponds to the plurality of enterprise data partitions.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “wherein the plurality of balanced training partitions of the balanced training dataset respectively corresponds to the plurality of enterprise data partitions.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 15
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A machine, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “wherein each of the plurality of balanced training partitions comprises a respective enterprise data partition and an equal portion of a respective domain-specific data partition.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “wherein each of the plurality of balanced training partitions comprises a respective enterprise data partition and an equal portion of a respective domain-specific data partition.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 16
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A machine, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “wherein the enterprise data source comprises a plurality of private documents accessible to an enterprise within a prediction domain and the one or more domain data sources comprise a plurality of public documents that are publicly accessible to a plurality of enterprises within the prediction domain.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “wherein the enterprise data source comprises a plurality of private documents accessible to an enterprise within a prediction domain and the one or more domain data sources comprise a plurality of public documents that are publicly accessible to a plurality of enterprises within the prediction domain.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 17
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A machine, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “wherein a size of the portion of the domain-specific data partition is based on a number of the plurality of public documents or a number of the plurality of enterprise data partitions.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “wherein a size of the portion of the domain-specific data partition is based on a number of the plurality of public documents or a number of the plurality of enterprise data partitions.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 18
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A machine, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “wherein the plurality of enterprise data partitions comprises a plurality of first non-overlapping text sequences extracted from the plurality of private documents and the plurality of domain-specific data partitions comprises a plurality of second non-overlapping text sequences extracted from the plurality of public documents.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “wherein the plurality of enterprise data partitions comprises a plurality of first non-overlapping text sequences extracted from the plurality of private documents and the plurality of domain-specific data partitions comprises a plurality of second non-overlapping text sequences extracted from the plurality of public documents.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 19
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
A machine, as above.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites the abstract ideas of the preceding claims from which it depends.
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “wherein a size of the plurality of first non-overlapping text sequences and the plurality of second non-overlapping text sequences is defined by predefined sequence length.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “wherein a size of the plurality of first non-overlapping text sequences and the plurality of second non-overlapping text sequences is defined by predefined sequence length.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim 20
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Claim 20 recites, "One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processors, cause the one or more processors to:" therefore it is directed to the statutory category of a machine.
Step 2A Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The claim recites, inter alia:
“generate a balanced training partition by appending a portion of the domain-specific data partition to the initial training partition; and” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating and observing data, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. A human is able to evaluate two sets of data and apply mental functions to them to combine or append them. The limitation is merely applying an abstract idea on generic computer system. See MPEP 2106.04(a)(2)(III)(c).
Step 2A Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
The claim recites the additional elements, “One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processors, cause the one or more processors to:” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“receive an enterprise data partition from a plurality of enterprise data partitions associated with an enterprise data source;” is an insignificant extra-solution activity required for any uses of the mental processes (see MPEP § 2106.05(g)) As such, the claim is ineligible.
“receive a domain-specific data partition from a plurality of domain-specific data partitions associated with one or more domain data sources that are different than the enterprise data source;” is an insignificant extra-solution activity required for any uses of the mental processes (see MPEP § 2106.05(g)) As such, the claim is ineligible.
“store the enterprise data partition as an initial training partition of a plurality of balanced training partitions within a balanced training dataset;” is an insignificant extra-solution activity required for any uses of the mental processes (see MPEP § 2106.05(g)) As such, the claim is ineligible.
“train a domain-specific language model based on the balanced training dataset.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements, “One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processors, cause the one or more processors to:” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
“receive an enterprise data partition from a plurality of enterprise data partitions associated with an enterprise data source;” is an insignificant extra-solution activity required for any uses of abstract ideas (see MPEP § 2106.05(g)), and is a well-understood, routine, conventional activity (see MPEP § 2106.05(d)(II)(i); “Receiving or transmitting data over a network, e.g., using the Internet to gather data”.
“receive a domain-specific data partition from a plurality of domain-specific data partitions associated with one or more domain data sources that are different than the enterprise data source;” is an insignificant extra-solution activity required for any uses of abstract ideas (see MPEP § 2106.05(g)), and is a well-understood, routine, conventional activity (see MPEP § 2106.05(d)(II)(i); “Receiving or transmitting data over a network, e.g., using the Internet to gather data”.
“store the enterprise data partition as an initial training partition of a plurality of balanced training partitions within a balanced training dataset;” is an insignificant extra-solution activity required for any uses of abstract ideas (see MPEP § 2106.05(g)), and is a well-understood, routine, conventional activity (see MPEP § 2106.05(d)(iv); “Storing and retrieving information in memory”.
“train a domain-specific language model based on the balanced training dataset.” amounts to generic computer components used as a tool to perform an existing process. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer (see MPEP § 2106.05(f)).
Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 13-15, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zheng et al, (Zheng et al, “Fine-tuning Large Language Models for Domain-specific Machine Translation”, Feb. 23rd, 2024, hereinafter “Zheng”) in view of Nguyen et al, (Nguyen et al, “Combining datasets to improve model fitting”, May 16th, 2023, hereinafter “Nguyen”).
Regarding claim 1, Zheng discloses, “A computer-implemented method comprising:” (Introduction, pp. 2; “To address the challenges, this paper proposes a novel prompt-oriented fine-tuning method, denoted as LlamaIT, for fine-tuning a general-purpose LLM for domain-specific MT tasks.” This article discloses a training process for a large language model to be implemented on a computer.)
“storing, by the one or more processors, the enterprise data partition as an initial training partition of a plurality of balanced training partitions within a balanced training dataset;” (Zero-shot Prompting with Instructions, pp. 4; “Fine-tuning an LLM directly on the XFIT24-ORI dataset can adapt its MT capabilities to the specific domain. However, its effectiveness is reported to be limited for MT tasks as it can struggle to respond to translation instructions without instruction-tuning [Wei et al., 2022a]. Thus, the need arises to construct a task-specific dataset, that is described via natural language instructions.” The system proposed in this article is able to combine data from different sources and generate a mixed dataset to fine tune a pretrained language model. This system generates a labeled dataset, XFIT24-ORI, and stores it so it is able to generate the second dataset.)
“training, by the one or more processors, a domain-specific language model based on the balanced training dataset.” (Datasets, pp. 5; “For fine-tuning, we employ the self-constructed dataset XFIT24, which contains the IT domain translation data (In Section 3.1). It mainly consists of 10, 000 English- Chinese translation pairs from publicly available datasets, and 200, 000 self-constructed translation pairs. We use test datasets from two general-domain datasets of Flores-101 [Goyal et al., 2022] and OPUS-100 [Zhang et al., 2020], and two domain-specific datasets of IT [Koehn and Knowles, 2017] and XFIT24 for translation inference.” This system is able to use a generated training dataset by combining data with domain specific data. XFIT24 is the resulting training data and it is used to further train a language model.)
Zheng fails to explicitly disclose:
“receiving, by the one or more processors, a domain-specific data partition from a plurality of domain-specific data partitions associated with one or more domain data sources that are different than the enterprise data source;”
“receiving, by one or more processors, an enterprise data partition from a plurality of enterprise data partitions associated with an enterprise data source;”
“generating, by the one or more processors, a balanced training partition by appending a portion of the domain-specific data partition to the initial training partition; and”
However, Nguyen discloses, “receiving, by the one or more processors, a domain-specific data partition from a plurality of domain-specific data partitions associated with one or more domain data sources that are different than the enterprise data source;” (ComImp Algorithm for Combining Datasets, pp. 3; “This section details our ComImp framework for combining datasets. The general ComImp framework is shown in Algorithm 1. Suppose that we have datasets
D
1
=
X
1
,
y
1
,
…
,
D
r
=
{
X
r
,
y
r
}
. In addition, assume that the
(
p
1
+
i
)
t
h
column of
D
1
and
i
t
h
column of
D
2
refer to the same feature.” This article discloses the combination of multiple datasets from different sources. The domain specific data is interpreted to be
D
2
and a partition is
D
2
=
X
2
,
y
2
.)
“receiving, by one or more processors, an enterprise data partition from a plurality of enterprise data partitions associated with an enterprise data source;” (ComImp Algorithm for Combining Datasets, pp. 3; “This section details our ComImp framework for combining datasets. The general ComImp framework is shown in Algorithm 1. Suppose that we have datasets
D
1
=
X
1
,
y
1
,
…
,
D
r
=
{
X
r
,
y
r
}
. In addition, assume that the
(
p
1
+
i
)
t
h
column of
D
1
and
i
t
h
column of
D
2
refer to the same feature.” This article discloses the combination of multiple datasets from different sources. The Enterprise data is interpreted to be
D
1
and a partition is
D
1
=
X
1
,
y
1
.)
“generating, by the one or more processors, a balanced training partition by appending a portion of the domain-specific data partition to the initial training partition; and” (Algorithm 1 ComImp algorithm, pp. 3; This algorithms discloses the generation of a new dataset by combining D1 with Dn to receive a final dataset D.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zheng and Nguyen. Zheng teaches a machine learning system that is able to take a pre-trained language model and generate a new dataset to further fine tune and retrain a pre-trained language model. Ngyuen teaches different methods to combine and mix datasets. The combined datasets are used for training models. One of ordinary skill would have motivation to combine a language model system that is able to train new language modes with a system that is able to combine and generate new datasets for training, “In this paper, we introduced a novel method, ComImp, to combine datasets. The proposed method increases the sample size and makes use of information from multiple datasets, and is highly beneficial for small sample datasets, as illustrated in the experiments. We also illustrated that ComImp could improve model fitting even more when being used with transfer learning. In addition, we introduced PCA-ComImp, a variant of ComImp, for combining datasets with a large number of features not in common.” (Nguyen, Conclusions, pp. 8).
Regarding claim 2, Nguyen discloses, “wherein the plurality of balanced training partitions of the balanced training dataset respectively corresponds to the plurality of enterprise data partitions.” (Algorithm 1, ComImp algorithm, pp. 3; The final resulting dataset D is the combination of two different datasets. The algorithm discloses how the different datasets are combined and how the datasets still correspond to their original data sources.)
Regarding claim 3, Nguyen discloses, “wherein each of the plurality of balanced training partitions comprises a respective enterprise data partition and an equal portion of a respective domain-specific data partition.” (ComImp Algorithm for Combining Datasets, pp. 3; “In addition, assume that the
(
p
1
+
i
)
t
h
column of
D
1
and
i
t
h
column of
D
2
refer to the same feature. Let F be the union of the sets of features in
X
1
,
X
2
,
…
,
X
r
in an ordered way (step 1 of the algorithm). Then, in step 2 of the algorithm, we rearrange features in each
X
i
to follow the order of the features in F and insert an empty column if a feature in F does not exist in
X
i
. This results in
X
i
*
for each i = 1, ..., r. Next, in step 3, we stack
X
i
, i = 1, .., r vertically and do similarly for the corresponding labels to get
X
*
and y. Finally, we impute the data
X
*
to fill in these entries and create a united, complete dataset.” As stated above, this method of combining dataset will combine them equally and pad missing columns in either dataset as to ensure they are added equal number of times.)
Regarding claim 13, Zheng discloses, “A system comprising memory and one or more processors communicatively coupled to the memory, the one or more processors configured to:” (Implementation Details, pp. 5; “In the fine-tuning experiment, the settings were as follows: a learning rate of 3e-4, a per-device training batch size of 2, weight decay of 0.00001, and a warmup ratio of 0.01. For efficient training, we employed the Deepspeed and Flash-Attention acceleration frameworks for fine-tuning with LoRA, the LoRA rank was set to 16. All experiments were conducted on 4 RTX 3090 for training and 2 A6000 GPUs.” As stated above, this process was implemented using a generic computing system containing processors coupled to memory.)
“store the enterprise data partition as an initial training partition of a plurality of balanced training partitions within a balanced training dataset;” (Zero-shot Prompting with Instructions, pp. 4; “Fine-tuning an LLM directly on the XFIT24-ORI dataset can adapt its MT capabilities to the specific domain. However, its effectiveness is reported to be limited for MT tasks as it can struggle to respond to translation instructions without instruction-tuning [Wei et al., 2022a]. Thus, the need arises to construct a task-specific dataset, that is described via natural language instructions.” The system proposed in this article is able to combine data from different sources and generate a mixed dataset to fine tune a pretrained language model. This system generates a labeled dataset, XFIT24-ORI, and stores it so it is able to generate the second dataset.)
“train a domain-specific language model based on the balanced training dataset.” (Datasets, pp. 5; “For fine-tuning, we employ the self-constructed dataset XFIT24, which contains the IT domain translation data (In Section 3.1). It mainly consists of 10, 000 English- Chinese translation pairs from publicly available datasets, and 200, 000 self-constructed translation pairs. We use test datasets from two general-domain datasets of Flores-101 [Goyal et al., 2022] and OPUS-100 [Zhang et al., 2020], and two domain-specific datasets of IT [Koehn and Knowles, 2017] and XFIT24 for translation inference.” This system is able to use a generated training dataset by combining data with domain specific data. XFIT24 is the resulting training data and it is used to further train a language model.)
Zheng fails to explicitly disclose:
“receive an enterprise data partition from a plurality of enterprise data partitions associated with an enterprise data source;”
“receive a domain-specific data partition from a plurality of domain-specific data partitions associated with one or more domain data sources that are different than the enterprise data source;”
“generate a balanced training partition by appending a portion of the domain-specific data partition to the initial training partition; and”
However, Nguyen discloses, “receive an enterprise data partition from a plurality of enterprise data partitions associated with an enterprise data source;” (ComImp Algorithm for Combining Datasets, pp. 3; “This section details our ComImp framework for combining datasets. The general ComImp framework is shown in Algorithm 1. Suppose that we have datasets
D
1
=
X
1
,
y
1
,
…
,
D
r
=
{
X
r
,
y
r
}
. In addition, assume that the
(
p
1
+
i
)
t
h
column of
D
1
and
i
t
h
column of
D
2
refer to the same feature.” This article discloses the combination of multiple datasets from different sources. The Enterprise data is interpreted to be
D
1
and a partition is
D
1
=
X
1
,
y
1
.)
“receive a domain-specific data partition from a plurality of domain-specific data partitions associated with one or more domain data sources that are different than the enterprise data source;” (ComImp Algorithm for Combining Datasets, pp. 3; “This section details our ComImp framework for combining datasets. The general ComImp framework is shown in Algorithm 1. Suppose that we have datasets
D
1
=
X
1
,
y
1
,
…
,
D
r
=
{
X
r
,
y
r
}
. In addition, assume that the
(
p
1
+
i
)
t
h
column of
D
1
and
i
t
h
column of
D
2
refer to the same feature.” This article discloses the combination of multiple datasets from different sources. The domain specific data is interpreted to be
D
2
and a partition is
D
2
=
X
2
,
y
2
.)
“generate a balanced training partition by appending a portion of the domain-specific data partition to the initial training partition; and” (Algorithm 1 ComImp algorithm, pp. 3; This algorithms discloses the generation of a new dataset by combining D1 with Dn to receive a final dataset D.)
Regarding claim 14, Nguyen discloses, “wherein the plurality of balanced training partitions of the balanced training dataset respectively corresponds to the plurality of enterprise data partitions.” (Algorithm 1, ComImp algorithm, pp. 3; The final resulting dataset D is the combination of two different datasets. The algorithm discloses how the different datasets are combined and how the datasets still correspond to their original data sources.)
Regarding claim 15, Nguyen discloses, “wherein each of the plurality of balanced training partitions comprises a respective enterprise data partition and an equal portion of a respective domain-specific data partition.” (ComImp Algorithm for Combining Datasets, pp. 3; “In addition, assume that the
(
p
1
+
i
)
t
h
column of
D
1
and
i
t
h
column of
D
2
refer to the same feature. Let F be the union of the sets of features in
X
1
,
X
2
,
…
,
X
r
in an ordered way (step 1 of the algorithm). Then, in step 2 of the algorithm, we rearrange features in each
X
i
to follow the order of the features in F and insert an empty column if a feature in F does not exist in
X
i
. This results in
X
i
*
for each i = 1, ..., r. Next, in step 3, we stack
X
i
, i = 1, .., r vertically and do similarly for the corresponding labels to get
X
*
and y. Finally, we impute the data
X
*
to fill in these entries and create a united, complete dataset.” As stated above, this method of combining dataset will combine them equally and pad missing columns in either dataset as to ensure they are added equal number of times.)
Regarding claim 20, Zheng discloses, “One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processors, cause the one or more processors to:” (Implementation Details, pp. 5; “In the fine-tuning experiment, the settings were as follows: a learning rate of 3e-4, a per-device training batch size of 2, weight decay of 0.00001, and a warmup ratio of 0.01. For efficient training, we employed the Deepspeed and Flash-Attention acceleration frameworks for fine-tuning with LoRA, the LoRA rank was set to 16. All experiments were conducted on 4 RTX 3090 for training and 2 A6000 GPUs.” As stated above, this process was implemented using a generic computing system containing processors coupled to memory.)
“store the enterprise data partition as an initial training partition of a plurality of balanced training partitions within a balanced training dataset;” (Zero-shot Prompting with Instructions, pp. 4; “Fine-tuning an LLM directly on the XFIT24-ORI dataset can adapt its MT capabilities to the specific domain. However, its effectiveness is reported to be limited for MT tasks as it can struggle to respond to translation instructions without instruction-tuning [Wei et al., 2022a]. Thus, the need arises to construct a task-specific dataset, that is described via natural language instructions.” The system proposed in this article is able to combine data from different sources and generate a mixed dataset to fine tune a pretrained language model. This system generates a labeled dataset, XFIT24-ORI, and stores it so it is able to generate the second dataset.)
“train a domain-specific language model based on the balanced training dataset.” (Datasets, pp. 5; “For fine-tuning, we employ the self-constructed dataset XFIT24, which contains the IT domain translation data (In Section 3.1). It mainly consists of 10, 000 English- Chinese translation pairs from publicly available datasets, and 200, 000 self-constructed translation pairs. We use test datasets from two general-domain datasets of Flores-101 [Goyal et al., 2022] and OPUS-100 [Zhang et al., 2020], and two domain-specific datasets of IT [Koehn and Knowles, 2017] and XFIT24 for translation inference.” This system is able to use a generated training dataset by combining data with domain specific data. XFIT24 is the resulting training data and it is used to further train a language model.)
Zheng fails to explicitly disclose:
“receive an enterprise data partition from a plurality of enterprise data partitions associated with an enterprise data source;”
“receive a domain-specific data partition from a plurality of domain-specific data partitions associated with one or more domain data sources that are different than the enterprise data source;”
“generate a balanced training partition by appending a portion of the domain-specific data partition to the initial training partition; and”
However, Nguyen discloses, “receive an enterprise data partition from a plurality of enterprise data partitions associated with an enterprise data source;” (ComImp Algorithm for Combining Datasets, pp. 3; “This section details our ComImp framework for combining datasets. The general ComImp framework is shown in Algorithm 1. Suppose that we have datasets
D
1
=
X
1
,
y
1
,
…
,
D
r
=
{
X
r
,
y
r
}
. In addition, assume that the
(
p
1
+
i
)
t
h
column of
D
1
and
i
t
h
column of
D
2
refer to the same feature.” This article discloses the combination of multiple datasets from different sources. The Enterprise data is interpreted to be
D
1
and a partition is
D
1
=
X
1
,
y
1
.)
“receive a domain-specific data partition from a plurality of domain-specific data partitions associated with one or more domain data sources that are different than the enterprise data source;” (ComImp Algorithm for Combining Datasets, pp. 3; “This section details our ComImp framework for combining datasets. The general ComImp framework is shown in Algorithm 1. Suppose that we have datasets
D
1
=
X
1
,
y
1
,
…
,
D
r
=
{
X
r
,
y
r
}
. In addition, assume that the
(
p
1
+
i
)
t
h
column of
D
1
and
i
t
h
column of
D
2
refer to the same feature.” This article discloses the combination of multiple datasets from different sources. The domain specific data is interpreted to be
D
2
and a partition is
D
2
=
X
2
,
y
2
.)
“generate a balanced training partition by appending a portion of the domain-specific data partition to the initial training partition; and” (Algorithm 1 ComImp algorithm, pp. 3; This algorithms discloses the generation of a new dataset by combining D1 with Dn to receive a final dataset D.)
Claims 4-7, 9, and 16-19 are rejected under 35 U.S.C. 103 as being unpatentable over Zheng and Nguyen in view of Shen et al, (Shen et al, “SlimPajama-DC: Understanding Data Combinations for LLM Training”, May 9th, 2024, hereinafter “Shen”).
Regarding claim 4, Shen discloses, “wherein the enterprise data source comprises a plurality of private documents accessible to an enterprise within a prediction domain and the one or more domain data sources comprise a plurality of public documents that are publicly accessible to a plurality of enterprises within the prediction domain.” (Introduction, pp. 3 “Consequently, the composition of the pretraining data domains, such as Github, Wikipedia, books, and web text like CommonCrawl, plays a critical role in the performance of large language models. In our research, we delve into the domain/source weightings of training data. Leveraging SlimPajama-DC, we investigate two primary areas: (1) global-level and local-level deduplication, and (2) the efficacy of various combinations of thoroughly deduplicated datasets. The first emphasis basically encourages the model to be trained on all sources as no cross-domain overlaps inside, and the second helps us understand how to manage the integration and proportions of diverse domains, especially as datasets for LLM training continue to expand in variety.” This article discloses a method that is able to combine different datasets for training language models. This will use document data from different sources and combine them using different combination schemes.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zheng, Nguyen, and Shen. Zheng teaches a machine learning system that is able to take a pre-trained language model and generate a new dataset to further fine tune and retrain a pre-trained language model. Ngyuen teaches different methods to combine and mix datasets. The combined datasets are used for training models. Shen teaches a Language model system that is able to combine and mix datasets for training. One of ordinary skill would have motivation to combine a language model system that is able to train new language modes with a system that is able to combine and generate new datasets for training with a system that fine tunes language models as well as combines datasets for training, “This unified framework facilitates the evaluation of generative language models across a broad scope of tasks. Specifically, our tests comprised: (1) AI2 Reasoning Challenge (25-shot) [6]; (2) HellaSwag (10-shot) [42]; (3) MMLU (5-shot) [17]; (4) TruthfulQA (0-shot) [24]. As shown in Table 5, with the exception of DC-4, our average results are all better than RedPajama- 1.3B which is also trained on 330B tokens. Among our combinations, the DC-1 (which relies solely on SlimPajama Commoncrawl) achieves the highest scores for ARC and MMLU among all tested configurations. Yet, its performance on TruthfulQA ranks at the bottom. On the other hand, DC-6 obtains the top average accuracy across all SlimPajama data combinations, as well as standing out with the best results on HellaSwag. From DC-1, 2, 5, and 6, it is clear that more domain combinations with diverse training data bring better overall accuracy. Moreover, a potential strategy to harness the strengths of each configuration might involve a sequential training process on DC-1, DC-6, and DC-7.” (Shen, Huggingface Leaderboard Evaluation with Harness, pp. 10)
Regarding claim 5, Nguyen discloses, “wherein a size of the portion of the domain-specific data partition is based on a number of the plurality of public documents or a number of the plurality of enterprise data partitions.” (ComImp Algorithm for Combining Datasets, pp. 3; “In addition, assume that the
(
p
1
+
i
)
t
h
column of
D
1
and
i
t
h
column of
D
2
refer to the same feature. Let F be the union of the sets of features in
X
1
,
X
2
,
…
,
X
r
in an ordered way (step 1 of the algorithm). Then, in step 2 of the algorithm, we rearrange features in each
X
i
to follow the order of the features in F and insert an empty column if a feature in F does not exist in
X
i
. This results in
X
i
*
for each i = 1, ..., r. Next, in step 3, we stack
X
i
, i = 1, .., r vertically and do similarly for the corresponding labels to get
X
*
and y. Finally, we impute the data
X
*
to fill in these entries and create a united, complete dataset.” As stated above, this method of combining dataset will combine them equally and pad missing columns in either dataset as to ensure they are added equal number of times. This would also teach each partition size is limited the number of data partitions one or the other entity has.)
Regarding claim 6, Shen discloses, “wherein the plurality of enterprise data partitions comprises a plurality of first non-overlapping text sequences extracted from the plurality of private documents and the plurality of domain-specific data partitions comprises a plurality of second non-overlapping text sequences extracted from the plurality of public documents.” (Dataset Processing Procedure, pp. 5; “SlimPajama is constructed by filtering low-length documents and applying MinHashLSH deduplication to the 1.2T token RedPajama dataset to reduce it to 627B tokens. RefinedWeb [28] shows that training on deduplicated data improves training compute efficiency and decreases the chance of LLMs generating memorized text from the dataset. By removing duplicate and low-length examples, it significantly improves the training compute efficiency and model performance.” This article discloses a dataset combination scheme. This model will manipulate different data sets and remove duplicate items. This would lead to no overlapping, or duplicate, sections of text.)
Regarding claim 7, Shen discloses, “wherein a size of the plurality of first non-overlapping text sequences and the plurality of second non-overlapping text sequences is defined by predefined sequence length.” (Global Deduplication, pp. 7; “Specifically, global MinHashLSH deduplication is performed using a Jaccard similarity threshold of 0.8, document signatures constructed with preprocessed lowercase 13-grams, and schema following [23]. To unify a representation of the same content, punctuation, consecutive spaces, newlines, tabs, and leading or trailing escape characters are removed. The level of deduplication performed per data source is presented in Table 2.” This model will perform deduplication on the individual datasets and then once again when the datasets are combined. This teaches a process of determining similarity to the duplication items and would remove the designated section based on a value.)
Regarding claim 9, Shen discloses, “wherein a partition size of the balanced training partition is defined by a predefined hardware constraint.” (Global Deduplication, pp. 7; “The vanilla implementation of MinHashLSH did not scale to trillion token datasets like RedPajama without running out of memory. This is overcome by optimizing the memory usage and parallelization to perform deduplication on 64 CPU cores with 1.4TB peak memory, which can be easily decreased by creating multiple MinHashLSH objects to query.” The proposed system is able to handle certain processes based on the constraints of its own hardware. This teaches a system that is hardware aware.)
Regarding claim 16, Shen discloses, “wherein the enterprise data source comprises a plurality of private documents accessible to an enterprise within a prediction domain and the one or more domain data sources comprise a plurality of public documents that are publicly accessible to a plurality of enterprises within the prediction domain.” (Introduction, pp. 3 “Consequently, the composition of the pretraining data domains, such as Github, Wikipedia, books, and web text like CommonCrawl, plays a critical role in the performance of large language models. In our research, we delve into the domain/source weightings of training data. Leveraging SlimPajama-DC, we investigate two primary areas: (1) global-level and local-level deduplication, and (2) the efficacy of various combinations of thoroughly deduplicated datasets. The first emphasis basically encourages the model to be trained on all sources as no cross-domain overlaps inside, and the second helps us understand how to manage the integration and proportions of diverse domains, especially as datasets for LLM training continue to expand in variety.” This article discloses a method that is able to combine different datasets for training language models. This will use document data from different sources and combine them using different combination schemes.)
Regarding claim 17, Nguyen discloses, “wherein a size of the portion of the domain-specific data partition is based on a number of the plurality of public documents or a number of the plurality of enterprise data partitions.” (ComImp Algorithm for Combining Datasets, pp. 3; “In addition, assume that the
(
p
1
+
i
)
t
h
column of
D
1
and
i
t
h
column of
D
2
refer to the same feature. Let F be the union of the sets of features in
X
1
,
X
2
,
…
,
X
r
in an ordered way (step 1 of the algorithm). Then, in step 2 of the algorithm, we rearrange features in each
X
i
to follow the order of the features in F and insert an empty column if a feature in F does not exist in
X
i
. This results in
X
i
*
for each i = 1, ..., r. Next, in step 3, we stack
X
i
, i = 1, .., r vertically and do similarly for the corresponding labels to get
X
*
and y. Finally, we impute the data
X
*
to fill in these entries and create a united, complete dataset.” As stated above, this method of combining dataset will combine them equally and pad missing columns in either dataset as to ensure they are added equal number of times. This would also teach each partition size is limited the number of data partitions one or the other entity has.)
Regarding claim 18, Shen discloses, “wherein the plurality of enterprise data partitions comprises a plurality of first non-overlapping text sequences extracted from the plurality of private documents and the plurality of domain-specific data partitions comprises a plurality of second non-overlapping text sequences extracted from the plurality of public documents.” (Dataset Processing Procedure, pp. 5; “SlimPajama is constructed by filtering low-length documents and applying MinHashLSH deduplication to the 1.2T token RedPajama dataset to reduce it to 627B tokens. RefinedWeb [28] shows that training on deduplicated data improves training compute efficiency and decreases the chance of LLMs generating memorized text from the dataset. By removing duplicate and low-length examples, it significantly improves the training compute efficiency and model performance.” This article discloses a dataset combination scheme. This model will manipulate different data sets and remove duplicate items. This would lead to no overlapping, or duplicate, sections of text.)
Regarding claim 19, Shen discloses, “wherein a size of the plurality of first non-overlapping text sequences and the plurality of second non-overlapping text sequences is defined by predefined sequence length.” (Global Deduplication, pp. 7; “Specifically, global MinHashLSH deduplication is performed using a Jaccard similarity threshold of 0.8, document signatures constructed with preprocessed lowercase 13-grams, and schema following [23]. To unify a representation of the same content, punctuation, consecutive spaces, newlines, tabs, and leading or trailing escape characters are removed. The level of deduplication performed per data source is presented in Table 2.” This model will perform deduplication on the individual datasets and then once again when the datasets are combined. This teaches a process of determining similarity to the duplication items and would remove the designated section based on a value.)
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Zheng and Nguyen in view of Harris et al, (Harris et al, “Array programming with NumPy” (NumPy User Guide), 2020, hereinafter “Harris”).
Regarding claim 8, Nguyen discloses, “wherein each of the plurality of balanced training partitions is stored at an indexed position within the balanced training dataset, and” (Algorithm 1 ComImp algorithm, pp. 3; The algorithm will return a combined set of datasets. As is known in the art, a set is an indexed data structure.)
Zheng and Nguyen fail to explicitly disclose:
“the computer-implemented method further comprises modifying the balanced training dataset by rearranging a plurality of indexed positions of the plurality of balanced training partitions within the balanced training dataset.”
However, Harris discloses, “the computer-implemented method further comprises modifying the balanced training dataset by rearranging a plurality of indexed positions of the plurality of balanced training partitions within the balanced training dataset.” (NumPy User Guide, Generating random number, pp. 48; “The use of random number generation is an important part of the configuration and evaluation of many numerical and machine learning algorithms. Whether you need to randomly initialize weights in an artificial neural network, split data into random sets, or randomly shuffle your dataset, being able to generate random numbers (actually, repeatable pseudorandom numbers) is essential.” This user manual discloses the randomization of indexes of array to rearrange the locations of the elements contained within the array.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zheng, Nguyen, and Harris. Zheng teaches a machine learning system that is able to take a pre-trained language model and generate a new dataset to further fine tune and retrain a pre-trained language model. Ngyuen teaches different methods to combine and mix datasets. The combined datasets are used for training models. Harris teaches a popular software package that is able to handle and manipulate data withing computing systems. One of ordinary skill would have motivation to combine a language model system programming and dataset combination and implement them in a computing system using popular programming framework called NumPy, “NumPy combines the expressive power of array programming, the performance of C, and the readability, usability and versatility of Python in a mature, well tested, well documented and community-developed library. Libraries in the scientific Python ecosystem provide fast implementations of most important algorithms. Where extreme optimization is warranted, compiled languages can be used, such as Cython43, Numba44 and Pythran45; these languages extend Python and transparently accelerate bottlenecks. Owing to NumPy’s simple memory model, it is easy to write low-level, hand-optimized code, usually in C or Fortran, to manipulate NumPy arrays and pass them back to Python. Furthermore, using array protocols, it is possible to utilize the full spectrum of specialized hardware acceleration with minimal changes to existing code.” (Harris, Discussion, pp. 361)
Claims 10-12 are rejected under 35 U.S.C. 103 as being unpatentable over Zheng and Nguyen in view of Aghaei et al, (Aghaei et al, “SecureBERT: A Domain-Specific Language Model for Cybersecurity”, 2023, hereinafter “Aghaei”).
Regarding claim 10, Aghaei discloses, “wherein the domain-specific language model comprises a bidirectional encoder representation from transformers model.” (Introduction, pp. 40-41; “In this study, we address this critical cybersecurity problem by introducing a new language model called SecureBERT by employing the state-of-the-art NLP architecture called BERT [12], which is capable of processing texts with cybersecurity implications effectively. SecureBERT is generic enough to be applied in a variety of cybersecurity tasks, such as phishing detection [10], code and malware analysis [24], intrusion detection [2], etc. SecureBERT is a pre-trained cybersecurity language model that have the fundamental understanding of both the word-level and sentence-level semantics, which is an essential building block for any cybersecurity report.” This article proposed a pretrained BERT based language model. This is domain specific model as well.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zheng, Nguyen, and Aghaei. Zheng teaches a machine learning system that is able to take a pre-trained language model and generate a new dataset to further fine tune and retrain a pre-trained language model. Ngyuen teaches different methods to combine and mix datasets. The combined datasets are used for training models. Aghaei teaches a domain specific language model for cyber security. One of ordinary skill would have motivation to combine a language model system that is able to train new language modes with a system that is able to combine and generate new datasets for training with a system that generates a pre-trained domain specific model such as a BERT based model, “This study introduces SecureBERT, a transformer-based language model for processing cybersecurity text language based on RoBERTa. We presented two practical ways for developing a successful model that can capture contextual relationships and semantic meanings in cybersecurity text by designing a customized tokenization tool on top of RoBERTa’s tokenizer and altering the pretrained weights. SecureBERT is trained to utilize a corpus of 1.1 billion words collected from a range of online cybersecurity resources. SecureBERT has been evaluated using the standard Masked Language Model (MLM) as well as the named entity recognition (NER) task. The evaluation outcomes demonstrated promising results in grasping cybersecurity language. (Aghaei, Conclusions and Future Works, pp. 54)
Regarding claim 11, Aghaei discloses, “wherein the domain-specific language model is trained using continued masked language modelling based on the balanced training dataset.” (Masked Language Model, pp. 46-47; “In this study, we address this critical cybersecurity problem by introducing a new language model called SecureBERT by employing the state-of-the-art NLP architecture called BERT [12], which is capable of processing texts with cybersecurity implications effectively. SecureBERT is generic enough to be applied in a variety of cybersecurity tasks, such as phishing detection [10], code and malware analysis [24], intrusion detection [2], etc. SecureBERT is a pre-trained SecureBERT 41 cybersecurity language model that have the fundamental understanding of both the word-level and sentence-level semantics, which is an essential building block for any cybersecurity report. which are not included in pre-training dataset. Rather than masking an arbitrary word in a sentence, as in RoBERTa, we masked only the verb or noun in the sentence because a verb denotes an action and a noun denotes an object, both of which are important for understanding the sentence’s semantics in a cybersecurity context. Our testing dataset contains 17, 341 records, with 12, 721 records containing a masked noun (2, 213 unique nouns) and 4, 620 records containing a masked verb (888 unique masked verbs in total).” This article discloses a masking language model task performed by the trained model. This also discloses the testing and training using masked works.)
Regarding claim 12, Aghaei discloses, “generating a byte-pair encoding subword for the balanced training dataset; and” (Customized Tokenizer, pp. 44; “For building the tokenizer, we employ a byte pair encoding (BPE) [25] method to build a vocabulary of words and subwords from the cybersecurity corpora, as it is proven to have better performance versus word-based tokenizer. Character based encoding used in BPE allows for the learning of a small subword vocabulary that can encode any input text without introducing any "unknown" tokens [22].” This system discloses the use of byte-pair encoding for tokenization. This process is performed using the datasets during training.)
“training the domain-specific language model based on the byte-pair encoding subword.” (Customized Tokenizer, pp. 44; “Hence, we choose all mutual terms and assign their original indices, while the remainder new tokens are assigned random indices with no conflict, where the original indices refers to the indices in RoBERTa’s tokenizer, to build our tokenizer. Ultimately, we develop a customized tokenizer with a vocabulary size similar to that of the original model, which includes tokens commonly seen in cybersecurity corpora in addition to cross-domain tokens. Our tokenizer encodes mutual tokens Vmutual as original model, ensuring that the model returns the appropriate pretrained weights, while for new terms Vdistinct the indices and accordingly the weights would be random.” As stated above, this system uses a BPE scheme which is used for training their BERT based system.)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAUL MICHAEL GALVIN-SIEBENALER whose telephone number is (571)272-1257. The examiner can normally be reached Monday - Friday 8AM to 5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PAUL M GALVIN-SIEBENALER/Examiner, Art Unit 2147
/HASSAN MRABI/Primary Examiner, Art Unit 2147
Prosecution Insights