DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 6/3/2026 has been entered.
Response to Amendment
The Amendments filed 6/3/2026 have been entered. Claims 1, 3, 10, and 19 were amended. Claims 1-20 are pending in the application.
Drawings
The drawings are objected to because the text label "Classifictation" appearing in each of blocks 410 and 416 of FIG. 4 is misspelled and should read "Classification". Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as "amended." If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either "Replacement Sheet" or "New Sheet" pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: 420 (FIG. 4, designating the output labeled "Model vx"). Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either "Replacement Sheet" or "New Sheet" pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities:
(a) [0007] ends in an incomplete sentence ("In other domains, other differences may exist relative to general language that similarly"), and the sentence should be completed;
(b) [0008] "such as documents titles" should read --such as document titles--;
(c) [0013] "to score the responsive of documents" should read --to score the responsiveness of documents--;
(d) [0016] FIG. 2 is described as illustrating "an example method 300" and "the method 300", whereas FIG. 2 and [0017]-[0029] designate that method as method 200; reference numeral 300 should read --200-- in [0016];
(e) [0017] "may be referred to in this method 200 as the method first version" should read --may be referred to in this method 200 as the model first version--;
(f) [0018] "information regarding products and services may be provide" should read --information regarding products and services may be provided--;
(g) [0024] "Accordingly, training at block 202 may teach the model to classify within the second type of information" should refer to --block 210--, because block 202 is the step of receiving the randomly-initialized model and the training described in that paragraph is conducted at block 210;
(h) [0026] describes block 214 as "conducting second training on the machine learning model second version using a second type of information" and states that the training at block 214 "may enable the model to predict the classification of a document within the second type of information", whereas FIG. 2 (block 214) and [0027] describe third training on the machine learning model third version using a third type of information; the references to the second training, the second version and the second type of information in [0026] should be conformed accordingly;
(i) [0030] FIG. 3 is described as illustrating "an example method 200" and "the method 200", whereas FIG. 3 and [0031]-[0036] designate that method as method 300; reference numeral 200 should read --300-- in [0030];
(j) [0038] "a further classification layer may be added to the model 410" should read --model version 408--, because reference numeral 410 designates a training block rather than a model, and "Training at block 418 may include classification training" should read --Training at block 416--, because reference numeral 418 designates the third training data rather than a training block;
(k) [0042] "one or more applications programs 528" should read --one or more application programs 528--;
(l) [0044] "via a further processing device, such a network router 548" should read --such as a network router 548--.
Appropriate correction is required.
The specification is objected to as failing to provide proper antecedent basis for the claimed subject matter. See 37 CFR 1.75(d)(1) and MPEP 608.01(o). Correction of the following is required: claims 1 and 10 recite "a plurality of first types of data" and "a plurality of second types of data", and claim 19 recites "a plurality of first types of information" and "a plurality of second types of information"; the specification describes "multiple types of information respective of a set of documents" ([0018]-[0019]) and a first, second and third type of information ([0021]-[0027]), but does not describe a plurality of first types and a separate plurality of second types of data or information, and does not use the term "types of data" at all. The terms of the claims should find clear support or antecedent basis in the description so that their meaning may be ascertainable by reference to the description, or the description should be amended, without introducing new matter, to provide such antecedent basis.
Claim Objections
Claims 9 and 18 are objected to because of the following informalities:
Claims 9 and 18 each recite the conjunction "and" following "the first version of the machine learning model is randomly-initialized;" should be deleted, because the three wherein clauses of each claim are already joined by the conjunction "and" that precedes the final clause ("the first training and the second training comprise use of training data selected from the plurality of documents").
Appropriate correction is required.
Claim Rejections - 35 U.S.C. 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3-4, 6-8, and 10-18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 3, claim 3 recites the limitation "the respective set of documents" in the last line of the claim. There is insufficient antecedent basis for this limitation in the claim. Claim 1, from which claim 3 depends, introduces a single "set of documents" that holds both the plurality of first types of data of the first training data and the plurality of second types of data of the second training data. No "respective" sets of documents are recited anywhere in the claim chain, and the word "respective" connotes a separate set of documents corresponding to each of the first training data and the second training data, which conflicts with the single set of documents recited in claim 1. As a result, claim 3 in the limitation "wherein the first training data is different from the second training data in the respective set of documents" can be read to require (i) that the first training data and the second training data come from different sets of documents, contrary to claim 1, (ii) that they come from one shared set, or (iii) only that the two training data differ from each other within the set of documents of claim 1. Because these readings differ in scope and the claim does not indicate which is intended, one of ordinary skill in the art would not be reasonably apprised of the scope of claim 3. For purposes of examination, claim 3 is interpreted as requiring that the first training data differ from the second training data, both being drawn from the single set of documents recited in claim 1, consistent with [0039] and [0048] of the specification.
Regarding claim 6, the claim recites "adding a second layer to the machine learning model second version to create a machine learning model third version" and "wherein the deploying comprises deploying the machine learning model after the third training." Claim 1, from which claim 6 depends, recites "deploying the machine learning model second version after the second training." Claim 6 further limits that same deploying step, so the deploying must still deploy the machine learning model second version, yet claim 6 states that the deploying occurs after third training of a different, third version that is created by adding a second layer to the second version. The generic recitation "the machine learning model" in claim 6 could refer to the machine learning model second version required by claim 1 or to the machine learning model third version introduced in claim 6. It is therefore unclear whether claim 6 requires deploying the second version, which lacks the second layer and the third training, deploying the third version, or deploying both. Because the claims are inconsistent as to which model is deployed, one of ordinary skill in the art would not be reasonably apprised of the scope of the deploying step. For purposes of examination, the deploying of claim 6 is interpreted as deploying the machine learning model as trained through the third training, that is, the machine learning model third version, consistent with [0028] and [0050] of the specification (additional layers added and trained before the trained model is deployed; see also [0035]).
Regarding claim 10, the claim recites "the machine learning model first version trained to predict a first type of data of a plurality of first types of data in a set of documents" and then, in the final wherein clause, recites "wherein the first training data comprises the plurality of first types of data in a set of documents and the second training data comprises the plurality of second types of data in the set of documents." The final wherein clause introduces "a set of documents" a second time with the indefinite article, after a set of documents was already introduced earlier in the claim. It is therefore unclear whether the claim requires a single set of documents or two different sets of documents, and the subsequent recitation "in the set of documents" at the end of the claim could refer to either set. Where two elements of the same name are recited earlier in a claim, a later reference to "the" element is unclear when it is uncertain which of the two was intended. MPEP 2173.05(e); In re Packard, 751 F.3d 1307, 1314 (Fed. Cir. 2014). The ambiguity is not merely formal: it determines whether the first training data and the second training data must be drawn from the same set of documents in which the first type of data is predicted, or may come from an unrelated collection of documents, so one of ordinary skill in the art would not be reasonably apprised of the scope of the claim. Nautilus, Inc. v. Biosig Instruments, Inc., 572 U.S. 898, 901 (2014). For purposes of examination, every recitation of "a set of documents" and "the set of documents" in claim 10 is interpreted as referring to one and the same set of documents, consistent with the disclosure that the training data may be the set of documents or a subset of the set of documents ([0020]) and that the training data used for each training stage may all be associated with a single set of documents ( [0039]).
Claim 4 depends from claim 3, claims 7 and 8 depend from claim 6, and claims 11-18 depend from claim 10; they incorporate and do not cure the respective defects identified above, and are rejected for the same reasons.
Claim Rejections - 35 USC 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-4, 6-8, 10, 12, 13, and 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Hashimoto et al. (hereinafter Hashimoto), US 2018/0121799 A1 in view of Zhang et al. (hereinafter Zhang), US 2020/0175362 A1.
Zhang was disclosed in an IDS dated 6/13/2023.
Regarding independent claim 1, Hashimoto teaches a computing system for training a machine learning model, the system comprising (Hashimoto: [0246], "FIG. 11 is a simplified block diagram of a computer system 1100 that can be used to implement the joint many-task neural network model 100"; the computer system 1100 (a computing system) implements the joint many-task neural network model 100 (a machine learning model), whose training is described at [0136]): a processor (Hashimoto: [0246], "Computer system 1100 typically includes one or more CPU processors 1120"; the CPU processors 1120 (a processor) are the processors of the computer system 1100); and a memory storing instructions that, when executed by the processor, cause the computing system to perform operations comprising (Hashimoto: [0250], "Storage subsystem 1110 stores programming and data constructs that provide the functionality of some or all of the modules and methods described herein. These software modules are generally executed by CPU processors 1120"; the software modules held in the storage subsystem 1110 are executed by the CPU processors 1120 to provide the described methods, including the training of the model 100; [0251], "Memory subsystem 1112 in the storage subsystem can include a number of memories including a main random access memory (RAM) 1116 for storage of instructions and data during program execution"; the memory subsystem 1112 (a memory) holds the instructions executed during program execution); conducting first training on the machine learning model first version based on a first training data (Hashimoto: [0121], "One training corpus for the POS layer 400 is the Wall Street Journal (WSJ) portion of Penn Treebank"; the POS layer 400 of the model 100 is trained on the WSJ corpus, which is a first training of the model; [0136], "training the first layer using first layer training examples (e.g., POS data)"; the POS data (a first training data) are the first layer training examples on which that training is based), the machine learning model first version trained to predict a first type of data of a plurality of first types of data in a document (Hashimoto: [0077], "the POS label embedding layer 400 of the model 100 is a bi-directional LSTM 410, whose hidden states are used to predict POS tags"; the POS layer is trained to predict POS tags (a first type of data); [0078], "fed into an exponential normalizer with a single ReLU layer which outputs the probability vector y(pos) for each of the POS tags"; the layer outputs a probability for each of the POS tags, which are a plurality of first types of data, for each word of the text of a WSJ document (a document)); conducting second training on the machine learning model second version based on a second training data (Hashimoto: [0124], "To train the chunking layer 500, the WSJ corpus was used in which the chunks are labelled"; the model 100 with the chunking layer 500 stacked on the POS layer is trained on the chunk-labelled WSJ corpus, which is a second training of the model; [0136], "training the second layer using second layer training examples (e.g., chunking data)"; the chunking data (a second training data) are the second layer training examples on which that training is based), the machine learning model second version trained to predict a second type of data of a plurality of second types of data (Hashimoto: [0082], "Chunking is also a word-level classification task which assigns a chunking tag (B-NP, I-VP, etc.) for each word"; the chunking layer is trained to assign chunking tags (a second type of data), each being one of the several chunking tags such as B-NP and I-VP, which are a plurality of second types of data) based on the prediction of the first type of data in the document (Hashimoto: [0084], "The probability values are automatically predicted by the POS label embedding layer working like a built-in POS tagger, and thus no gold POS tags are needed"; the POS label embedding layer predicts, for each word of the text, the probability of each POS tag, and the POS label embedding weighted by those predicted probabilities is an input of the chunking layer; [0042], "It processes at least the word embedding vectors, the POS label embedding vectors and the POS state vectors, to produce chunk label embeddings"; the chunk label embeddings are produced from the POS label embedding vectors, so the chunking tags are predicted based on the POS tag predictions for the words of the WSJ document), wherein the first training data comprises the plurality of first types of data in a set of documents (Hashimoto: [0121], "One training corpus for the POS layer 400 is the Wall Street Journal (WSJ) portion of Penn Treebank. This corpus includes tokens labelled with POS tags"; the POS data (the first training data) comprise the POS tags of the tokens of the WSJ corpus (a set of documents), the Wall Street Journal (WSJ) portion of Penn Treebank) and the second training data comprises the plurality of second types of data in the set of documents (Hashimoto: [0124], "To train the chunking layer 500, the WSJ corpus was used in which the chunks are labelled"; the chunking data (the second training data) comprise the chunking tags labelled in the same WSJ corpus (the set of documents)).
Hashimoto does not expressly teach receiving a first version of a machine learning model; adding a layer to the machine learning model first version after conducting the first training to create a machine learning model second version; and deploying the machine learning model second version after the second training.
However, Zhang teaches receiving a first version of a machine learning model (Zhang: [0055], "With a given small network, the system 500 learns an initial model on a first given task"; the given small network (a first version of a machine learning model) is provided to the system 500, which learns an initial model on the first task from it); adding a layer to the machine learning model first version after conducting the first training to create a machine learning model second version (Zhang: [0055], "the system 500 then searches for the best network architecture by network transformation based AutoML for the upcoming new tasks without access to the old tasks' data using an MWC algorithm and compresses the expanded network to the initial network's size"; after the initial model is learned on the first task, the network architecture is expanded for the upcoming new tasks, and the expanded network is then compressed; [0053], "The deeper operator 510 adds layer(s) to the existing network architecture to perform the new task"; the deeper operator adds a layer to the existing network architecture of the trained model, the added layer (a layer) being learned as a task-specific layer for the new task; [0053], "The child network architectures 520 are expanded versions of the original/prior/existing network architecture that are generated to perform the new tasks"; the resulting expanded version (a machine learning model second version) of the existing network architecture performs the new task); and deploying the machine learning model second version after the second training (Zhang: [0077], "the system may train the ML model to perform the new task using training data for the new task without access to the training data for the old task"; in step 710 the expanded model is trained using the training data of the new task, which Zhang can do without access to the training data of the old task; [0080], "The system then uses the adapted ML model to perform both existing and new tasks (step 715)"; after that training, the adapted model is put into use to perform the existing and new tasks, which is deploying it after the second training).
Because Hashimoto and Zhang are analogous art with both addressing the training of multi-layer neural network models to perform additional tasks while avoiding catastrophic forgetting of the tasks already learned, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Zhang's learning of an initial model on a first task from a given network, followed by the addition of a task-specific layer to the trained network for a new task, the training of the expanded network, and the use of the adapted model, to the computer system of Hashimoto, with a reasonable expectation of success, by training the POS layer of a given network on the POS-tagged WSJ corpus, then adding the chunking layer on top of the trained POS layer as the task-specific layer for the new chunking task, training the expanded model on the chunk-labelled WSJ corpus with the predicted POS label embeddings supplied to the added layer and with successive regularization holding the trained POS-layer parameters near their trained values (Hashimoto: [0124]-[0125]), and then using the adapted model to perform both the POS tagging and the chunking, to teach receiving a first version of a machine learning model; adding a layer to the machine learning model first version after conducting the first training to create a machine learning model second version; and deploying the machine learning model second version after the second training. This modification would have been motivated by the desire to expand the neural network architecture so that the model learns the new task with improved model capability and performance while overcoming the catastrophic forgetting of the task already learned (Zhang: [0021]). That is the aim Hashimoto itself pursues for its model, which Hashimoto seeks to allow "to grow in depth while avoiding catastrophic forgetting" (Hashimoto: [0035]), by stacking a layer for each successively higher task on the layers beneath it.
Regarding dependent claim 3, Hashimoto, in view of Zhang, teach the computing system of claim 1, wherein the first training data is different from the second training data in the [[respective]] (interpreted per the 35 U.S.C. 112(b) rejection set forth above) set of documents (Hashimoto: [0136], "training the first layer using first layer training examples (e.g., POS data), training the second layer using second layer training examples (e.g., chunking data)"; the POS data (the first training data) and the chunking data (the second training data) are different training examples, the POS tags and the chunking tags labelled in the same WSJ corpus (the set of documents)).
Regarding dependent claim 4, Hashimoto, in view of Zhang, teach the computing system of claim 3, wherein the first training data comprises a first type of information respective of a plurality of entities (Hashimoto: [0121], "One training corpus for the POS layer 400 is the Wall Street Journal (WSJ) portion of Penn Treebank. This corpus includes tokens labelled with POS tags"; the POS data (the first training data) comprise the POS tags (a first type of information), each labelling a token of the text of one of the WSJ documents (a plurality of entities), so the POS tags of each document's tokens are information relating to that document) and the second training data comprises a second type of information respective of the plurality of entities. (Hashimoto: [0124], "To train the chunking layer 500, the WSJ corpus was used in which the chunks are labelled"; the chunking data (the second training data) comprise the chunking tags (a second type of information) labelled in the same WSJ corpus; [0082], "The tag specifies the region of major phrases (or chunks) in the sentence"; each chunking tag is assigned to a word and marks the region of a major phrase in a sentence of the text of one of the WSJ documents (the plurality of entities), so the chunking tags of each document's words are information relating to the same documents)
Regarding dependent claim 6, Hashimoto, in view of Zhang, teach the computing system of claim 1, wherein the layer is a first layer (Hashimoto: [0084], "In model 100, chunking is performed using a second bi-LSTM layer 510 on top of the POS layer"; counting from the POS layer, the second bi-LSTM layer 510 is the first layer stacked on it, the chunking layer 500 (a first layer)), wherein the operations further comprise: adding a second layer to the machine learning model second version to create a machine learning model third version (Zhang: [0054], "Thereafter, the system 500 repeats the process for additional new tasks that are identified to provide continual or lifelong learning for multiple tasks"; the process of expanding the trained network for a new task, which Zhang follows with compression, is repeated for each additional new task, so another added layer (a second layer) is added for the next task to produce a further expanded version (a machine learning model third version)); and conducting third training on the machine learning model third version (Hashimoto: [0127], "To train the dependency parsing layer 600, the WSJ corpus with dependency labels was used"; the model with the dependency parsing layer 600 stacked on the chunking layer is trained on the dependency-labelled WSJ corpus, which is a third training of the model; [0136], "training the third layer using third layer training examples (e.g., dependency data)"; the third training uses the dependency data); wherein the deploying comprises deploying the machine learning model third version (interpreted per the 35 U.S.C. 112(b) rejection set forth above) after the third training (Zhang: [0080], "The system then uses the adapted ML model to perform both existing and new tasks (step 715). Also, after step 715, the system may return to step 710 to continuously adapt the network architecture for additional new tasks for the lifelong learning for each new task identified"; the adapted model is put into use after the training for a new task, and the process returns to step 710 for each additional new task, so the further expanded version is again put into use after the training for the next task, which is deploying it after the third training).
Regarding dependent claim 7, Hashimoto, in view of Zhang, teach the computing system of claim 6, wherein the first training comprises first training data and the second training comprises second training data and the third training comprises third training data (Hashimoto: [0136], "training the first layer using first layer training examples (e.g., POS data), training the second layer using second layer training examples (e.g., chunking data)"; the first training uses the POS data (the first training data) and the second training uses the chunking data (the second training data); [0136], "training the third layer using third layer training examples (e.g., dependency data)"; the third training uses the dependency data (the third training data)), wherein the first training data, the second training data, and the third training data are different from one another (Hashimoto: [0136], "The training includes training the first layer using first layer training examples (e.g., POS data), training the second layer using second layer training examples (e.g., chunking data) with regularized pass down training to the first layer, and training the third layer using third layer training examples (e.g., dependency data) with regularized pass down training to the first and second layers"; the POS data, the chunking data and the dependency data are the separate training examples directed to the first, second and third layers, which are different from one another).
Regarding dependent claim 8, Hashimoto, in view of Zhang, teach the computing system of claim 7, wherein: the first training data comprises a first type of information respective of a plurality of entities (Hashimoto: [0121], "One training corpus for the POS layer 400 is the Wall Street Journal (WSJ) portion of Penn Treebank. This corpus includes tokens labelled with POS tags"; the POS data (the first training data) comprise the POS tags (a first type of information), each labelling a token of the text of one of the WSJ documents (a plurality of entities), so the POS tags of each document's tokens are information relating to that document); the second training data comprises a second type of information respective of the plurality of entities (Hashimoto: [0124], "To train the chunking layer 500, the WSJ corpus was used in which the chunks are labelled"; the chunking data (the second training data) comprise the chunking tags (a second type of information) labelled in the same WSJ corpus; [0082], "The tag specifies the region of major phrases (or chunks) in the sentence"; each chunking tag is assigned to a word and marks the region of a major phrase in a sentence of the text of one of the WSJ documents (the plurality of entities), so the chunking tags of each document's words are information relating to the same documents); and the third training data comprises a third type of information respective of the plurality of entities (Hashimoto: [0127], "To train the dependency parsing layer 600, the WSJ corpus with dependency labels was used"; the dependency data (the third training data) comprise the dependency labels (a third type of information) of the same WSJ corpus; [0089], "Dependency parsing identifies syntactic relationships (such as an adjective modifying a noun) between pairs of words in a sentence"; each dependency label identifies a syntactic relationship between a pair of words of a sentence of the text of one of the WSJ documents (the plurality of entities), so the dependency labels of each document's sentences are information relating to the same documents).
Regarding independent claim 10, Hashimoto teaches a method comprising (Hashimoto: [0136], "FIG. 9A shows one implementation of training a stacked LSTM sequence processor that is stacked with at least three layers according to an analytical hierarchy"; the training of the stacked LSTM sequence processor of the model 100 (a machine learning model) is a method); conducting first training on the machine learning model first version based on a first training data (Hashimoto: [0121], "One training corpus for the POS layer 400 is the Wall Street Journal (WSJ) portion of Penn Treebank"; the POS layer 400 of the model 100 is trained on the WSJ corpus, which is a first training of the model; [0136], "training the first layer using first layer training examples (e.g., POS data)"; the POS data (a first training data) are the first layer training examples on which that training is based), the machine learning model first version trained to predict a first type of data of a plurality of first types of data in a set of documents (Hashimoto: [0077], "the POS label embedding layer 400 of the model 100 is a bi-directional LSTM 410, whose hidden states are used to predict POS tags"; the POS layer is trained to predict POS tags (a first type of data); [0078], "fed into an exponential normalizer with a single ReLU layer which outputs the probability vector y(pos) for each of the POS tags"; the layer outputs a probability for each of the POS tags, which are a plurality of first types of data, for each word of the texts of the WSJ corpus (a set of documents)); conducting second training on the machine learning model second version based on a second training data (Hashimoto: [0124], "To train the chunking layer 500, the WSJ corpus was used in which the chunks are labelled"; the model 100 with the chunking layer 500 stacked on the POS layer is trained on the chunk-labelled WSJ corpus, which is a second training of the model; [0136], "training the second layer using second layer training examples (e.g., chunking data)"; the chunking data (a second training data) are the second layer training examples on which that training is based), the machine learning model second version trained to predict a second type of data of a plurality of second types of data (Hashimoto: [0082], "Chunking is also a word-level classification task which assigns a chunking tag (B-NP, I-VP, etc.) for each word"; the chunking layer is trained to assign chunking tags (a second type of data), each being one of the several chunking tags such as B-NP and I-VP, which are a plurality of second types of data) based on the prediction of the first type of data in the set of documents (Hashimoto: [0084], "The probability values are automatically predicted by the POS label embedding layer working like a built-in POS tagger, and thus no gold POS tags are needed"; the POS label embedding layer predicts, for each word of the text, the probability of each POS tag, and the POS label embedding weighted by those predicted probabilities is an input of the chunking layer; [0042], "It processes at least the word embedding vectors, the POS label embedding vectors and the POS state vectors, to produce chunk label embeddings"; the chunk label embeddings are produced from the POS label embedding vectors, so the chunking tags are predicted based on the POS tag predictions for the words of the WSJ corpus), wherein the first training data comprises the plurality of first types of data in [[a]] the (interpreted per the 35 U.S.C. 112(b) rejection set forth above) set of documents (Hashimoto: [0121], "One training corpus for the POS layer 400 is the Wall Street Journal (WSJ) portion of Penn Treebank. This corpus includes tokens labelled with POS tags"; the POS data (the first training data) comprise the POS tags of the tokens of the WSJ corpus (the set of documents), the Wall Street Journal (WSJ) portion of Penn Treebank) and the second training data comprises the plurality of second types of data in the set of documents (Hashimoto: [0124], "To train the chunking layer 500, the WSJ corpus was used in which the chunks are labelled"; the chunking data (the second training data) comprise the chunking tags labelled in the same WSJ corpus (the set of documents)).
Hashimoto does not expressly teach receiving a first version of a machine learning model; adding a layer to the machine learning model first version after conducting the first training to create a machine learning model second version; and deploying the machine learning model after the second training.
However, Zhang teaches receiving a first version of a machine learning model (Zhang: [0055], "With a given small network, the system 500 learns an initial model on a first given task"; the given small network (a first version of a machine learning model) is provided to the system 500, which learns an initial model on the first task from it); adding a layer to the machine learning model first version after conducting the first training to create a machine learning model second version (Zhang: [0055], "the system 500 then searches for the best network architecture by network transformation based AutoML for the upcoming new tasks without access to the old tasks' data using an MWC algorithm and compresses the expanded network to the initial network's size"; after the initial model is learned on the first task, the network architecture is expanded for the upcoming new tasks, and the expanded network is then compressed; [0053], "The deeper operator 510 adds layer(s) to the existing network architecture to perform the new task"; the deeper operator adds a layer to the existing network architecture of the trained model, the added layer (a layer) being learned as a task-specific layer for the new task; [0053], "The child network architectures 520 are expanded versions of the original/prior/existing network architecture that are generated to perform the new tasks"; the resulting expanded version (a machine learning model second version) of the existing network architecture performs the new task); and deploying the machine learning model after the second training (Zhang: [0077], "the system may train the ML model to perform the new task using training data for the new task without access to the training data for the old task"; in step 710 the expanded model is trained using the training data of the new task, which Zhang can do without access to the training data of the old task; [0080], "The system then uses the adapted ML model to perform both existing and new tasks (step 715)"; after that training, the adapted model is put into use to perform the existing and new tasks, which is deploying it after the second training).
Because Hashimoto and Zhang are analogous art with both addressing the training of multi-layer neural network models to perform additional tasks while avoiding catastrophic forgetting of the tasks already learned, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Zhang's learning of an initial model on a first task from a given network, followed by the addition of a task-specific layer to the trained network for a new task, the training of the expanded network, and the use of the adapted model, to the method of Hashimoto, with a reasonable expectation of success, by training the POS layer of a given network on the POS-tagged WSJ corpus, then adding the chunking layer on top of the trained POS layer as the task-specific layer for the new chunking task, training the expanded model on the chunk-labelled WSJ corpus with the predicted POS label embeddings supplied to the added layer and with successive regularization holding the trained POS-layer parameters near their trained values (Hashimoto: [0124]-[0125]), and then using the adapted model to perform both the POS tagging and the chunking, to teach receiving a first version of a machine learning model; adding a layer to the machine learning model first version after conducting the first training to create a machine learning model second version; and deploying the machine learning model after the second training. This modification would have been motivated by the desire to expand the neural network architecture so that the model learns the new task with improved model capability and performance while overcoming the catastrophic forgetting of the task already learned (Zhang: [0021]). That is the aim Hashimoto itself pursues for its model, which Hashimoto seeks to allow "to grow in depth while avoiding catastrophic forgetting" (Hashimoto: [0035]), by stacking a layer for each successively higher task on the layers beneath it.
Regarding dependent claim 12, Hashimoto, in view of Zhang, teach the method of claim 10, wherein the first training data comprises a first type of data and the second training data comprises a second type of data (Hashimoto: [0136], "training the first layer using first layer training examples (e.g., POS data), training the second layer using second layer training examples (e.g., chunking data)"; the POS data (the first training data) comprise POS tags (a first type of data) and the chunking data (the second training data) comprise chunking tags (a second type of data)), wherein the first training data is different from the second training data (Hashimoto: [0121], "This corpus includes tokens labelled with POS tags"; the POS data label the tokens with POS tags; [0124], "To train the chunking layer 500, the WSJ corpus was used in which the chunks are labelled"; the chunking data label the chunks, so the two training data differ).
Regarding dependent claim 13, Hashimoto, in view of Zhang, teach the method of claim 12, wherein the first training data comprises a first type of information respective of a plurality of entities (Hashimoto: [0121], "One training corpus for the POS layer 400 is the Wall Street Journal (WSJ) portion of Penn Treebank. This corpus includes tokens labelled with POS tags"; the POS data (the first training data) comprise the POS tags (a first type of information), each labelling a token of the text of one of the WSJ documents (a plurality of entities), so the POS tags of each document's tokens are information relating to that document) and the second training data comprises a second type of information respective of the plurality of entities (Hashimoto: [0124], "To train the chunking layer 500, the WSJ corpus was used in which the chunks are labelled"; the chunking data (the second training data) comprise the chunking tags (a second type of information) labelled in the same WSJ corpus; [0082], "The tag specifies the region of major phrases (or chunks) in the sentence"; each chunking tag is assigned to a word and marks the region of a major phrase in a sentence of the text of one of the WSJ documents (the plurality of entities), so the chunking tags of each document's words are information relating to the same documents).
Regarding dependent claim 15, Hashimoto, in view of Zhang, teach the method of claim 10, wherein the layer is a first layer (Hashimoto: [0084], "In model 100, chunking is performed using a second bi-LSTM layer 510 on top of the POS layer"; counting from the POS layer, the second bi-LSTM layer 510 is the first layer stacked on it, the chunking layer 500 (a first layer)), wherein the method further comprises: adding a second layer to the machine learning model second version to create a machine learning model third version (Zhang: [0054], "Thereafter, the system 500 repeats the process for additional new tasks that are identified to provide continual or lifelong learning for multiple tasks"; the process of expanding the trained network for a new task, which Zhang follows with compression, is repeated for each additional new task, so another added layer (a second layer) is added for the next task to produce a further expanded version (a machine learning model third version)); and conducting third training on the machine learning model third version (Hashimoto: [0127], "To train the dependency parsing layer 600, the WSJ corpus with dependency labels was used"; the model with the dependency parsing layer 600 stacked on the chunking layer is trained on the dependency-labelled WSJ corpus, which is a third training of the model; [0136], "training the third layer using third layer training examples (e.g., dependency data)"; the third training uses the dependency data); wherein the deploying comprises deploying the machine learning model after the third training (Zhang: [0080], "The system then uses the adapted ML model to perform both existing and new tasks (step 715). Also, after step 715, the system may return to step 710 to continuously adapt the network architecture for additional new tasks for the lifelong learning for each new task identified"; the adapted model is put into use after the training for a new task, and the process returns to step 710 for each additional new task, so the further expanded version is again put into use after the training for the next task, which is deploying it after the third training).
Regarding dependent claim 16, Hashimoto, in view of Zhang, teach the method of claim 15, wherein the first training comprises first training data and the second training comprises second training data and the third training comprises third training data (Hashimoto: [0136], "training the first layer using first layer training examples (e.g., POS data), training the second layer using second layer training examples (e.g., chunking data)"; the first training uses the POS data (the first training data) and the second training uses the chunking data (the second training data); [0136], "training the third layer using third layer training examples (e.g., dependency data)"; the third training uses the dependency data (the third training data)), wherein the first training data, the second training data, and the third training data are different from one another (Hashimoto: [0136], "The training includes training the first layer using first layer training examples (e.g., POS data), training the second layer using second layer training examples (e.g., chunking data) with regularized pass down training to the first layer, and training the third layer using third layer training examples (e.g., dependency data) with regularized pass down training to the first and second layers"; the POS data, the chunking data and the dependency data are the separate training examples directed to the first, second and third layers, which are different from one another).
Regarding dependent claim 17, Hashimoto, in view of Zhang, teach the method of claim 16, wherein: the first training data comprises a first type of information respective of a plurality of entities (Hashimoto: [0121], "One training corpus for the POS layer 400 is the Wall Street Journal (WSJ) portion of Penn Treebank. This corpus includes tokens labelled with POS tags"; the POS data (the first training data) comprise the POS tags (a first type of information), each labelling a token of the text of one of the WSJ documents (a plurality of entities), so the POS tags of each document's tokens are information relating to that document); the second training data comprises a second type of information respective of the plurality of entities (Hashimoto: [0124], "To train the chunking layer 500, the WSJ corpus was used in which the chunks are labelled"; the chunking data (the second training data) comprise the chunking tags (a second type of information) labelled in the same WSJ corpus; [0082], "The tag specifies the region of major phrases (or chunks) in the sentence"; each chunking tag is assigned to a word and marks the region of a major phrase in a sentence of the text of one of the WSJ documents (the plurality of entities), so the chunking tags of each document's words are information relating to the same documents); and the third training data comprises a third type of information respective of the plurality of entities (Hashimoto: [0127], "To train the dependency parsing layer 600, the WSJ corpus with dependency labels was used"; the dependency data (the third training data) comprise the dependency labels (a third type of information) of the same WSJ corpus; [0089], "Dependency parsing identifies syntactic relationships (such as an adjective modifying a noun) between pairs of words in a sentence"; each dependency label identifies a syntactic relationship between a pair of words of a sentence of the text of one of the WSJ documents (the plurality of entities), so the dependency labels of each document's sentences are information relating to the same documents).
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Song et al. (hereinafter Song), US 2022/0138572 A1 in view of Kalyan et al. (hereinafter Kalyan) "AMMUS: A Survey of Transformer-based Pretrained Models in Natural Language Processing" 2021.
Regarding independent claim 19, Song teaches a method comprising (Song: [0006], "a computer implemented method for classifying documents is provided that includes: training, by a computer device, a machine learning model"; the computer implemented method trains a machine learning model and uses it to classify documents): receiving a first version of a machine learning model (Song: [0074], "For our continued pre-training, we use the RoBERTa Large model from Huggingface"; the RoBERTa Large model (a first version of a machine learning model) is obtained from Huggingface as the model on which training is then performed); conducting first training on the machine learning model first version using first training data (Song: [0085], "the out-of-the- box RoBERTa Large model was pre-trained on the Train split of the corresponding datasets for one epoch"; the domain-specific pre-training of the RoBERTa Large model is a first training conducted on the model as received; [0096], "domain transfer via masked language modeling over the individual paragraphs in the training set"; the individual paragraphs (first training data) are the paragraphs of the training set on which the pre-training is conducted), the first training data comprising a first type of information of a plurality of first types of information respective of a plurality of documents (Song: [0021], "the masked language modeling comprises randomly selecting tokens from the original document and replacing a portion of the selected tokens with a mask token"; the masked tokens of each training document have token values (a first type of information) drawn from a vocabulary of many values, information respective of the legal documents of the Train split (a plurality of documents)); adding a layer to the machine learning model first version after conducting the first training to create a machine learning model second version (Song: [0107], "Preferably this involves pretraining 1304 an out-of-the box model, such as the RoBERTa model, with a domain-specific dataset, such as the POSTURE50K, as discussed above. The pretrained model may be fine-tuned at 1306"; fine-tuning 1306 follows the pretraining 1304 of the out-of-the box model; [0076], "the output of the domain-specific RoBERTa model 322 is also sent to a dense layer (Dense_All 308) in order to classify a sample to the label space (Dense Output 312)"; the dense layer (a layer) Dense_All 308 receives the output of the already domain-pretrained RoBERTa model 322, so the dense layer is placed on the pretrained model after the pre-training, and the pretrained model with the dense layer is the fine tuned model (a machine learning model second version) that is used at 1312); conducting second training on the machine learning model second version using second training data (Song: [0006], "training, by a computer device, a machine learning model with a domain specific dataset comprising a plurality of documents each annotated with at least one label selected from a plurality of predefined labels for a given domain"; the fine-tuning trains the pretrained model with the domain specific dataset (a second training data) of labeled documents, which is a second training conducted on the model that carries the dense layer), the second training data comprising a second type of information of a plurality of second types of information respective of the plurality of documents (Song: [0064], "postures (a list of manually tagged procedural postures for this case)" and TABLE 2; the fine-tuning data comprise the procedural posture (a second type of information) categories tagged to each case of the same Train split (respective of the plurality of documents), one of 256 posture categories (a plurality of second types of information)); and deploying the machine learning model after the second training (Song: [0107], "Once trained, the model may be applied to classify documents. Specifically, the system and/or server may receive a document to be classified at 1310"; after the fine-tuning is complete, the trained model is put into use by the system to classify received documents, which is deploying the model after the second training), wherein the machine learning model first version is trained to predict the first type of information and the machine learning model second version is trained to predict the second type of information (Song: [0021], "the machine learning model predicts token values based on tokens surrounding the mask token"; the pre-trained model is trained to predict the masked token values; [0076], "the output of the domain-specific RoBERTa model 322 is also sent to a dense layer (Dense_All 308) in order to classify a sample to the label space (Dense Output 312)"; the fine tuned model with the dense layer is trained to predict the procedural posture labels).
Song does not expressly teach randomly-initialized.
However, Kalyan teaches randomly-initialized (Kalyan: page 6, Section 3.2.1 Pretraining from Scratch (PTS), "Models like BERT, RoBERTa, ELECTRA, and T5 are pretrained from scratch on large volumes of unlabeled text"; a RoBERTa model is pretrained from scratch on unlabeled text; page 7, left column, "All these layer parameters are randomly initialized and then learned during pretraining by minimizing the losses of one or more pretraining tasks. For example, BERT model is pretrained from scratch using MLM and NSP"; the model that is pretrained from scratch starts the pretraining with all of its layer parameters randomly initialized).
Because Song and Kalyan are analogous art with both addressing the pre-training of transformer-based neural network language models on unlabeled domain text before their use for downstream natural language predictions, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Kalyan's pretraining from scratch of a language model whose layer parameters are randomly initialized to the method of Song, with a reasonable expectation of success, by randomly initializing the layer parameters of the RoBERTa model received for the masked language model pre-training, instead of loading its out-of-the-box pre-trained weights, and then conducting the masked language model pre-training from those random values on a corpus comprising the paragraphs of the legal documents, with a vocabulary learned over that corpus, as Kalyan's domain-specific models, including the legal-domain LegalBERT, are pretrained from scratch (Kalyan: page 17, Section 4.1.4), the legal paragraphs being combined with general-domain text, with the legal text up-sampled, to the extent the available legal text is limited, as Kalyan's simultaneous pretraining provides (Kalyan: page 8, Section 3.2.3), and then adding the dense layer and fine-tuning as before, to teach receiving a randomly-initialized first version of a machine learning model. This modification would have been motivated by the desire to give the domain-specific language model a vocabulary learned over the domain text, because a model initialized from an existing general domain model keeps the general vocabulary, in which many domain-specific words are missing and are not represented properly, which hinders model learning (Kalyan: page 17, Section 4.1.4).
Claims 2 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Hashimoto in view of Zhang, as applied in the rejections of claims 1 and 10, respectively above, and further in view of Devlin et al. (hereinafter Devlin) "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" 2019.
Regarding dependent claim 2, Hashimoto, in view of Zhang, teach the computing system of claim 1.
Hashimoto and Zhang do not expressly teach wherein the machine learning model comprises a plurality of multi-directional transformer encoders.
However, Devlin teaches wherein the machine learning model comprises a plurality of multi-directional transformer encoders (Devlin: page 3, 3 BERT Model Architecture, "BERT's model architecture is a multi-layer bidirectional Transformer encoder"; BERT (the machine learning model) is built from a multi-layer Transformer encoder that attends in both directions; page 3, 3 BERT Model Architecture, "we denote the number of layers (i.e., Transformer blocks) as L"; the encoder consists of L stacked Transformer blocks (a plurality of multi-directional transformer encoders), twelve in BERTBASE and twenty-four in BERTLARGE, each using bidirectional self-attention).
Because Hashimoto, in view of Zhang, and Devlin are analogous art with all three addressing the training of multi-layer neural network models in which task-specific layers operate on the output of an underlying trained network, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Devlin's multi-layer bidirectional Transformer encoder to the computer system of Hashimoto and Zhang, with a reasonable expectation of success, by implementing the bi-directional LSTM that encodes the words of the input text for the POS layer, which Hashimoto permits to be any other conventional neural network (Hashimoto: [0145]), as the stack of bidirectional Transformer encoder blocks of Devlin, with the POS classifier and the later-added chunking layer receiving the encoder outputs as before, to teach wherein the machine learning model comprises a plurality of multi-directional transformer encoders. This modification would have been motivated by the desire to obtain a pre-trained model that can be fine-tuned with one additional output layer to create state-of-the-art models for a wide range of tasks (Devlin: page 1, Abstract).
Regarding dependent claim 11, Hashimoto, in view of Zhang, teach the method of claim 10.
Hashimoto and Zhang do not expressly teach wherein the machine learning model comprises a plurality of multi-directional transformer encoders.
However, Devlin teaches wherein the machine learning model comprises a plurality of multi-directional transformer encoders (Devlin: page 3, 3 BERT Model Architecture, "BERT's model architecture is a multi-layer bidirectional Transformer encoder"; BERT (the machine learning model) is built from a multi-layer Transformer encoder that attends in both directions; page 3, 3 BERT Model Architecture, "we denote the number of layers (i.e., Transformer blocks) as L"; the encoder consists of L stacked Transformer blocks (a plurality of multi-directional transformer encoders), twelve in BERTBASE and twenty-four in BERTLARGE, each using bidirectional self-attention).
Because Hashimoto, in view of Zhang, and Devlin are analogous art with all three addressing the training of multi-layer neural network models in which task-specific layers operate on the output of an underlying trained network, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Devlin's multi-layer bidirectional Transformer encoder to the method of Hashimoto and Zhang, with a reasonable expectation of success, by implementing the bi-directional LSTM that encodes the words of the input text for the POS layer, which Hashimoto permits to be any other conventional neural network (Hashimoto: [0145]), as the stack of bidirectional Transformer encoder blocks of Devlin, with the POS classifier and the later-added chunking layer receiving the encoder outputs as before, to teach wherein the machine learning model comprises a plurality of multi-directional transformer encoders. This modification would have been motivated by the desire to obtain a pre-trained model that can be fine-tuned with one additional output layer to create state-of-the-art models for a wide range of tasks (Devlin: page 1, Abstract).
Claims 5 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Hashimoto in view of Zhang, as applied in the rejections of claims 1 and 10, respectively above, and further in view of Song.
Regarding dependent claim 5, Hashimoto, in view of Zhang, teach the computing system of claim 1.
Hashimoto and Zhang do not expressly teach wherein the layer comprises a fully-connected layer.
However, Song teaches wherein the layer comprises a fully-connected layer (Song: [0076], "the output of the domain-specific RoBERTa model 322 is also sent to a dense layer (Dense_All 308)"; the layer placed on the pretrained model is a dense layer (a fully-connected layer), a dense layer being a layer in which each output unit is connected to every input of the layer).
Because Hashimoto, in view of Zhang, and Song are analogous art with all three addressing the training of multi-layer neural network models whose task-specific layers classify the output of an underlying network into a label space, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Song's dense layer that receives the output of the pretrained model and classifies it to the label space to the computer system of Hashimoto and Zhang, with a reasonable expectation of success, by implementing the classifier of the added chunking layer, which maps the chunking-layer hidden states onto the chunk label space (Hashimoto: [0069], [0085]), as such a dense layer, to teach wherein the layer comprises a fully-connected layer. This modification would have been obvious as the simple substitution of one known element, a dense layer that classifies the output of a network to a label space, for another known element that performs the same function, the classifier of the chunking layer, to obtain predictable results (KSR Int'l Co. v. Teleflex Inc., 550 U.S. 398, 416 (2007); MPEP 2143(I)(B)).
Regarding dependent claim 14, Hashimoto, in view of Zhang, teach the method of claim 10.
Hashimoto and Zhang do not expressly teach wherein the layer comprises a fully-connected layer.
However, Song teaches wherein the layer comprises a fully-connected layer (Song: [0076], "the output of the domain-specific RoBERTa model 322 is also sent to a dense layer (Dense_All 308)"; the layer placed on the pretrained model is a dense layer (a fully-connected layer), a dense layer being a layer in which each output unit is connected to every input of the layer).
Because Hashimoto, in view of Zhang, and Song are analogous art with all three addressing the training of multi-layer neural network models whose task-specific layers classify the output of an underlying network into a label space, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Song's dense layer that receives the output of the pretrained model and classifies it to the label space to the method of Hashimoto and Zhang, with a reasonable expectation of success, by implementing the classifier of the added chunking layer, which maps the chunking-layer hidden states onto the chunk label space (Hashimoto: [0069], [0085]), as such a dense layer, to teach wherein the layer comprises a fully-connected layer. This modification would have been obvious as the simple substitution of one known element, a dense layer that classifies the output of a network to a label space, for another known element that performs the same function, the classifier of the chunking layer, to obtain predictable results (KSR Int'l Co. v. Teleflex Inc., 550 U.S. 398, 416 (2007); MPEP 2143(I)(B)).
Claims 9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Hashimoto in view of Zhang, as applied in the rejections of claims 1 and 10, respectively above, and further in view of Kalyan and Song.
Regarding dependent claim 9, Hashimoto, in view of Zhang, teach the computing system of claim 1, wherein: the first training and the second training comprise use of training data selected from the plurality of documents (Hashimoto: [0121], "One training corpus for the POS layer 400 is the Wall Street Journal (WSJ) portion of Penn Treebank"; the POS data of the first training are selected from the Wall Street Journal (WSJ) portion of Penn Treebank, the WSJ corpus (the plurality of documents); [0124], "To train the chunking layer 500, the WSJ corpus was used in which the chunks are labelled"; the chunking data of the second training are selected from the same WSJ corpus).
Hashimoto and Zhang do not expressly teach the first version of the machine learning model is randomly-initialized.
However, Kalyan teaches the first version of the machine learning model is randomly-initialized (Kalyan: page 6, Section 3.2.1 Pretraining from Scratch (PTS), "Models like BERT, RoBERTa, ELECTRA, and T5 are pretrained from scratch on large volumes of unlabeled text"; a RoBERTa model is pretrained from scratch on unlabeled text; page 7, left column, "All these layer parameters are randomly initialized and then learned during pretraining by minimizing the losses of one or more pretraining tasks. For example, BERT model is pretrained from scratch using MLM and NSP"; the BERT model (the first version of the machine learning model) that is pretrained from scratch has all of its layer parameters randomly initialized before the pretraining learns them).
Because Hashimoto, in view of Zhang, and Kalyan are analogous art with all three addressing the training of multi-layer neural network models, Hashimoto and Kalyan specifically of neural language models trained on text, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Kalyan's pretraining from scratch of a language model whose layer parameters are randomly initialized to the computer system of Hashimoto and Zhang, with a reasonable expectation of success, by randomly initializing the parameters of the given network, including its word and character n-gram embedding parameters, and learning the embeddings from scratch over text of the domain of the WSJ documents, with a vocabulary learned over that text rather than over the general English Wikipedia text on which Hashimoto builds its character n-gram vocabulary (Hashimoto: [0063]), the domain text being combined with general text, with the domain text up-sampled, to the extent the available domain text is limited, as Kalyan's simultaneous pretraining provides (Kalyan: page 8, Section 3.2.3), before the POS training, the addition of the chunking layer, and the chunk training proceed as before, as Kalyan's domain-specific models, such as FinBERT and LegalBERT, are pretrained from scratch (Kalyan: page 17, Section 4.1.4), to teach the first version of the machine learning model is randomly-initialized. This modification would have been motivated by the desire to give the model a vocabulary learned over the domain text, because a model that keeps the vocabulary learned over general domain text is missing many domain-specific words, which are then not represented properly, which hinders model learning (Kalyan: page 17, Section 4.1.4).
Hashimoto, Zhang, and Kalyan do not expressly teach [[and]] (interpreted per the Claim Objections set forth above) deploying the machine learning model comprises deploying the machine learning model in association with a plurality of documents.
However, Song teaches deploying the machine learning model comprises deploying the machine learning model in association with a plurality of documents (Song: [0107], "the system and/or server may receive a document to be classified at 1310. The system may then classify the document using the trained/fine tuned model at 1312"; the trained model is put into use by classifying received legal documents; [0103], "The databases 1204 preferably include case law databases and a statutes database, which respectively include judicial opinions and statutes"; the classified documents are the judicial opinions held in the case law databases of the service, a plurality of documents with which the deployed model is used).
Because Hashimoto, in view of Zhang and Kalyan, and Song are analogous art with all four addressing the training of multi-layer neural network models and the use of the trained models to make predictions from input data, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Song's use of a trained model in a service that classifies received documents of the document collections held in the service databases to the computer system of Hashimoto, Zhang, and Kalyan, with a reasonable expectation of success, by deploying the trained tagging model in a service whose document collection holds the WSJ documents from which the POS and chunking training data are selected together with further documents that carry no gold POS or chunking annotations, and applying the model to the documents of that collection, which are tagged and stored for the service, as the model Song trains on legal opinions (Song: [0048]) classifies the judicial opinions held in the case law databases of the service (Song: [0103]), to teach deploying the machine learning model comprises deploying the machine learning model in association with a plurality of documents. This modification would have been motivated by the desire to tag the documents with the trained model and provide the tagged documents to the service for use therein (Song: [0107]).
Regarding dependent claim 18, Hashimoto, in view of Zhang, teach the method of claim 10, wherein: the first training and the second training comprise use of training data selected from the plurality of documents (Hashimoto: [0121], "One training corpus for the POS layer 400 is the Wall Street Journal (WSJ) portion of Penn Treebank"; the POS data of the first training are selected from the Wall Street Journal (WSJ) portion of Penn Treebank, the WSJ corpus (the plurality of documents); [0124], "To train the chunking layer 500, the WSJ corpus was used in which the chunks are labelled"; the chunking data of the second training are selected from the same WSJ corpus).
Hashimoto and Zhang do not expressly teach the first version of the machine learning model is randomly-initialized.
However, Kalyan teaches the first version of the machine learning model is randomly-initialized (Kalyan: page 6, Section 3.2.1 Pretraining from Scratch (PTS), "Models like BERT, RoBERTa, ELECTRA, and T5 are pretrained from scratch on large volumes of unlabeled text"; a RoBERTa model is pretrained from scratch on unlabeled text; page 7, left column, "All these layer parameters are randomly initialized and then learned during pretraining by minimizing the losses of one or more pretraining tasks. For example, BERT model is pretrained from scratch using MLM and NSP"; the BERT model (the first version of the machine learning model) that is pretrained from scratch has all of its layer parameters randomly initialized before the pretraining learns them).
Because Hashimoto, in view of Zhang, and Kalyan are analogous art with all three addressing the training of multi-layer neural network models, Hashimoto and Kalyan specifically of neural language models trained on text, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Kalyan's pretraining from scratch of a language model whose layer parameters are randomly initialized to the method of Hashimoto and Zhang, with a reasonable expectation of success, by randomly initializing the parameters of the given network, including its word and character n-gram embedding parameters, and learning the embeddings from scratch over text of the domain of the WSJ documents, with a vocabulary learned over that text rather than over the general English Wikipedia text on which Hashimoto builds its character n-gram vocabulary (Hashimoto: [0063]), the domain text being combined with general text, with the domain text up-sampled, to the extent the available domain text is limited, as Kalyan's simultaneous pretraining provides (Kalyan: page 8, Section 3.2.3), before the POS training, the addition of the chunking layer, and the chunk training proceed as before, as Kalyan's domain-specific models, such as FinBERT and LegalBERT, are pretrained from scratch (Kalyan: page 17, Section 4.1.4), to teach the first version of the machine learning model is randomly-initialized. This modification would have been motivated by the desire to give the model a vocabulary learned over the domain text, because a model that keeps the vocabulary learned over general domain text is missing many domain-specific words, which are then not represented properly, which hinders model learning (Kalyan: page 17, Section 4.1.4).
Hashimoto, Zhang, and Kalyan do not expressly teach [[and]] (interpreted per the Claim Objections set forth above) deploying the machine learning model comprises deploying the machine learning model in association with a plurality of documents.
However, Song teaches deploying the machine learning model comprises deploying the machine learning model in association with a plurality of documents (Song: [0107], "the system and/or server may receive a document to be classified at 1310. The system may then classify the document using the trained/fine tuned model at 1312"; the trained model is put into use by classifying received legal documents; [0103], "The databases 1204 preferably include case law databases and a statutes database, which respectively include judicial opinions and statutes"; the classified documents are the judicial opinions held in the case law databases of the service, a plurality of documents with which the deployed model is used).
Because Hashimoto, in view of Zhang and Kalyan, and Song are analogous art with all four addressing the training of multi-layer neural network models and the use of the trained models to make predictions from input data, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Song's use of a trained model in a service that classifies received documents of the document collections held in the service databases to the method of Hashimoto, Zhang, and Kalyan, with a reasonable expectation of success, by deploying the trained tagging model in a service whose document collection holds the WSJ documents from which the POS and chunking training data are selected together with further documents that carry no gold POS or chunking annotations, and applying the model to the documents of that collection, which are tagged and stored for the service, as the model Song trains on legal opinions (Song: [0048]) classifies the judicial opinions held in the case law databases of the service (Song: [0103]), to teach deploying the machine learning model comprises deploying the machine learning model in association with a plurality of documents. This modification would have been motivated by the desire to tag the documents with the trained model and provide the tagged documents to the service for use therein (Song: [0107]).
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Song in view of Kalyan, as applied in the rejection of claim 19 above, and further in view of Zhang.
Regarding dependent claim 20, Song, in view of Kalyan, teach the method of claim 19, wherein the layer is a first layer (Song: [0076], "the output of the domain-specific RoBERTa model 322 is also sent to a dense layer (Dense_All 308)"; the dense layer Dense_All 308, which is placed on the pretrained model in the rejection of claim 19 set forth above, is the layer of claim 19, a first layer) and the third training data comprising a third type of information respective of the plurality of documents (Song: [0064], "Each section further consists of a headtext (the title of a section) and paragraphs (a list of textual paragraphs in the section) while not all sections have a headtext. Here, the sections may be Facts, Analysis, Discussion, etc."; besides the opinion text and the procedural postures, each case record of the Train split (the third training data comprising the plurality of documents) carries the section titles (a third type of information respective of) of its opinion, which name the type of each section, such as Facts, Analysis, and Discussion, and are information relating to that case).
Song and Kalyan do not expressly teach wherein the method further comprises: adding a second layer to the machine learning model second version to create a machine learning model third version; and conducting third training on the machine learning model third version using third training data, wherein the deploying comprises deploying the machine learning model after the third training.
However, Zhang teaches adding a second layer to the machine learning model second version to create a machine learning model third version; (Zhang: [0078], "The expanded network architecture may include adding a layer to the network architecture", [0077], "identify the added layer as a task-specific layer for the new task", [0054], "Thereafter, the system 500 repeats the process for additional new tasks that are identified to provide continual or lifelong learning for multiple tasks"; the network of a trained model is expanded for a new task by adding a layer, which is learned as a task-specific layer for that task, and for each additional new task the expanded version, which Zhang may but need not compress ([0078], "After network expansion, the system may compress the expanded network architecture to generate the adapted ML model"), is expanded again, so another added layer (a second layer) is added to the expanded version (a machine learning model second version) for the next task to produce a further expanded version (a machine learning model third version)) and conducting third training on the machine learning model third version using third training data, (Zhang: [0077], "the system may train the ML model to perform the new task using training data for the new task"; the further expanded model is trained using the training data for the additional task (third training data), which is a third training; [0056], "Using MWC, the system 500 considers the discriminative weight subset by incorporating inherent correlations between old tasks and new task and learns the newly added or widened layer as a task-specific layer for the new task"; the training learns the added layer for the new task while taking into account the weights that matter to the earlier tasks) wherein the deploying comprises deploying the machine learning model after the third training (Zhang: [0080], "The system then uses the adapted ML model to perform both existing and new tasks (step 715). Also, after step 715, the system may return to step 710 to continuously adapt the network architecture for additional new tasks for the lifelong learning for each new task identified"; the adapted model is put into use after the training for a new task, and the process returns to step 710 for each additional new task, so the model is again put into use after the training for the next task).
Because Song, in view of Kalyan, and Zhang are analogous art with all three addressing the incremental training of multi-layer neural network models for successive prediction tasks, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Zhang's addition of a task-specific layer for each additional new task that is identified, its training of the expanded model with multi-task weight consolidation, and its use of the adapted model to the method of Song and Kalyan, with a reasonable expectation of success, by identifying, after the posture classifier is deployed, an additional task of classifying the paragraphs of a case opinion by the type of section to which they belong, adding to the fine-tuned model a further dense layer that receives the RoBERTa output alongside the dense layer Dense_All 308 (Song: [0078]), the dense layer for the posture task being retained, training the further expanded model with the MWC loss (Zhang: [0056]) on the paragraphs of the sections of the Train-split case records whose section titles name one of the section types that Song names, Facts, Analysis, and Discussion (Song: [0064]), with those types as the labels, so that the posture classification already learned is maintained, and then applying the model again to classify received documents by posture and their paragraphs by section type (Song: [0107]), to teach adding a second layer to the machine learning model second version to create a machine learning model third version; and conducting third training on the machine learning model third version using third training data, the third training data comprising a third type of information respective of the plurality of documents; wherein the deploying comprises deploying the machine learning model after the third training. This modification would have been motivated by the desire to learn new tasks while maintaining the performance on the previously learned tasks (Zhang: [0003]).
Response to Arguments
Applicant's arguments in Remarks filed 06/03/2026 have been fully considered. The arguments directed to Zhang (US 2020/0175362 A1) are not persuasive for the reasons set forth below. The arguments that depend on Zhu (US 2021/0374518 A1) are moot because Zhu is not relied upon in this action; each argument directed to Zhang itself is answered on its merits because Zhang is again applied, now as a secondary reference to Hashimoto (US 2018/0121799 A1) and, for claim 20, as a further reference to Song (US 2022/0138572 A1) in view of Kalyan.
Regarding the rejection of claims 1, 5-7, 10, 12, and 14-16 under 35 U.S.C. 102 as anticipated by Zhang, Applicant amended claims 1 and 10 and asserts that the rejection is moot. In view of the amendments, the rejection under 35 U.S.C. 102 over Zhang is withdrawn. No claim is rejected in this action as anticipated by Zhang. The amended claims are rejected under 35 U.S.C. 103 over Hashimoto in view of Zhang, alone or with further references, as set forth above, and the teaching for which Zhu was previously cited, first and second training data made of different types of data of one set of documents, is now supplied by Hashimoto, whose POS data and chunking data are different annotations of the same Wall Street Journal (WSJ) corpus (Hashimoto: [0121], [0124], [0136]).
Applicant argues that Zhang does not teach conducting first training on the first version trained to predict a first type of data of a plurality of first types of data in a document, or conducting second training on the second version trained to predict a second type of data based on the prediction of the first type of data in the document, because Zhang's system 500 does not train the model versions to predict different types of data with respect to given documents and Zhang's tasks are computer vision tasks such as image classification. This argument is not persuasive. The rejection does not rely on Zhang for these limitations. Hashimoto teaches them: its POS layer is trained on the POS-tagged WSJ corpus to predict POS tags (Hashimoto: [0077], [0121]), and the chunking layer stacked on the POS layer is trained on the chunk-labelled WSJ corpus to predict chunking tags (Hashimoto: [0082], [0124]) from the POS label embeddings weighted by the predicted POS probabilities, so the second prediction is made based on the first (Hashimoto: [0042], [0084]). In the rejections of claims 1 and 10, Zhang is relied upon only for receiving a given network, adding a layer to the network after its training on the first task, and using the adapted model after the training for the new task (Zhang: [0053]-[0055], [0077], [0080]). In the rejection of claim 20, Zhang is relied upon only for the addition, training, and use of a further layer for an additional task (Zhang: [0054], [0056], [0077], [0078], [0080]); the limitations Applicant discusses are met there by Song in view of Kalyan, as set forth in the rejection of claim 19. One cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. In re Keller, 642 F.2d 413, 426 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 1097 (Fed. Cir. 1986); MPEP 2145(IV). Nor is it required that Zhang's image tasks be bodily incorporated into Hashimoto; the test is what the combined teachings of the references would have suggested to one of ordinary skill in the art. In re Keller, 642 F.2d at 425; MPEP 2145(III).
The computer vision tasks do not remove Zhang from the pertinent art. Zhang presents them as examples, stating that "Tasks may include image classification, image segmentation, object detection and/or many other computer vision tasks" (Zhang: [0024]) and that "the principles of the present disclosure may be implemented in any suitably-arranged system or device" (Zhang: [0020]). Zhang is in the field of training neural network models to learn additional tasks, and it is reasonably pertinent to the problem the claims address, adding a layer to an already trained model so that the model learns a further prediction task: Zhang identifies "two primary obstacles for lifelong learning of deep neural networks: catastrophic forgetting and capacity limitations" (Zhang: [0021]), the problem Hashimoto faces when it allows its model "to grow in depth while avoiding catastrophic forgetting" (Hashimoto: [0035]). In re Bigio, 381 F.3d 1320, 1325 (Fed. Cir. 2004); In re Klein, 647 F.3d 1343, 1348 (Fed. Cir. 2011); MPEP 2141.01(a).
Applicant further argues that Zhang does not teach that the first training data comprises the plurality of first types of data in a set of documents and the second training data comprises the plurality of second types of data in the set of documents, because Zhang trains the new task without access to the old training database (Zhang: [0051], [0055], [0074]). This argument is not persuasive. The common set of documents is taught by Hashimoto, whose POS layer and chunking layer are trained on the POS annotations and the chunk annotations of the same WSJ corpus (Hashimoto: [0121], [0124]); Zhang is not relied upon for the training data. In re Keller; In re Merck. Moreover, Zhang's ability to learn a new task without the old task's data is a capability for the situation in which that data cannot be obtained, as Zhang explains: "If the initial model is pre-trained by a third-party, access to the training data may not be available" (Zhang: [0073]). Zhang does not require that available data be withheld, and its framework "includes at least one of the three stages" (Zhang: [0021]). As stated in the rejections of claims 1 and 10, only Zhang's addition of a task-specific layer after the first task is learned, the training of the expanded network with the training data for the new task, and the use of the adapted model are applied there; Zhang's multi-task weight consolidation for learning without the old tasks' data, including its use as the loss of the AutoML expansion (Zhang: [0065]), and its later compression of the expanded network (Zhang: [0078]) are not imported into those rejections. Nor do the claims require that the second training use the first training data; they require that the first and second training data comprise their types of data in one set of documents. After Hashimoto's POS training, the chunking layer is added and the expanded model is trained on the chunk annotations of the same WSJ corpus with Hashimoto's own chunking objective, successive regularization holding the trained POS-layer parameters near their trained values (Hashimoto: [0124], [0125]), which is consistent with Zhang's training of the new task "using training data for the new task" (Zhang: [0077]). As to claim 20, the common set of documents is supplied by Song's case records, which carry the opinion text, the procedural postures, and the section titles of each case (Song: [0064]), not by Zhang. That rejection does use Zhang's multi-task weight consolidation, which learns the added layer for the section-type task while taking into account the weights that matter to the posture task already learned (Zhang: [0056]). The section-type task is trained on the data for that task, the section titles and the paragraphs they head, as Zhang trains each new task using the training data for that task (Zhang: [0077]), and Zhang's ability to do so without the old task's data is a capability, not a requirement (Zhang: [0051], [0073]).
Applicant also argues that Zhang teaches away from training the model for the old and new tasks on different types of data from a common set of documents, because a benefit of Zhang's approach is that the model "can retain old task performance without the original training data for the old tasks" (Zhang: [0051]). This argument is not persuasive. A reference teaches away when a person of ordinary skill, upon reading it, would be discouraged from following the path set out in the reference, or would be led in a direction divergent from the path taken by the applicant. In re Gurley, 27 F.3d 551, 553 (Fed. Cir. 1994). The disclosure of an alternative or a preferred mode does not teach away where the reference does not criticize, discredit, or otherwise discourage the claimed solution. In re Fulton, 391 F.3d 1195, 1201 (Fed. Cir. 2004); MPEP 2141.02(VI); MPEP 2145(X)(D)(1). Zhang nowhere criticizes, discredits, or discourages training on data that are available, or drawing the data for successive tasks from one document collection; it describes retaining old-task performance without the old data as an advantage when that data cannot be accessed (Zhang: [0051], [0073]). A technique that remains usable when data is missing does not teach that the data should not be used when it is present. The combinations as proposed also do not change Zhang's handling of training data. For claims 1 and 10, they apply Zhang's addition of a task-specific layer to Hashimoto, whose tasks are already defined on one annotated corpus. For claim 20, the added layer is trained with Zhang's multi-task weight consolidation on the data of the new task, the section titles and the paragraphs they head in Song's case records (Song: [0064]; Zhang: [0056], [0077]), which is how Zhang learns each new task.
Applicant argues that the combination lacks an articulated reason with rational underpinning and that the Office relied on Applicant's specification as a blueprint, which is impermissible hindsight. This argument is not persuasive as to the combination applied in this action. The reason to combine is drawn from the references themselves. Hashimoto seeks to allow its model "to grow in depth while avoiding catastrophic forgetting" (Hashimoto: [0035]); Zhang identifies catastrophic forgetting and capacity limitations as the obstacles to learning new tasks (Zhang: [0021]) and expands the network architecture so that the model learns the new tasks "with improved model capability and performance" (Zhang: [0021]), learning the added layer as a task-specific layer for the new task (Zhang: [0053], [0056]). The rejections of claims 1 and 10 apply that teaching to the next task of Hashimoto's own task hierarchy, with a reasonable expectation of success because Hashimoto already trains each overlying layer on the outputs of the trained layer beneath it, and each of those rejections states the reason with citations to Hashimoto [0035] and Zhang [0021]. The rejection of claim 20 applies Zhang's addition of a task-specific layer for an additional task, trained with multi-task weight consolidation, to the model of Song and Kalyan for the purpose Zhang states, to "learn new tasks while maintaining the performance on the previously learned tasks" (Zhang: [0003]); the new task is drawn from the section titles that Song's own case records carry, and Song notes that "not all sections have a headtext" (Song: [0064]); and Song already places more than one task-specific dense layer on the output of its pretrained model (Song: [0078]). The rejections also explain why the sequential addition does not change Hashimoto's principle of operation, although Hashimoto trains its model jointly over all datasets in each epoch (Hashimoto: [0119], [0125]). The rationale cites no passage of Applicant's specification. Any judgment on obviousness is in a sense necessarily a reconstruction based upon hindsight reasoning, but so long as it takes into account only knowledge within the level of ordinary skill at the time the claimed invention was made and does not include knowledge gleaned only from the applicant's disclosure, such a reconstruction is proper. In re McLaughlin, 443 F.2d 1392, 1395 (CCPA 1971); MPEP 2145(X)(A). The rejections articulate reasoning with rational underpinning, as KSR requires. KSR Int'l Co. v. Teleflex Inc., 550 U.S. 398, 418 (2007); MPEP 2142, 2143.
Applicant argues that independent claims 10 and 19 are patentable for reasons similar to those given for claim 1, and that the dependent claims are patentable at least by their dependency. As to claim 10 and the dependent claims rejected over Hashimoto in view of Zhang, alone or with Devlin, Song, or Kalyan, the arguments are not persuasive for the reasons given above, and no separate argument is presented for the limitations those dependent claims add. Claim 19 is rejected over Song (US 2022/0138572 A1) in view of Kalyan (arXiv:2108.05542v1), a ground that relies on neither Zhang nor Zhu, so the reasons Applicant incorporates for claim 19 do not address that ground. The limitations Applicant relies on, a first training directed to a first type of information and a second training directed to a second type of information, both respective of one plurality of documents, are met by Song's pre-training on the paragraphs of the legal opinions of the training split and its fine-tuning on the procedural postures carried by the same case records (Song: [0021], [0064], [0096]), and the random initialization is taught by Kalyan, as set forth in the rejection of claim 19 above. Claim 20 is rejected over Song in view of Kalyan and further in view of Zhang, a ground that relies on Zhang only for the addition of a task-specific layer for an additional task, its training with multi-task weight consolidation, and the use of the adapted model (Zhang: [0054], [0056], [0077], [0078], [0080]); the arguments directed to Zhang are not persuasive as to that ground for the reasons given above, and no separate argument is presented for the limitations claim 20 adds. The dependent claims are not allowable by virtue of their dependency because their parent claims stand rejected.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Collobert et al., “Natural Language Processing (almost) from Scratch”, 2 March 2011 (Abstract: We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including: part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KUANG FU CHEN whose telephone number is (571)272-1393. The examiner can normally be reached M-F 9:00-5:30pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached on (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KC CHEN/Primary Patent Examiner, Art Unit 2143
