Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in instant Application No. 18/278,298, filed on 8/22/2023. As such the effective filing date for claims 1-9 is 2/9/2022.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 8/22/2023 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Status
Claims 1-9 are pending.
Claims 1-9 are rejected.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation is: “a task module for performing task prediction” in claim 1.
Because this/these claim limitation is being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it is being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof. The specification provides a description of the “task module” on page 11 of the specification beginning in line 4, “the task module includes a task mapping layer corresponding to each type of protein conformation, and each task mapping layer maps protein embedding under each type of prompt identifier to different task spaces by using different mapping functions, that is, each task mapping layer performs different task predictions. Preferably, the task mapping layer may
perform different task predictions by using a dual-layer MLP as a mapping function”.
If applicant does not intend to have this limitation interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation to avoid it being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation recites sufficient structure to perform the claimed function so as to avoid it being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, and 6-9 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract ideas without significantly more. The claims recite a method of perceptual representation learning for protein conformations. This judicial exception is not integrated into a practical application because while claims 1, and 6-9 attempt to integrate the exception into a practical application, said practical application is a generically recited computer element that does not add meaningful limitations to the abstract idea as it is simply implementing the abstract idea on a computer. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the computer elements only store and retrieve information in memory as well as perform basic calculations that are known to be well-understood, routine and conventional computer functions as recognized by the decisions listed in MPEP § 2106.05(d).
Framework with which to Analyze Subject Matter Eligibility:
Step 1: Are the claims directed to a category of statutory subject matter (a process, machine manufacture, or composition of matter)? [see MPEP § 2106.03]
Claims are directed to statutory subject matter, specifically a method (1, and 6-9).
Step 2A Prong One: Do the claims recite a judicially recognized exception, i.e., an abstract idea, a law of nature, or a natural phenomenon? [see MPEP § 2106.04(a)]
With respect to the Step 2A Prong One evaluation, the instant claims are found herein to recite abstract ideas that fall into the grouping of mental processes and mathematical concepts.
The following claims recite abstract ideas (mental processes and mathematical concepts):
Claim 1: Building different data sets, defining a prompt for each type of protein conformation, building a representation learning module, building a task module, building a loss function, updating model parameters, and extracting the representation learning module are processes of comparing/contrasting, identifying, selecting, and calculating information that can be done via pen and paper or within the human mind and are therefore abstract ideas, specifically mental processes. Building a loss function is a verbal articulation of a mathematical process and is therefore an abstract idea, specifically a mathematical concept.
Claim 6: The loss function minimizing an error of the task prediction result and the tag is a process of calculating information that can be done via pen and paper or within the human mind and is therefore an abstract idea, specifically a mental process. The loss function minimizing an error of the task prediction result and the tag is a verbal articulation of a mathematical process and is therefore an abstract idea, specifically a mathematical concept.
Claim 7: Applying the module to a prediction task, embedding representations, and the embedding representations being configured to predict the protein structure/function are processes of comparing/contrasting, and calculating information that can be done via pen and paper or within the human mind and are therefore abstract ideas, specifically mental processes.
Claim 8: Splicing the embedding representations, and performing fusion on an obtained spliced representation are processes of comparing/contrasting, and calculating information that can be done via pen and paper or within the human mind and are therefore abstract ideas, specifically mental processes. The fusion comprising the specified methods is merely further limiting the data itself which is an abstract idea, specifically a mental process.
Claim 9: The protein conformation comprising the specified states, the natural folding state comprising the specified information, and the interaction state comprising the specified information are merely further limiting the data itself which is an abstract idea, specifically a mental process.
Step 2A Prong Two: If the claims recite a judicial exception under prong one, then is the judicial exception integrated into a practical application? [see MPEP § 2106.04(d)]
Because the claims do recite judicial exceptions, direction under Step 2A Prong Two provides that the claims must be examined further to determine whether they integrate the abstract ideas into a practical application.
The following claims recite the following additional elements in the form of non-abstract elements:
Claim 1: Obtaining a protein made up of an amino acid sequences is an insignificant extra solution activity, specifically mere data gathering (See Performing clinical tests on individuals to obtain input for an equation, In re Grams, 888 F.2d 835, 839- 40; 12 USPQ2d 1824, 1827-28 (Fed. Cir. 1989) and Determining the level of a biomarker in blood, Mayo, 566 U.S. at 79, 101 USPQ2d at 1968. See also PerkinElmer, Inc. v. Intema Ltd., 496 Fed. App'x 65, 73, 105 USPQ2d 1960, 1966 (Fed. Cir. 2012) (assessing or measuring data derived from an ultrasound scan, to be used in a diagnosis)) [See MPEP § 2106.05(g)].
Step 2B: If the claims do not integrate the judicial exception, do the claims provide an inventive concept? [see MPEP § 2106.05]
Because the additional claim elements do not integrate the abstract ideas into a practical application, the claims are further examined under Step 2B, which evaluates whether the additional elements, individually and in combination, amount to significantly more than the judicial exception itself by providing an inventive concept.
The claims do not recite additional elements that are sufficient to amount to significantly more than the judicial exceptions because the claims recite additional elements that are generic, conventional, nonspecific, or insignificant extra solution activity. These additional elements include:
The additional element of obtaining a protein made up of an amino acid sequences is an insignificant extra solution activity, specifically mere data gathering, that are recognized as well understood, routine and conventional by the courts (See Analyzing DNA to provide sequence information or detect allelic variants, Genetic Techs. Ltd., 818 F.3d at 1377; 118 USPQ2d at 1546, Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93, Electronic recordkeeping, Alice Corp. Pty. Ltd. v. CLS Bank Int'l, 573 U.S. 208, 225, 110 USPQ2d 1984 (2014) (creating and maintaining "shadow accounts"); Ultramercial, 772 F.3d at 716, 112 USPQ2d at 1755 (updating an activity log), and Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information)) [See MPEP § 2106.05(g)]. Therefore, taken both individually and as whole, the additional elements do not amount to significantly more than the judicial exception by providing an inventive concept.
Therefore, claims 1, and 6-9, when the limitations are considered individually and as a whole, are rejected under 35 U.S.C. § 101 as being directed to non-statutory subject matter.
Claim 2, and thus claims 3-5 as being dependent, is not rejected under 35 U.S.C. 101 for being directed to abstract ideas without significantly more, as the claim recites sufficient structure so as to be directed to a particular machine under 35 U.S.C. 101 [See MPEP § 2106.04(d) and 2106.05(b)]. More specifically, the first claim is directed to building a method (building a model, building a module, building a loss function, and extracting parameters) which on their own are abstract ideas, however the building a specific model, with a specific architecture is akin to a particular machine, especially as though the additional elements of claim 2 might be conventional components on their own, but are not conventional in their arrangement and connection to each other as described within the claim. As such, claims 2-5 are not rejected under 35 U.S.C. 101 for being directed to non-statutory subject matter.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (Proceedings of the 2021 conference on empirical methods in natural language processing (2021) 2792-2802) and Einaggar et al. (IEEE transactions on pattern analysis and machine intelligence (2021) 7112-7127).
Claim 1 is directed to a method for predicting protein conformations based on a language model using prompt-based learning.
Wang et al. teaches in the abstract “Based on continuous prompt embeddings, we propose TransPrompt, a transferable prompting framework for few-shot learning across similar tasks. In TransPrompt, we employ a multitask meta-knowledge acquisition procedure to train a meta-learner that captures cross-task transferable knowledge. Two de-biasing techniques are further designed to make it more task-agnostic and unbiased towards any tasks. After that, the meta-learner can be adapted to target tasks with high accuracy”, on page 2792, column 1, paragraph 1 “Fine-tuning Pre-trained Language Models (PLMs) has become the standard practice to train models for a majority of NLP tasks… Inspired by these works, Gao et al. (2020) propose a prompt-based approach to fine-tune BERTstyle PLMs in a few-shot learning setting, which adapts PLMs into producing specific tokens corresponding to each class, instead of learning the prediction head”, on page 2792, column 2, paragraph 2 “A natural question arises: how can we design a prompting framework for BERT-style models that captures transferable knowledge across similar NLP tasks to improve the performance of few-shot learning”, on page 2795, column 1, paragraph 1 “The average pooled results from both task-specific and universal prompt encoders are treated as the prompt embedding…As prompt parameters are fully differentiable, during back propagation, they effectively capture the task-specific and universal knowledge”, on page 2797, column 2, paragraph 1 “Specifically, we employ separate prompts (either discrete prompts or continuous prompt embeddings)”, and on page 2800, column 1, paragraph 2 “we present the TransPrompt framework for few-shot learning across similar NLP tasks based on continuous prompt embeddings”, reading on building, based on a pre-trained language model, a representation learning module for fusing an embedding representation of each type of the prompt into an embedding representation of the protein, so as to obtain a protein embedding representation under a prompt identifier. Wang et al. teaches on page 2796, column 1, paragraph 1 “we consider the model prediction entropy H(Dm) over Dm: where yhat(x) is the predicted probability of x being assigned to the class yhat = Y”, reading on building a task module for performing task prediction on a task corresponding to each type of protein conformation based on the protein embedding representation under the prompt identifier. Wang et al teaches on page 2796, column 1, paragraph 1 “we consider the model prediction entropy H(Dm) over Dm: where yhat(x) is the predicted probability of x being assigned to the class yhat = Y…minimizing L’ (θ) is a non-trivial problem. This is because when we calculate s(x), we must obtain model parameters of the PLM beforehand…We employ a dual optimization process to solve the problem of L’ (θ). In the initial stage, all s(x)s are uniformly initialized. Next, we fix s(x)s as constants to minimize l(x; y; θ) in L’ (θ)”, on page 2796, column 2, paragraph 2 “The entire model is trained over the dataset D, with the loss function L(θ)”, reading on building a loss function for each type of task based on a task prediction result and a tag, and updating model parameters of the representation learning module and the task module in combination with loss functions of all types of tasks and the different data sets.
Wang et al. does not teach the application of their method to amino acid sequences or protein conformation prediction.
Einaggar et al. teaches in the abstract “we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids. The protein LMs (pLMs) were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores. Dimensionality reduction revealed that the raw pLM-embeddings from unlabeled data captured some biophysical features of protein sequences. We validated the advantage of using the embeddings as exclusive input for several subsequent tasks: (1) a per-residue (per-token) prediction of protein secondary structure”, and on page 7119, column 2, paragraph 2 “To ease comparability, we evaluated all models on standard performance measures (Q3/Q8: three/eight-state per-residue accuracy, i.e., percentage of residues predicted correctly in either of the 3/8 secondary structure states) and on standard data sets (CASP12, TS115, CB513). To increase the validity, we added a novel, non-redundant test set (dubbed NEW364). For simplicity, we only presented values for Q3 on CASP12 and NEW364 (TS115 and CB513 contain substantial redundancy; Q8 results brought little novelty; SOM Tables 7, 6, available online)”, reading on a perceptual representation learning method for protein conformations based on a pre-trained language model, comprising the following steps: obtaining a protein made up of an amino acid sequence, building different data sets according to protein conformations, and defining a prompt for each type of protein conformation. Einaggar et al. teaches on page 7119, column 2, paragraph 4 “Prot-Bert outperformed other models trained on the same corpus (SOM Tables 7, 6, available online). For ProtTXL and Prot-Bert, we could analyze the influence of database size upon performance”, reading on after the model parameters are updated, extracting the representation learning module as a protein representation module.
It would have been obvious at the time of first filing to have modified the teachings of Wang et al. for the use of prompt learning in PLMs with the teachings of Einaggar et al. for the use of a BERT model in structure prediction as the former explicitly teaches on page 2792, column 2, paragraph 2 “natural question arises: how can we design a prompting framework for BERT-style models that captures transferable knowledge across similar NLP tasks to improve the performance of few-shot learning” and within the abstract “Extensive experiments show that TransPrompt outperforms single-task and cross-task strong baselines over multiple NLP tasks and datasets. We further show that the meta-learner can effectively improve the performance on previously unseen tasks. TransPrompt also outperforms strong fine-tuning baselines when learning with full training sets”, while the latter teaches in the abstract “For secondary structure, the most informative embeddings (ProtT5) for the first time outperformed the state-of-the-art without multiple sequence alignments (MSAs) or evolutionary information thereby bypassing expensive database searches. Taken together, the results implied that pLMs learned some of the grammar of the language of life”. One would have had a reasonable expectation of success given that the latter uses a BERT architecture and the former is explicitly designed to work with said architecture to enhance the performance of the model through prompt based learning. Therefore, it would have been obvious at the time of first filing to have modified the teachings of each and to be successful.
Claim 6 is directed to the method of claim 1 but further specifies that the loss function is to minimize the error or the task prediction and tag.
Wang et al teaches on page 2796, column 1, paragraph 1 “we consider the model prediction entropy H(Dm) over Dm: where yhat(x) is the predicted probability of x being assigned to the class yhat = Y…minimizing L’ (θ) is a non-trivial problem. This is because when we calculate s(x), we must obtain model parameters of the PLM beforehand…We employ a dual optimization process to solve the problem of L’ (θ). In the initial stage, all s(x)s are uniformly initialized. Next, we fix s(x)s as constants to minimize l(x; y; θ) in L’ (θ)”, reading on wherein the loss function for each type of prediction task is to minimize an error of the task prediction result and the tag.
Claims 2-4, and 7-8 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (Proceedings of the 2021 conference on empirical methods in natural language processing (2021) 2792-2802) and Einaggar et al. (IEEE transactions on pattern analysis and machine intelligence (2021) 7112-7127) as applied to claims 1 and 6 above, and further in view of Qin et al. (IEEE International Conference on Bioinformatics and Biomedicine (2020) 708-715).
Claim 2 is directed to the method of claim 1 but further specifies the exact architecture of the language model, including an prompt embedding layer, amino acid embedding layer, and fusion layer.
Wang et al. and Einaggar et al. teach the method of claim 1 as previously described.
Wang et al. teaches on page 2795, column 1, paragraph 1 “The average pooled results from both task-specific and universal prompt encoders are treated as the prompt embedding…As prompt parameters are fully differentiable, during back propagation, they effectively capture the task-specific and universal knowledge”, on page 2797, column 2, paragraph 1 “Specifically, we employ separate prompts (either discrete prompts or continuous prompt embeddings)”, and on page 2800, column 1, paragraph 2 “we present the TransPrompt framework for few-shot learning across similar NLP tasks based on continuous prompt embeddings”, reading on wherein the prompt embedding layer is configured to learn the embedding representation of each type of prompt.
Einaggar et al. teaches on page 7118, column 2, paragraph 2 “Applying t-SNE to the uncontextualized token embedding layer visualized information extracted by the pLMs for individual amino acids independent of their context”, reading on the amino acid embedding layer is configured to learn the embedding representation of the protein.
Wang et al. and Einaggar et al. do not teach the use of a fusion layer or the use of a mapping layer.
Qin et al. teaches in the abstract “We pre-train BERT on large-scale chemical interaction corpora and re-define it as ChemicalBERT to generate high-quality contextual representation, and employ AGGCN to capture syntactic graph information of the sentence. Finally, the contextual representation and syntactic graph representation are merged into a fusion layer and then fed into the fully-connected softmax layer to extract CPIs”, and on page 709, column 1, paragraph 6 “we feed the features generated by ChemicalBERT and AGGCN into the fusion layer in parallel and then employ a fully-connected softmax layer”, which in view of Wang et al. and Einaggar et al. reads on wherein the representation learning module comprises a prompt embedding layer, an amino acid embedding layer, a fusion layer, and the pre-trained language model, and the fusion layer is configured to fuse the embedding representation of each type of the prompt and the embedding representation of protein to obtain a fused representation, and the pre-trained language model is configured to perform representation learning on the fused representation to obtain the protein embedding representation under each type of the prompt identifier. Qin et al. teaches on page 712, column 2, paragraph 1 “Once we obtain hidden representations of all tokens, we apply a max pooling function that maps N from output vectors to 1 sentence vector. And we get the sentence representation”, reading on the task module comprises a task mapping layer corresponding to each type of protein conformation, and each task mapping layer is configured to perform task prediction based on the protein embedding representation under each type of the prompt identifier.
It would have been obvious at the time of first filing to have modified the teachings of Wang et al. and Einaggar et al. for the method of claim 1, with the teachings of Qin et al. for the design of their BERT model as the latter is also examining sequence based information to predict binding information, which is inherently based on protein structure, which Einaggar et al. is attempting to capture, and Qin et al. teaches in the abstract “We achieve state-of-the-art results for the CPI extraction with a microaveraged F1-score of 80.21%. To further demonstrate the efficacy of the proposed model, we have also conducted experiments on the DDIExtraction 2013 corpus and obtained a micro-averaged F1-score of 82.88%, which is also the highest score compared to the existing models. Experimental results show that our proposed model can adequately capture semantic and syntactic information by parallelly extracting sentence features from different views”. One would have had a reasonable expectation of success given that the latter is a BERT model, which Wang et al. is designed to work with, and is attempting to capture much of the same information as Einaggar et al. using a different subset of layers within their model, making it a substitution of one known method for another. Therefore, it would have been obvious at the time of first filing to have modified the teachings of each and to be successful.
Claim 3 is directed to the method of claim 2 but further specifies that the amino acid embedding layer comprises the specified layer.
Wang et al. and Einaggar et al. teach the method of claim 1 as previously described.
Einaggar et al. teaches on page 7114, column 2, paragraph 4 “As self-attention is a set-operation and thus order-independent, Transformers require explicit positional encoding”, and on page 7118, column 2, paragraph 2 “Applying t-SNE to the uncontextualized token embedding layer visualized information extracted by the pLMs for individual amino acids independent of their context”, reading on wherein the amino acid embedding layer comprises an amino acid information embedding layer and an amino acid position embedding layer which are configured to extract an amino acid information representation and an amino acid position representation according to the amino acid sequence, respectively, and the amino acid information representation and the amino acid position representation are superimposed to obtain the embedding representation of the protein.
Claim 4 is directed to the method of claim 2 but further specifies that the language model comprises one of those specified.
Wang et al. and Einaggar et al. teach the method of claim 1 as previously described.
Einaggar et al. teaches in the abstract “we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids”, reading on wherein the pre-trained language model is a pluggable masked pre-trained language model, wherein the masked pre-trained language model comprises BERT, RoBERTa, ALBERT, and XLNet.
Claim 7 is directed to the method of claim 1 but further specifies the method is applied to protein structure/function prediction.
Wang et al. and Einaggar et al. teach the method of claim 1 as previously described.
Einaggar et al. teaches in the abstract “we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids. The protein LMs (pLMs) were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores. Dimensionality reduction revealed that the raw pLM-embeddings from unlabeled data captured some biophysical features of protein sequences. We validated the advantage of using the embeddings as exclusive input for several subsequent tasks: (1) a per-residue (per-token) prediction of protein secondary structure”.
Wang et al. and Einaggar et al. do not teach the use of a fusion layer.
Qin et al. teaches in the abstract “We pre-train BERT on large-scale chemical interaction corpora and re-define it as ChemicalBERT to generate high-quality contextual representation, and employ AGGCN to capture syntactic graph information of the sentence. Finally, the contextual representation and syntactic graph representation are merged into a fusion layer and then fed into the fully-connected softmax layer to extract CPIs”, and on page 709, column 1, paragraph 6 “we feed the features generated by ChemicalBERT and AGGCN into the fusion layer in parallel and then employ a fully-connected softmax layer”, which in view of Wang et al. and Einaggar et al. reads on wherein the protein representation module is applied to a prediction task for a protein structure and/or a protein function; during application, in the protein representation module, embedding representations of all types of the prompts are simultaneously fused into the embedding representation of the protein, so as to obtain protein embedding representations under all the prompt identifiers; and the protein embedding representations under all the prompt identifiers are configured to predict the protein structure and/or the protein function.
Claim 8 is directed to the method of claim 7 but further specifies splicing the embedding representations of all the prompts first then performing fusion, full connection mapping and convolutional mapping.
Wang et al. and Einaggar et al. teach the method of claim 1 as previously described.
Wang et al. and Einaggar et al. do not teach the splicing of the embedding representations of all the prompts first then performing fusion, full connection mapping and convolutional mapping.
Qin et al. teaches in the abstract “We pre-train BERT on large-scale chemical interaction corpora and re-define it as ChemicalBERT to generate high-quality contextual representation, and employ AGGCN to capture syntactic graph information of the sentence. Finally, the contextual representation and syntactic graph representation are merged into a fusion layer and then fed into the fully-connected softmax layer to extract CPIs”, on page 712, column 2, paragraph 1 “In the fusion layer, we concatenate the output of ChemicalBERT representation”, and on page 712, column 1, paragraph 1 “For each A˜(t), it is fed into the later graph convolutional layer”, reading on splicing the embedding representations of all the types of the prompts first, and then performing fusion on an obtained spliced representation and the embedding representation of the protein, wherein the fusion comprises splicing, full connection mapping, and convolutional mapping.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (Proceedings of the 2021 conference on empirical methods in natural language processing (2021) 2792-2802), Einaggar et al. (IEEE transactions on pattern analysis and machine intelligence (2021) 7112-7127), and Qin et al. (IEEE International Conference on Bioinformatics and Biomedicine (2020) 708-715) as applied to claims 2-7, and 7-8 above, and further in view of Gao et al. (Proceedings of the 2021 conference on empirical methods in natural language processing (2021) 6894-6910).
Claim 5 is directed to the method of claim 2 but further specifies that the task mapping layer comprises a dual-layer MLP for task prediction.
Wang et al., Einaggar et al., and Qin et al. teach the method of claim 2 as previously described.
Wang et al., Einaggar et al., and Qin et al. do not teach the task mapping layer comprises a dual-layer MLP for task prediction.
Gao et al. teaches in the abstract “This paper presents SimCSE, a simple contrastive learning framework that greatly advances the state-of-the-art sentence embeddings. We first describe an unsupervised approach, which takes an input sentence and predicts itself in a contrastive objective, with only standard dropout used as noise. This simple method works surprisingly well, performing on par with previous supervised counterparts. We find that dropout acts as minimal data augmentation and removing it leads to a representation collapse. Then, we propose a supervised approach, which incorporates annotated pairs from natural language inference datasets into our contrastive learning framework, by using “entailment” pairs as positives and “contradiction” pairs as hard negatives”, and on page 6906, column 1, paragraph 2 “For both unsupervised and supervised SimCSE, we take the [CLS] representation with an MLP layer on top of it as the sentence representation. Specially, for unsupervised SimCSE, we discard the MLP layer and only use the [CLS] output during test, since we find that it leads to better performance”, reading on wherein the task mapping layer comprises a dual-layer MLP for performing task prediction based on the protein embedding representation under each type of the prompt identifier.
It would have been obvious at the time of first filing to have modified the teachings of Wang et al., Einaggar et al., and Qin et al. for the method of claim 2, with the teachings of Gao et al. for the use of a MLP layer as a mapping layer for embeddings, as the latter teaches in the abstract “our unsupervised and supervised models using BERTbase achieve an average of 76.3% and 81.6% Spearman’s correlation respectively, a 4.2% and 2.2% improvement compared to previous best results. We also show—both theoretically and empirically—that contrastive learning objective regularizes pre-trained embeddings’ anisotropic space to be more uniform, and it better aligns positive pairs when supervised signals are available”. One would have had a reasonable expectation of success given that the SimCSE of Gao et al. is merely a training framework for the BERT architecture, which is used by Einaggar et al. and Qin et al., and the prompt learning of Wang et al. is designed for, leading to a mere substitution of known methods. Therefore, it would have been obvious at the time of first filing to have modified the teachings of each and to be successful.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (Proceedings of the 2021 conference on empirical methods in natural language processing (2021) 2792-2802) and Einaggar et al. (IEEE transactions on pattern analysis and machine intelligence (2021) 7112-7127) as applied to claims 1 and 6 above, and further in view of Mintseris et al. (Proteins: Structure, Function, and Bioinformatics (2005) 214-216).
Claim 9 is directed to the method of claim 1 but further specifies that the protein conformation comprises a natural and interaction folding states with corresponding prompts and tags.
Wang et al., Einaggar et al., and Qin et al. teach the method of claim 2 as previously described.
Wang et al., Einaggar et al., and Qin et al. do not teach the protein conformation comprises a natural and interaction folding states with corresponding prompts and tags.
Mintseris et al. teaches in the abstract “We present a new version of the Protein–Protein Docking Benchmark, reconstructed from the bottom up to include more complexes, particularly focusing on more unbound–unbound test cases. SCOP (Structural Classification of Proteins) was used to assess redundancy between the complexes in this version. The new benchmark consists of 72 unbound–unbound cases, with 52 rigid-body cases, 13 medium-difficulty cases, and 7 high-difficulty cases with substantial conformational change…The new benchmark provides a platform for evaluating the progress of docking methods on a wide variety of targets”, it would therefore be obvious that the data would include conformations based upon a natural (unbound) state and an interaction state (bound state) with all corresponding information required by the model, thereby reading on wherein the protein conformation comprises a natural folding state and an interaction state; for the natural folding state, an amino acid sequence with a mask is taken as a sample and a normal amino acid sequence is taken as a sample tag to constitute a data set, a corresponding prompt in the natural folding state is a sequence prompt, and a corresponding task is a prediction task for a masked amino acid; and for the interaction state, at least two amino acid sequences are taken as samples and a protein reaction type is taken as a sample tag to constitute a data set, a corresponding prompt in the interaction state is an interaction prompt, and a corresponding task is a prediction task for a protein interaction.
It would have been obvious at the time of first filing to have modified the teachings of Wang et al. and Einaggar et al. for the method of claim 1, with the teachings of Mintseris et al. for the use of data corresponding to different states of protein conformations (i.e. bound vs unbound) as the latter is a benchmark for evaluation of protein structure. One would have had a reasonable expectation of success given that the latter is merely providing additional information and does not require adjustment of the method merely the integration of different subsets of the same information. Therefore, it would have been obvious at the time of first filing to have modified the teachings of each and to be successful.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KEENAN NEIL ANDERSON-FEARS whose telephone number is (571)272-0108. The examiner can normally be reached M-Th, alternate F, 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Karlheinz Skowronek can be reached at 571-272-9047. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.N.A./Examiner, Art Unit 1687
/LARRY D RIGGS II/Supervisory Patent Examiner, Art Unit 1686
Prosecution Insights