DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Applicant's response, filed on 01/28/2026, has been fully considered. The following rejections and/or objections are either reiterated or newly applied. They constitute the complete set presently being applied to the instant application.
Status of claims
Canceled:
4
Amended:
1
New:
24
Pending:
1-3, 5-24
Withdrawn:
none
Examined:
1-3, 5-24
Independent:
1, 23
Allowable:
none
Priority
As detailed on the 03/01/2024 filing receipt, this application claims priority to as early as 02/17/2023.
Drawings
The drawings filed 02/16/2024 are accepted.
Information Disclosure Statement
The Information Disclosure Statements filed on 03/11/2025 and 04/10/2026 are in compliance with the provisions of 37 CFR 1.97 and have been considered in full. A signed copy of the list of references cited from each IDS is included with this Office Action.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 24 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 24 recites "wherein the output generated by the language model has a context length that is longer than any context length seen by the language model during training," in which "longer than any context length seen" is a phrase of relative or vague degree or form of association, neither defined in the specification nor having well-known and sufficiently particular definitions in the art. (MPEP 2173.05(b) pertains.).
Claim 24 has no dependent claims.
Claim rejections - 101
35 USC 101 reads:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
For each rejection below, dependent claims are rejected similarly as not remedying the rejection, unless otherwise noted.
Judicial exceptions (JEs) to 101 patentability
Claims 1-3 and 5-24 are rejected under 35 USC 101 because the claimed inventions are not directed to patent eligible subject matter. After consideration of relevant factors with respect to each claim as a whole, each claim is directed to one or more judicially-recognized exceptions to patentability (JEs), i.e. an abstract idea, a natural phenomenon, a law of nature and/or a product of nature, as identified below. As set forth below, it is not clear that any element or combination of elements in addition to the JE(s), i.e. and "additional elements," either integrate the identified JE(s) into a practical application and/or is a non-conventional additional element, such that it is not clear that any claim is directed to significantly more than the identified JE(s).
MPEP 2106 organizes JE analysis into Steps 1, 2A (1st prong & 2nd prong) and 2B as analyzed below. MPEP 2106 and the following USPTO website provide further explanation and case law citations: www.uspto.gov/patent/laws-and-regulations/examination-policy/examination-guidance-and-training-materials.
Analysis of claims 1-3 and 5-24
Step 1: Are the claims directed to a 101 process, machine, manufacture, or composition of matter (MPEP 2106.03)?
Independent claim 1 is directed to a 101 process, here a "training a language model," with process steps such as "training... and performing..."
Independent claim 23 is directed to a 101 machine or manufacture, here an "apparatus, comprising: a hardware processor," with non-transitory elements such as "hardware processor."
[Step 1: claims 1-3 and 5-24: YES]
Step 2A, 1st prong: Do the claims recite a judicially-recognized exception (JE), e.g. a law of nature, a natural phenomenon or product, or an abstract idea (MPEP 2106.04.II.A.1 & .04(a))?
The MPEP at 2106.I, 2nd para. explains that JEs have been court-recognized as occurring in at least four types: abstract ideas, laws of nature and natural phenomena (including natural products).
MPEP § 2106.04(a)(2) further explains that abstract ideas may be grouped as:
• mathematical concepts (mathematical formulas or equations, mathematical relationships and mathematical calculations);
• certain methods of organizing human activity (fundamental economic practices or principles, managing personal behavior or relationships or interactions between people);
and/or
• mental processes (procedures for observing, evaluating, analyzing/ judging and
organizing information).
Regarding the instant claims and with respect to Step 2A, 1st prong, at least preliminarily these claims recite JEs in the form of abstract ideas and certain methods of organizing human activity as follows.
Mental processes recited include:
Claims 1 and 23 recite: language model and …the given protein sequence is updated by the first attention network based on the first attention network attending only to other tokens within the given protein sequence, and wherein the second attention network is a between-protein sequence network in which the representation at each position is updated by the second attention network based on the second attention network attending to the outputs from the set of first attention networks… is interpreted to involve evaluating, analyzing and comparing token positions within a protein sequence and protein sequences outputted by the first and second attention network to provide an updated protein sequence. Evaluating, analyzing and comparing would fall under the abstract idea of mental processes. The language model is a mental process because the language model is used to determine probabilities and to compare sequences to evaluate and assess patterns. As stated in paragraph [0031] of the specification, a language model is a probabilistic model of sequences that can take on the form of column frequencies with multiple sequences aligned to each other.
Claim 8 recites: assigns a score to each protein sequence... The process of assigning involves evaluating, analyzing and organizing sequence data that could be practically performed in the human mind and/or with pen and paper.
Claim 10 recites: identifying a specific property of interest... Identifying is an act of evaluating and analyzing data that could be practically performed in the human mind and/or with pen and paper.
Claim 15 recites: configuring a prompt..., applying the prompt... and replaces masked tokens..., which is involved with evaluating and organizing data that could be practically performed in the human mind and/or with pen and paper.
Claim 17 recites: mapping the reduced embeddings to measures..., which is involved with evaluating and organizing data that could be practically performed in the human mind and/or with pen and paper.
Claim 19 recites: 3D protein structure prediction... and ...prediction model... which are involved with evaluating and analyzing data that could be practically performed in the human mind and/or with pen and paper.
Mathematical concepts recited include:
Claims 1 and 23 recite: training a language model, attention networks, transformer networks and embeddings. The language model falls under mathematical concepts because the language model is composed of mathematical formulas used to determine probabilities and compares sequences to evaluate and assess patterns. As stated in paragraph [0031] of the specification, a language model is a probabilistic model of sequences that can take on the form of column frequencies with multiple sequences aligned to each other. Transformer also is a mathematical concept because it is a mathematical algorithm.
Claim 2 recites: language model. A language model is a mathematical concept.
Claim 3 recites: transformer network... and apply a transformation to the set of embeddings, the transformation generating a set of probabilities corresponding to the set of embeddings. A transformer network, applying a transformation to the embeddings and probabilities are mathematical concepts.
Claim 8 recites: log likelihood, which is a mathematical formula and a mathematical concept. It requires carrying out a series of mathematical calculations.
Claim 11 recites: probabilities, which is a mathematical formula and a mathematical concept. It requires carrying out a series of mathematical calculations.
Claim 17 recites: model embeddings and supervised machine learning algorithm, which are mathematical formulas and mathematical concepts.
Claim 20 recites: deep structure prediction model... and embeddings, which are mathematical formulas and mathematical concepts.
Claim 21 recites: encoder, decoder and cross-attention function, which are mathematical formulas and mathematical concepts.
Claim 22 recites: language model. A language model is a mathematical concept.
Certain methods of organizing human activity recited include:
Claims 1 and 23 recite: using input training data... and performing a protein engineering workflow using the trained language model. These processes are involved with providing rules or instructions to be followed.
Claim 15 recites: configuring a prompt..., applying the prompt... and replaces masked tokens... These processes are involved with providing rules or instructions to be followed.
[Step 2A, 1st prong: claims 1-3 and 5-24: YES]
Step 2A, 2nd prong: Are the above-identified JEs integrated into a practical application (MPEP 2106.04.II.A.2 & .04(d))?
Generally regarding Step 2A, 2nd prong
MPEP 2106.04(d).I lists the following considerations for evaluating whether additional elements integrate a judicial exception into a practical application:
An improvement in the functioning of a computer, or an improvement to other technology or technical field, as discussed in MPEP §§ 2106.04(d)(1) and 2106.05(a);
Applying or using a judicial exception to affect a particular treatment or prophylaxis for a disease or medical condition, as discussed in MPEP § 2106.04(d)(2);
Implementing a judicial exception with, or using a judicial exception in conjunction with, a particular machine or manufacture that is integral to the claim, as discussed in MPEP § 2106.05(b);
Effecting a transformation or reduction of a particular article to a different state or thing, as discussed in MPEP § 2106.05(c); and
Applying or using the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception, as discussed in MPEP § 2106.05(e).
Additionally, the courts have also identified limitations that did not integrate a judicial exception into a practical application:
Merely reciting a phrase such as "apply it" (or an equivalent) along with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea, as discussed in MPEP 2106.05(f);
Adding insignificant extra-solution activity to the judicial exception, as discussed in MPEP 2106.05(g); and
Generally linking the use of a judicial exception to a particular technological environment or field of use, as discussed in MPEP 2106.05(h).
In Step 2A, 1st prong above, claim steps and/or elements were identified as part of one or more judicial exceptions (JEs).
In Step 2B below, any remaining steps and/or elements are therefore in addition to the identified JE(s). Any such additional steps and additional elements are further discussed in Step 2B.
Here in Step 2A, 2nd prong, no additional step or element clearly demonstrates integration of the JE(s) into a practical application.
At this point in examination it is not yet the case that any of the Step 2A, 2nd prong considerations enumerated above clearly demonstrates integration of the identified JE(s) into a practical application. Referring to the considerations above, none of 1. an improvement, 2. treatment, 3. a particular machine or 4. a transformation is clear in the record.
For example, regarding the first consideration at MPEP 2106.04(d)(1), the record does not yet clearly disclose an explanation of improvement over the previous state of the technology field. An explanation of improvement requires detailed explanation applicable to all embodiments reasonably within the claim scope. In particular, such an explanation of improvement over the previous state of technology may include: identification of the technology field, the particular improvement, as particular as possible identification of any asserted improvements, explanation of a clear difference from the technology field, explanation that reasonably all embodiments within the claim scope result in the asserted improvement, and an extension of the explanation as far as possible to include the result of an identified practical application. The claims do not yet clearly result in such an improvement (e.g. specification: para. [0006]). See MPEP 2106.04(d) and (d)(1).
[Step 2A, 2nd prong: claims 1-3 and 5-24: NO]
Step 2B: Do the claims recite a non-conventional arrangement of additional elements (i.e. elements in addition to any identified JE) (MPEP 2106.05)?
All elements of claims 1-3 and 5-24 are part of one or more identified JEs (as described above), except for elements identified here as conventional elements in addition to the above JEs:
Elements of the following claims are additional elements but nonetheless are conventional elements of a laboratory or computing environment, conventional data gathering elements or conventional post-processing elements:
claim 1: the recited "computer implemented…input training data...," “…a second attention network configured to receive outputs from the set of first attention networks…,” “…second attention network attending to the outputs from the set of first attention networks…,” “…applying a set of input protein sequences to the language model and receiving, as output, a set of embeddings…,” "each protein family is generated..." and "performing a protein engineering workflow..." step/element, as evidenced by MPEP 2106.05(g), e.g. "insignificant extra solution activity" since the recitation is a conventional element of a laboratory and/or computing environment, conventional data gathering/input elements, and/or conventional post-processing or output elements.
claim 5: the recited "using sets of homologous protein sequences..." step/element, as evidenced by MPEP 2106.05(g), e.g. "insignificant extra solution activity" and “insignificant computer implementation” (penultimate para.), since the recitation is a conventional element of a laboratory and/or computing environment, conventional data gathering/input elements, and/or conventional post-processing or output elements.
claim 9: the recited "generates one or more protein sequences..." step/element, as evidenced by MPEP 2106.05(g), e.g. "insignificant extra solution activity" and “insignificant computer implementation” (penultimate para.), since the recitation is a conventional element of a laboratory and/or computing environment, conventional data gathering/input elements, and/or conventional post-processing or output elements.
claim 11: the recited "sampling from the model..." step/element, as evidenced by MPEP 2106.05(g), e.g. "insignificant extra solution activity" and “insignificant computer implementation” (penultimate para.), since the recitation is a conventional element of a laboratory and/or computing environment, conventional data gathering/input elements, and/or conventional post-processing or output elements.
claim 13: the recited "output from the model..." step/element, as evidenced by MPEP 2106.05(g), e.g. "insignificant extra solution activity" and “insignificant computer implementation” (penultimate para.), since the recitation is a conventional element of a laboratory and/or computing environment, conventional data gathering/input elements, and/or conventional post-processing or output elements.
claim 21: the recited "receives the output of the encoder portion..." step/element, as evidenced by MPEP 2106.05(g), e.g. "insignificant extra solution activity" and “insignificant computer implementation” (penultimate para.), since the recitation is a conventional element of a laboratory and/or computing environment, conventional data gathering/input elements, and/or conventional post-processing or output elements.
claim 23: the recited "hardware processor...," "computer memory...," "computer program code...," "input training data...," "input training data...," “…a second attention network configured to receive outputs from the set of first attention networks…,” “…second attention network attending to the outputs from the set of first attention networks…,” “…applying a set of input protein sequences to the language model and receiving, as output, a set of embeddings…” and "performing a protein engineering workflow..." step/element, as evidenced by MPEP 2106.05(g), e.g. "insignificant extra solution activity" since the recitation is a conventional element of a laboratory and/or computing environment, conventional data gathering/input elements, and/or conventional post-processing or output elements.
Claims 6-7, 12, 16 and 18 are providing information on what the data represents and do not change the character of the data obtaining step beyond mere data gathering activity.
The additional elements of claims 1, 5, 9, 11, 13, 21 and 23 equate to necessary data gathering and outputting, (i.e., all uses of the recited judicial exception require such data gathering or data output) (See MPEP 2106.05(g) and See Mayo, 566 U.S. at 79, 101 USPQ2d at 1968; OIP Techs., Inc. v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1092-93 (Fed. Cir. 2015)). Data input and data output steps are insignificant extra solution activities (See MPEP 2106.05(g)). As explained by the Supreme Court, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood or conventional (see MPEP 2106.05(g)). Data gathering steps are not an abstract idea, they are extra-solution activity, as they collect the data needed to carry out the abstract idea. Data gathering does not impose any meaningful limitation on the abstract idea, or how the abstract idea is performed. Data gathering steps are not sufficient to integrate an abstract idea into a practical application. (MPEP 2106.05(g)). Claim 1 recites a computer-implemented method and Claim 23 includes limitations that equate to mere data gathering and outputting via generic computer components, such as receiving data at a computer or outputting data, amount to insignificant extra-solution activity as set forth by the courts in Mayo, 566 U.S. at 79, 101 USPQ2d at 1968 and OIP Techs., Inc, v, Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1092-93 (Fed. Cir. 2015). Also, storing and retrieving information in memory were identified by the courts as well-understood, routine and conventional in Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93. The use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more as identified by the courts in Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Additionally, using language models with attention-based transformers to process protein sequences are known methods as discussed by Aggarwal ("A review of deep learning techniques for protein function prediction." arXiv preprint arXiv:2211.09705 (2022); as cited on the attached 892 form) (page 3, section C titled Protein Function prediction using sequence only, bullet 3 titled Transformers for Protein Sequence Classification). Overall, the additional elements do not comprise an inventive concept when considered individually or as an ordered combination that transforms the claimed judicial exception into a patent-eligible application of the judicial exception.
[Step 2B: claims 1-3 and 5-24: NO]
Summary and conclusion regarding claims 1-3 and 5-24
Summing up the above 101 JE analysis of claims 1-3 and 5-24, each viewed as a whole and considering all elements individually and in combination, no claim recites limitations that transform the claim, finally interpreted as directed to the above-identified JE(s), into patent eligible subject matter.
The claims have all been examined to identify the presence of one or more judicial exceptions. Each additional element in the claims has been addressed, alone and in combination, to determine whether the additional elements integrate the judicial exception into a practical application. Each additional limitation in the claims has been addressed, alone and in combination, to determine whether those additional limitations provide an inventive concept which provides significantly more than those exceptions. Individually, the limitations of the claims and the claims as a whole have been found to be patent ineligible under 35 U.S.C. 101.
Response to 35 USC § 101 (Remarks received 1/28/2026, pages 8-13)
Applicant amended claim 1 and added new claim 24. Claim 4 is cancelled.
Applicant states that the Examiner repeatedly cites paragraph [0031] of the Specification as referring to a "language model" as just being a "probabilistic model of sequences that can take on the form of column frequencies with multiple sequences aligned to each other." (See, e.g., pages 4, 7-8 and 17, emphasis supplied). Based on this wording, the Examiner finds the claimed structural embodiment of the language model (namely, the "transformer network" and its component elements) is not a physical construct but instead just a mental process or mathematical concept or algorithm.
In response, the reason for citing paragraph [0031] of the Specification is for the purpose of explaining the reason for the language model falling under the mathematical concepts grouping of abstract ideas and not physical entities. As Applicant states, the claimed subject matter provides software-based improvements to computer technology (page 12 of 1/28/2026 remarks, para. 3) and software are not physical structures.
Applicant refers to paragraphs [0003]-[0008] of the Specification that describe multiple protein language models used in the prior art, and their various deficiencies and limitations in general and with respect to use thereof for protein sequence engineering workflows in particular. Applicant states that this "technical problem" is solved by the "technical solution" of Applicant's claimed subject matter, which involves both the structural aspects of the model as well as how it is trained.
Applicant states that the tiered approach of the claimed subject matter and depicted in Fig. 4 ensures capture of long-range dependencies between sequences and uniquely allows the model to extrapolate to much longer context lengths than used during training, thereby improving sequence generation and performance on downstream tasks.
In response, Applicant’s remarks are not persuasive because the cited paragraphs of the specification provide a general overview of existing technologies and Applicant does not clearly state which of the listed deficiencies of the prior art is the claimed invention trying to solve. The specification also does not identify the specific improvement to the technology.
Applicant also states that the model is able to extrapolate to much longer context lengths than used during training, thereby improving sequence generation and performance on downstream tasks. Applicant’s remarks are not persuasive because the argument of improvement is a bare assertion of an improvement without the detail necessary to be apparent to a person of ordinary skill in the art. It is not clear how one would gauge the improvement since there are no metrics for comparison between the claimed technology and previous technologies. Applicant also states that the claimed subject matter provides software-based improvements to computer technology, but does not state how it improves the computer technology or its function. Although the concept in Ex Parte Desjardins, Appeal No. 2024-0005676 (PTAB, September 26, 2025, Appeals Review Panel (ARP) Decision) of training a machine learning model is similar to the claimed subject matter, the claimed subject matter does not clearly solve a problem or improve the functioning of a computer or the technological field. In Ex Parte Desjardins, the improvement is in solving the problem of “catastrophic forgetting” encountered in continual learning systems and reducing storage, reducing system complexity and streamlining, and preserving of performance attributes associated with earlier tasks during subsequent computational tasks that were disclosed in the specification. Also, the limitation of training probabilistic models and a machine-learning classifier are mathematical concepts as determined by the court in Recentive Analytics, Inc. V. Fox Corp. In Recentive Analytics, Inc. V. Fox Corp., the court found that Machine Learning Training are directed to abstract ideas at step one of Alice.
Response to 35 USC § 101 (Supplementary Remarks received 2/10/2026, pages 1-5)
Under prong 2A, step 1 of the 101 analysis, Applicant states that the claims do not recite a "mental process" or "mathematical concepts" because "training" a transformer network of the
type specified, then using that trained model to generate "embeddings" and carry out a "protein
engineering workflow" are highly complex tasks that "cannot practically be performed in the
human mind." (See, the August 4, 2025 Memorandum to Technology Center 2100 from Deputy
Commissioner Kim regarding the "mental process" grouping).
Applicant refers to paragraphs [0003]-[0008] of the Specification that describe multiple protein language models used in the prior art, and their various deficiencies and limitations in general and with respect to use thereof for protein sequence engineering workflows in particular. Applicant states that this "technical problem" is solved by the "technical solution" of Applicant's claimed subject matter, which involves both the structural aspects of the model as well as how it is trained.
Applicant states that the tiered approach of the claimed subject matter and depicted in Fig. 4 ensures capture of long-range dependencies between sequences and uniquely allows the model to extrapolate to much longer context lengths than used during training, thereby improving sequence generation and performance on downstream tasks.
It is noted that the response to the remarks above are also applicable for the response to the supplemental remarks.
In response, Applicant’s remarks are not persuasive because the cited paragraphs of the specification provide a general overview of existing technologies and Applicant does not clearly state which of the listed deficiencies of the prior art is the claimed invention trying to solve. The specification also does not identify the specific improvement to the technology.
Applicant also states that the model is able to extrapolate to much longer context lengths than used during training, thereby improving sequence generation and performance on downstream tasks. Applicant’s remarks are not persuasive because the argument of improvement is a bare assertion of an improvement without the detail necessary to be apparent to a person of ordinary skill in the art. It is not clear how one would gauge the improvement since there are no metrics for comparison between the claimed technology and previous technologies. Applicant also states that the claimed subject matter provides software-based improvements to computer technology, but does not state how it improves the computer technology or its function. Although the concept in Ex Parte Desjardins, Appeal No. 2024-0005676 (PTAB, September 26, 2025, Appeals Review Panel (ARP) Decision) of training a machine learning model is similar to the claimed subject matter, the claimed subject matter does not clearly solve a problem or improve the functioning of a computer or the technological field. In Ex Parte Desjardins, the improvement is in solving the problem of “catastrophic forgetting” encountered in continual learning systems and reducing storage, reducing system complexity and streamlining, and preserving of performance attributes associated with earlier tasks during subsequent computational tasks that were disclosed in the specification. Also, the limitation of training probabilistic models and a machine-learning classifier are mathematical concepts as determined by the court in Recentive Analytics, Inc. V. Fox Corp. In Recentive Analytics, Inc. V. Fox Corp., the court found that Machine Learning Training are directed to abstract ideas at step one of Alice.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-3, 9-11, 18 and 21-24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wu ("Improving protein secondary structure prediction by deep language models and transformer networks." bioRxiv (2022): 2022-11, published 2022; cited on the 05/08/2024 “Notice of References Cited” form 892) in view of Essaghir (WO2022185179A1, published 09/09/2022; cited on the 05/08/2024 “Notice of References Cited” form 892).
Regarding independent claims 1 and 23,
Wu teaches the claim limitation of training a language model with “We first randomly selected 5% of the targets to create the TransPross test set and the rest of the targets were used for training and validation” (Page 6, para. 3).
Wu teaches the claim limitation of the language model comprising a transformer network comprising a set of first attention networks that are independent from one another, and a second attention network configured to receive outputs from the set of first attention networks wherein each first attention network is a within-sequence network associated with a given protein sequence of a specific protein family and in which a representation at each position of the given protein sequence is updated by the first attention network based on the first attention network attending only to other tokens within the given protein sequence, and wherein the second attention network is a between-protein sequence network in which the representation at each position is updated by the second attention network based on the second attention network attending to the outputs from the set of first attention networks; following training, applying a set of input protein sequences to the language model and receiving, as output, a set of embeddings with Fig. 3: “The architecture of TransPross. It has two main modules: an encoder and a decoder. The input for the encoder is the MSA of a protein (M). The input for the decoder includes both the secondary structures prior to the current position under consideration and the output of the encoder. From the input, the decoder predicts the three-class secondary structure for each position in the protein sequence.” (page 7) and with section 3.2 “Protein sequence language model and transformer architecture” (page 7 to page 9).
The recited “wherein each first attention network is a within-sequence network associated with a given protein sequence” is interpreted to correspond to “We apply a transformer network with the attention mechanism that has achieved success in natural language translation to detect the relevant sequence context across the entire protein sequence for each amino acid position…” (page 7, para. 2) as taught by Wu. Wu teaches analyzing the protein sequence for each amino acid position within the sequence.
The recited “wherein the second attention network is a between-protein sequence network in which the representation at each position is updated by the second attention network based on the second attention network attending to the outputs from the set of first attention networks” is interpreted to correspond to “A sinusoidal position encoding vector is added on top of the embedding vector, which is then fed into the first self-attention component, followed by the cross-attention component.” (page 8, para. 2) as taught by Wu. It is noted that the recited “between-protein sequence network” is interpreted to correspond to “cross-attention component” as taught by Wu.
Wu does not explicitly teach using input training data comprising sets of protein sequences each comprising a protein family and performing a protein engineering workflow using the set of embeddings of claims 1 and 23. Wu also does not explicitly teach a hardware processor; and computer memory holding computer program code executed by the hardware processor, the computer program code of claim 23. However, these limitations are taught by Essaghir.
Essaghir teaches the claim limitation of using input training data comprising sets of protein sequences each comprising a protein family with FIGs. 5A-5D show operations associated with training and/or utilizing the protein language NLP system. FIG. 5A is a flow diagram for operations involving generating a training data set for training a protein language NLP system according to embodiments of the present disclosure. At operation 510, a library of protein sequences is ingested. The libraries may include any suitable protein repository including but not limited to public databases such as UniProt (uniprot.org), EMBL (ebi.ac.uk), PFam (pfam.xfam.org), private databases such as internal databases, or any combination thereof. The protein sequences may be provided in any suitable format including FASTA, SAM, etc. At operation 520, the protein sequences undergo tokenization, for example, individual amino acid-level tokenization, n-mer tokenization or sub word tokenization. At operation 530, the tokenized sequences are subjected to a masking process, for example, in which about 15% of the amino acids are masked (e.g., 5-25%, 10- 20%, 12-17%, 15%). At operation 540, the training dataset is generated for input into the first neural network. Label “A” from FIG. 5 A continues to Label “A” on FIG. 5B. (Page 30, para. 5 to page 31, para. 1). It is interpreted that pfam corresponds to the recited “sets of protein sequences each comprising a protein family”
Essaghir teaches the claim limitation of performing a protein engineering workflow using the set of embeddings with “In aspects, the library of therapeutic candidates may be hypothetical (not yet synthesized) and the output of the protein language NLP system may select therapeutic candidates predicted to have certain properties. These candidates may be synthesized and experimentally validated.” (page 37, para. 4) and “The protein language NLP system may be customized to a variety of applications, including crystallization, binding, antibody optimization, protein expression, protein stability, TCR-epitope binding affinity, microbiome analysis, enzyme engineering, etc. Present techniques may further be applied to a wide variety of applications in vaccine development including protein-related aspects of this process, such as solubility, productivity, aggregation, homogeneity, integrity, structural stability, protein-protein interactions, TCR- and BCR-epitope recognition, etc. These techniques may be used to prioritize vaccine antigens based upon a categorization and/or a ranking provided by the protein language NLP system. Present techniques may also be used to generate and explore binding predictions for novel TCR and epitope sequences as well as interrogate and visualize which amino acid residues contribute to the prediction. Additionally, the protein language NLP system, after a first phase of training, may be used as an off-the-shelf product that is customized (fine-tuned) for a particular application.” (page 19, para. 1).
Essaghir teaches the recited a hardware processor; and computer memory holding computer program code executed by the hardware processor, the computer program code of independent claim 23 at least with "The various functions of the computer or other processing systems may be distributed in any manner among any number of software and/or hardware modules or units, processing or computer systems and/or circuitry, where the computer or processing systems may be disposed locally or remotely of each other and communicate via any suitable communications medium (e.g., LAN, WAN, Intranet, Internet, hardwire, modem connection, wireless, etc.)" (Page 39, Para. 2) and with "Memory 1460 stores programming instructions/logic and data constructs that provide the functionality of some or all of the software modules/programs described herein. For example, memory 1460 may include the programming instructions/logic and data constructs associated with the protein language NLP system to perform aspects of the methods described herein. The programming instructions/logic may be executed by one or more processor(s) 1435 to implement one or more software modules as described herein. In embodiments, computing device 1400 may have multiple processors 1435, and/or multiple cores per processor" (Page 40, Para. 1).
Rationale for combining
It would have been prima facia obvious to combine the teachings of Wu and Essaghir to arrive at the claimed invention. A person of ordinary skill in the art would have been motivated to modify the method of Wu to include using training data of sets of protein sequences each comprising a protein family as input into the language model as taught by Essaghir to better predict protein sequences associated with a protein family. A person of ordinary skill in the art would have also been motivated to modify the method of Wu to include performing a protein engineering workflow using the set of embeddings as taught by Essaghir to experimentally validate the predicted protein sequences properties. A person of ordinary skill in the art would have also been motivated to modify the method of Wu to include utilizing a computer system with hardware and memory to store instructions, models and data for the automation of protein sequence prediction process. There would have been a reasonable expectation of success, since both Wu and Essaghir teach methods that pertain to the use of language models with transformers to predict protein sequences.
Regarding claim 3, Wu teaches the claim limitation of wherein the transformer with “In order to extract more important amino acids in each column of MSA, we also apply the column attention to the input MSA embedding vector by using the linear transformation and softmax function. The attention weight has the dimension of L × M × 8.” (Page 8, para. 2).
Regarding claim 10, Wu teaches the claim limitation of wherein the one or more protein sequences are generated by identifying a specific property of interest, and conditioning the model on only a subset of relevant homologs that are known to or are predicted to display the specific property of interest with “TransPross performed better in 8 of 12 cases, demonstrating its capability of accurately predicting secondary structure for hard targets with a small number of sequence homologs.” (page 6, para. 1).
Regarding claim 11, Wu teaches the claim limitation of wherein the one or more protein sequences are generated by prompting the model with a known set of protein sequences, and then sampling from the model by using predicted next token probabilities to determine a sequence of amino acids with “The first self-attention component consists of a stack of N = 6 identical layers, similar to the attention component in the encoder except that it applies the masked self-attention mechanism to ensure that the prediction for the current position only depends on the output tokens prior to this position in the training phase.” (page 8, para. 2)
Regarding claim 18, Wu teaches the claim limitation of wherein the protein engineering workflow is per-residue sequence annotation with “The performance of the methods is evaluated by the Q3 accuracy score, which is the percent of residues of the proteins in a dataset that are correctly assigned into three categories (i.e., helix, beta-strand, and coil).” (Page 4, para. 1) and “The objective is to predict the secondary structure label for each residue of a protein from its sequence.” (page 7, para. 1).
Regarding claim 21, Wu teaches the claim limitation of wherein the language model comprises an encoder portion, and a decoder portion, the decoder portion having a cross- attention function that receives the output of the encoder portion with “Similar to the encoder, the decoder also consists of the embedding component and two attention components. It starts from the output tokens (i.e., true 3-state secondary structure in the training phase or predicted secondary structure in the inference phase) of the previous positions in a target protein, which are converted to an embedding vector of the dimension L × 512. A sinusoidal position encoding vector is added on top of the embedding vector, which is then fed into the first self-attention component, followed by the cross-attention component.” (page 8, para.2).
Wu does not teach wherein the language model models the distribution P(X = x), where x = s1, s2, ... , sn is a concatenation of n protein sequences si from a same family, and wherein each protein sequence si= si,1, si,2, ... , si,Li is a sequence of Li amino acids padded by a start token, and an end token of claim 2 and wherein the protein engineering workflow generates one or more protein sequences with a given function of claim 9. Wu also does not teach wherein the output generated by the language model has a context length that is longer than any context length seen by the language model during training in claim 24. However, these limitations are taught by Essaghir.
Regarding claim 2 , Essaghir teaches the recited wherein the language model models the distribution P(X = x), where x = s1, s2, ... , sn is a concatenation of n protein sequences si from a same family, and wherein each protein sequence si= si,1, si,2, ... , si,Li is a sequence of Li amino acids padded by a start token, and an end token at least with "In aspects, the input to the language model for the second phase comprises concatenated representations of sequence and categorical feature embeddings (e.g., gene, family variables)" (Page 14, Para. 3) and with Fig. 3A and Fig. 3C. The recited "start token" reads on <s> and "end token" reads on </s> of Fig. 3A and Fig. 3C of Essaghir.
Regarding claim 9, Essaghir teaches the claim limitation of wherein the protein engineering workflow generates one or more protein sequences with a given function with “In aspects, the library of therapeutic candidates may be hypothetical (not yet synthesized) and the output of the protein language NLP system may select therapeutic candidates predicted to have certain properties. These candidates may be synthesized and experimentally validated.” (page 37, para. 4).
Regarding claim 22, Essaghir teaches the recited wherein the language model implements one of: a prefix language modeling objective, a masked language modeling objective, and a combination of a prefix language model objective and a masked language modeling objective at least with "Amino acids of proteins are tokenized and masked" (Abstract).
Regarding claim 24, Esseghir teaches wherein the output generated by the language model has a context length that is longer than any context length seen by the language model during training. Esseghir’s claim 20 teaches “The computer-implemented method of claim 18 or 19, further comprising generating concatenated sequence and categorical embeddings from the first phase of training and providing the concatenated sequence and categorical embeddings to the second neural network for the second phase of training.” Esseghir also teaches “…an output comprising a prediction including one or more biophysiochemical properties for the candidate amino acid sequence. In aspects, the output is displayed (optionally), on a display screen of a device, the output comprising the predicted one or more biophysiochemical properties for the candidate amino acid sequence.” Esseghir teaches training the model with concatenated sequences and the output are candidate sequences.
Rationale for combining
It would have been prima facia obvious to combine the teachings of Wu and Essaghir to arrive at the claimed invention. A person of ordinary skill in the art would have been motivated to modify the method of Wu to include the concatenation of protein sequences as taught by Essaghir to better facilitate information transfer from the first phase of training to the second phase of training (Page 4, para. 2). A person of ordinary skill in the art would have also been motivated to modify the method of Wu to include performing a protein engineering workflow using the set of embeddings as taught by Essaghir to experimentally validate the predicted protein sequences properties. A person of ordinary skill in the art would have also been motivated to modify the method of Wu to include masking of amino acids of protein sequences as taught by Essaghir to better predict protein sequences. Furthermore, there would have been a reasonable expectation of success, since both Essaghir and Wu teach methods that pertain to the use of language models with transformers to predict protein sequences.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Wu ("Improving protein secondary structure prediction by deep language models and transformer networks." bioRxiv (2022): 2022-11, published 2022; cited on the 05/08/2024 “Notice of References Cited” form 892) in view of Essaghir (WO2022185179A1, published 09/09/2022; cited on the 05/08/2024 “Notice of References Cited” form 892) as applied to claims 1-3, 9-11, 18 and 21-24 above and further in view Yamaguchi (Briefings in Bioinformatics 22.6: bbab234., published 2021; cited on the 05/08/2024 “Notice of References Cited” form 892).
Wu and Essaghir are applied to claims 1-3, 9-11, 18 and 21-24 as discussed above.
Wu does not teach the language model is trained using sets of homologous sequences of claim 5. However, this limitation was known in the art at the time of the effective filing date of the invention, as taught by Yamaguchi.
Regarding claim 5, Yamaguchi teaches the recited wherein the language model is trained using sets of homologous protein sequences at least with "A Transformer model was trained from scratch with about 32 million primary sequences in Pfam [32] to capture general properties of proteins without reference to the domain annotation data; (b) fine-tuning the pretrained model with homologous sequences for each target protein to incorporate evolutionary information specific to the target; (c) embedding each wild-type or mutagenized sequence through the fine-tuned model to obtain the vectorized feature representation of the whole sequence" (Page 2, Col. 2, Para. 2).
Rationale for combining
It would have been prima facia obvious to combine the teachings of Wu and Yamaguchi to arrive at the claimed invention. Yamaguchi’s method of DA-aware sequence filtering strategies effectively improved prediction accuracy (Page 7, Col. 1, Para. 4). A person of ordinary skill in the art would have been motivated to modify the method of Wu by training the language model with homologous sequences as taught by Yamaguchi to improve prediction accuracy. Furthermore, there would have been a reasonable expectation of success, since both Wu and Yamaguchi teach methods that pertain to the use of language models with transformers to predict protein sequences.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Wu ("Improving protein secondary structure prediction by deep language models and transformer networks." bioRxiv (2022): 2022-11, published 2022; cited on the 05/08/2024 “Notice of References Cited” form 892) in view of Essaghir (WO2022185179A1, published 09/09/2022; cited on the 05/08/2024 “Notice of References Cited” form 892) as applied to claims 1-3, 9-11, 18 and 21-24 above and further in view of Rao (International Conference on Machine Learning. PMLR, published 2021; cited on the 03/25/2024 IDS Document).
Wu and Essaghir are applied to claims 1-3, 9-11, 18 and 21-24 as discussed above.
Wu does not teach following training, an order of protein sequences in a sequence-of-protein sequences is random of claim 6. However, this limitation was known in the art at the time of the effective filing date of the invention, as taught by Rao.
Regarding claim 6, Rao teaches the recited wherein, following training, an order of protein sequences in a sequence-of-protein sequences is random at least with "To remove sequence patterns seen during training, we randomly permute the order of positions (columns) in the MSA. This preserves all covariance information between pairs of columns, but results in a scrambled input dissimilar to a real protein" (Page 8, Col. 2, Para. 3).
Rationale for combining
It would have been prima facia obvious to combine the teachings of Wu and Rao to arrive at the claimed invention. Rao demonstrated that by taking sets of sequences as input, the model gains the ability to extract information during inference, which improves the parameter efficiency (Page 2, Col. 1, Para. 2). A person of ordinary skill in the art would have been motivated to modify the method of Wu by randomly ordering the sequences as taught by Rao to improve parameter efficiency. Furthermore, there would have been a reasonable expectation of success, since both Wu and Rao teach methods that pertain to the use of language models with transformers to predict protein sequences.
Claims 7-8 are rejected under 35 U.S.C. 103 as being unpatentable over Wu ("Improving protein secondary structure prediction by deep language models and transformer networks." bioRxiv (2022): 2022-11, published 2022; cited on the 05/08/2024 “Notice of References Cited” form 892) in view of Essaghir (WO2022185179A1, published 09/09/2022; cited on the 05/08/2024 “Notice of References Cited” form 892) as applied to claims 1-3, 9-11, 18 and 21-24 above and further in view of Lou (Nature communications 12.1: 5743, published 2021; cited on the 05/08/2024 “Notice of References Cited” form 892) and Notin (International Conference on Machine Learning. PMLR, published 2022; cited on the 03/25/2024 IDS Document).
Wu and Essaghir are applied to claims 1-3, 9-11, 18 and 21-24 as discussed above.
Wu does not teach wherein the protein engineering workflow is variant prioritization of claim 7 and wherein variant prioritization assigns a score to each sequence in a set of variants {v1,v2, ... , vn} of a target sequence t that accurately reflects a relative fitness of the variants, and predicts the fitness of a variant as a conditioned log-likelihood of the variant vi given a set of protein sequences S homologous to the target t of claim 8. However, these limitations were known in the art at the time of the effective filing date of the invention, as taught by Luo and Notin.
Regarding claim 7, Luo teaches the recited wherein the protein engineering workflow is variant prioritization at least with "As such, the model can fully leverage the fitness data of screened low-order variants and prioritize higher order variants that are likely to exhibit improved properties for the next round of directed evolution" (Page 5, Col. 2, Para. 2) and with "We trained an ECNet model using DMS data of low-order TEM-1 variants and used the model to prioritize new high-order variants that were likely to have enhanced fitness" (Page 11, Col. 2, Para. 5).
Regarding claim 8, Notin teaches the recited wherein variant prioritization assigns a score to each protein sequence in a set of variants {v1,v2, ... , vn} of a target protein sequence t that accurately reflects a relative fitness of the variants, and predicts the fitness of a variant as a conditioned log-likelihood of the variant vi given a set of protein sequences S homologous to the target t at least with "Several modeling approaches have been introduced in recent years, offering various trade-offs in terms of performance, diversity of proteins which can be modelled and types of sequence variation which can be scored" (Page 1 of 28 of PDF, Col. 2, Para. 1) and with "Finally, we estimate the log likelihood log P(x) for a protein sequence x by a weighted arithmetic average of the log likelihood log PA(x) from the autoregressive inference mode and the log likelihood log PR(x) obtained from the retrieval inference mode. This can be equivalently viewed as a weighted geometric average in probability space, and form a proper probability distribution up to a normalization constant" (Page 5 of 28 of PDF, Col. 2, Para. 1).
Rationale for combining
It would have been prima facia obvious to combine the teachings of Wu, Luo and Notin to achieve the claimed invention. Luo demonstrated that ECNet predicts the sequence-function relationship more accurately as compared to existing machine learning algorithms by using ~50 deep mutational scanning and random mutagenesis datasets (Abstract). While Notin's approach is robust across taxa and protein families, making it well suited to a broad range of tasks, such as predicting disease causing variants in humans (Page 9 of 28 of PDF, Col. 1, Para. 5). A person of ordinary skill in the art would have been motivated to modify the method of Wu to include variant prioritization as taught by Luo and scoring variants as taught by Notin to accurately predict variants and sequence-function relationships. Furthermore, there would have been a reasonable expectation of success, since both Wu, Luo and Notin teach methods that pertain to the use of language models with transformers to predict protein sequences.
Claims 12-16 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Wu ("Improving protein secondary structure prediction by deep language models and transformer networks." bioRxiv (2022): 2022-11, published 2022; cited on the 05/08/2024 “Notice of References Cited” form 892) in view of Essaghir (WO2022185179A1, published 09/09/2022; cited on the 05/08/2024 “Notice of References Cited” form 892) as applied to claims 1-3, 9-11, 18 and 21-24 above and further in view of Zhang (arXiv preprint arXiv:2202.02944, published 2022; cited on the 05/08/2024 “Notice of References Cited” form 892).
Wu and Essaghir are applied to claims 1-3, 9-11, 18 and 21-24 as discussed above.
Wu does not teach wherein the one or more protein sequences are generated by prompting the model with a prompt that is augmented to include a natural language description of claim 12; wherein the prompt concatenates a natural language description of a protein, a sequence of the protein, and a natural language description of a target protein to be output from the model, wherein the target protein has a degree of similarity to the protein of claim 13; wherein the protein engineering workflow is sequence infilling of claim 14; sequence infilling comprises: configuring a prompt with at least one region of a protein sequence masked with a masking token; and applying the prompt to the model to generate the protein sequence, wherein the model replaces the masking token with one or more amino acid sequences of claim 15; wherein the protein engineering workflow is homology augmented learning of claim 16; wherein the protein engineering workflow is 3D protein structure prediction of claim 19; and wherein the 3D protein structure prediction uses a deep structure prediction model augmented to use per-residue embeddings generated by the language model of claim 20. However, these limitations were known in the art at the time of the effective filing date of the invention, as taught by Zhang.
Regarding claim 12, Zhang teaches the recited wherein the one or more protein sequences are generated by prompting the model with a prompt that is augmented to include a natural language description at least with "Specifically, ConfProtein has two learnable prompts for the properties of the protein itself and the interaction conformation in protein pairs respectively. As for the properties of the protein itself can be mined by the sequence of amino acids, we leverage the masked language modeling (MLM) task (Devlin et al., 2019) to learn this prompt, which is called the sequence prompt (Seq prompt). For conformations that exist in interaction pairs, an interaction conformation prompt (IC prompt) is learned with protein protein interaction prediction (PPI) tasks. Two prompts can be learned in a multitask setting. " (Page 2 of 13 of PDF, Col. 1, Para. 1).
Regarding claim 13, Zhang teaches the recited wherein the prompt concatenates a natural language description of a protein, a sequence of the protein, and a natural language description of a target protein to be output from the model, wherein the target protein has a degree of similarity to the protein at least with Figure 3 "Overview of our proposed ConfProtein" (Page 4 of 13 of PDF, Figure 3). Figure 3 depicts combining prompt with amino acid sequence with Prompt-Amino Acides Fusion step." Zhang also teaches whether the output protein is similar to target protein with Tables 2-4. Tables 2-4 compare the results of the predicted tasks with other models, such as ProtoBert and MSA Transformer.
Regarding claim 14, Zhang teaches the recited wherein the protein engineering workflow is sequence infilling at least with Figure 4. "The knowledge-injection based attention masks. The original input tokens are denoted as blue circles while the prompt tokens are denoted as pink circles." (Page 4 of 13 of PDF, Figure 4). The recited "sequence filling" reads on "knowledge-injection" of Zhang.
Regarding claim 15, Zhang teaches the recited sequence infilling comprises: configuring a prompt with at least one region of a sequence masked with a masking token; and applying the prompt to the model to generate the sequence, wherein the model replaces the masking token with one or more amino acid sequences at least with "Specifically, ConfProtein has two learnable prompts for the properties of the protein itself and the interaction conformation in protein pairs respectively. As for the properties of the protein itself can be mined by the sequence of amino acids, we leverage the masked language modeling (MLM) task (Devlin et al., 2019) to learn this prompt, which is called the sequence prompt (Seq prompt). For conformations that exist in interaction pairs, an interaction conformation prompt (IC prompt) is learned with protein-protein interaction prediction (PPI) tasks. Two prompts can be learned in a multitask setting" (Page 2 of 13 of PDF, Col. 1, Para. 1).
Regarding claim 16, Zhang teaches the recited wherein the protein engineering workflow is homology augmented learning at least with "TAPE is a benchmark designed to evaluate the generalization of protein models. There are three major aspects that the benchmark involves: structure prediction, detection of remote homologs, and protein engineering. With TAPE, we can analyze and discuss the learned protein prompts" (Page 5 of 13 of PDF, Col. 2, Para. 3).
Regarding claim 19, Zhang teaches the recited wherein the protein engineering workflow is 3D protein structure prediction at least with " The experimental results show that PTPMs with the Seq prompt can only acquire knowledge about amino acid sequences and relevant secondary structures, while those with the IC prompt can effectively acquire 3D structural knowledge" (Page 2 of 13 of PDF, Col. 1, Para. 2).
Regarding claim 20, Zhang teaches the recited wherein the 3D protein structure prediction uses a deep structure prediction model augmented to use per-residue embeddings generated by the language model at least with "The first three baselines use different deep learning architectures (CNN, RCNN, and LSTM) to convert amino acid embeddings to protein embeddings, and use linear classifiers to predict whether two proteins have an interaction relationship" (Page 6 of 13 of PDF, Col. 1, Para. 2).
Rationale for combining
It would have been prima facia obvious to combine the teachings of Wu and Zhang to achieve the claimed invention. Zhang demonstrated that using the Seq prompt does not hurt (Pre-trained protein models) PTPMs’ performance on sequence-related tasks while incorporating the IC prompt significantly improves PTPMs’ performance on tasks where interaction conformational knowledge counts (Abstract). A person of ordinary skill in the art would have been motivated to modify the method of Wu to include prompts and homology augmented learning as taught by Zhang to improve the models' performance. Furthermore, there would have been a reasonable expectation of success, since both Wu and Zhang teach methods that pertain to the use of language models with transformers to predict protein sequences.
Claims 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Wu ("Improving protein secondary structure prediction by deep language models and transformer networks." bioRxiv (2022): 2022-11, published 2022; cited on the 05/08/2024 “Notice of References Cited” form 892) in view of Essaghir (WO2022185179A1, published 09/09/2022; cited on the 05/08/2024 “Notice of References Cited” form 892) as applied to claims 1-3, 9-11, 18 and 21-24 above and further in view of Zhang (arXiv preprint arXiv:2202.02944, published 2022; cited on the 05/08/2024 “Notice of References Cited” form 892) and Littmann (Sci Rep 11, 23916, published 2021; cited on the 05/08/2024 “Notice of References Cited” form 892).
Wu and Essaghir are applied to claims 1-3, 9-11, 18 and 21-24 as discussed above.
Wu does not teach wherein the homology augmented learning comprises mapping each of a set of protein sequences to a sequence of per-residue model embeddings, reducing each sequence of per-residue model embeddings to a fixed length vector to generate a reduced embedding, and training a supervised machine learning algorithm to learn a function mapping the reduced embeddings to measures of a given function of claim 17 and wherein the homology augmented learning comprises mapping each of a set of protein sequences to a sequence of per-residue model embeddings, reducing each sequence of per-residue model embeddings to a fixed length vector to generate a reduced embedding, and training a supervised machine learning algorithm to learn a function mapping the reduced embeddings to measures of a given function of claim 18. However, these limitations were known in the art at the time of the effective filing date of the invention, as taught by Zhang and Littmann.
Regarding claim 17, Zhang teaches the recited wherein the homology augmented learning comprises mapping each of a set of protein sequences to a sequence of per-residue model embeddings at least with "A pre-trained model M conducts a mapping f: X → H, where X is the space of the input sequence embeddings and H is the space of the returned representations h = f(Xin). It is h that represents the input sequence as a dense vector" (Page 3 of 13 of PDF, Col. 2, Para. 2).
Littmann teaches the recited wherein the homology augmented learning comprises mapping each of a set of protein sequences to a sequence of per-residue model embeddings, reducing each sequence of per-residue model embeddings to a fixed length vector to generate a reduced embedding, and training a supervised machine learning algorithm to learn a function mapping the reduced embeddings to measures of a given function at least with "We used ProtT5-XL-UniRef5028 (in the following ProtT5) to create fixed-length vector representations for each residue in a protein sequence" (Page 11, Para. 4) and with "For bindEmbed21DL, we realized the supervised learning through a relatively shallow (few free parameters) two-layer Convolutional Neural Network (CNN; Supplementary Fig. S13, Step 2). The CNN was implemented in PyTorch46 and trained with the following settings: Adamax optimizer, learning rate = 0.01, early stopping, and a batch size of 406 (resulting in two batches). ProtT5 embeddings (from the last layer of ProtT5, 1024-dimensional vector per residue) were used as the only input." (Page 11, Para. 5).
Regarding claim 18, Littmann teaches the recited wherein the protein engineering workflow is per residue sequence annotation at least with "Overall, BioLiP annotates 13 residues in 5T5K as DNA-binding, 10 of those were correctly predicted (77% recall; Fig. 5A, lighter red). With respect to the three remaining: although our sequence-based method clearly did not reach remotely the power of X-ray crystallography, at least some of the parts of the proteins seemingly bridged over by the major grove (Fig. 5A: dark blue) might, indeed not bind DNA" (Page 7, Para. 1) and with "We observed similar results for the ribonuclease P protein component (UniProt ID: Q9X1H4): The PDB structure 6MAX31,34 (1.42 Å) annotated this protein with seven residues binding to a small molecule; none of the those were predicted at p ≥ 0.95. Indeed, the available functional annotations clearly suggest nucleic acid binding; the small molecule bound in 6MAX seems to mainly inhibit RNA-binding34" (Page 7, Para. 2).
Rationale for combining
It would have been prima facia obvious to combine the teachings of Wu and Zhang to achieve the claimed invention. Zhang demonstrated that using the Seq prompt does not hurt (Pre-trained protein models) PTPMs’ performance on sequence-related tasks while incorporating the IC prompt significantly improves PTPMs’ performance on tasks where interaction conformational knowledge counts (Abstract). Littmann's method bindEmbed21 is fast, simple, and broadly applicable—neither using structure nor MSAs (Abstract). A person of ordinary skill in the art would have been motivated to modify the method of Wu to include homology learning as taught by Zhang and fixed length vector as taught by Littmann to improve the language models' performance. Furthermore, there would have been a reasonable expectation of success, since both Wu, Zhang and Littmann teach methods that pertain to the use of language models with transformers to predict protein sequences.
Response to 35 USC § 103 (Remarks received 1/28/2026, pages 14-19)
Applicant amended claim 1 and added new claim 24. Claim 4 is cancelled.
Applicant states that in Wu, the attention networks in the Encoder or the attention networks in the Decoder do not ever apply attention either to (1) a single sequence at once (described as "a within sequence network associated with a given protein sequence of a specific protein family") or (2) multiple sequences at once (described as a "between-protein sequence ... "). Applicant also states that notion of "within-sequence" and "between-sequence" as applied to an "attention network" is positively recited, and it is not disclosed or suggested by the Wu paper.
In response, Applicant’s remarks are not persuasive because Wu teaches the claim limitations in section 3.2 “Protein sequence language model and transformer architecture” (page 7 to page 9) as indicated with the following. The recited “wherein each first attention network is a within-sequence network associated with a given protein sequence” is interpreted to correspond to “…a transformer network with the attention mechanism that has achieved success in natural language translation to detect the relevant sequence context across the entire protein sequence for each amino acid position…” (page 7, para. 2) as taught by Wu. Wu teaches analyzing the protein sequence for each amino acid position within the sequence.
The recited “wherein the second attention network is a between-protein sequence network in which the representation at each position is updated by the second attention network based on the second attention network attending to the outputs from the set of first attention networks” is interpreted to correspond to “A sinusoidal position encoding vector is added on top of the embedding vector, which is then fed into the first self-attention component, followed by the cross-attention component.” (page 8, para. 2) as taught by Wu. It is noted that the recited “between-protein sequence network” is interpreted to correspond to “cross-attention component” as taught by Wu.
Applicant states that Essaghir does not teach the claimed subject matter. Applicant states that Essaghir does not train a language model "using input training data comprising sets of protein sequences each comprising a protein family," Essaghir’s neural network is not generative of the protein sequence or the auxiliary labels and Essaghir’s modeling is not used for a "protein engineering workflow.
In response, Applicant’s arguments are not persuasive because Esseghir teaches protein families and Wu provides the attention-based networks. It would have been obvious to the skilled artisan to create models based on families that provide an ability to analyze and assess importance of conserved, modified, or maintained amino acids among the family members. Also, Esseghir teaches protein engineering with selecting and synthesizing protein candidates from the outputs of the NPL model (page 37, para. 4) and enzyme engineering (page 19, para. 1).
Response to IDS (Remarks received 1/28/2026, page 20)
Regarding Applicant’s remarks on the non-consideration of the IDS submitted on March 11,
2025. Applicant’s arguments have been found persuasive and the submitted IDSs are considered in full and a signed copy of the IDS is included with this Office Action.
Conclusion
No claims are allowed.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KETTIP KRIANGCHAIVECH whose telephone number is (571)272-1735. The examiner can normally be reached 8:30am-5:00pm EDT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Larry D. Riggs can be reached on (571) 270-3062. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.K./Examiner, Art Unit 1686 /LARRY D RIGGS II/Supervisory Patent Examiner, Art Unit 1686
Prosecution Insights