Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant's claim for foreign priority based on an application filed in EP on 02/05/2024. It is noted, however, that applicant has not filed a certified copy of the EP24155883.2 application as required by 37 CFR 1.55.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 08/04/2025 has been entered.
Status of the Claims
Claims 1 and 14-15 have been amended. Claims 2-4 have been cancelled. Claims 1 and 5-15 are currently pending and have been considered by the Examiner.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “a normalizer” in claim 1, line 8; and in claim 14, line 8.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
Claims 1, 5-6, 8, and 11-15 are rejected under 35 U.S.C. 103 as being unpatentable over Sodhani et al. (US 20190188463 A1, cited in PTO-892 issued 08/20/2024) in view of Patwari et al. (US 20230297824 A1, cited in PTO-892 issued 01/21/2025).
Regarding claim 1, Sodhani teaches: A system comprising a long short-term memory (LSTM) implemented on a data processing apparatus comprising one or more processors, wherein the LSTM comprises a memory cell stored in a memory of the data processing apparatus, an input gate, an output gate, and a forget gate, wherein the input gate comprises at least one input gate activation function which is the exponential function t based on a sigmoid activation function, an output gate, and a forget gate. Since a sigmoid function contains an exponential term e-z, the input gate activation function is an exponential function. [0136], lines 1-12 and [0139], lines 1-7 disclose a data processing apparatus comprising a processor and a memory of the data processing apparatus.)
wherein the LSTM further comprises a normalizer configured to stabilize the input gate to avoid overflows caused by the input gate activation function, wherein the normalizer provides a normalizer state which sums up a product of the input gate times the forget gate from subsequent time steps in a sequence. ([0078]-[0080] disclose equations for the forget gate, input gate, vector of new candidate values C~t, and cell state Ct.
At time t = 1, the cell state C1 = f1*C0 + i1*C~1.
At time t = 2, the cell state C2 = f2*C1 + i2*C~2.
f2*C1 above expands to f2*(f1*C0 + i1*C~1), which sums up a product of the input gate times the forget gate from subsequent time steps in a sequence, and thus indicates a normalizer state has been provided by a normalizer. Avoiding overflows caused by the input gate activation function is the reason for applying the normalizer.)
However, Sodhani does not explicitly teach: exp(x) = ex
But Patwari teaches: at least one x. (Crossed-out terms are not explicitly taught. All of [0004] discloses a non-linear layer of a neural network. [0058], lines 1-3 describes Table 1. In the second row of Table 1 on page 5, the first and second columns disclose a non-linear activation function that is an exponential function f(x) = ex.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have replaced the sigmoid activation function in Sodhani’s input gate with Patwari’s exponential activation function f(x) = ex because a sigmoid activation function is an alternative to an exponential activation function according to Patwari’s Table 1. Chae et al. (US 20200257981 A1, cited in PTO-892 issued 04/04/2025) provides evidentiary support at [0049], line 1 that the sigmoid function is a non-linear function. Since Sodhani’s sigmoid activation function σ in [0077] is a non-linear activation function, a neural network layer that implements the sigmoid activation function is a non-linear layer. A motivation for the combination is that the non-linear layers (including those using an exponential activation function) facilitate learning of parameters during training and higher accuracy in during inference. (Patwari, [0004])
Regarding claim 5, the combination of Sodhani and Patwari teaches: The system of claim 1,
Sodhani teaches: wherein the memory cell of the LSTM is configured to store a scalar value, thereby forming a scalar LSTM (sLSTM) comprising a scalar memory cell stored in the memory of the data processing apparatus. ([0078], lines 5-9 discloses weight Wf is some scalar constant. The weight is stored as part of the memory cell.)
Regarding claim 6, the combination of Sodhani and Patwari teaches: The system of claim 5,
Sodhani teaches: wherein the sLSTM comprises a plurality of scalar memory cells stored in the memory of the data processing apparatus, wherein the sLSTM is configured for memory mixing across the plurality of scalar memory cells. ([0076], lines 7-12, Fig. 4c and Fig. 5a disclose a series of LSTM cells. [0079]-[0080] discloses the cell states Ct are forward propagated across the memory cells which corresponds to the limitation “memory mixing”.)
Regarding claim 8, the combination of Sodhani and Patwari teaches: The system of claim 1,
Sodhani teaches: wherein the memory cell of the LSTM is configured to store a matrix of values, thereby forming a vectorized LSTM (mLSTM) comprising a matrix memory cell stored in the memory of the data processing apparatus. ([0077] and [0080], line 8 discloses an LSTM maintains a cell state Ct which is the result of a Hadamard product. A Hadamard product generates a matrix. [0079], lines 9-11 discloses C~t is a vector of candidate values, and a vector is a 1xN or Nx1 matrix and is used to compute Ct. Each LSTM cell is a matrix memory cell.)
Regarding claim 11, the combination of Sodhani and Patwari teaches: The system of claim 8,
Sodhani teaches: wherein the mLSTM comprises a plurality of matrix memory cells stored in the memory of the data processing apparatus. ([0076], lines 7-12, Fig. 4c and Fig. 5a disclose a series of LSTM cells. The series of LSTM cells is a plurality of matrix memory cells.)
Regarding claim 12, the combination of Sodhani and Patwari teaches: The system of claim 1,
Sodhani teaches: further comprising a residual block comprising the LSTM to form an extended LSTM (xLSTM) block. ([0076], lines 7-12, Fig. 4c and Fig. 5a disclose a series of LSTM cells. Each LSTM cell is an xLSTM block.)
Regarding claim 13, the combination of Sodhani and Patwari teaches: The system of claim 12,
Sodhani teaches: wherein a plurality of xLSTM blocks are arranged in a stacked arrangement to form an xLSTM architecture. ([0076], lines 7-12, Fig. 4c and Fig. 5a disclose a series of LSTM cells (xLSTM blocks). The cells are arranged in a series to form an xLSTM architecture.)
Claim 14 recites an apparatus that implements similar features as the system of claim 1 and is therefore rejected for at least the same reasons.
Sodhani teaches: A data processing apparatus comprising one or more processors and configured for storing and executing a long short-term memory (LSTM), ([0077] discloses an LSTM. [0136], lines 1-12 and [0139], lines 1-7 disclose a data processing apparatus as claimed.)
Claim 15 recites a product that implements similar features as the system of claim 1 and is therefore rejected for at least the same reasons.
Sodhani teaches: A non-transitory computer-readable medium having stored thereon a computer program, the computer program comprising instructions which, when the program is executed by a computer, cause the computer to implement a long short-term memory (LSTM) ([0077] discloses an LSTM. [0136], lines 1-12, [0139], lines 1-7, and [0140] disclose a non- transitory computer-readable medium and a computer as claimed.)
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Sodhani et al. (US 20190188463 A1, cited in PTO-892 issued 08/20/2024) in view of Patwari et al. (US 20230297824 A1, cited in PTO-892 issued 01/21/2025) and Palaniappan et al. (US 11862298 B1, cited in PTO-892 issued 08/20/2024).
Regarding claim 7, the combination of Sodhani and Patwari teaches: The system of claim 6,
Sodhani teaches: wherein the sLSTM comprises a [head]
However, Sodhani and Patwari do not explicitly teach: a plurality of heads
But Palaniappan teaches: a plurality of heads (C. 9, L. 16-24, 27-42. The forward direction is one head, and the backward direction is another head.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have used Palaniappan’s bidirectional LSTM for Sodhani’s LSTM in the combination of Sodhani and Patwari. A motivation for the combination is that for every point in a given sequence, the LSTM network has complete, sequential information about all points before and after it. (Palaniappan, C. 9, L. 38-40)
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Sodhani et al. (US 20190188463 A1, cited in PTO-892 issued 08/20/2024) in view of Patwari et al. (US 20230297824 A1, cited in PTO-892 issued 01/21/2025) and Kosko (“Bidirectional Associative Memories”, cited in PTO-892 issued 08/20/2024).
Regarding claim 9, Sodhani and Patwari teaches: The system of claim 8,
Sodhani teaches: wherein the matrix memory cell is configured as a t.)
However, Sodhani and Patwari do not explicitly teach: a Bidirectional Associative Memory (BAM).
But Kosko teaches: a Bidirectional Associative Memory (BAM). (P. 52, equations 15 and 16 in the right column.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have configured the memory cell in the combination of Sodhani and Patwari as a BAM because the BAM illustrates rapid convergence and accurate pattern completion. (Kosko, p. 54, left column, first sentence in bottom paragraph)
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Sodhani et al. (US 20190188463 A1, cited in PTO-892 issued 08/20/2024) in view of Patwari et al. (US 20230297824 A1, cited in PTO-892 issued 01/21/2025) and Ba et al. (US 20230251646 A1, cited in PTO-892 issued 08/20/2024).
Regarding claim 10, Sodhani and Patwari teaches: The system of claim 8,
Sodhani teaches: wherein the mLSTM is configured to apply a t)
However, Sodhani and Patwari do not explicitly teach: covariance
But Ba teaches: covariance update rule ([0076]-[0077], [0086] discloses a covariance update rule for an LSTM)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated Ba’s covariance update into Sodhani’s LSTM in the combination of Sodhani and Patwari. A motivation for the combination is to monitor and detect anomalies for a plurality of processes of an industrial system using a machine learning operation, wherein the one or more anomalies are localized. (Ba, [0021], lines 1-8 and [0088], lines 1-4)
Response to Arguments
Examiner responds to Applicant’s arguments filed 08/04/2025. On pages 8-12, Applicant generally argues that neither Sodhani nor Patwari, individually or in combination, discloses or suggests the limitations from claim 1, from “wherein the LSTM” in line 7 to the end of line 12. Applicant's arguments have been fully considered but they are not persuasive. Detailed arguments are addressed below.
In the remarks, page 8 argues that claim 1 recites “an input gate” and “a normalizer” as distinct structural elements, page 9 lists reasons that Sodhani allegedly does not provide overflow prevention capability, and pages 9-10 argues Sodhani allegedly does not teach the summing function of the normalizer.
In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., “an input gate” and “a normalizer” are distinct structural elements) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). Furthermore, the current Office action does not equate the normalizer and input gate activation function.
The current Office action has shown how the summation as claimed can be derived from the equations in Sodhani, [0078]-[0080]. At time t = 2, the cell state C2 contains the term f2*i1*C~1 plus other terms, which sums up a product of the input gate value at t = 1 multiplied by the forget time value at t = 2. The summation indicates a normalizer state has been provided by a normalizer. Avoiding overflows caused by the input gate activation function is the reason for applying the normalizer.
Pages 10-12 argue that Patwari contributes only the exponential function f(x)=ex as an activation function alternative, and explicitly teaches away from using any form of normalization or scaling with exponential activation functions.
In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). Examiner agrees with Applicant’s remark that Patwari contributes only the exponential function f(x)=ex as an activation function alternative. Sodhani, not Patwari, was relied upon for teaching a normalizer.
In arguing that Patawari teaches away from the claimed invention, Applicant relies upon Patawari, [0059] which discloses that the exponential non-linear activation function “may be approximated without requiring pre- or post-scaling”. This disclosure is only relevant to the third column in Table 1 called “Approximation method (degree = 4)”. The rejection of pending claim 1 relied upon the first two columns in Table 1 to show that the exponential function f(x)=ex may be used as an activation function alternative.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Asher H. Jablon whose telephone number is (571)270-7648. The examiner can normally be reached Monday - Friday, 9:00 am - 6:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Al Kawsar can be reached at (571)270-3169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/A.H.J./Examiner, Art Unit 2127
/ABDULLAH AL KAWSAR/Supervisory Patent Examiner, Art Unit 2127