DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are pending.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Analysis of Independent Claims (1, 10, and 20):
Step 1: Statutory Categories: Claim 1 is a machine (apparatus). Claim 10 is a process (method). Claim 20 is an article of manufacture (computer-readable medium). All fall within statutory categories.
Step 2A, Prong 1: Does the claim recite an abstract idea? Yes. The claims are directed to generating a "target approximate polynomial" for a "neural network operation" by calculating maximum and minimum values, and generating a "neural network operation result." These limitations recite mathematical concepts (mathematical formulas, relationships, and calculations) and mental processes (evaluating limits and executing mathematical approximations).
Step 2A, Prong 2: Is the abstract idea integrated into a practical application? To satisfy Prong 2, the claim must improve the functioning of a computer or apply the abstract idea to a specific technological environment in a non-conventional way. The title of the patent and the specification heavily emphasize performing neural network operations on homomorphically encrypted data. The specification details that applying standard ReLU functions to encrypted data is mathematically difficult, thereby necessitating the claimed dynamic polynomial approximation to avoid overflow while maintaining a low polynomial degree. However, "encrypted data" and "homomorphic encryption" are completely absent from Claims 1, 10, and 20. The claims as currently drafted broadly cover optimizing a standard neural network activation function (like a ReLU) by swapping it with an approximate polynomial based on min/max input bounds. Without the encrypted data limitation, the claim reads as a purely mathematical optimization of a mathematical model. Modifying mathematical operations to yield a more accurate or computationally efficient mathematical result is generally not considered a practical application on its own unless explicitly tied to a specific hardware improvement or physical world application.
Step 2B: Do the claims recite "significantly more" (an inventive concept)? No. The recited hardware in Claim 1 ("one or more processors," "one or more memories") and the inherent computer execution in Claims 10 and 20 are invoked at a high level of generality. Merely implementing a new mathematical algorithm on generic computer components is insufficient to transform an abstract idea into patent-eligible subject matter.
The claims are directed to mathematical algorithms and lack the critical "homomorphic encryption" limitation that provides the technological context and practical application described in the specification.
Claims 2, 3, and 11, 12: Implementing other portions of the NN and generating a final result. Executing subsequent mathematical layers of a neural network model to reach a final numerical output is merely completing the abstract mathematical process. It does not integrate the concept into a practical application.
Claims 4, 13: Updating the approximate region of the first polynomial. This recites the mathematical mechanics of generating the polynomial. It adds further detail to the abstract idea itself but does not integrate it into a technological application.
Claims 5, 14: Setting regions based on a total number of ReLU portions and total input data samples. Defining mathematical bounds using sample set sizes is a standard statistical/mathematical technique. It remains an abstract concept.
Claims 6, 15, 19: Specifically reciting a Rectified Linear Unit (ReLU) and calculating absolute values. A ReLU is simply a specific mathematical function ($f(x) = \max(0, x)$). Calculating absolute values to bound this function is purely mathematical.
Claims 7, 16: Applying different approximation regions to different layers. Varying the variables of a mathematical equation across different iterations (layers) of an algorithm does not confer eligibility.
Claims 8, 17: Utilizing a precision parameter and precision threshold to bound the absolute value of the error. This is a recitation of mathematical error-bounding and limits. Ensuring an approximation stays within a mathematical tolerance (precision threshold) does not provide an inventive concept or practical application outside the realm of mathematics.
Claims 9, 18: Iteratively incrementing the update of the interim region until a calculated accuracy meets a threshold. This recites a standard computational loop ("until a calculated accuracy... meets an accuracy threshold"). While iterative loops are foundational to programming, claiming the logical structure of mathematical iteration is generally treated as an abstract mental process or algorithm.
None of the dependent claims cure the § 101 defect of the independent claims. They entirely focus on refining the mathematical algorithm (bounding, error checking, iterating) rather than applying the algorithm to a specific technological challenge (like processing homomorphically encrypted data) or improving the physical execution of the hardware.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1 and 10 recite the limitation "the portion of the neural network layer" in lines 10-11. There is insufficient antecedent basis for this limitation in the claim. It is unclear if this is the same or different from “a portion of a neural network model” in line 7.
Claims 1 and 10 recite “…generated based on parameters corresponding to a generation of the first approximate polynomial…”. This phrasing is circular and ambiguous. It essential states: “The polynomial is generated based on the parameters used to generate it.” The parameters and limits of the scope are unclear. Claims 8 and 17 attempt to clarify this by stating the parameters include a “precision parameter”. However, the vague language in claims 1 and 10 fail to distinctly claim the invention.
Claims 8 and 17 recite “a precision parameter to control a precision” and “calculate a precision threshold based on the precision parameter”. The claims do not specific the mathematical relationship between the parameter and the threshold. The claims do not reasonably identify the boundaries of the claim.
Claims 1 and 10 recite “…using a determined target approximation region for the target approximate polynomial based on a first approximate polynomial…”. It is unclear how the “determined target approximation region” relates to the rest of the claim. The claim does not actively recite a step of determining the region, nor does it clearly link how the maximum and minimum values are used to shape this region. The claim does not specify how the region is determined, and the actual boundaries of the claim are difficult to ascertain. The claims do not reasonably inform a person of ordinary skill in the art of the metes and bounds of the claimed “target approximation region” and the relationship among the first approximate polynomial, parameters and maximum/minimum values.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-20 are rejected under 35 U.S.C. 102(a)(1)/(a)(2) as being anticipated by CHEON et al, US 2022/0337390.
Regarding claim 1, CHEON teaches a computing apparatus, comprising: one or more processors configured to execute instructions; and
one or more memories storing the instructions, wherein the execution of the instructions by the one or more processors configures the one or more processors (Fig. 2, paragraph 0016, 0068-0083: an “operation device 400: comprising a “memory 420” configured to store at least one instruction and a “processor 450” configured to execute the instruction) to:
generate a target approximate polynomial, approximating a neural network operation of a portion of a neural network model (0016 and 0063: the processor “computes an approximate polynomial function corresponding to the non-polynomial operation”. 0006-0008: applied to “machine learning (ML)” algorithms and neural network functions. 0015: explicitly lists ReLU-n functions), using a determined target approximation region for the target approximate polynomial based on a first approximate polynomial generated based on parameters corresponding to a generation of the first approximate polynomial (generating a “second approximate polynomial function” (the target approximate polynomial) obtained by “extending a first approximate polynomial function to have a second range wider than the first range” (target approximation region) having a preset accuracy, 0010, 0017, 0125-0126, 0142), a maximum value of input data to the portion of the neural network layer, and a minimum value of the input data (calculates the limits of the input data to determine the new extended range. Calculates the “input absolute value” to find its upper bound limit. By defining the approximation interval as for the target polynomial, explicitly sets the region based on the calculated minimum and maximum values of the input data, 0219.); and
generate a neural network operation result using the target approximate polynomial and the input data (performs the “operation of the homomorphic ciphertext using the computed polynomial function, and outputting the operated homomorphic ciphertext”; 0010, 0061-0063.).
Regarding claim 2, CHEON teaches the computing apparatus of claim 1, wherein the execution of the instructions by the one or more processors configures the one or more processors to: generate the input data by implementing another portion of the neural network model (models (e.g. logistic regression) where intermediate vectors and inputs are generated sequentially from previous iterations, attributes and weights within the machine learning model, 0213-0214).
Regarding claim 3, CHEON teaches the computing apparatus of claim 2, wherein the execution of the instructions by the one or more processors configures the one or more processors to: perform the generation of the target approximate polynomial and generation of the neural network operation result for each of plural portions of the neural network model; and generate a result of the neural network model dependent on each of the generated neural network operation results (computes extended approximation intervals and polynomials for “each non-polynomial function in the machine learning model” 0023, 0220; to calculate the models final classification 0130).
Regarding claim 4, CHEON teaches the computing apparatus of claim 1, wherein the one or more processors are configured to: set the approximation region based on the maximum value and the minimum value; and generate the target approximate polynomial by updating an approximate region of the first approximate polynomial based on the set approximation region (calculates the bounds based on the maximum limit of the input and computes the second approximate polynomial “that has a value corresponding to a computation value of the first approximate polynomial… in an end of the first range with the second range”. 0011, 0018, 0219).
Regarding claim 5, CHEON teaches the computing apparatus of claim 4, wherein the one or more processors are configured to: set respective approximate regions of respective approximate polynomials, of plural rectified linear unit (ReLU) portions of the neural network model, based on a total number of the ReLU portions of the neural network model and a total number of input data samples input to the neural network model (calculates the expanded boundary for the second range based on variables including the “mini batch size is n” (number of data samples) and “attributes of the data d” across iterative layers t. 0013-0014, 0213-0215, 0219).
Regarding claim 6, CHEON teaches the computing apparatus of claim 1, wherein the neural network operation comprises a rectified linear unit (ReLU), and wherein the one or more processors are configured to: calculate absolute values of data determined for input to the neural network operation; and calculate the maximum value based on the calculated absolute values (explicitly identifies ReLU-n functions as target non-polynomial functions. 0015, 0022, 0126. Calculates the absolute value of the input data to determine the maximum boundary for the interval 0214-0215).
Regarding claim 7, CHEON teaches the computing apparatus of claim 1, wherein the execution of the instructions by the one or more processors configures the one or more processors to: perform the generation of the target approximate polynomial and generation of the neural network operation result for each of plural portions of the neural network model; and wherein a first approximation region generated corresponding to a first layer of the plural portions is different from a second approximation region generated corresponding to a second layer of the plural portions (the approximate interval is “carefully selected in each iteration” 0210, 0220, Updating the interval dynamically as input magnitudes shift across iterations/layers is explicitly taught).
Regarding claim 8, CHEON teaches the computing apparatus of claim 1, wherein the parameters comprise a precision parameter to control a precision of the target approximate polynomial with respect to the approximate region, and wherein, for the generation of the target approximate polynomials, the one or more processors are configured to: calculate a precision threshold based on the precision parameter; and generate the first approximate polynomial such that an absolute value of an error between the neural network operation and the first approximate polynomial is equal to or less than the precision threshold (first approximate polynomial has a “preset accuracy…within a first range” 0010, 0017. Evaluated to maintain a specified maximum error (e.g. a “supremum norm error”) such as keeping error < 0.05, which acts as the precision threshold, 0164, 0172, 0222).
Regarding claim 9, CHEON teaches the computing apparatus of claim 1, wherein, for the determination of the target approximate region, the one or more processors are configured to: calculate an accuracy of an interim target approximate polynomial based on the first approximate polynomial; until a calculated accuracy of an updated interim target approximate polynomial meets an accuracy threshold: increment an update of an interim approximation region of the interim target approximate polynomial to generate the updated interim target approximate polynomial; and calculate the accuracy of the updated interim target approximate polynomial, wherein, when the updated interim target approximate polynomial meets the accuracy threshold, the updated interim target approximate polynomial is the generated target approximate polynomial (determining a “number of iterations and a basic function…and iteratively composing the basic function and a scaling basic function…by the determined number of iterations to compute a second approximate polynomial” 0012, 0019. Checks error bounds across these iterations to ensure the composite polynomial meets the targeted error thresholds, 0151-0153, 0164-0165).
Regarding claim 19, CHEON teaches the method of claim 12, wherein the neural network operation is a ReLU function (ReLu-n functions as specific target operations for the polynomial extension method, 0015, 0022, 0126).
As per claims 10-18 and 20, this is a method and medium version of the claimed apparatus discussed above in claims 1-9 wherein all claimed limitations have also been addressed and/or cited as set forth above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 2022/0085972 teaches a server can receive from a client, a public key of a fully homomorphic encryption scheme and one or more encrypted numbers, and perform a set of operations comprising a square root function, a rectified linear activation function (ReLU), or a multiplicative inverse function on the one or more encrypted numbers to generate an encrypted operational result. The encrypted operational result generated by the set of operations can be an approximation of performing an algebraic operation on the one or more encrypted number. The server can further transmit to the client the encrypted operational result.
US 9,946,970 teaches performing neural network computations on encrypted data. Encrypted data is received from a user. The encrypted data is encrypted with an encryption scheme that allows for computations on the ciphertext to generate encrypted results data. Neural network computations are performed on the encrypted data, using approximations of neural network functions to generate encrypted neural network results data from encrypted data. The approximations of neural network functions can approximate activation functions, where the activation functions are approximated using polynomial expressions. The encrypted neural network results data are communicated to the user associated with the encrypted data such that the user decrypts the encrypted data based on the encryption scheme. The functionality of the neural network system can be provided using a cloud computing platform that supports restricted access to particular neural networks.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AUBREY H WYSZYNSKI whose telephone number is (571)272-8155. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ALI SHAYANFAR can be reached at 571-270-1050. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AUBREY H WYSZYNSKI/Primary Examiner, Art Unit 2434
Prosecution Insights