DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This communication is responsive to application filed on 07/14/2023.
Claims 1-20 are presented for examination.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/19/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over US Publication No. 2020/0311300 A1 issued to CALLCUT et al in view of US Publication No. 2019/0228533 A1 issued to Giurgica-Tiron et al.
Claim 1. CALLCUT et al discloses a computerized method of performance modeling of an algorithm in a sequestered computing node comprising:
receiving an algorithm and a data set within a secure computing node (Abstract, receiving an algorithm and input data requirements associated with the algorithm; par [0010] receiving, at the data processing system, the algorithm and input data requirements associated with the algorithm, where the input data requirements include optimization and/or validation selection criteria for data assets to be run on the algorithm);
processing the data set using the algorithm to generate an algorithm output (See: Abstract, receiving an algorithm and input data requirements associated with the algorithm, identifying data assets as being available from a data host based on the input data requirements, curating the data assets within a data storage structure that is within infrastructure of the data host, and integrating the algorithm into a secure capsule computing framework. The secure capsule computing framework serves the algorithm to the data assets within the data storage structure in a secure manner that preserves privacy of the data assets and the algorithm. The computer implemented method further includes running the data assets through the algorithm to obtain an inference; par [0018] The method also includes executing, by the data processing system, a validation workflow on the algorithm, where the validation workflow takes as input the data assets, finds patterns in the data assets using learned parameters, and outputs an inference);
generating a raw performance model by regression modeling the algorithm output (See: par [0007] creating multiple instances of the model, splitting the data assets into sets of training data and one or more sets of testing data, training the multiple instances of the model on the sets of training data, integrating results from the training each of the multiple instances of the model into a fully federated model, running the one or more sets of testing data through the fully federated model, and computing performance of the fully federated model based on the running of the one or more sets of testing data. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.; par [0069] The input data requirements may include optimization and/or validation selection criteria for data assets to be run on the algorithm or model. The optimization and/or validation selection criteria define characteristics, data formats, and requirements for input data (e.g., external data) to be usable in the models. The characteristics and requirements for the input data refer to the characteristics and requirements of data such that the data is usable to optimize and/validate the model….the characteristics and requirements of the input data are defined based on: (i) the environment of the model, (ii) distribution of examples such as 50% male and 50% female, (iii) parameters and types of devices generating data (e.g., image data) and/or measurements, (iv) variance versus bias—models with high variance can easily fit into training data and welcome complexity but are sensitive to noise; whereas models with high bias are more rigid, less sensitive to variations in data and noise, and prone to missing complexities, (v) the task(s) implemented by the models such as classification, clustering, regression, ranking, and the like);
encrypting the final performance model (See: [0008] Implementations may include one or more of the following features. The method where the secure capsule computing framework is provisioned within a computing infrastructure configured to accept encrypted code required to run the algorithm, and where the provisioning the computing infrastructure includes instantiating the secure capsule computing framework on the computing infrastructure); and
routing the encrypted final performance model to an algorithm developer for further analysis (See: par [0009] The method also includes when the performance of the fully federated algorithm does satisfy the algorithm termination criteria, providing, by the data processing system, the performance of the fully federated algorithm and the aggregated parameters to an algorithm developer of the algorithm; par [0014] The method also includes when the performance of the fully federated algorithm or model does satisfy the algorithm termination criteria, providing, by the data processing system, the performance of the fully federated algorithm or model and the aggregated parameters to an algorithm developer of the algorithm or model. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods).
CALLCUT et al does not specify but Giurgica-Tiron et al discloses smoothing the raw performance model (See: Abstract, provide as input to a trained statistical model, the plurality of neuromuscular signals and temporally smooth in real-time an output of the trained statistical model. The system is also programmed to determine, based on the smoothed output of the trained statistical model; par [0057] In one example implementation, a simple non-linear Kalman filter may be used to smooth model prediction outputs. According to one aspect, performance of the Kalman filter may be modified in real-time responsive to how fast the output changes and where the system believes the model is at in real-time (e.g., modify loss function to the model)).
It would have been obvious before the effective filing date to combine the real-time smoothing as taught by Giurgica-Tiron et al to distributed privacy-preserving computing on protected data method of CALLCUT et al would be to train the statistical model with a penalization term to promote smoothness in the model outputs (Giurgica-Tiron et al, par [0008]).
Claim 2. CALLCUT et al discloses the method of claim 1, wherein the performance model models at least one of algorithm accuracy, F1 score accuracy, precision, recall, dice score, ROC (receiver operator characteristic) curve/area, log loss, Jaccard index, error, R2 or by some combination thereof (See: par [0068] common error metrics such as mean percentage error and R2 score are not always indicative of accuracy of a model, and thus the algorithm developer may want to define additional metrics and criteria for a more in depth look at accuracy of the model. For example, if a chosen data set includes time series data which tends to be correlated in time and often exhibits a significant autocorrelation (when evaluating the model in terms of its ability of predicting the value directly, common error metrics such as mean percentage error and R.sup.2 (coefficient of determination) score both indicate a false high prediction accuracy), then the algorithm developer may desire to monitor the autocorrelation using one or more additional metrics or criteria).
Claim 3. Giurgica-Tiron et al discloses the method of claim 1, wherein the regression modeling includes linear least squares, logistic regression, deep learning or some combination thereof (See: par [0083] the statistical model may be a neural network and, for example, may be a recurrent neural network. In some embodiments, the recurrent neural network may be a long short-term memory (LSTM) neural network).
Claim 4. Giurgica-Tiron et al discloses the method of claim 1, wherein the smoothing includes identifying portions of the raw performance model which are highly variable (See: [0054] FIG. 4 illustrates a schematic diagram of a computer-based process 400 for temporally smoothing an output in real-time according to some embodiments. According to one aspect, it is appreciated that the best estimate of the model at any particular time point may not be the correct or most accurate representation. Thus, in a system where sensor outputs are received in real-time, including noise, high variation outputs (e.g., highly variant movement data), or other signals that may drastically change the state of the statistical model, it may be beneficial to provide some type of smoothing function (e.g. a smoothing function correspond to a non-linear Kalmaan filter) that reduces errors and/or otherwise improves the reliability that the output response models the user's intended movement).
Claim 5. Giurgica-Tiron et al discloses the method of claim 4, wherein the smoothing includes best fit transform, moving averages and application of filters, Loess smoothing, kernel smoothing, wavelets, splines or some combination thereof (See: par [0006] According to some embodiments, temporally smoothing the output of the trained statistical model comprises processing the output of the trained statistical model using at least one filter. According to some embodiments, the at least one filter comprises at least one filter from a group comprising an exponential filter, a Kalman filter, a non-linear Kalman filter, a particle filter and a Bayesian filter. According to some embodiments, the at least one computer processor is programmed to determine an accuracy of the trained statistical model, and temporally smooth, responsive to the determined accuracy of the trained statistical model, the output of the trained statistical model in real-time; par [0087] For example, the sensor data may be analyzed as time series data using wavelet analysis techniques (e.g., continuous wavelet transform, discrete-time wavelet transform, etc.), Fourier-analytic techniques (e.g., short-time Fourier transform, Fourier transform, etc.), and/or any other suitable type of time-frequency analysis technique).
Claim 6. Giurgica-Tiron et al discloses the method of claim 5, wherein the smoothing weights the data points of the raw performance model by instances of the algorithm’s input variables (See: Abstract, provide as input to a trained statistical model, the plurality of neuromuscular signals and temporally smooth in real-time an output of the trained statistical model. The system is also programmed to determine, based on the smoothed output of the trained statistical model; par [0057] In one example implementation, a simple non-linear Kalman filter may be used to smooth model prediction outputs. According to one aspect, performance of the Kalman filter may be modified in real-time responsive to how fast the output changes and where the system believes the model is at in real-time (e.g., modify loss function to the model)).
Claim 7. CALLCUT et al discloses the method of claim 1, wherein the algorithm developer receives multiple final performance models from the algorithm operating on a plurality of data sets (See: 0007] Implementations may include one or more of the following features. The method where the algorithm and input data requirements are received from an algorithm developer, which is a different entity from the data host, and the optimization and/or validation selection criteria define characteristics, formats and requirements for the data assets to be run on the algorithm).
Claim 8. CALLCUT et al discloses the method of claim 7, wherein the further analysis includes identifying at least one perturbation in the multiple final performance models (See: par [0007] The method where the characteristics and the requirements of the data assets are defined based on: (i) the environment of the algorithm, (ii) distribution of examples in the input data, (iii) parameters and types of devices generating the input data, (iv) variance versus bias, (v) tasks implemented by the algorithm, or (vi) any combination thereof. The method further including onboarding, by the data processing system, the data host, where the onboarding includes confirming that the use of the data assets with the algorithm is in compliance with data privacy requirements. The method where the preparing the data assets includes applying one or more transforms to the data assets, annotating the data assets, harmonizing the data assets, or a combination thereof. The method where the running the data assets through the algorithm includes executing a training workflow that includes: creating multiple instances of the model, splitting the data assets into sets of training data and one or more sets of testing data, training the multiple instances of the model on the sets of training data, integrating results from the training each of the multiple instances of the model into a fully federated model, running the one or more sets of testing data through the fully federated model, and computing performance of the fully federated model based on the running of the one or more sets of testing data).
Claim 9. CALLCUT et al discloses the method of claim 1, wherein the further analysis includes identifying portions of the final performance model with lower performance and provides feedback to a data steward to generate more training data for variables in the data set associated with said portions (See: par [0009] The method also includes when the performance of the fully federated algorithm does not satisfy the algorithm termination criteria, replacing, by the data processing system, each instance of the algorithm with the fully federated algorithm and re-executing the federated training workflow on each instance of the fully federated algorithm. The method also includes when the performance of the fully federated algorithm does satisfy the algorithm termination criteria, providing, by the data processing system, the performance of the fully federated algorithm and the aggregated parameters to an algorithm developer of the algorithm).
Claim 10. CALLCUT et al discloses the method of claim 7, further comprising performing training on the algorithm in response to the feedback (See: par [0009] The method also includes when the performance of the fully federated algorithm does not satisfy the algorithm termination criteria, replacing, by the data processing system, each instance of the algorithm with the fully federated algorithm and re-executing the federated training workflow on each instance of the fully federated algorithm. The method also includes when the performance of the fully federated algorithm does satisfy the algorithm termination criteria, providing, by the data processing system, the performance of the fully federated algorithm and the aggregated parameters to an algorithm developer of the algorithm).
As per Claims 11-20: The instant claims recite substantially same limitation as the above rejected claims 1-10 and therefore rejected under the same rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KIBROM K GEBRESILASSIE whose telephone number is (571)272-8571. The examiner can normally be reached M-F 9:00 AM-5:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Rehana Perveen can be reached at 571 272 3676. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
KIBROM K. GEBRESILASSIE
Primary Examiner
Art Unit 2189
/KIBROM K GEBRESILASSIE/Primary Examiner, Art Unit 2189 08/25/2026
Prosecution Insights