Detailed Action
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This is the initial office action based on the application filed on January 06th, 2023, which claim 1-20 have been presented for examination.
Status of Claims
2. Claims 1-20 are pending in the application, of which claim 1 is in independent form and these claims (1-20) are subject to following rejection(s) and/or objection(s) set forth in the following Office Action below.
Examiner Notes
3. (A). Information Disclosure Statement (IDS): The information disclosure statements filed on 02/22/2024 comply with the provisions of 37 CFR 1.97, 1.98. They have been placed in the application file and the information referred to therein has been considered as to the merits.
(B). Priority: Effective filing date considered for the following office action is 06/03/2019.
Double Patenting
4. The claims 1-20 are rejected on the ground of Statutory double patenting of the claims in United States Patent No. 11,599,752.
A rejection based on double patenting of the “same invention” type finds its support in the language of 35 U.S.C. 101 which states that “whoever invents or discovers any new and useful process... may obtain a patent therefor...” (Emphasis added). Thus, the term “same invention,” in this context, means an invention drawn to identical subject matter. See Miller v. Eagle Mfg. Co., 151 U.S. 186 (1894); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); and In re Ockert, 245 F.2d 467, 114 USPQ 330 (CCPA 1957).
A statutory type (35 U.S.C. 101) double patenting rejection can be overcome by canceling or amending the claims that are directed to the same invention so they are no longer coextensive in scope. The filing of a terminal disclaimer cannot overcome a double patenting rejection based upon 35 U.S.C. 101.
Instant Application No. 18/094,279
US Patent No. 11,599,752
1. A tangible, non-transitory, machine-readable medium storing instructions that when executed by one or more processors in a computing system effectuate operations to execute quality management of modelling methods for implementation of a machine learning design in an object-oriented modeling (OOM) framework, the operations comprising: writing, with the computing system, modelling-object classes using object-oriented modelling of the modelling methods, the modelling-object classes being members of a set of class libraries; writing, with the computing system, quality-management classes using object-oriented modelling of quality management, the quality-management classes being members of the set of class libraries; scanning, with the computing system, modelling-object classes in the set of class libraries to determine modelling-object class definition information; scanning, with the computing system, quality-management classes in the set of class libraries to determine quality-management class definition information; using, with the computing system, the modelling-object class definition information and the quality-management class definition information to produce object manipulation functions that allow a quality management system to access methods and attributes of modelling-object classes to manipulate objects of the modelling-object classes; and using, with the computing system, the modelling-object class definition information and the quality-management class definition information to produce access to the object manipulation functions.
1. A tangible, non-transitory, machine-readable medium storing instructions that when executed by one or more processors in a computing system effectuate operations to execute quality management of modelling methods for implementation of a machine learning design in an object-oriented modelling (OOM) framework, the operations comprising: writing, with the computing system, modelling-object classes using OOM of the modelling methods, the modelling-object classes being members of a set of class libraries; writing, with the computing system, quality-management classes using OOM of quality management, the quality-management classes being members of the set of class libraries; scanning, with the computing system, the modelling-object classes in the set of class libraries to determine modelling-object class definition information; scanning, with the computing system, the quality-management classes in the set of class libraries to determine quality-management class definition information; using, with the computing system, the modelling-object class definition information and the quality-management class definition information to produce object manipulation functions that allow a quality management system to access methods and attributes of the modelling-object classes to manipulate objects of the modelling-object classes; and using, with the computing system, the modelling-object class definition information and the quality-management class definition information to produce access to the object manipulation functions.
2. The medium of claim 1, wherein: executing quality management comprises executing a process that integrates raw data ingestion, manipulation, transformation, composition, and storage for building artificial intelligence models.
2. The medium of claim 1, wherein: executing the quality management of modelling methods comprises executing a process that integrates raw data ingestion, manipulation, transformation, composition, and storage for building artificial intelligence models.
3. The medium of claim 1, wherein: the modeled quality management comprises management of extract, transform, and load (ETL) phases of a machine learning model designed in the OOM framework.
3. The medium of claim 1, wherein: the quality management of modelling methods comprises management of extract, transform, and load (ETL) phases of a machine learning model designed in the OOM framework.
4. The medium of claim 1, wherein: the modeled quality management comprises reporting of model performance of a machine learning model designed in the OOM framework.
4. The medium of claim 1, wherein: the quality management of modelling methods comprises reporting of model performance of a machine learning model designed in the OOM framework.
5. The medium of claim 4, wherein: model performance is measured by recall, precision, or F1 score.
5. The medium of claim 4, wherein: the model performance is measured by recall, precision, or F1-score.
6. The medium of claim 1, wherein: the modeled quality management comprises data quality monitoring (DQM).
6. The medium of claim 1, wherein: the quality management of modelling methods comprises data quality monitoring (DQM).
7. The medium of claim 6, wherein: DQM comprises monitoring data sources to detect a new or missing table or data element, data element counts, data element null count and unique counts, or datatype changes.
7. The medium of claim 6, wherein: the DQM comprises monitoring data sources to detect a new or missing table or data element, data element counts, data element null count and unique counts, or datatype changes.
8. The medium of claim 1, wherein: the modeled quality management comprises model quality monitoring (MQM) of a machine learning model designed in the OOM framework.
8. The medium of claim 1, wherein: the quality management of modelling methods comprises model quality monitoring (MQM) of a machine learning model designed in the OOM framework.
9. The medium of claim 8, wherein: MQM comprises measuring a model-based metric and causing model retraining responsive to detecting more than a threshold amount of drift in the model-based metric.
9. The medium of claim 8, wherein: the MQM comprises measuring a model-based metric and causing model retraining responsive to detecting more than a threshold amount of drift in the model-based metric.
10. The medium of claim 1, wherein: the modeled quality management comprises score quality monitoring (SQM) of a machine learning model designed in the OOM framework.
10. The medium of claim 1, wherein: the quality management of modelling methods comprises score quality monitoring (SQM) of a machine learning model designed in the OOM framework.
11. The medium of claim 10, wherein: SQM comprises performing a model hypothesis test.
11. The medium of claim 10, wherein: the SQM comprises performing a model hypothesis test.
12. The medium of claim 10, wherein: SQM comprises computing a lift table or a decile table.
12. The medium of claim 10, wherein: the SQM comprises computing a lift table or a decile table.
13. The medium of claim 1, wherein: the modeled quality management comprises label quality monitoring (LQM) of a machine learning model designed in the OOM framework.
13. The medium of claim 1, wherein: the quality management of modelling methods comprises label quality monitoring (LQM) of a machine learning model designed in the OOM framework.
14. The medium of claim 13, wherein: LQM comprises determining which data sources among a plurality of data sources are more leverageable or impactful on model performance than other data sources among the plurality of data sources.
14. The medium of claim 13, wherein: the LQM comprises determining which data sources among a plurality of data sources are more leverageable or impactful on model performance than other data sources among the plurality of data sources.
15. The medium of claim 1, wherein: the modeled quality management comprises bias quality monitoring (BQM) of a machine learning model designed in the OOM framework.
15. The medium of claim 1, wherein: the quality management of modelling methods comprises bias quality monitoring (BQM) of a machine learning model designed in the OOM framework.
16. The medium of claim 15, wherein BQM comprises detecting information bias, selection bias, or confounding by the machine learning model designed in the OOM framework.
16. The medium of claim 15, wherein the BQM comprises detecting information bias, selection bias, or confounding by the machine learning model designed in the OOM framework.
17. The medium of claim 1, wherein: the modeled quality management comprises privacy quality monitoring (PQM) of a machine learning model designed in the OOM framework.
17. The medium of claim 1, wherein: the quality management of modelling methods comprises privacy quality monitoring (PQM) of a machine learning model designed in the OOM framework.
18. The medium of claim 1, wherein: the modeled quality management comprises data quality monitoring (DQM) of a machine learning model designed in the object-oriented modeling (OOM) framework; DQM comprises monitoring data sources to detect a new or missing table or data element, data element counts, data element null count and unique counts, and datatype changes; the modeled quality management comprises model quality monitoring (MQM) of the machine learning model designed in the object-oriented modeling (OOM) framework; MQM comprises measuring a model-based metric and causing model retraining responsive to detecting more than a threshold amount of drift in the model-based metric; the model-based metric is indicative of an F 1 score, accuracy, precision, mean error, media error, distance measure, or recall; the modeled quality management comprises score quality monitoring (SQM) of the machine learning model designed in the object-oriented modeling (OOM) framework; SQM comprises performing a model hypothesis test and computing a lift table and a decile table based on predicted probability of positive class membership, based on a cumulative distribution function of positive cases; the model hypothesis test comprises a Welch's t-test, Kolmogorov-Smirnov test, or a Mann-Whitney U-test; the modeled quality management comprises label quality monitoring (LQM) of the machine learning model designed in the object-oriented modeling (OOM) framework; LQM comprises determining which data sources among a plurality of data sources are more leverageable or impactful on model performance than other data sources among the plurality of data sources; the modeled quality management comprises bias quality monitoring (BQM) of the machine learning model designed in the object-oriented modeling (OOM) framework; BQM comprises detecting information bias, selection bias, and confounding by the machine learning model designed in the object-oriented modeling (OOM) framework; the modeled quality management comprises privacy quality monitoring (PQM) of the machine learning model designed in the OOM framework.
18. The medium of claim 1, wherein: the quality management of modelling methods comprises data quality monitoring (DQM) of a machine learning model designed in the OOM framework; the DQM comprises monitoring data sources to detect a new or missing table or data element, data element counts, data element null count and unique counts, and datatype changes; the quality management of modelling methods comprises model quality monitoring (MQM) of the machine learning model designed in the OOM framework; the MQM comprises measuring a model-based metric and causing model retraining responsive to detecting more than a threshold amount of drift in the model-based metric; the model-based metric is indicative of F1-score, accuracy, precision, mean error, media error, distance measure, or recall; the quality management of modelling methods comprises score quality monitoring (SQM) of the machine learning model designed in the OOM framework; the SQM comprises performing a model hypothesis test and computing a lift table and a decile table based on predicted probability of positive class membership, based on a cumulative distribution function of positive cases; the model hypothesis test comprises a Welch's t-test, Kolmogorov-Smirnov test, or a Mann-Whitney U-test; the quality management of modelling methods comprises label quality monitoring (LQM) of the machine learning model designed in the OOM framework; the LQM comprises determining which data sources among a plurality of data sources are more leverageable or impactful on model performance than other data sources among the plurality of data sources; the quality management of modelling methods comprises bias quality monitoring (BQM) of the machine learning model designed in the OOM framework; the BQM comprises detecting information bias, selection bias, and confounding by the machine learning model designed in the OOM framework; and the quality management of modelling methods comprises privacy quality monitoring (PQM) of the machine learning model designed in the OOM framework.
19. The medium of claim 1, wherein: the modeled quality management comprises a process to determine data source reliability.
19. The medium of claim 1, wherein: the quality management of modelling methods comprises a process to determine data source reliability.
20. The medium of claim 1, wherein: an attribute of a quality-management object in one of the quality-management classes comprise means for characterizing quality with the attribute of the quality-management object.
20. The medium of claim 1, wherein: an attribute of a quality-management object in one of the quality-management classes comprise means for characterizing quality with the attribute of the quality-management object.
Examiner anticipate that the claims 1-20 would be extensively/ sufficiently amended, or claims 1-20 would be canceled and provided with a distinct set of claims; therefore, for the current office action prior art rejections to claims 1-20 have been put in abeyance until a formal response and a new set or amended claims are received. Applicants are also reminded that the next office action may possibly contain a non-statutory (or obviousness type) double patenting rejection along with the prior art rejection. And following office action would be made final.
CONCLUSION
5. Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZIAUL A. CHOWDHURY whose telephone number is (571)270-7750. The examiner can normally be reached on 9:30PM 6:30PM Monday -Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hyung S. Sough can be reached on 571-272-6799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Status information for published applications may be obtained from Patent Public Search tool (for all users) – A link to the Patent Public Search Tool is available at www. Uspto.gov/PatentPublicSearch. To find a U.S. patent or U.S. patent application publication, open the Patent Public Search tool by selecting “Start search”. Type the U.S. patent or U.S. patent application publication number in the “Search” panel without any punctuation and followed by an”.pn.”.
Should you have questions on access to the system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZIAUL A CHOWDHURY/ Primary Examiner, Art Unit 2192
09/23/2025