DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in responsive to communication(s): original application filed on 03/17/2023, said application claims a priority filing date of 03/25/2022. Claims 1-18 are pending. Claims 1, 5, 9, and 13 are independent.
Drawings
The drawings are objected to because (1) FIG. 5 (e.g., readMOI(List(MLTrainingJobs))) is inconsistent with description of FIG. 5 in ¶ [0021] that "FIG. 5 illustrates an example of modifying the characteristics of one or more ongoing or completed MLTrainingJobs …"; (2) FIG. 6 (e.g., modifyMOIAttributes(List(MLTrainingJobs), attributes)) is inconsistent with description of FIG. 6 in ¶ [0022] that "FIG. 6 illustrates an example of reading the characteristics of one or more ongoing or completed MLTrainingJobs …"; (3) FIG. 14 (e.g., readMOI(List(MLTrainingReporting))) is inconsistent with description of FIG. 14 in ¶ [0030] that "FIG. 14 illustrates an example of reading the characteristics of one or more ongoing or completed MLTrainingJobs …"; (4) FIG. 15 (e.g., modifyMOIAttributes(List(MLTrainingReporting), attributes)) is inconsistent with description of FIG. 15 in ¶ [0031] that "FIG. 15 illustrates an example of reading the characteristics of one or more ongoing or completed MLTrainingJobs …"; (5) FIG. 16 (e.g., deleteMOI (List(MLTrainingReportingID))) is inconsistent with description of FIG. 16 in ¶ [0032] that "FIG. 16 illustrates an example of deleting the characteristics of one or more ongoing or completed MLTrainingJobs …"; (6) FIG. 17 (e.g., readMOI (List(MLTrainingReportingRequests))) is inconsistent with description of FIG. 17 in ¶ [0033] that "FIG. 17 illustrates an example of reading the characteristics of one or more ongoing or completed MLTrainingJobs …"; (7) FIG. 18 (e.g., modifyMOIAttributes(List(MLTrainingReportingRequestsID), attributes)) is inconsistent with description of FIG. 18 in ¶ [0034] that "FIG. 18 illustrates an example of modifying the characteristics of one or more ongoing or completed MLTrainingJobs …"; and (8) FIG. 19 (e.g., deleteMOI (List(MLTrainingReportingRequestsID))) is inconsistent with description of FIG. 19 in ¶ [0035] that "FIG. 19 illustrates an example of deleting the characteristics of one or more ongoing or completed MLTrainingJobs …". Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities:
in ¶ [0021], "FIG. 5 illustrates an example of modifying …" appears to be "FIG. 5 illustrates an example of reading …";
in ¶ [0022], "FIG. 6 illustrates an example of reading …" appears to be "FIG. 5 illustrates an example of modifying …";
in ¶ [0031], "FIG. 15 illustrates an example of reading …" appears to be "FIG. 15 illustrates an example of modifying …";.
Appropriate correction is required.
Claim Objections
Claims 3-4, 7-8, and 10-16 are objected to because of the following informalities:
in Claim 3, lines 5-6, "… transmitting … a notification to the consumer of the list of modified machine learning training jobs identifiers" appears to be "… transmitting … a notification to the consumer indicating the list of modified machine learning training jobs identifiers" according to FIG. 6 and Claim 1;
in Claim 4, lines 5-6, "… transmitting … a notification to the consumer of the list of deleted machine learning training jobs identifiers" appears to be "… transmitting … a notification to the consumer indicating the list of deleted machine learning training jobs identifiers" according to FIG. 7 and Claim 1;
in Claim 7, lines 5-6, "… receiving … a notification from the machine learning training function of the list of modified machine learning training jobs identifiers" appears to be "… receiving … a notification from the machine learning training function indicating the list of modified machine learning training jobs identifiers" according to FIG. 6 and Claim 5;
in Claim 8, lines 5-6, "… receiving … a notification from the machine learning training function of the list of deleted machine learning training jobs identifiers" appears to be "… receiving … a notification from the machine learning training function indicating the list of deleted machine learning training jobs identifiers" according to FIG. 7 and Claim 5;
in Claims 10-12, lines 1-2, "… wherein the at least one memory and computer program code are further configured …" appears to be "… wherein the at least one memory and the computer program code are further configured …";
in Claim 11, lines 6-7, "… transmitting a notification to the consumer of the list of modified machine learning training jobs identifiers" appears to be "… transmitting a notification to the consumer indicating the list of modified machine learning training jobs identifiers" according to FIG. 6 and Claim 9;
in Claim 12, lines 6-7, "… transmitting a notification to the consumer of the list of deleted machine learning training jobs identifiers" appears to be "… transmitting a notification to the consumer indicating the list of deleted machine learning training jobs identifiers" according to FIG. 7 and Claim 9;
in Claim 13, line 8, "… receiving a notification form the machine learning training function …" appears to be "… receiving a notification from the machine learning training function …" according to Claim 5;
in Claims 14-16, lines 1-2, "… wherein the at least one memory and computer program code are further configured …" appears to be "… wherein the at least one memory and the computer program code are further configured …";
in Claim 15, lines 7-8, "… receiving … a notification from the machine learning training function of the list of modified machine learning training jobs identifiers" appears to be "… receiving … a notification from the machine learning training function indicating the list of modified machine learning training jobs identifiers" according to FIG. 6 and Claim 13;
in Claim 16, lines 7-8, "… receiving … a notification from the machine learning training function of the list of deleted machine learning training jobs identifiers" appears to be "… receiving … a notification from the machine learning training function indicating the list of deleted machine learning training jobs identifiers" according to FIG. 7 and Claim 13.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2, 6, 10, and 14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 2 recite the limitation "… receiving … a request from the consumer to read a managed object instance associated with a list of machine learning training jobs; and transmitting … a list of machine learning training jobs identifiers to the consumer" in lines 2-6, which rendering these claims indefinite because it is unclear whether "a list of machine learning training jobs identifiers" are identifiers for the same or different "list of machine learning training jobs" associated with a "managed object instance". For examination purpose, "… receiving … a request from the consumer to read a managed object instance associated with a list of machine learning training jobs; and transmitting … identifiers associated with the list of machine learning training jobs to the consumer" or "… receiving … a request from the consumer to read a managed object instance associated with a list of machine learning training jobs identifiers; and transmitting … the list of machine learning training jobs identifiers to the consumer" according to Claims 3 and 4.
Claim 6 recite the limitation "… transmitting … a request to the machine learning training function to read a managed object instance associated with a list of machine learning training jobs; and receiving … a list of machine learning training jobs identifiers from the machine learning training function" in lines 2-6, which rendering these claims indefinite because it is unclear whether "a list of machine learning training jobs identifiers" are identifiers for the same or different "list of machine learning training jobs" associated with a "managed object instance". For examination purpose, "… transmitting … a request to the machine learning training function to read a managed object instance associated with a list of machine learning training jobs; and receiving … identifiers associated with the list of machine learning training jobs from the machine learning training function" or "… transmitting … a request to the machine learning training function to read a managed object instance associated with a list of machine learning training jobs identifiers; and receiving … the list of machine learning training jobs identifiers from the machine learning training function" according to Claims 7 and 8.
Claim 10 recite the limitation "… receiving a request from the consumer to read a managed object instance associated with a list of machine learning training jobs; and transmitting a list of machine learning training jobs identifiers to the consumer" in lines 4-6, which rendering these claims indefinite because it is unclear whether "a list of machine learning training jobs identifiers" are identifiers for the same or different "list of machine learning training jobs" associated with a "managed object instance". For examination purpose, "… receiving a request from the consumer to read a managed object instance associated with a list of machine learning training jobs; and transmitting identifiers associated with the list of machine learning training jobs to the consumer" or "… receiving a request from the consumer to read a managed object instance associated with a list of machine learning training jobs identifiers; and transmitting the list of machine learning training jobs identifiers to the consumer" according to Claims 11 and 12.
Claim 14 recite the limitation "… transmitting … a request to the machine learning training function to read a managed object instance associated with a list of machine learning training jobs; and receiving … a list of machine learning training jobs identifiers from the machine learning training function" in lines 4-8, which rendering these claims indefinite because it is unclear whether "a list of machine learning training jobs identifiers" are identifiers for the same or different "list of machine learning training jobs" associated with a "managed object instance". For examination purpose, "… transmitting … a request to the machine learning training function to read a managed object instance associated with a list of machine learning training jobs; and receiving … identifiers associated with the list of machine learning training jobs from the machine learning training function" or "… transmitting … a request to the machine learning training function to read a managed object instance associated with a list of machine learning training jobs identifiers; and receiving … the list of machine learning training jobs identifiers from the machine learning training function" according to Claims 15 and 16.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-18 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (US 2022/0108214 A1, filed on 08/13/2021), hereinafter Lee in view of DIRAC et al. (US 2015/0379424 A1, pub. date: 12/31/2015), hereinafter DIRAC.
Independent Claims 1 and 9
Lee discloses a method, comprising: receiving, by a machine learning training function, a request to instantiate a machine learning training job from a consumer; instantiating, by the machine learning training function, the requested machine learning training job (Lee, ¶¶ [0066]-[0102] with FIG. 1: a network data analytics function (NWDAF) device 101 may use a mechanism and interface specified with respect to a 5G core (5GC) network; the NWDAF device 101 may interact with different entities for different purposes: (a) data collection based on event subscriptions provided by an access and mobility management function (AMF), a session management function (SMF), a policy control function (PCF), unified data management (UDM), AF (directly or through NEF) and OAM (Operations, administration and management); (b) analytics and data collection using a data collection coordination function (DCCF); (c) information search from a datastore (for example, UDR through UDM for subscription interest-related information); (d) information storage and search using an analytics data repository function (ADRF); (e) analytics and data collection using a messaging framework adapter function (MF AF); (f) search of information on a network function (NF) (for example, using a network repository function (NRF) for NF-related information): (g) on-demand provision of analytics for a consumer; and (h) mass data provision to a consumer; a single instance or multiple instances of the NWDAF device 101 may be deployed in a PLMN (Public Land Mobile Network); when multiple NWDAF instances are deployed, an architecture may support deploying the NWDAF device 101 as a central NF device 102, a distributed NF collection, or a combination of the two; when the multiple NWDAF instances are deployed, the NWDAF device 101 may serve as an aggregation point (that is, an NWDAF device that performs aggregation of analytics, aggregation of a machine learning (ML) model of an untrained initial model or aggregation of a trained model); in addition, the NWDAF device 101 may generate aggregate analytics (per Analytic ID) by collecting analytics information from another NWDAF device capable of having another service area, or may perform federation learning or aggregation training (per Analytic ID) by training an ML model on each of the NWDAF devices; an instance of the NWDAF device 101 may be deployed with a 5GC NF; the NWDAF device 101 may provide analytics for the 5GC NF and the OAM, which may be disassembled as follows: (i) Analytics Logical Function (AnLF): (a) the NWDAF device 101 that performs an AnLF may perform inference and derive analytics information (that is, derive a statistic and/or prediction in response to an analytics consumer request); and (b) in addition, the NWDAF device 101 that performs the AnLF may expose an analytics service for network data (for example, Nnwdaf_AnalyticsSubscription or Nnwdaf_Analyticsinfo); and (ii) Model Training Logical Function (MTLF): (a) the NWDAF device 101 that performs an MTLF may train an ML model and expose a new training service (for example, provision of an untrained initial version of model or a trained model); the NWDAF device 101 may include each of the MTLF and the AnLF, or may support both functions; the NWDAF device 101 that performs the AnLF may set an ID and an Analytic ID of an NWDAF device that performs the MTLF so as to search for a trained ML model; the NWDAF device 101 that performs the AnLF may search for the NWDAF device that performs the MTLF using an MTLF ID; the NWDAF device 101 that performs the MTLF may discover and select an NWDAF device that supports the MTLF for federation learning; the NWDAF device 101 may perform local training and global training in federation learning; the NWDAF device 101 may generate a new model without input data related to the abnormal UE list during the observed time window and/or generate an analytics result for network data, and then may transmit the new model or the network data to the subscribed NWDAF device 101 or update the new model or the network data; the NF device 102 may include any one of the MTLF and the AnLF capable of providing a service operation (e.g., an analytic exposure operation, an ML model provisioning operation, or an ML model training operation) required for a type of analytics required, or an instance of each NWDAF device 101 may provide the following information so as to assist in searching for and selecting an instance of the NWDAF device 101 including both the MTLF and the AnLF; the NF device 102 that needs to search for an NWDAF instance that provides support for some specific service operations for a specific type of analytics may query the NRF with respect to the NWDAF device 101 that supports a required service operation and a required Analytic ID; the NWDAF device 101 that performs the MTLF may register an ML model provisioning service and a training service (that is, Nnwdaf_MLModelProvision, Nnwdaf_MLModelInfo, Nnwdaf_MLModelUpdate, Nnwdaf_MLModelTraining, and Nnwdaf_MLModelTraininginfo) when an ML model is providable and trainable with respect to the Analytic ID; the 5GC NF and the OAM, which are consumers, may determine how to use data analytics provided by the NWDAF device 101; interaction between the 5GC NF(s) and the NWDAF device 101 may occur within the PLMN; ¶¶ [0103]-[0109] with FIG. 2: allow an NWDAF device 201 including an AnLF to use a provisioning service operation and a training service operation with respect to an ML model of an untrained initial model or a trained ML model in another NWDAF device 202 that supports an MTLF; the AnLF and the MTLF may be defined as follows: (i) AnLF: the NWDAF device 101 including the AnLF may perform inference, derive analytics information (i.e., derive a statistic and/or prediction in response to an analytics consumer request), and expose an analytics service (e.g., Nwdaf_AnalyticsSubscription or Nnwdaf_Analyticsinfo); (ii) MTLF: the NWDAF device 101 including the MTLF may train an ML model and expose a new training service (e.g., trained model provision and model training); the AnLF may support Nnwdaf_Analyticsinfo (data analytics information) or Nnwdaf_AnalyticsSubscription (analytics subscription) service; the MTLF may support services such as Nnwdaf_MLModelProvision (ML model provisioning), Nnwdaf_MLModelInfo (ML model information request), Nnwdaf_MLModelUpdate (ML model update), Nnwdaf_MLModelTraining (ML model training), and Nnwdaf_MLModelTraininginfo (ML model training information); an Nnwdaf interface may be used to request and subscribe to a provisioning service for an untrained initial version of ML model or a trained ML model in an NWDAF; in addition, the Nnwdaf interface may be used by the NWDAF device 101 that supports the MTLF to request and subscribe to an ML model training service for model learning and cooperative learning; ¶¶ [0110]-[0113] with FIG. 3: an operation of an NWDAF device that performs an MTLF is described in FIG. 3(a); the NWDAF device may receive an initial version of ML model from a model provisioning server (operator), a model provisioning server (3rd party), or another NWDAF device that performs the MTLF; then, after training the initial version of ML model, the NWDAF device may provide the trained ML model to an NWDAF device that performs an AnLF or MTLF through an Nnwdaf_MLModelProvision service (model provisioning service) or Nnwdaf_MLModelInfo service (model information service); in addition, for update of the ML model, the NWDAF device may use an Nnwdaf_MLModelUpdate, Nnwdaf_MLModelTraining, or Nnwdaf_MLModelTraininginfo service; the NWDAF device illustrated in FIG. 3(b) may collect data from a DCCF device and a data source (NF device or ADRF device), and receive an ML model from an NWDAF device that performs an MTLF; then, the NWDAF device that performs the AnLF may analyze the collected data using the ML model; a data analytics result may be provided to a consumer NF device in a statistical or predictive manner; ¶¶ [0114]-[0156] with FIG. 4: an NWDAF device 401 (service consumer) may use an NWDAF search principle to select an NWDAF device that supports requested analytics information, required analytics function, and/or requested ML model information; in order to search for an NWDAF device that supports an AnLF or an NWDAF device that supports an MTLF using an NRF, the following conditions may need to be satisfied: when (i) an ML model to be provided/trained is related to UE(s) and (ii) an NWDAF service consumer (other than an NWDAF) is not capable of providing an area of interest for the requested ML model to be provided/trained, the NWDAF device 401 may select an NWDAF device 403 having a large service area from candidate NWDAF devices 403; e.g., in response to discovery, when the NWDAF device 401 receives NWDAF device(s) 403 having an aggregation capability (e.g., an ML model aggregation capability and an ML model update capability), the NWDAF device 401 may preferably select the NWDAF device 403 having an aggregation capability (for example, an ML model aggregation capability and an ML model update capability) with a large serving area; when the NWDAF device 401 is not capable of providing an ML model to be provided/trained for requested UE(s) (e.g., an NWDAF providing another service area), the NWDAF device 401 may reject a subscription or request for provision of the ML model to be provided/trained, or determine an AMF that serves an UE as specified; in order to request UE location information from the AMF and discover another target NWDAF that serves a region where the UE(s) is located, the NWDAF device 401 may query the NRF device 402 with a tracking area where the UE is located; during discovery of the NWDAF device 403 that supports the MTLF, the NRF device 402 may return instances of one or more candidate NWDAF devices 403 to an NF consumer, and an instance of each candidate NWDAF device 403 may include analytics filter information on an ML model trained for each Analytic ID; a selection function of the NWDAF device 403 that supports an MTLF of the NF consumer may select an NWDAF instance based an instance of the NWDAF device 403 that supports an available MTLF; for NWDAF selection, the NF consumer may consider at least one of the following items: a supportable service for each Analytic ID (e.g., an ML model provision/training service) or NWDAF service region information, that is, a list of TAIs to which an NWDAF may provide analytics, ML model provision, ML model training, and/or data; when selecting the NWDAF device 103 that supports an MTLF for ML model provisioning and model training, the NWDAF device 401 may consider the following additional factors: (a) An analytics filter for ML model trained for each Analytic ID; and (b) an ML model aggregation capability; ¶¶ [0135] and [0157]-[0209] with operation 1 in FIG. 5: the NWDAF device 501 that performs an AnLF, which is a service consumer, may subscribe or unsubscribe to the NWDAF device 502 that performs an MTLF; the NWDAF device 501 may simultaneously be a consumer of services provided by other NWDAF(s) and a provider of services to other NWDAF device(s) 502; in operation 1, the NWDAF device 501, which is a service consumer, may invoke a subscription service operation for provisioning of an ML model (Nnwdaf_MLModelProvision_Subscribe) or an unsubscription service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Unsubscribe) to subscribe, modify, or unsubscribe an ML model trained in the NWDAF device 502 that supports an MTLF connected to an Analytic ID; a parameter used by the NWDAF device 501 may include at least one of (i) an Analytic ID, (ii) S-NSSAI (Subscribed Network Slice Selection Assistance Information), (iii) a target area of interest, (iv) an application ID, (v) a target UE, (vi) an ML model target period, (vii) an expiry time, and (viii) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version; when a subscription to the trained ML model connected to the Analytic ID is received, the NWDAF device 502 including the MTLF may perform the following process: determine (i) whether an existing ML model is available for subscription, or (ii) whether to trigger additional training for the existing ML model with respect to subscription; the NWDAF device 502 that performs the MTLF may determine that additional training is required for an already subscribed ML model; when the NWDAF device 502 that performs the MTLF determines that additional training is required for the already subscribed ML model, the NWDAF device 502 may collect data required for training of the ML model from an NF device, DCCF device, or OAM device; ¶¶ [0210]-[0216] with operation 1 in FIG. 6: an NWDAF service consumer, that is, an NWDAF device 601 may request an NWDAF device 602 including MTLF ML model information using an Nnwdaf_MLModelInfo service; the NWDAF device 601 (e.g., NWDAF(MTLF+AnLF)) may simultaneously be a consumer of a service provided by another NWDAF device 602 and a provider of this service to other NWDAF(s); in operation 1, the NWDAF device 601 that supports an AnLF may invoke an ML model information request service operation (Nnwdaf_MLMoldelInfo_Request) to request ML model(s) connected to an Analytic ID from the NWDAF device 602 that supports an MTLF; a parameter used when the NWDAF device 601, which is an NWDAF service consumer, invokes an information request service operation, may include at least one of an Analytic ID, S-NSSAI, a target area of interest, an application ID, a target UE, an ML model target period, and an expiry time; when a request for ML model information for Analytics is received, the NWDAF device 602 that performs the MTLF may perform the following process: (i) determine whether an existing trained ML model is available for the request, or (ii) determine whether an additional training trigger for the existing trained ML model is required for the request; when the NWDAF device 602 that performs the MTLF determines that additional training is required for an already requested ML model, the NWDAF may start collecting data from an NF device, DCCF device, or OAM device required for ML model training; ¶¶ [0218]-[0222] with operation 3 in FIG. 7: in operation 3, the NWDAF device 701 may invoke, from the NWDAF device 702, a subscription service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Subscribe); at this time, a subscription ID included in the subscription service operation may be the same as a subscription ID used in operation 1; i.e., when invoking the subscription service operation of operation 3 again, the NWDAF device 701 may include a parameter same as that included when previously invoking the subscription service operation for provisioning of the ML model to request a new ML model different from a previous one or re-request an ML model previously provided through a subscription or request process from the NWDAF device 702; at this time, the NWDAF device 701 may incorporate an alternative ML model flag into the subscription service operation for provisioning of the ML model in operation 3 to request a new ML model different from a previous one or re-request a previously provided ML model from the NWDAF device 702; ¶¶ [0224]-[0247] with FIG. 8: the NWADF device 801 may perform local training in federation learning, and the NWDAF device 803 may perform global training in federation learning; the NWDAF device 801 (service consumer) may search for and select the NWDAF device 803 that provides requested analytics information, a required analytics function and/or a requested ML model, and supports ML model training; when model training is related to NF(s)/UE(s) and an NWDAF service consumer (NWDAF device 803) is not capable of providing an area of interest for requested model training, the NWDAF device 801 may select the NWDAF device 803 having a large service area from candidate NWDAF devices 803; e.g., in response to discovery, when the NWDAF device 801 receives the NWDAF device(s) 803 having an ML model update capability, the NWDAF device 801 may preferably select the NWDAF device 803 having a large serving area and an ML model update capability; in order to search for an NWDAF registered in a UDM with respect to a given UE, the NWDAF device 801 or other NWDAFs interested in providing and training UE-related data or an ML model may make a query to a UDM device to search for an instance of the NWDAF device 803 that is already providing a service to the UE; in order to search for the NWDAF device 803 that performs the MTLF, the NWDAF device 801 that performs the MTLF may include at least one of analytics filter information, a trainable and providable ML model ID, an ML model version, and an ML model aggregation capability with respect to an ML model that is trained per Analytic ID in response to a registration request for the NRF device 802; in operation 1, the NWDAF device 803 may invoke, from the NRF device 802, a registration service operation (Nnrf_NFManagement_NFRegister request) for the NWDAF device 803; in operation 2, the NRF device 802 may store a profile of an NF device of the NWDAF device 803; in operation 4, the NWDAF device 801 may invoke, from the NRF device 802, a request service operation (Nnrf_NFDiscovery_Request) for searching for the NWDAF device 803; in operation 7, the NWDAF device 801 may select an NWDAF device that performs an MTLF; ¶¶ [0248]-[00271] with FIG. 9: the NWDAF device 901 may perform local training in federation learning, and the NWDAF device 903 may perform global training in federation learning; the NWDAF device 903 (service consumer) may use an NWDAF search principle to select the NWDAF device 901 that supports requested analytics information, required analytics function, and/or requested ML model training; the NWDAF device 903 may support at least one of an ML model training service (Nnwdaf_MLModelTraining) or an ML model training information service (Nnwdaf_MLModelTraininginfo) to search for the NWDAF device 901; when an ML model to be trained is related to NF(s)/UE(s) and an NWDAF service consumer (other than an NWDAF) is not capable of providing an area of interest for requested ML model training, the NWDAF device 903 may select an NWDAF having a large service area from candidate NWDAFs; e.g., in response to discovery, when the NWDAF device 903 receives NWDAF(s) 901 having an ML model aggregation/update capability, the NWDAF device 903 may preferably select the NWDAF device 901 having an ML model aggregation/update capability for a large serving area; in order to search for the NWDAF device 901 that supports the MTLF, the NWDAF device 901 that supports the MTLF may include at least one of analytics filter information and a service providable for model training (i.e., an Nnwdaf_MLModelTraining service or an Nnwdaf_MLMode1Traininginfo service) with respect to an ML model that is trainable per Analytic ID in response to a registration request for the NRF device 902; in operation 1, the NWDAF device 901 may invoke, from the NRF device 902, a registration service operation (Nnrf_NFManagement_NFRegister request) for the NWDAF device 901; at this time, the registration service operation may include at least one of a list of supported Analytic IDs, per supported service (e.g., an Nnwdaf_MLModelTraining service or an Nnwdaf_MLModelTraininginfo service), a serving area where an ML model is provided, S-NSSAI, ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and a federation learning capability (ML model training capability); in operation 2, the NRF device 902 may store a profile of an NF device of the NWDAF device 901; in operation 4, the NWDAF device 903 may invoke, from the NRF device 902, a request service operation (Nnrf_NFDiscovery_Request) for searching for the NWDAF device 901; at this time, the request service operation may include at least one of an Analytic ID, per supported service (e.g., an Nnwdaf_MLModelTraining service or an Nnwdaf_MLModelTraininginfo service), a serving area where an ML model is provided, S-NSSAI, ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and a federation learning capability (for example, an ML model training capability); in operation 7, the NWDAF device 903 may select at least one NWDAF device 901 capable of learning a local ML model that performs the MTLF; ¶¶ [0272]-[0289] with operation 1 in FIG. 10: the NWDAF device 1001 may perform local training in federation learning, and the NWDAF device 1002 may perform global training in federation learning; the NWDAF device 1001 may be configured locally with ID(s) and Analytic ID(s) of an NWDAF device that performs an MTLF to search for an ML model of an untrained initial model or a trained ML model; the NWDAF device 1001 that performs an MTLF, which is a service consumer, may be used to subscribe or unsubscribe to the NWDAF device 1002 that performs the MTLF; the NWDAF device 1001 may simultaneously be a consumer of this service provided by other NWDAF(s) and a provider of this service to other NWDAF device(s) 1002; in operation 1, the NWDAF device 1001, which is a service consumer, may invoke a subscription service operation for provisioning of an ML model (Nnwdaf_MLModelProvision_Subscribe) or an unsubscription service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Unsubscribe) to subscribe, modify, or unsubscribe an ML model of an untrained initial model or a trained ML model connected to an Analytic ID; when a subscription to the ML model of the untrained initial model or the trained ML model connected to the Analytic ID is received, the NWDAF device 1002 including an MTLF may perform the following process: determine (i) whether an existing trained ML model is available for subscription or (ii) whether to trigger additional training for the existing trained ML model with respect to subscription; the NWDAF device 1002 that performs the MTLF may determine that additional training is required for the existing ML model; when the NWDAF device 1002 determines that additional training is required, the NWDAF device 1002 may collect data required for training of the ML model from an NF device, DCCF device, or OAM device; ¶¶ [0290]-[0298] with operation 1 in FIG. 6: the NWDAF device 1101 may perform local training in federation learning, and the NWDAF device 1102 may perform global training in federation learning; an NWDAF service consumer, that is, an NWDAF device 1101 may request an NWDAF device 1102 including ML model information using an Nnwdaf_MLModelInfo service operation; the NWDAF device 1101 (e.g., NWDAF(MTLF+AnLF)) may simultaneously be a consumer of a service provided by another NWDAF device 1102 and a provider of this service to other NWDAF(s); in operation 1, the NWDAF device 1101 may invoke an ML model information request service operation (Nnwdaf_MLMoldelInfo_Request) to request ML model(s) connected to an Analytic ID; when a request for ML model information for analytics is received, the NWDAF device 1102 that performs the MTLF may perform the following process: (i) determine whether an existing trained ML model is available for the request, or (ii) determine whether an additional training trigger for the existing trained ML model is required for the request; when the NWDAF device 1102 that performs the MTLF determines that additional training is required, this NWDAF may start collecting data from an NF device, DCCF device, or OAM device required for ML model training; ¶¶ [0299]-[0305] with FIGS. 10 and 12: operations 1 and 2 may be the same as operation 1 and 2 described with reference to FIG. 10; in operation 3, the NWDAF device 1201 may locally train the ML model; in operation 4, the NWDAF device 1201 may invoke an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) from the NWDAF device 1202 that performs a global update; in operation 5, the NWDAF device 1202 may globally update the ML model; here, globally updating the ML model may mean aggregating, by each of a plurality of NWDAF devices 1202, the locally trained ML model, and then changing the ML model by reflecting a gradient of the ML model expressed as a polynomial; ¶¶ [0306]-[0310] with FIGS. 11 and 13: operations 1 and 2 may be the same as operations 1 and 2 described with reference to FIG. 11; in operation 3, an NWDAF device 1301 may locally train the ML model; in operation 4, the NWDAF device 1301 may invoke an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) from an NWDAF device 1302 that performs a global update; in operation 5, the NWDAF device 1302 may globally update the ML model; ¶¶ [0311]-[0317] with FIGS.9 and 14: the NWDAF device 1401 may perform local training in federation learning, and the NWDAF device 1402 may perform global training in federation learning; in operation 1 of FIG. 14, the NWDAF device 1401, the NRF device 1402, and the NWDAF device 1403 may be the same as the discovery process of the NWDAF device 903 that performs the MTLF previously described with reference to FIG. 9; in operation 2, the NWDAF device 1403 may invoke, from the NWDAF device 1401, an ML model training subscription service operation (Nnwdaf_MLModelTraining_Subscribe); the training subscription service operation may include at least one of (i) an Analytic ID, (ii) S-NSSAI, (iii) a target area of interest, (iv) an application ID, (v) a target UE, (vi) an ML model target period, (vii) an expiry time, and (viii) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version; in addition, the training subscription service operation may further include at least one of a description of a requested parameter for ML model update and a description of a budget for an update reporting time (e.g., a target reporting time); in operation 3, the NWDAF device 1401 may locally train an ML model; in operation 4, the NWDAF device 1401 may invoke an ML model training notification service operation (Nnwdaf_MLModelTraining_Notify) from the NWDAF device 1402 that performs a global update; in operation 5, the NWDAF device 1402 may globally update the ML model; ¶¶ [0318]-[0324] with FIGS. 9 and 15: the NWDAF device 1501 may perform local training in federation learning, and the NWDAF device 1502 may perform global training in federation learning; in operation 1 of FIG. 15, the NWDAF device 1501, the NRF device 1502, and the NWDAF device 1503 may be the same as the discovery process of the NWDAF device 903 that performs the MTLF previously described with reference to FIG. 9; in operation 2, an NWDAF device 1503 may invoke, from the NWDAF device 1502, an ML model training request service operation (Nnwdaf_MLModelTraininginfo_request); at this time, the training request service operation may include at least one of (i) an Analytic ID, (ii) S-NSSAI, (iii) a target area of interest, (iv) an application ID, (v) a target UE, (vi) an ML model target period, (vii) an expiry time, and (viii) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version.; in addition, the training request service operation may further include at least one of a description of a requested parameter for ML model update and a description of a budget for an update reporting time (e.g., a top-k gradient, a threshold for sparsification of gradient, and the like); in operation 3, the NWDAF device 1501 may locally train an ML model; in operation 4, the NWDAF device 1501 may invoke an ML model training request response service operation (Nnwdaf_MLModelTraininginfo_request_response) from an NWDAF device 1502 that performs global training; in operation 5, the NWDAF device 1502 may globally update the ML model; ¶¶ [0325]-[0345] with Table 1: an Nnwdaf_MLModelTraining service operation or an Nnwdaf_MLModelTraininginfo service operation may need to include the following input and output: (i) Input: an Analytic ID, an expiry time, and ML model information (an ML model file, a model ID, and a model version); and (ii) Output: an Analytic ID, a requested parameter for ML model update (for example, a gradient), a time stamp, an ID and a version of an ML model targeted for training, and a training area (e.g., a list of TAs targeted for training, and the like); ¶¶ [0346]-[0359] with FIGS. 16-17: in operation 1, an NWDAF device 1 1601, which is an ML model provider, may register a function of providing an untrained initial version of model or a trained model (that is, an "MLModelProvision service operation" with a list of supported Analytic IDs) with an NRF device 1603 as part of a profile; in operation 2, an NRF device 1603 may store an NWDAF profile of the NWDAF device 1 1601; in operation 4, an NWDAF device 2 1602 may invoke, from the NRF device 1603, a discovery request service operation including a service parameter list (e.g., Analytic ID, and the like) so as to search for the NWDAF device 11601 that provides an "MLModelProvision service"; in operation 6, the consumer NWDAF device 2 1602 may invoke, from the NWDAF device 1 1601, a request service operation or subscription service operation of the "MLModelProvision service" using an instance of the searched provider NWDAF device 1 1601; in operation 7, the NWDAF device 1 1601 may invoke, from the NWDAF device 2 1602, a request response service operation or subscription notification service operation including a model parameter for an untrained initial version of model or a trained model; in operation 8, when the NWDAF device 2 1602 is capable of training an ML model, the NWDAF device 2 1602 may locally train the model and model parameter; in operation 9, the NWADF device 2 1602 may locally evaluate the ML model after training the ML model; in operation 10, when a subscription to the ML model is performed, the NWDAF device 2 1602 may invoke, from the NWDAF device 1 1601, an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) to transmit information on the locally trained ML model; in operation 11, the NWDAF device 1 1601 may aggregate the trained ML model transmitted from the NWDAF device 2 1602 to update the ML model based on a globally trained ML model; in operation 11, the NWDAF device 1 1601 may evaluate the ML model); and
transmitting, by the machine learning training function, a notification to the consumer indicating that the machine learning training job has been completed (Lee, ¶¶ [0088] and [0090] with FIG. 1; the NWDAF device 101 may generate a new model without input data related to the abnormal UE list during the observed time window and/or generate an analytics result for network data, and then may transmit the new model or the network data to the subscribed NWDAF device 101 or update the new model or the network data; ¶¶ [0162]-[0209] with FIG. 5: a notification may be received from the NWDAF device 502 that performs the MTLF using an ML model provisioning service (Nnwdaf_MLModelProvision); ML model information received through the notification may be used to output analytics from the NWDAF device 501 that performs the AnLF; the provisioning service for the ML model may be used to modify an existing ML model subscription in the NWDAF device 501; the NWDAF device 501 may simultaneously be a consumer of services provided by other NWDAF(s) and a provider of services to other NWDAF device(s) 502; in operation 2, when the NWDAF device 501 subscribes to the trained ML model(s) connected to the Analytic ID(s), the NWDAF device 502 that performs the MTLF may invoke a notification service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Notify) to transmit trained ML model information (e.g., a file address of the trained ML model) to the NWDAF device 501; the NWDAF device 502 that performs the MTLF may invoke an Nnwdaf_MLModelProvision_Notify service operation to notify an available retrained ML model when the NWDAF device 502 determines that retraining is required for a previously provided trained ML model; when a process of operation 1 is performed for subscription modification (i.e., including a subscription correlation ID), the NWDAF device 502 that performs the MTLF may invoke the notification service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Notify) to provide a new learned ML model different from that previously provided or provide a relearned ML model; <Nnwdaf_MLModelProvision Service – ML Model Provisioning Service> a service description: this service may allow a consumer to be notified when an ML model corresponding to a subscription parameter becomes available; an NWDAF may notify ML model information to a consumer instance subscribed to a specific NWDAF service; ¶ [0217] with operation 2 in FIG. 6: in operation 2, invoke an ML model information request response service operation (Nnwdaf_MLModelInfo_Request_response) to respond to the NWDAF device 601 (service consumer) with ML model information (including an ML model file address); the NWDAF device 103 that performs the MTLF may invoke an ML model information request response service operation including at least one of (i) ML model information, (ii) a validity period, and (iii) a spatial validity; at this time, the ML model information may include at least one of an ML model file address, an ML model file, a model ID, and a model version; ¶¶ [0223] with operation 4 in FIG. 7: in operation 4, the NWDAF device 702 may invoke, from an NWDAF device, a notification service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Notify); at this time, the notification service operation may include at least one of ML model information different from the ML model provided in operation 1 (e.g., including at least one of an ML model file, an ML model file address, a model version, or a model ID), a validity period, and a spatial validity; ¶¶ [0240]-[0247] with FIG. 8: during discovery of the NWDAF device 803 that performs the MTLF, the NRF device 802 may return instances of one or more candidate NWDAF devices 803 to an NF consumer, and an instance of each candidate NWDAF device 803 may include analytics filter information on an ML model of an initial model that is untrained or an ML Model that is trained for each Analytic ID; in operation 3, the NRF device 802 may invoke, from the NWDAF device 803, a registration response service operation (Nnrf_NFManagement_NFRegister_response); in operation 5, the NRF device 802 may invoke, from the NWDAF device 801, a discovery request response service operation (Nnrf_NFDiscovery_Request_response); here, the response service operation may include a list and an address of an instance ID of the NWDAF device 803; ¶¶ [0264]-[0271] with FIG. 9: during discovery of the NWDAF device 901 that supports the MTLF, the NRF device 902 may return instances of one or more candidate NWDAF devices 901 to an NF consumer, and an instance of each candidate NWDAF device 901 may include analytics filter information for an ML model that is trainable for each Analytic ID; in operation 3, the NRF device 902 may invoke an Nnrf_NFManagement_NFRegister response from the NWDAF device 901; in operation 5, the NRF device 902 may invoke, from the NWDAF device 903, a response service operation (Nnrf_NFDiscovery_Request_response); here, the response service operation may include a list and an address of an instance ID of the NWDAF device 901; ¶¶ [0272]-[0289] with operation 2 in FIG. 10: the NWDAF device 1001 may receive a notification from the NWDAF device 1002 that performs the MTLF using an ML model provisioning service (Nnwdaf_MLModelProvision) with respect to ML model information in related analytics; the ML model information received through the notification may be used by the NWDAF device 1001 to train the ML model; the provisioning service may be also used by the NWDAF device 1001 to modify an existing ML model subscription; the NWDAF device 1001 may simultaneously be a consumer of this service provided by other NWDAF(s) and a provider of this service to other NWDAF device(s) 1002; in operation 2, when the NWDAF device 1001 subscribes to the ML model of the untrained initial model or the trained ML model(s) connected to the Analytic ID(s), the NWDAF device 1002 may invoke an Nnwdaf_MLModelProvision_Notify service operation including information on the ML model of the untrained initial model or information on the trained ML model to transmit a file address of the ML model of the untrained initial model or the trained ML model; the NWDAF device 1002 that performs the MTLF may invoke a notification service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Notify) to notify an available retrained ML model when the NWDAF device 1002 determines that retraining is required for an ML model of a previously provided untrained initial model or a trained ML model; when a process of operation 1 is performed for subscription modification (i.e., including a subscription correlation ID), the NWDAF device 1002 that performs the MTLF may provide a new learned ML model different from that previously provided, or may provide a relearned ML model by invoking the Nnwdaf_MLModelProvision_Notify service operation; ¶ [0298] with operation 2 in FIG. 11: in operation 2, the NWDAF device 1102 that performs the MTLF may invoke an ML model information request response service operation (Nnwdaf_MLModelInfo_Request_response) for an ML information request service operation (Nnwdaf_MLMode1Info_Request) to respond to the NWDAF device 1101 (service consumer), including at least one of ML model information, a validity period, and a spatial validity. ML model information that may be provided by an NWDAF that performs the MTLF may include at least one of an ML model file address, an ML model file, a model ID, and a model version; ¶¶ [0299]-[0305] with FIG. 12: in operation 4, the NWDAF device 1201 may invoke an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) from the NWDAF device 1202 that performs a global update; in operation 6, the NWDAF device 1202 may invoke, from the NWDAF device 1201, a notification service operation for provisioning of the ML model; at this time, the notification service operation may include an updated model; ¶¶ [0311]-[0317] with FIG. 14: in operation 4, the NWDAF device 1401 may invoke an ML model training notification service operation (Nnwdaf_MLModelTraining_Notify) from the NWDAF device 1402 that performs a global update; the training notification service operation may include at least one of (i) an Analytic ID, (ii) a requested parameter for ML model update (e.g., a gradient), (iii) a time stamp, (iv) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and (v) a training area (for example, a list of TAs targeted for training, and the like); ¶¶ [0318]-[0324] with FIGS. 9 and 15: in operation 4, the NWDAF device 1501 may invoke an ML model training request response service operation (Nnwdaf_MLModelTraininginfo_request_response) from an NWDAF device 1502 that performs global training; the training response service operation may include at least one of an Analytic ID, a requested parameter for ML model update (e.g., a gradient), a time stamp, a training area, an ML model ID, and an ML model version; ¶¶ [0325]-[0345] with Table 1: the Nnwdaf_MLModelUpdate_Notify service operation may need to include the following input and output: (i) Input: an Analytic ID, a model update request parameter (e.g., a gradient), an update time stamp, a target model ID, and an update version; and (ii) Output: display of success or failure; when the ML model provider NWDAF provides a model to a consumer NWDAF by invoking Nnwdaf_ModelProvision_Notify, the provider NWDAF may (implicitly) subscribe to an Nnwdaf_ModelUpdate service operation of the consumer NWDAF to obtain a result of a locally updated model parameter when the consumer NWDAF is capable of training the model; ¶¶ [0346]-[0359] with FIGS. 16-17: in operation 3, the NRF device 1603 may invoke, from the NWDAF device 1 1601, a registration response service operation; in operation 5, the NRF device 1603 may invoke, from the NWDAF device 2 1602, a discovery request response service operation including an instance of the NWDAF device 11601 that provides the "MLModelProvision service"; in operation 7, the NWDAF device 1 1601 may invoke, from the NWDAF device 2 1602, a request response service operation or subscription notification service operation including a model parameter for an untrained initial version of model or a trained model; in operation 10, when a subscription to the ML model is performed, the NWDAF device 2 1602 may invoke, from the NWDAF device 1 1601, an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) to transmit information on the locally trained ML model; in operation 13, the NWDAF device 1 1601 may transmit the updated ML model to the NWDAF device 2 1602 through a notification service operation for provisioning of the ML model);
Lee further discloses an apparatus (Lee, ¶ [0379] with FIG. 1: NWDAF device 101; ¶ [0382]: data processing apparatus), comprising: at least one processor (Lee, ¶ [0380]: at least one digital signal processor (DSP), a processor, a controller, an application-specific integrated circuit (ASIC), a programmable logic element, such as a field programmable gate array (FPGA), other electronic devices, or combinations thereof; ¶ [0383]: processors suitable for processing of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer); and at least one memory including computer program code (Lee, ¶ [0383]: one or more memory devices for storing instructions and data; one or more mass storage devices for storing data), wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform a method described above (Lee, ¶¶ [0382]-[0383]: various techniques described herein may be implemented in digital electronic circuitry, computer hardware, firmware, software, or combinations thereof; a computer program may be deployed to be processed on one computer or multiple computers at one site or distributed across multiple sites and interconnected by a communication network; at least one processor for executing instructions).
Lee fails to explicitly disclose transmitting a notification to the consumer indicating that the machine learning training job has been instantiated.
DIRAC teaches a system and a method for providing machine learning service (DIRAC, ¶ [0024]), wherein transmitting a notification to the consumer indicating that the machine learning training job has been instantiated (DIRAC, ¶¶ [0024]-[0031]: a customizable, easy-to-use machine learning service (MLS) designed to support large numbers of users and a wide variety of algorithms and problem sizes; allow non-experts to rely on default settings or parameters for various aspects of the procedures used for building, training and using machine learning models, where the defaults are derived from the accumulated experience of other practitioners addressing similar types of machine learning problems; at the same time, expert users may customize the parameters or settings they wish to use for various types of machine learning tasks, such as input record handling, feature processing, model building, execution and evaluation; in addition to or instead of using predefined libraries implementing various types of machine learning tasks, MLS clients may be able to extend the built-in capabilities of the service, e.g., by registering their own customized functions with the service; the term "MLS control plane" may be used herein to refer to a collection of hardware and/or software entities that are responsible for implementing various types of machine learning functionality on behalf of clients of the MLS; the term "MLS data plane" may refer to the pathways and resources used for the processing, transfer, and storage of the input data used for client-requested operations, as well as the processing, transfer and storage of output data produced as a result of client-requested operations; a number of different types of entities related to machine learning tasks may be generated, modified, read, executed, and/or queried/searched via MLS programmatic interfaces; the MLS programmatic interfaces may enable users to submit respective requests for several related tasks of a given machine learning workflow, such as tasks for extracting records from data sources, generating statistics on the records, feature processing, model training, prediction, and so on; a queue of job objects may be used for storing internal representations of requested tasks; the term "task", as used herein, refers to a set of logical operations corresponding to a given request from a client, while the term "job" refers to the internal representation of a task within the MLS; the MLS may be responsible for ensuring that the dependencies of a given job have been met before the corresponding operations are initiated; the MLS may also be responsible for generating a processing plan for each job, identifying the appropriate set of resources (e.g., CPUs/cores, storage or memory) for the plan, scheduling the execution of the plan, gathering results, providing/saving the results in an appropriate destination, and at least in some cases for providing status updates or responses to the requesting clients; not all client API requests may be implemented using jobs – e.g., a relatively short or lightweight task may be performed synchronously with respect to the corresponding request, without incurring the overhead of job creation and asynchronous job scheduling; the APIs implemented by the MLS may allow clients to submit requests to create, query the attributes of, read, update/modify, search, or delete an instance of at least some of the various entity types supported; e.g., for the entity type "DataSource", respective APIs similar to "createDataSource", "describeDataSource" (to obtain the values of attributes of the data source), "updateDataSource", "searchForDataSource", and "deleteDataSource" may be supported by the MLS; a similar set of APIs may be supported for recipes, models, and so on; some entity types may also have APIs for executing or running the entities, such as "executeModel" or "executeRecipe"; some machine learning models may be created and trained, e.g., by a group of model developers or data scientists using the MLS APIs, and then published for use by another community of users; an alias may comprise an immutable name and a pointer to a model (i.e., an internal identifier generated for the model by the MLS) that has already been created and stored in an MLS artifact repository; members of a business analyst group may be allowed to run the model using its alias name, but may not be allowed to change the pointer, while model developers may be allowed to modify the pointer and/or modify the underlying model; the model developers may continue to experiment with various algorithms, parameters and/or input data sets to obtain improved versions of the underlying model, and may be able to change the pointer to point to an enhanced version to improve the quality of predictions obtained by the business analysts; as part of developing the processing plan for a job, the MLS may select a workload distribution strategy for the job; after the workload strategy is selected, the actual set of resources to be used may be identified in accordance with the strategy, and the job's operations may be scheduled on the identified resources; a pool of compute servers and/or storage servers may be pre-configured for the MLS, and the resources for a given job may be selected from such a pool; ¶¶ [0032]-[0036] with FIG. 1: the MLS may implement a set of programmatic interfaces 161 (e.g., APIs, command-line tools, web pages, or standalone GUIs) that can be used by clients 164 (e.g., hardware or software entities owned by or assigned to customers of the MLS) to submit requests 111 for a variety of machine learning tasks or operations; the administrative or control plane portion of the MLS may include MLS request handler 180, which accepts the client requests 111 and inserts corresponding job objects into MLS job queue 142, as indicated by arrow 112; i.e., job objects (e.g., model training job objects) corresponding to client requests 111 are instantiated into MLS job queue 142 in response to requests 111 submitted by clients 164; the control plane of the MLS may comprise a plurality of components (including the request handler, workload distribution strategy selectors, one or more job schedulers, metrics collectors, and modules that act as interfaces with other services) which may also be referred to collectively as the MLS manager; each job object may indicate one or more operations that are to be performed as a result of the invocation of a programmatic interface 161, and the scheduling of a given job may in some cases depend upon the successful completion of at least a subset of the operations of an earlier-generated job; job queue 142 may be managed as a first-in-first-out (FIFO) queue, with the further constraint that the dependency requirements of a given job must have been met in order for that job to be removed from the queue; asynchronously with respect to the submission of the requests 111, the next job whose dependency requirements have been met may be removed from job queue 142 as indicated by arrow 113, and a processing plan comprising a workload distribution strategy may be identified for it; the workload distribution strategy layer 175 may determine the manner in which the lower level operations of the job are to be distributed among one or more compute servers (e.g., servers selected from pool 185), and/or the manner in which the data analyzed or manipulated for the job is to be distributed among one or more storage devices or servers; results of some jobs may be stored as MLS artifacts within repository 120 as indicated by arrow 142; some relatively simple types of client requests 111 may result in the immediate generation, retrieval, storage, or modification of corresponding artifacts within MLS artifact repository 120 by the MLS request handler 180 (as indicated by arrow 141); thus, the insertion of a job object in job queue 142 may not be required for all types of client requests; clients 164maybeabletoview at least a subset of the artifacts stored in repository 120, e.g., by issuing read requests 118 via programmatic interfaces 161; a client request 111 may indicate one or more parameters that may be used by the MLS to perform the operations, such as a data source definition 150, a feature processing transformation recipe 152, or parameters 154 to be used for a particular machine learning algorithm; the output 116 of the feature processing transformations may in tum be used as input for a selected machine learning algorithm 166, which may be executed in accordance with algorithm parameters 154 using yet another set of resources from pool 185; a wide variety of machine learning algorithms may be supported natively by the MLS libraries, including for example random forest algorithms, neural network algorithms, stochastic gradient descent algorithms, and the like; the MLS may maintain knowledge base 122 containing information on best practices for various machine learning tasks; entries may be added into the best practices KB 122 by various control-plane components of the MLS, e.g., based on metrics collected from server pools 185, feedback provided by clients 164, and so on; clients 164 may be able to search for and retrieve KB entries via programmatic interfaces 161, as indicated by arrow 117, and may use the information contained in the entries to select parameters (such as specific recipes or algorithms to be used) for their request submissions; ¶¶ [0037]-[0041] with FIG. 2: the MLS utilizes storage service 202, computing service 258, and database service 255 of provider network 202; MLS gateway 222 may be established to receive client requests 210 submitted over external network 206 (such as portions of the Internet) by clients 164; MLS customers may be provided an SDK (software development kit) 204 for local installation at client computing devices, and the requests 210 may be submitted from within programs written in conformance with the SDK; a client may also or instead access MLS functions from a compute server 262 of computing service 262 that has been allocated to the client; in response to at least some client requests 210, the MLS request handler 180 may generate and store corresponding job objects within a job queue 142; the job queue 142 may itself be represented by a database object (e.g., a table) stored at database service 255; a job scheduler 272 may retrieve a job from queue 142, e.g., after checking that the job's dependency requirements have been met, and identify one or more servers 262 from computing service 258 to execute the job's computational operations; ¶¶ [0046]-[0048] with FIG. 4: the MLS control plane may be responsible for generating processing plans corresponding to each of the job objects generated in response to client requests; for each processing plan, a corresponding set of resources may then have to be identified to execute the plan, e.g., based on the workload distribution strategy selected for the plan, the available resources, and so on; MLS job queue 142 comprises five jobs, each corresponding to the invocation of a respective API by a client; corresponding to job J1, an input data cleansing plan 422 may be generated, and the plan may be executed using resource set RS1; corresponding to job J2, a statistics generation plan 424 may be generated, and subsequently executed on resource set RS2; a recipe-based feature processing plan 426 corresponding to job J3 (and APB) may be generated, and executed on resource set RS3; Job J4 may result in the generation of a model training plan 428 (which may in tum involve several iterations of training, e.g., with different sets of parameters); the model training may be performed using resource set RS4; model execution plan 430 may correspond to job JS (resulting from the client's invocation of API5), and the model may eventually be executed using resource set RS5; the same set of resources (or an overlapping set of resources) may be used for performing several or all of a client's jobs – e.g., the resource sets RS1-RS5 may not necessarily differ from one another; ¶¶ [0055]-[0061] with FIG. 6: MLS artifacts 601 may include, among others, data sources 602, statistics 603, feature processing recipes 606, model predictions 608, evaluations 610, modifiable or in-development models 630, and published models or aliases 640; the MLS may generate a respective; unique identifier for each instance of at least some of the types of artifacts shown and provide the identifiers to the clients; the identifiers may subsequently be used by clients to refer to the artifact (e.g., in subsequent API calls, in status queries, and so on); at least two types of artifacts representing machine learning models or predictors may be generated and stored; the process of developing and refining a model may take a long time, as the developer may try to improve the accuracy of the predictions using a variety of data sets and a variety of parameters; the artifacts representing models may belong to one of two categories modifiable models 630, and published models or aliases 640; the phrase "publishing a model" refers to making a particular version of a model executable by a set of users by reference to an alias name or identifier; nonexpert users 678 may be granted read and execute permissions to the aliases, while model developers 676 may also be allowed to modify models 630 (and/or the pointers of the aliases 640); after model developers 676 improve the accuracy and/or performance characteristics of a newer version of a model 630 relative to an older version for which an alias 640 has been created, they may switch the pointer of the alias so that it now points to the improved version; submit a query to learn when the underlying model was last changed, or may be notified when they request an execution of an alias that the underlying model has been changes since the last execution; the MLS may support recurring scheduling of related jobs; e.g. a client may create an artifact such as a model, and may want that same model to be re-trained and/or re-executed for different input data sets (e.g., using the same configuration of resources for each of the training or prediction iterations) at specified points in time; a respective job may be placed in the MLS job queue for each recurring training or execution iteration; in addition to the artifact types shown in FIG. 6, pipeline artifacts may be stored in the MLS artifact repository with each instance of a pipeline artifact representing a named set of recurring operations requested via such APIs; ¶¶ [0067]-[0071] with FIGS. 9a-9b: in element 901 of FIG. 9a, the MLS may receive a request from a client via a programmatic interface (such as an API, a command-line tool, a web page, or a custom GUI) to perform a particular operation on an entity belonging to a set of supported entity types of the MLS; the entity types may include, e.g., data sources, statistics, feature processing recipes, models, aliases, predictions, and/or evaluations; the operations requested may include, e.g., create, read (or describe the attributes of), modify/update attributes, execute, search, or delete operations; the request may next be validated in accordance with various rules or policies of the MLS (element 904); if the request passes the validation checks, a decision may be made as to whether a job object is to be created for the request; if an analysis of the request indicates that a job is required (as detected in element 907), a job object may be generated, indicating the nature of the lower-level operations to be performed at the MLS as well as any dependencies on other jobs, and the job object may be placed in a queue (element 913); the requesting client may be notified that the request has been accepted for execution (e.g., by indicating to the client that a job has been queued for later execution); i.e., when a job object (e.g., model training job object) is instantiated into a queue, the client requesting a machine learning task associated with the job object is notified; if the job does not have any dependencies that have yet to be met, and meets other criteria for immediate or in-line execution (as also determined in element 907), the requested operation may be performed without creating a job object (element 910) and the results may optionally be provided to the requesting client; operations corresponding to elements 901-913 may be performed for each request that is received via the MLS programmatic interface; at some point after a particular job Jk is placed in the queue, Jk may be identified (e.g., by a job scheduler component of the MLS control plane) as the next job to be implemented (element 951 of FIG. 9b); to identify the next job to be implemented, the scheduler may, e.g., start from the head of the queue (the earliest-inserted job that has not yet been executed) and search for jobs whose dependencies (if any are specified) have been met; in element 952 of FIG. 9b, one or more types of validation checks may be performed on the job Jk identified in element 951; a workload distribution strategy and processing plan may be identified for Jk – e.g., the number of processing passes or phases to be used, the degree of parallelism to be used, an iterative convergence criterion to be used for completing Jk (element 954); in accordance with the selected distribution strategy and processing plan, a set of resources may be identified for Jk (element 957); JK's operations may then be performed on the identified resources (element 960), and the client on whose behalf Jk was created may optionally be notified when the operations complete (or in the event of a failure that prevents completion of the operations); ¶¶ [0072]-[0078] with FIGS. 10a-10b: Given the asynchronous manner in which client requests are handled, clients may sometimes end up submitting the same request multiple times; such multiple submissions may occur because the client is unaware whether the previous submission was accepted or not (e.g., because the client failed to notice an indication that the previous submission was accepted, or because such an indication was lost); a duplicate request may be received because the client has assumed that since the expected results of completing the requested task have not been provided for a long time, the previous request must have failed; in order to avoid such problematic scenarios, in at least one embodiment one or more of the programmatic interfaces supported by the MLS may be designed to be idempotent, such that the re-submission of a duplicate request by the same client does not have negative consequences; as shown in element 1001, a request to create a new instance of an entity type ET1 may be received from a client C1 at the MLS via a programmatic interface such as a particular APL; the MLS may generate a representation IPR1 of the input parameters included in the client's invocation of the programmatic interface (element 1004); the MLS may check, e.g., via a lookup in the artifact repository, whether an instance of entity type ET1, with instance identifier ID1 and client identifier Cl already exists in the repository; if no such instance is found (as detected in element 1007), a new instance of type ET1 with the identifier ID1, input parameter representation IPR1 and client identifier C1 may be inserted into the repository (element 1007); a success response to the client's request (element 1016) may be generated; if, in operations corresponding to element 1007, a pre-existing instance with the same instance identifier ID1 and client identifier C1 is found in the repository, the MLS may check whether the input parameter representation of the pre-existing instance also matches IPR1 (element 1013); if the input parameter representations also match, the MLS may assume that the client's request is a (harmless) duplicate, and no new work needs to be performed; accordingly, the MLS may also indicate success to the client (either explicitly or implicitly) if such a duplicate request is found (element 1016); thus, if the client had inadvertently resubmitted the same request, the creation of a new job object and the associated resource usage may be avoided; if the client request is found to be an exact duplicate of an earlier request using the methodology described, an indication may be provided to the client that the request, while not being designated as an error, was in fact identified as a duplicate; if the input parameter representation of the pre-existing instance does not match that of the client's request, an error message may be returned to the client (element 1019), e.g., indicating that there is a pre-existing instance of the same entity type ET1 with the same identifier; as shown in element 1051, at least some of the artifacts (such as recipes and models) generated at the MLS as a result of client requests may be classified into groups based on problem domains-e. g., some artifacts may be used for financial analysis, others for computer vision applications, others for bioinformatics, and so on; the MLS control plane may comprise a set of monitoring agents that collect performance and other metrics from the resources used for the various phases of machine learning operations (element 1054); based on the collected metrics and/or feedback, respective sets of best practices for various phases of machine learning workflows may be identified (element 1057); representations or summaries of the best practices identified may be stored in a knowledge base of the MLS; access (e.g., via a browser or a search tool) to the knowledge base may be provided to MLS users (element 1060)).
Lee and DIRAC are analogous art because they are from the same field of endeavor, a system and a method for providing machine learning service. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of DIRAC to Lee. Motivation for doing so would provide a proper feedback for a requesting client to prevent resubmitting the same request multiple times and avoid their associated resources usage (DIRAC, ¶¶ [0072] and [0075]).
Claims 2 and 10
Lee in view of DIRAC discloses all the elements as stated in Claims 1 and 9 respectively and further discloses receiving, by the machine learning training function, a request from the consumer to read a managed object instance associated with a list of machine learning training jobs; and transmitting, by the machine learning training function, a list of machine learning training jobs identifiers to the consumer (Lee, ¶¶ [0094]-[0095] with FIG. 1: the NF device 102 that needs to search for an NWDAF instance that provides support for some specific service operations for a specific type of analytics may query the NRF with respect to the NWDAF device 101 that supports a required service operation and a required Analytic ID; the NWDAF device 101 that performs the MTLF may register an ML model provisioning service and a training service (that is, Nnwdaf_MLModelProvision, Nnwdaf_MLModelInfo, Nnwdaf_MLModelUpdate, Nnwdaf_MLModelTraining, and Nnwdaf_MLModelTraininginfo) when an ML model is providable and trainable with respect to the Analytic ID; ¶¶ [0241]-[0247] with FIG. 8: in operation 4, the NWDAF device 801 may invoke, from the NRF device 802, a request service operation (Nnrf_NFDiscovery_Request) for searching for the NWDAF device 803; at this time, the request service operation may include at least one of (i) a list of supported Analytic IDs, (ii) a service supported by the NWDAF device 803 (e.g., an Nnwdaf_MLModelProvision service and an Nnwdaf_MLModelInfo service), (iii) a serving area where an ML model is provided, (iv) S-NSSAI, (v) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and (vi) a federation learning capability (aggregation capability for a result of training an ML model (e.g., an ML model update capability)); in operation 5, the NRF device 802 may invoke, from the NWDAF device 801, a discovery request response service operation (Nnrf_NFDiscovery_Request_response); here, the response service operation may include a list and an address of an instance ID of the NWDAF device 803; ¶¶ [0265]-[0271] with FIG. 9: the NWDAF device 901 may invoke, from the NRF device 902, a registration service operation (Nnrf_NFManagement_NFRegister request) for the NWDAF device 901; at this time, the registration service operation may include at least one of a list of supported Analytic IDs, per supported service (e.g., an Nnwdaf_MLModelTraining service or an Nnwdaf_MLModelTraininginfo service), a serving area where an ML model is provided, S-NSSAI, ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and a federation learning capability (ML model training capability); the NWDAF device 903 may invoke, from the NRF device 902, a request service operation (Nnrf_NFDiscovery_Request) for searching for the NWDAF device 901; at this time, the request service operation may include at least one of an Analytic ID, per supported service (e.g., an Nnwdaf_MLModelTraining service or an Nnwdaf_MLModelTraininginfo service), a serving area where an ML model is provided, S-NSSAI, ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and a federation learning capability (e.g., an ML model training capability); the NRF device 902 may invoke, from the NWDAF device 903, a response service operation (Nnrf_NFDiscovery_Request_response); here, the response service operation may include a list and an address of an instance ID of the NWDAF device 901; ¶¶ [0325]-[0345] with Table 1: an Nnwdaf_MLModelProvision service operation or an Nnwdaf_MLModelInfo service operation may include the following input and output: (i) Input: an analytic ID or configured ML model information (e.g., specific purpose or pre-configured ML model ID) (if available); and (ii) Output: (a) a description of a corresponding ML model with a model parameter with respect to each requested Analytic ID, wherein the description of the ML model with the model parameter may include at least one of an ML model type (e.g., a neural network), an ML model structure (e.g., a weight matrix describing a weight), and a connection and hierarchy for each node of a neural network; (b) a model ID and model version, wherein the model ID and model version may be local information rather than global unique information of an ML model provider NWDAF; and (c) optional output: a description of a requested parameter for model update and a description of a budget for an update reporting time, wherein the description of the requested parameter for model update may include a requested parameter (e.g., a gradient and a specific method for specifying a gradient (e.g., a top-k gradient, a threshold for sparsation of gradient, and the like); an Nnwdaf_MLModelTraining service operation or an Nnwdaf_MLModelTraininginfo service operation may need to include the following input and output: (i) Input: (a) an Analytic ID, an expiry time, and ML model information (an ML model file, a model ID, and a model version); and (b) optional input: a description of a requested parameter for model update and a description of a budget for an update reporting time, wherein the description of the requested parameter for model update may include a requested parameter (e.g., a gradient and a specific method for specifying a gradient (e.g., a top-k gradient, a threshold for sparsation of gradient, and the like), S-NSSAI, a target area of interest, an application ID, a target UE, and an ML model target period; and (ii) Output: an Analytic ID, a requested parameter for ML model update (for example, a gradient), a time stamp, an ID and a version of an ML model targeted for training, and a training area (e.g., a list of TAs targeted for training, and the like)) (DIRAC, ¶¶ [0026]-[0029]: a number of different types of entities related to machine learning tasks may be generated, modified, read, executed, and/or queried/searched via MLS programmatic interfaces; the MLS programmatic interfaces may enable users to submit respective requests for several related tasks of a given machine learning workflow, such as tasks for extracting records from data sources, generating statistics on the records, feature processing, model training, prediction, and so on; the APIs implemented by the MLS may allow clients to submit requests to create, query the attributes of, read, update/modify, search, or delete an instance of at least some of the various entity types supported; ¶ [0034] with FIG. 1: clients 164 may be able to view at least a subset of the artifacts stored in repository 120, e.g., by issuing read requests 118 via programmatic interfaces 161; ¶¶ [0046]-[0048] and [0051] with FIGS. 4-5: the MLS control plane may be responsible for generating processing plans corresponding to each of the job objects generated in response to client requests; for each processing plan, a corresponding set of resources may then have to be identified to execute the plan, e.g., based on the workload distribution strategy selected for the plan, the available resources, and so on; Job J4 (i.e., job identifier) may result in the generation of a model training plan 428 (which may in tum involve several iterations of training, e.g., with different sets of parameters); the model training may be performed using resource set RS4; model execution plan 430 may correspond to job JS (resulting from the client's invocation of APIS), and the model may eventually be executed using resource set RSS; ¶¶ [0067]-[0069] with FIGS. 9a-9b: in element 901 of FIG. 9a, the MLS may receive a request from a client via a programmatic interface (such as an API, a command-line tool, a web page, or a custom GUI) to perform a particular operation on an entity belonging to a set of supported entity types of the MLS; the operations requested may include, e.g., create, read (or describe the attributes of), modify/update attributes, execute, search, or delete operations; the requesting client may be notified that the request has been accepted for execution (e.g., by indicating to the client that a job has been queued for later execution); at some point after a particular job Jk is placed in the queue, Jk may be identified (e.g., by a job scheduler component of the MLS control plane) as the next job to be implemented (element 951 of FIG. 9b); to identify the next job to be implemented, the scheduler may, e.g., start from the head of the queue (the earliest-inserted job that has not yet been executed) and search for jobs whose dependencies (if any are specified) have been met; ¶ [0074] with FIG. 10a: a success response to the client's request (element 1016) may be generated; ¶ [0120]: the particular operation comprises one or more of: (a) a creation of the instance, (b) a read operation to obtain respective values of one or more attributes of the instance, (c) a modification of an attribute of the instance, (d) a deletion of the instance, (e) a search operation, or (f) an execute operation).
Claims 3 and 11
Lee in view of DIRAC discloses all the elements as stated in Claims 1 and 9 respectively and further discloses receiving, by the machine learning training function, a request from the consumer to modify managed object instance attributes associated with a list of machine learning training jobs identifiers; and transmitting, by the machine learning training function, a notification to the consumer of the list of modified machine learning training jobs identifiers (Lee, ¶¶ [0023]-[0024]: locally training the ML model, invoking, from the second NWDAF device, an ML model update notification service operation, and invoking, from the second NWDAF device that globally updates the ML model, a notification service operation for provisioning of the ML model; locally training the ML model, and invoking, from the second NWDAF device that globally updates the ML model, an ML model update notification service operation; ¶¶ [0090]-[0095] with FIG. 1: the NWDAF device 101 may generate a new model without input data related to the abnormal UE list during the observed time window and/or generate an analytics result for network data, and then may transmit the new model or the network data to the subscribed NWDAF device 101 or update the new model or the network data; the NWDAF device 101 that performs the MTLF may register an ML model provisioning service and a training service (that is, Nnwdaf_MLModelProvision, Nnwdaf_MLModelInfo, Nnwdaf_MLModelUpdate, Nnwdaf_MLModelTraining, and Nnwdaf_MLModelTraininginfo) when an ML model is providable and trainable with respect to the Analytic ID; ¶ [0111] with FIG. 3: for update of the ML model, the NWDAF device may use an Nnwdaf_MLModelUpdate, Nnwdaf_MLModelTraining, or Nnwdaf_MLModelTraininginfo service; ¶¶ [0157]-[0209] with FIG. 5: the provisioning service for the ML model may be used to modify an existing ML model subscription in the NWDAF device 501; in operation 1, the NWDAF device 501, which is a service consumer, may invoke a subscription service operation for provisioning of an ML model (Nnwdaf_MLModelProvision_Subscribe) or an unsubscription service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Unsubscribe) to subscribe, modify, or unsubscribe an ML model trained in the NWDAF device 502 that supports an MTLF connected to an Analytic ID; a parameter used by the NWDAF device 501 may include at least one of (i) an Analytic ID, (ii) S-NSSAI, (iii) a target area of interest, (iv) an application ID, (v) a target UE, (vi) an ML model target period, (vii) an expiry time, and (viii) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version; when invocation of a service operation of the NWDAF device 501 is for subscription modification or unsubscription, the NWDAF device 501 may include an identifier (subscription correlation ID) to be modified in invocation of a subscription service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Subscribe); when a process of operation 1 is performed for subscription modification (that is, including a subscription correlation ID), the NWDAF device 502 that performs the MTLF may invoke the notification service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Notify) to provide a new learned ML model different from that previously provided or provide a relearned ML model; a list of Analytic IDs used to identify analytics for which an ML model is used; when a subscription is accepted by an NWDAF, a consumer NF device receives an identifier (subscription correlation ID) from the NWDAF so as to additionally manage (modify and delete) this subscription; a modification to an ML model subscription may be performed by the NWDAF based on an operator policy and a configuration; a model for analyzing a subscription correlation ID (when modifying an ML model subscription); ¶¶ [0274]-[0289] with FIG. 10: in operation 1, the NWDAF device 1001, which is a service consumer, may invoke a subscription service operation for provisioning of an ML model (Nnwdaf_MLModelProvision_Subscribe) or an unsubscription service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Unsubscribe) to subscribe, modify, or unsubscribe an ML model of an untrained initial model or a trained ML model connected to an Analytic ID; when invocation of a service operation of the NWDAF device 1001 is for subscription modification or unsubscription, the NWDAF device 1001 may include an identifier (subscription correlation ID) to be modified in invocation of Nnwdaf_MLModelProvision_Subscribe; when a process of operation 1 is performed for subscription modification (that is, including a subscription correlation ID), the NWDAF device 1002 that performs the MTLF may provide a new learned ML model different from that previously provided, or may provide a relearned ML model by invoking the Nnwdaf_MLModelProvision_Notify service operation; ¶¶ [0299]-[0305] with FIG. 12: the NWDAF device 1201 may invoke an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) from the NWDAF device 1202 that performs a global update; the notification service operation may include at least one of (i) an Analytic ID, (ii) a requested parameter for ML model update (e.g., a gradient), (iii) a time stamp, (iv) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and (v) a training area (for example, a list of target areas (TAs) targeted for training, and the like); in operation 5, the NWDAF device 1202 may globally update the ML model, wherein globally updating the ML model may mean aggregating, by each of a plurality of NWDAF devices 1202, the locally trained ML model, and then changing the ML model by reflecting a gradient of the ML model expressed as a polynomial; ¶¶ [0306]-[0310] with FIG. 13: in operation 4, the NWDAF device 1301 may invoke an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) from an NWDAF device 1302 that performs a global update; the update notification service operation may include at least one of (i) an Analytic ID, (ii) a requested parameter for ML model update (e.g., a gradient), (iii) a time stamp, (iv) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and (v) a training area (e.g., a list of TAs targeted for training, and the like); in operation 5, the NWDAF device 1302 may globally update the ML model; ¶¶ [0311]-[0317] with FIG. 14: in operation 4, the NWDAF device 1401 may invoke an ML model training notification service operation (Nnwdaf_MLModelTraining_Notify) from the NWDAF device 1402 that performs a global update; the training notification service operation may include at least one of (i) an Analytic ID, (ii) a requested parameter for ML model update (for example, a gradient), (iii) a time stamp, (iv) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and (v) a training area (for example, a list of TAs targeted for training, and the like); in operation 5, the NWDAF device 1402 may globally update the ML model; ¶¶ [0318]-[0324] with FIG. 15: in operation 4, the NWDAF device 1501 may invoke an ML model training request response service operation (Nnwdaf_MLModelTraininginfo_request response) from an NWDAF device 1502 that performs global training; the training response service operation may include at least one of an Analytic ID, a requested parameter for ML model update (e.g., a gradient), a time stamp, a training area, an ML model ID, and an ML model version; in operation 5, the NWDAF device 1502 may globally update the ML model; ¶¶ [0325]-[0345] with Table 1: the Nnwdaf_MLModelUpdate_Notify service operation may need to include the following input and output: (i) Input: (a) an Analytic ID, a model update request parameter (e.g., a gradient), an update time stamp, a target model ID, and an update version; and (b) optional input: evaluation of an updated model if available; and (ii) Output: [0338] display of success or failure; when the ML model provider NWDAF provides a model to a consumer NWDAF by invoking Nnwdaf_ModelProvision_Notify, the provider NWDAF may (implicitly) subscribe to an Nnwdaf_ModelUpdate service operation of the consumer NWDAF to obtain a result of a locally updated model parameter when the consumer NWDAF is capable of training the model; ¶¶ [0356]-[0359] with FIG. 17: in operation 10, when a subscription to the ML model is performed, the NWDAF device 2 1602 may invoke, from the NWDAF device 1 1601, an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) to transmit information on the locally trained ML model; in operation 11, the NWDAF device 1 1601 may aggregate the trained ML model transmitted from the NWDAF device 2 1602 to update the ML model based on a globally trained ML model; in operation 12, the NWDAF device 1 1601 may transmit the updated ML model to the NWDAF device 2 1602 through a notification service operation for provisioning of the ML model; ¶ [0379] with FIG. 1: the NWDAF device 101 may update an analytics model used when generating the analytics information of the first network data (for example, optimizing or additionally training the analytics model)) (DIRAC, ¶¶ [0026]-[0030]: a number of different types of entities related to machine learning tasks may be generated, modified, read, executed, and/or queried/searched via MLS programmatic interfaces; the MLS programmatic interfaces may enable users to submit respective requests for several related tasks of a given machine learning workflow, such as tasks for extracting records from data sources, generating statistics on the records, feature processing, model training, prediction, and so on; the MLS may also be responsible for generating a processing plan for each job, identifying the appropriate set of resources (e.g., CPUs/cores, storage or memory) for the plan, scheduling the execution of the plan, gathering results, providing/saving the results in an appropriate destination, and at least in some cases for providing status updates or responses to the requesting clients; the APIs implemented by the MLS may allow clients to submit requests to create, query the attributes of, read, update/modify, search, or delete an instance of at least some of the various entity types supported; model developers may be allowed to modify the pointer and/or modify the underlying model; ¶ [0034] with FIG. 1: some relatively simple types of client requests 111 may result in the immediate generation, retrieval, storage, or modification of corresponding artifacts within MLS artifact repository 120 by the MLS request handler 180 (as indicated by arrow 141); ¶ [0059] with FIG. 6: model developers 676 may also be allowed to modify models 630 (and/or the pointers of the aliases 640); submit a query to learn when the underlying model was last changed, or may be notified when they request an execution of an alias that the underlying model has been changes since the last execution; ¶¶ [0067]-[0069] with FIG. 9a: in element 901 of FIG. 9a, the MLS may receive a request from a client via a programmatic interface (such as an API, a command-line tool, a web page, or a custom GUI) to perform a particular operation on an entity belonging to a set of supported entity types of the MLS; the operations requested may include, e.g., create, read (or describe the attributes of), modify/update attributes, execute, search, or delete operations; the requesting client may be notified that the request has been accepted for execution (e.g., by indicating to the client that a job has been queued for later execution); ¶ [0074] with FIG. 10a: a success response to the client's request (element 1016) may be generated; ¶ [0120]: the particular operation comprises one or more of: (a) a creation of the instance, (b) a read operation to obtain respective values of one or more attributes of the instance, (c) a modification of an attribute of the instance, (d) a deletion of the instance, (e) a search operation, or (f) an execute operation).
Claims 4 and 12
Lee in view of DIRAC discloses all the elements as stated in Claims 1 and 9 respectively and further discloses receiving, by the machine learning training function, a request from the consumer to delete managed object instance attributes associated with a list of machine learning training jobs identifiers; and transmitting, by the machine learning training function, a notification to the consumer of the list of deleted machine learning training jobs identifiers (Lee, ¶¶ [0023]-[0024]: locally training the ML model, invoking, from the second NWDAF device, an ML model update notification service operation, and invoking, from the second NWDAF device that globally updates the ML model, a notification service operation for provisioning of the ML model; locally training the ML model, and invoking, from the second NWDAF device that globally updates the ML model, an ML model update notification service operation; ¶¶ [0090]-[0095] with FIG. 1: the NWDAF device 101 may generate a new model without input data related to the abnormal UE list during the observed time window and/or generate an analytics result for network data, and then may transmit the new model or the network data to the subscribed NWDAF device 101 or update the new model or the network data; the NWDAF device 101 that performs the MTLF may register an ML model provisioning service and a training service (that is, Nnwdaf_MLModelProvision, Nnwdaf_MLModelInfo, Nnwdaf_MLModelUpdate, Nnwdaf_MLModelTraining, and Nnwdaf_MLModelTraininginfo) when an ML model is providable and trainable with respect to the Analytic ID; ¶ [0111] with FIG. 3: for update of the ML model, the NWDAF device may use an Nnwdaf_MLModelUpdate, Nnwdaf_MLModelTraining, or Nnwdaf_MLModelTraininginfo service; ¶¶ [0157]-[0209] with FIG. 5: the provisioning service for the ML model may be used to modify an existing ML model subscription in the NWDAF device 501; in operation 1, the NWDAF device 501, which is a service consumer, may invoke a subscription service operation for provisioning of an ML model (Nnwdaf_MLModelProvision_Subscribe) or an unsubscription service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Unsubscribe) to subscribe, modify, or unsubscribe an ML model trained in the NWDAF device 502 that supports an MTLF connected to an Analytic ID; a parameter used by the NWDAF device 501 may include at least one of (i) an Analytic ID, (ii) S-NSSAI, (iii) a target area of interest, (iv) an application ID, (v) a target UE, (vi) an ML model target period, (vii) an expiry time, and (viii) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version; when invocation of a service operation of the NWDAF device 501 is for subscription modification or unsubscription, the NWDAF device 501 may include an identifier (subscription correlation ID) to be modified in invocation of a subscription service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Subscribe); when a process of operation 1 is performed for subscription modification (that is, including a subscription correlation ID), the NWDAF device 502 that performs the MTLF may invoke the notification service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Notify) to provide a new learned ML model different from that previously provided or provide a relearned ML model; a list of Analytic IDs used to identify analytics for which an ML model is used; when a subscription is accepted by an NWDAF, a consumer NF device receives an identifier (subscription correlation ID) from the NWDAF so as to additionally manage (modify and delete) this subscription; a modification to an ML model subscription may be performed by the NWDAF based on an operator policy and a configuration; a model for analyzing a subscription correlation ID (when modifying an ML model subscription); ¶¶ [0274]-[0289] with FIG. 10: in operation 1, the NWDAF device 1001, which is a service consumer, may invoke a subscription service operation for provisioning of an ML model (Nnwdaf_MLModelProvision_Subscribe) or an unsubscription service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Unsubscribe) to subscribe, modify, or unsubscribe an ML model of an untrained initial model or a trained ML model connected to an Analytic ID; when invocation of a service operation of the NWDAF device 1001 is for subscription modification or unsubscription, the NWDAF device 1001 may include an identifier (subscription correlation ID) to be modified in invocation of Nnwdaf_MLModelProvision_Subscribe; when a process of operation 1 is performed for subscription modification (that is, including a subscription correlation ID), the NWDAF device 1002 that performs the MTLF may provide a new learned ML model different from that previously provided, or may provide a relearned ML model by invoking the Nnwdaf_MLModelProvision_Notify service operation; ¶¶ [0299]-[0305] with FIG. 12: the NWDAF device 1201 may invoke an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) from the NWDAF device 1202 that performs a global update; the notification service operation may include at least one of (i) an Analytic ID, (ii) a requested parameter for ML model update (e.g., a gradient), (iii) a time stamp, (iv) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and (v) a training area (for example, a list of target areas (TAs) targeted for training, and the like); in operation 5, the NWDAF device 1202 may globally update the ML model, wherein globally updating the ML model may mean aggregating, by each of a plurality of NWDAF devices 1202, the locally trained ML model, and then changing the ML model by reflecting a gradient of the ML model expressed as a polynomial; ¶¶ [0306]-[0310] with FIG. 13: in operation 4, the NWDAF device 1301 may invoke an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) from an NWDAF device 1302 that performs a global update; the update notification service operation may include at least one of (i) an Analytic ID, (ii) a requested parameter for ML model update (e.g., a gradient), (iii) a time stamp, (iv) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and (v) a training area (e.g., a list of TAs targeted for training, and the like); in operation 5, the NWDAF device 1302 may globally update the ML model; ¶¶ [0311]-[0317] with FIG. 14: in operation 4, the NWDAF device 1401 may invoke an ML model training notification service operation (Nnwdaf_MLModelTraining_Notify) from the NWDAF device 1402 that performs a global update; the training notification service operation may include at least one of (i) an Analytic ID, (ii) a requested parameter for ML model update (for example, a gradient), (iii) a time stamp, (iv) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and (v) a training area (for example, a list of TAs targeted for training, and the like); in operation 5, the NWDAF device 1402 may globally update the ML model; ¶¶ [0318]-[0324] with FIG. 15: in operation 4, the NWDAF device 1501 may invoke an ML model training request response service operation (Nnwdaf_MLModelTraininginfo_request response) from an NWDAF device 1502 that performs global training; the training response service operation may include at least one of an Analytic ID, a requested parameter for ML model update (e.g., a gradient), a time stamp, a training area, an ML model ID, and an ML model version; in operation 5, the NWDAF device 1502 may globally update the ML model; ¶¶ [0325]-[0345] with Table 1: the Nnwdaf_MLModelUpdate_Notify service operation may need to include the following input and output: (i) Input: (a) an Analytic ID, a model update request parameter (e.g., a gradient), an update time stamp, a target model ID, and an update version; and (b) optional input: evaluation of an updated model if available; and (ii) Output: [0338] display of success or failure; when the ML model provider NWDAF provides a model to a consumer NWDAF by invoking Nnwdaf_ModelProvision_Notify, the provider NWDAF may (implicitly) subscribe to an Nnwdaf_ModelUpdate service operation of the consumer NWDAF to obtain a result of a locally updated model parameter when the consumer NWDAF is capable of training the model; ¶¶ [0356]-[0359] with FIG. 17: in operation 10, when a subscription to the ML model is performed, the NWDAF device 2 1602 may invoke, from the NWDAF device 1 1601, an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) to transmit information on the locally trained ML model; in operation 11, the NWDAF device 1 1601 may aggregate the trained ML model transmitted from the NWDAF device 2 1602 to update the ML model based on a globally trained ML model; in operation 12, the NWDAF device 1 1601 may transmit the updated ML model to the NWDAF device 2 1602 through a notification service operation for provisioning of the ML model; ¶ [0379] with FIG. 1: the NWDAF device 101 may update an analytics model used when generating the analytics information of the first network data (for example, optimizing or additionally training the analytics model)) (DIRAC, ¶¶ [0026]-[0030]: a number of different types of entities related to machine learning tasks may be generated, modified, read, executed, and/or queried/searched via MLS programmatic interfaces; the MLS programmatic interfaces may enable users to submit respective requests for several related tasks of a given machine learning workflow, such as tasks for extracting records from data sources, generating statistics on the records, feature processing, model training, prediction, and so on; the MLS may also be responsible for generating a processing plan for each job, identifying the appropriate set of resources (e.g., CPUs/cores, storage or memory) for the plan, scheduling the execution of the plan, gathering results, providing/saving the results in an appropriate destination, and at least in some cases for providing status updates or responses to the requesting clients; the APIs implemented by the MLS may allow clients to submit requests to create, query the attributes of, read, update/modify, search, or delete an instance of at least some of the various entity types supported; model developers may be allowed to modify the pointer and/or modify the underlying model; ¶ [0034] with FIG. 1: some relatively simple types of client requests 111 may result in the immediate generation, retrieval, storage, or modification of corresponding artifacts within MLS artifact repository 120 by the MLS request handler 180 (as indicated by arrow 141); ¶ [0059] with FIG. 6: model developers 676 may also be allowed to modify models 630 (and/or the pointers of the aliases 640); submit a query to learn when the underlying model was last changed, or may be notified when they request an execution of an alias that the underlying model has been changes since the last execution; ¶¶ [0067]-[0069] with FIG. 9a: in element 901 of FIG. 9a, the MLS may receive a request from a client via a programmatic interface (such as an API, a command-line tool, a web page, or a custom GUI) to perform a particular operation on an entity belonging to a set of supported entity types of the MLS; the operations requested may include, e.g., create, read (or describe the attributes of), modify/update attributes, execute, search, or delete operations; the requesting client may be notified that the request has been accepted for execution (e.g., by indicating to the client that a job has been queued for later execution); ¶ [0073]: although idempotency may be especially useful for programmatic interfaces that involve creation of artifacts such as data sources and models, idempotent interfaces may also be supported for other types of operations (e.g., deletes or executes); ¶ [0074] with FIG. 10a: a success response to the client's request (element 1016) may be generated; ¶ [0120]: the particular operation comprises one or more of: (a) a creation of the instance, (b) a read operation to obtain respective values of one or more attributes of the instance, (c) a modification of an attribute of the instance, (d) a deletion of the instance, (e) a search operation, or (f) an execute operation).
Independent Claims 5 and 13
Lee discloses a method, comprising: transmitting, by a consumer, a request to instantiate a machine learning training job to a machine learning training function (Lee, ¶¶ [0066]-[0102] with FIG. 1: a network data analytics function (NWDAF) device 101 may use a mechanism and interface specified with respect to a 5G core (5GC) network; the NWDAF device 101 may interact with different entities for different purposes: (a) data collection based on event subscriptions provided by an access and mobility management function (AMF), a session management function (SMF), a policy control function (PCF), unified data management (UDM), AF (directly or through NEF) and OAM (Operations, administration and management); (b) analytics and data collection using a data collection coordination function (DCCF); (c) information search from a datastore (for example, UDR through UDM for subscription interest-related information); (d) information storage and search using an analytics data repository function (ADRF); (e) analytics and data collection using a messaging framework adapter function (MF AF); (f) search of information on a network function (NF) (for example, using a network repository function (NRF) for NF-related information): (g) on-demand provision of analytics for a consumer; and (h) mass data provision to a consumer; a single instance or multiple instances of the NWDAF device 101 may be deployed in a PLMN (Public Land Mobile Network); when multiple NWDAF instances are deployed, an architecture may support deploying the NWDAF device 101 as a central NF device 102, a distributed NF collection, or a combination of the two; when the multiple NWDAF instances are deployed, the NWDAF device 101 may serve as an aggregation point (that is, an NWDAF device that performs aggregation of analytics, aggregation of a machine learning (ML) model of an untrained initial model or aggregation of a trained model); in addition, the NWDAF device 101 may generate aggregate analytics (per Analytic ID) by collecting analytics information from another NWDAF device capable of having another service area, or may perform federation learning or aggregation training (per Analytic ID) by training an ML model on each of the NWDAF devices; an instance of the NWDAF device 101 may be deployed with a 5GC NF; the NWDAF device 101 may provide analytics for the 5GC NF and the OAM, which may be disassembled as follows: (i) Analytics Logical Function (AnLF): (a) the NWDAF device 101 that performs an AnLF may perform inference and derive analytics information (that is, derive a statistic and/or prediction in response to an analytics consumer request); and (b) in addition, the NWDAF device 101 that performs the AnLF may expose an analytics service for network data (for example, Nnwdaf_AnalyticsSubscription or Nnwdaf_Analyticsinfo); and (ii) Model Training Logical Function (MTLF): (a) the NWDAF device 101 that performs an MTLF may train an ML model and expose a new training service (for example, provision of an untrained initial version of model or a trained model); the NWDAF device 101 may include each of the MTLF and the AnLF, or may support both functions; the NWDAF device 101 that performs the AnLF may set an ID and an Analytic ID of an NWDAF device that performs the MTLF so as to search for a trained ML model; the NWDAF device 101 that performs the AnLF may search for the NWDAF device that performs the MTLF using an MTLF ID; the NWDAF device 101 that performs the MTLF may discover and select an NWDAF device that supports the MTLF for federation learning; the NWDAF device 101 may perform local training and global training in federation learning; the NWDAF device 101 may generate a new model without input data related to the abnormal UE list during the observed time window and/or generate an analytics result for network data, and then may transmit the new model or the network data to the subscribed NWDAF device 101 or update the new model or the network data; the NF device 102 may include any one of the MTLF and the AnLF capable of providing a service operation (e.g., an analytic exposure operation, an ML model provisioning operation, or an ML model training operation) required for a type of analytics required, or an instance of each NWDAF device 101 may provide the following information so as to assist in searching for and selecting an instance of the NWDAF device 101 including both the MTLF and the AnLF; the NF device 102 that needs to search for an NWDAF instance that provides support for some specific service operations for a specific type of analytics may query the NRF with respect to the NWDAF device 101 that supports a required service operation and a required Analytic ID; the NWDAF device 101 that performs the MTLF may register an ML model provisioning service and a training service (that is, Nnwdaf_MLModelProvision, Nnwdaf_MLModelInfo, Nnwdaf_MLModelUpdate, Nnwdaf_MLModelTraining, and Nnwdaf_MLModelTraininginfo) when an ML model is providable and trainable with respect to the Analytic ID; the 5GC NF and the OAM, which are consumers, may determine how to use data analytics provided by the NWDAF device 101; interaction between the 5GC NF(s) and the NWDAF device 101 may occur within the PLMN; ¶¶ [0103]-[0109] with FIG. 2: allow an NWDAF device 201 including an AnLF to use a provisioning service operation and a training service operation with respect to an ML model of an untrained initial model or a trained ML model in another NWDAF device 202 that supports an MTLF; the AnLF and the MTLF may be defined as follows: (i) AnLF: the NWDAF device 101 including the AnLF may perform inference, derive analytics information (i.e., derive a statistic and/or prediction in response to an analytics consumer request), and expose an analytics service (e.g., Nwdaf_AnalyticsSubscription or Nnwdaf_Analyticsinfo); (ii) MTLF: the NWDAF device 101 including the MTLF may train an ML model and expose a new training service (e.g., trained model provision and model training); the AnLF may support Nnwdaf_Analyticsinfo (data analytics information) or Nnwdaf_AnalyticsSubscription (analytics subscription) service; the MTLF may support services such as Nnwdaf_MLModelProvision (ML model provisioning), Nnwdaf_MLModelInfo (ML model information request), Nnwdaf_MLModelUpdate (ML model update), Nnwdaf_MLModelTraining (ML model training), and Nnwdaf_MLModelTraininginfo (ML model training information); an Nnwdaf interface may be used to request and subscribe to a provisioning service for an untrained initial version of ML model or a trained ML model in an NWDAF; in addition, the Nnwdaf interface may be used by the NWDAF device 101 that supports the MTLF to request and subscribe to an ML model training service for model learning and cooperative learning; ¶¶ [0110]-[0113] with FIG. 3: an operation of an NWDAF device that performs an MTLF is described in FIG. 3(a); the NWDAF device may receive an initial version of ML model from a model provisioning server (operator), a model provisioning server (3rd party), or another NWDAF device that performs the MTLF; then, after training the initial version of ML model, the NWDAF device may provide the trained ML model to an NWDAF device that performs an AnLF or MTLF through an Nnwdaf_MLModelProvision service (model provisioning service) or Nnwdaf_MLModelInfo service (model information service); in addition, for update of the ML model, the NWDAF device may use an Nnwdaf_MLModelUpdate, Nnwdaf_MLModelTraining, or Nnwdaf_MLModelTraininginfo service; the NWDAF device illustrated in FIG. 3(b) may collect data from a DCCF device and a data source (NF device or ADRF device), and receive an ML model from an NWDAF device that performs an MTLF; then, the NWDAF device that performs the AnLF may analyze the collected data using the ML model; a data analytics result may be provided to a consumer NF device in a statistical or predictive manner; ¶¶ [0114]-[0156] with FIG. 4: an NWDAF device 401 (service consumer) may use an NWDAF search principle to select an NWDAF device that supports requested analytics information, required analytics function, and/or requested ML model information; in order to search for an NWDAF device that supports an AnLF or an NWDAF device that supports an MTLF using an NRF, the following conditions may need to be satisfied: when (i) an ML model to be provided/trained is related to UE(s) and (ii) an NWDAF service consumer (other than an NWDAF) is not capable of providing an area of interest for the requested ML model to be provided/trained, the NWDAF device 401 may select an NWDAF device 403 having a large service area from candidate NWDAF devices 403; e.g., in response to discovery, when the NWDAF device 401 receives NWDAF device(s) 403 having an aggregation capability (e.g., an ML model aggregation capability and an ML model update capability), the NWDAF device 401 may preferably select the NWDAF device 403 having an aggregation capability (for example, an ML model aggregation capability and an ML model update capability) with a large serving area; when the NWDAF device 401 is not capable of providing an ML model to be provided/trained for requested UE(s) (e.g., an NWDAF providing another service area), the NWDAF device 401 may reject a subscription or request for provision of the ML model to be provided/trained, or determine an AMF that serves an UE as specified; in order to request UE location information from the AMF and discover another target NWDAF that serves a region where the UE(s) is located, the NWDAF device 401 may query the NRF device 402 with a tracking area where the UE is located; during discovery of the NWDAF device 403 that supports the MTLF, the NRF device 402 may return instances of one or more candidate NWDAF devices 403 to an NF consumer, and an instance of each candidate NWDAF device 403 may include analytics filter information on an ML model trained for each Analytic ID; a selection function of the NWDAF device 403 that supports an MTLF of the NF consumer may select an NWDAF instance based an instance of the NWDAF device 403 that supports an available MTLF; for NWDAF selection, the NF consumer may consider at least one of the following items: a supportable service for each Analytic ID (e.g., an ML model provision/training service) or NWDAF service region information, that is, a list of TAIs to which an NWDAF may provide analytics, ML model provision, ML model training, and/or data; when selecting the NWDAF device 103 that supports an MTLF for ML model provisioning and model training, the NWDAF device 401 may consider the following additional factors: (a) An analytics filter for ML model trained for each Analytic ID; and (b) an ML model aggregation capability; ¶¶ [0135] and [0157]-[0209] with operation 1 in FIG. 5: the NWDAF device 501 that performs an AnLF, which is a service consumer, may subscribe or unsubscribe to the NWDAF device 502 that performs an MTLF; the NWDAF device 501 may simultaneously be a consumer of services provided by other NWDAF(s) and a provider of services to other NWDAF device(s) 502; in operation 1, the NWDAF device 501, which is a service consumer, may invoke a subscription service operation for provisioning of an ML model (Nnwdaf_MLModelProvision_Subscribe) or an unsubscription service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Unsubscribe) to subscribe, modify, or unsubscribe an ML model trained in the NWDAF device 502 that supports an MTLF connected to an Analytic ID; a parameter used by the NWDAF device 501 may include at least one of (i) an Analytic ID, (ii) S-NSSAI (Subscribed Network Slice Selection Assistance Information), (iii) a target area of interest, (iv) an application ID, (v) a target UE, (vi) an ML model target period, (vii) an expiry time, and (viii) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version; when a subscription to the trained ML model connected to the Analytic ID is received, the NWDAF device 502 including the MTLF may perform the following process: determine (i) whether an existing ML model is available for subscription, or (ii) whether to trigger additional training for the existing ML model with respect to subscription; the NWDAF device 502 that performs the MTLF may determine that additional training is required for an already subscribed ML model; when the NWDAF device 502 that performs the MTLF determines that additional training is required for the already subscribed ML model, the NWDAF device 502 may collect data required for training of the ML model from an NF device, DCCF device, or OAM device; ¶¶ [0210]-[0216] with operation 1 in FIG. 6: an NWDAF service consumer, that is, an NWDAF device 601 may request an NWDAF device 602 including MTLF ML model information using an Nnwdaf_MLModelInfo service; the NWDAF device 601 (e.g., NWDAF(MTLF+AnLF)) may simultaneously be a consumer of a service provided by another NWDAF device 602 and a provider of this service to other NWDAF(s); in operation 1, the NWDAF device 601 that supports an AnLF may invoke an ML model information request service operation (Nnwdaf_MLMoldelInfo_Request) to request ML model(s) connected to an Analytic ID from the NWDAF device 602 that supports an MTLF; a parameter used when the NWDAF device 601, which is an NWDAF service consumer, invokes an information request service operation, may include at least one of an Analytic ID, S-NSSAI, a target area of interest, an application ID, a target UE, an ML model target period, and an expiry time; when a request for ML model information for Analytics is received, the NWDAF device 602 that performs the MTLF may perform the following process: (i) determine whether an existing trained ML model is available for the request, or (ii) determine whether an additional training trigger for the existing trained ML model is required for the request; when the NWDAF device 602 that performs the MTLF determines that additional training is required for an already requested ML model, the NWDAF may start collecting data from an NF device, DCCF device, or OAM device required for ML model training; ¶¶ [0218]-[0222] with operation 3 in FIG. 7: in operation 3, the NWDAF device 701 may invoke, from the NWDAF device 702, a subscription service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Subscribe); at this time, a subscription ID included in the subscription service operation may be the same as a subscription ID used in operation 1; i.e., when invoking the subscription service operation of operation 3 again, the NWDAF device 701 may include a parameter same as that included when previously invoking the subscription service operation for provisioning of the ML model to request a new ML model different from a previous one or re-request an ML model previously provided through a subscription or request process from the NWDAF device 702; at this time, the NWDAF device 701 may incorporate an alternative ML model flag into the subscription service operation for provisioning of the ML model in operation 3 to request a new ML model different from a previous one or re-request a previously provided ML model from the NWDAF device 702; ¶¶ [0224]-[0247] with FIG. 8: the NWADF device 801 may perform local training in federation learning, and the NWDAF device 803 may perform global training in federation learning; the NWDAF device 801 (service consumer) may search for and select the NWDAF device 803 that provides requested analytics information, a required analytics function and/or a requested ML model, and supports ML model training; when model training is related to NF(s)/UE(s) and an NWDAF service consumer (NWDAF device 803) is not capable of providing an area of interest for requested model training, the NWDAF device 801 may select the NWDAF device 803 having a large service area from candidate NWDAF devices 803; e.g., in response to discovery, when the NWDAF device 801 receives the NWDAF device(s) 803 having an ML model update capability, the NWDAF device 801 may preferably select the NWDAF device 803 having a large serving area and an ML model update capability; in order to search for an NWDAF registered in a UDM with respect to a given UE, the NWDAF device 801 or other NWDAFs interested in providing and training UE-related data or an ML model may make a query to a UDM device to search for an instance of the NWDAF device 803 that is already providing a service to the UE; in order to search for the NWDAF device 803 that performs the MTLF, the NWDAF device 801 that performs the MTLF may include at least one of analytics filter information, a trainable and providable ML model ID, an ML model version, and an ML model aggregation capability with respect to an ML model that is trained per Analytic ID in response to a registration request for the NRF device 802; in operation 1, the NWDAF device 803 may invoke, from the NRF device 802, a registration service operation (Nnrf_NFManagement_NFRegister request) for the NWDAF device 803; in operation 2, the NRF device 802 may store a profile of an NF device of the NWDAF device 803; in operation 4, the NWDAF device 801 may invoke, from the NRF device 802, a request service operation (Nnrf_NFDiscovery_Request) for searching for the NWDAF device 803; in operation 7, the NWDAF device 801 may select an NWDAF device that performs an MTLF; ¶¶ [0248]-[00271] with FIG. 9: the NWDAF device 901 may perform local training in federation learning, and the NWDAF device 903 may perform global training in federation learning; the NWDAF device 903 (service consumer) may use an NWDAF search principle to select the NWDAF device 901 that supports requested analytics information, required analytics function, and/or requested ML model training; the NWDAF device 903 may support at least one of an ML model training service (Nnwdaf_MLModelTraining) or an ML model training information service (Nnwdaf_MLModelTraininginfo) to search for the NWDAF device 901; when an ML model to be trained is related to NF(s)/UE(s) and an NWDAF service consumer (other than an NWDAF) is not capable of providing an area of interest for requested ML model training, the NWDAF device 903 may select an NWDAF having a large service area from candidate NWDAFs; e.g., in response to discovery, when the NWDAF device 903 receives NWDAF(s) 901 having an ML model aggregation/update capability, the NWDAF device 903 may preferably select the NWDAF device 901 having an ML model aggregation/update capability for a large serving area; in order to search for the NWDAF device 901 that supports the MTLF, the NWDAF device 901 that supports the MTLF may include at least one of analytics filter information and a service providable for model training (i.e., an Nnwdaf_MLModelTraining service or an Nnwdaf_MLMode1Traininginfo service) with respect to an ML model that is trainable per Analytic ID in response to a registration request for the NRF device 902; in operation 1, the NWDAF device 901 may invoke, from the NRF device 902, a registration service operation (Nnrf_NFManagement_NFRegister request) for the NWDAF device 901; at this time, the registration service operation may include at least one of a list of supported Analytic IDs, per supported service (e.g., an Nnwdaf_MLModelTraining service or an Nnwdaf_MLModelTraininginfo service), a serving area where an ML model is provided, S-NSSAI, ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and a federation learning capability (ML model training capability); in operation 2, the NRF device 902 may store a profile of an NF device of the NWDAF device 901; in operation 4, the NWDAF device 903 may invoke, from the NRF device 902, a request service operation (Nnrf_NFDiscovery_Request) for searching for the NWDAF device 901; at this time, the request service operation may include at least one of an Analytic ID, per supported service (e.g., an Nnwdaf_MLModelTraining service or an Nnwdaf_MLModelTraininginfo service), a serving area where an ML model is provided, S-NSSAI, ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and a federation learning capability (for example, an ML model training capability); in operation 7, the NWDAF device 903 may select at least one NWDAF device 901 capable of learning a local ML model that performs the MTLF; ¶¶ [0272]-[0289] with operation 1 in FIG. 10: the NWDAF device 1001 may perform local training in federation learning, and the NWDAF device 1002 may perform global training in federation learning; the NWDAF device 1001 may be configured locally with ID(s) and Analytic ID(s) of an NWDAF device that performs an MTLF to search for an ML model of an untrained initial model or a trained ML model; the NWDAF device 1001 that performs an MTLF, which is a service consumer, may be used to subscribe or unsubscribe to the NWDAF device 1002 that performs the MTLF; the NWDAF device 1001 may simultaneously be a consumer of this service provided by other NWDAF(s) and a provider of this service to other NWDAF device(s) 1002; in operation 1, the NWDAF device 1001, which is a service consumer, may invoke a subscription service operation for provisioning of an ML model (Nnwdaf_MLModelProvision_Subscribe) or an unsubscription service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Unsubscribe) to subscribe, modify, or unsubscribe an ML model of an untrained initial model or a trained ML model connected to an Analytic ID; when a subscription to the ML model of the untrained initial model or the trained ML model connected to the Analytic ID is received, the NWDAF device 1002 including an MTLF may perform the following process: determine (i) whether an existing trained ML model is available for subscription or (ii) whether to trigger additional training for the existing trained ML model with respect to subscription; the NWDAF device 1002 that performs the MTLF may determine that additional training is required for the existing ML model; when the NWDAF device 1002 determines that additional training is required, the NWDAF device 1002 may collect data required for training of the ML model from an NF device, DCCF device, or OAM device; ¶¶ [0290]-[0298] with operation 1 in FIG. 6: the NWDAF device 1101 may perform local training in federation learning, and the NWDAF device 1102 may perform global training in federation learning; an NWDAF service consumer, that is, an NWDAF device 1101 may request an NWDAF device 1102 including ML model information using an Nnwdaf_MLModelInfo service operation; the NWDAF device 1101 (e.g., NWDAF(MTLF+AnLF)) may simultaneously be a consumer of a service provided by another NWDAF device 1102 and a provider of this service to other NWDAF(s); in operation 1, the NWDAF device 1101 may invoke an ML model information request service operation (Nnwdaf_MLMoldelInfo_Request) to request ML model(s) connected to an Analytic ID; when a request for ML model information for analytics is received, the NWDAF device 1102 that performs the MTLF may perform the following process: (i) determine whether an existing trained ML model is available for the request, or (ii) determine whether an additional training trigger for the existing trained ML model is required for the request; when the NWDAF device 1102 that performs the MTLF determines that additional training is required, this NWDAF may start collecting data from an NF device, DCCF device, or OAM device required for ML model training; ¶¶ [0299]-[0305] with FIGS. 10 and 12: operations 1 and 2 may be the same as operation 1 and 2 described with reference to FIG. 10; in operation 3, the NWDAF device 1201 may locally train the ML model; in operation 4, the NWDAF device 1201 may invoke an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) from the NWDAF device 1202 that performs a global update; in operation 5, the NWDAF device 1202 may globally update the ML model; here, globally updating the ML model may mean aggregating, by each of a plurality of NWDAF devices 1202, the locally trained ML model, and then changing the ML model by reflecting a gradient of the ML model expressed as a polynomial; ¶¶ [0306]-[0310] with FIGS. 11 and 13: operations 1 and 2 may be the same as operations 1 and 2 described with reference to FIG. 11; in operation 3, an NWDAF device 1301 may locally train the ML model; in operation 4, the NWDAF device 1301 may invoke an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) from an NWDAF device 1302 that performs a global update; in operation 5, the NWDAF device 1302 may globally update the ML model; ¶¶ [0311]-[0317] with FIGS.9 and 14: the NWDAF device 1401 may perform local training in federation learning, and the NWDAF device 1402 may perform global training in federation learning; in operation 1 of FIG. 14, the NWDAF device 1401, the NRF device 1402, and the NWDAF device 1403 may be the same as the discovery process of the NWDAF device 903 that performs the MTLF previously described with reference to FIG. 9; in operation 2, the NWDAF device 1403 may invoke, from the NWDAF device 1401, an ML model training subscription service operation (Nnwdaf_MLModelTraining_Subscribe); the training subscription service operation may include at least one of (i) an Analytic ID, (ii) S-NSSAI, (iii) a target area of interest, (iv) an application ID, (v) a target UE, (vi) an ML model target period, (vii) an expiry time, and (viii) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version; in addition, the training subscription service operation may further include at least one of a description of a requested parameter for ML model update and a description of a budget for an update reporting time (e.g., a target reporting time); in operation 3, the NWDAF device 1401 may locally train an ML model; in operation 4, the NWDAF device 1401 may invoke an ML model training notification service operation (Nnwdaf_MLModelTraining_Notify) from the NWDAF device 1402 that performs a global update; in operation 5, the NWDAF device 1402 may globally update the ML model; ¶¶ [0318]-[0324] with FIGS. 9 and 15: the NWDAF device 1501 may perform local training in federation learning, and the NWDAF device 1502 may perform global training in federation learning; in operation 1 of FIG. 15, the NWDAF device 1501, the NRF device 1502, and the NWDAF device 1503 may be the same as the discovery process of the NWDAF device 903 that performs the MTLF previously described with reference to FIG. 9; in operation 2, an NWDAF device 1503 may invoke, from the NWDAF device 1502, an ML model training request service operation (Nnwdaf_MLModelTraininginfo_request); at this time, the training request service operation may include at least one of (i) an Analytic ID, (ii) S-NSSAI, (iii) a target area of interest, (iv) an application ID, (v) a target UE, (vi) an ML model target period, (vii) an expiry time, and (viii) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version.; in addition, the training request service operation may further include at least one of a description of a requested parameter for ML model update and a description of a budget for an update reporting time (e.g., a top-k gradient, a threshold for sparsification of gradient, and the like); in operation 3, the NWDAF device 1501 may locally train an ML model; in operation 4, the NWDAF device 1501 may invoke an ML model training request response service operation (Nnwdaf_MLModelTraininginfo_request_response) from an NWDAF device 1502 that performs global training; in operation 5, the NWDAF device 1502 may globally update the ML model; ¶¶ [0325]-[0345] with Table 1: an Nnwdaf_MLModelTraining service operation or an Nnwdaf_MLModelTraininginfo service operation may need to include the following input and output: (i) Input: an Analytic ID, an expiry time, and ML model information (an ML model file, a model ID, and a model version); and (ii) Output: an Analytic ID, a requested parameter for ML model update (for example, a gradient), a time stamp, an ID and a version of an ML model targeted for training, and a training area (e.g., a list of TAs targeted for training, and the like); ¶¶ [0346]-[0359] with FIGS. 16-17: in operation 1, an NWDAF device 1 1601, which is an ML model provider, may register a function of providing an untrained initial version of model or a trained model (that is, an "MLModelProvision service operation" with a list of supported Analytic IDs) with an NRF device 1603 as part of a profile; in operation 2, an NRF device 1603 may store an NWDAF profile of the NWDAF device 1 1601; in operation 4, an NWDAF device 2 1602 may invoke, from the NRF device 1603, a discovery request service operation including a service parameter list (e.g., Analytic ID, and the like) so as to search for the NWDAF device 11601 that provides an "MLModelProvision service"; in operation 6, the consumer NWDAF device 2 1602 may invoke, from the NWDAF device 1 1601, a request service operation or subscription service operation of the "MLModelProvision service" using an instance of the searched provider NWDAF device 1 1601; in operation 7, the NWDAF device 1 1601 may invoke, from the NWDAF device 2 1602, a request response service operation or subscription notification service operation including a model parameter for an untrained initial version of model or a trained model; in operation 8, when the NWDAF device 2 1602 is capable of training an ML model, the NWDAF device 2 1602 may locally train the model and model parameter; in operation 9, the NWADF device 2 1602 may locally evaluate the ML model after training the ML model; in operation 10, when a subscription to the ML model is performed, the NWDAF device 2 1602 may invoke, from the NWDAF device 1 1601, an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) to transmit information on the locally trained ML model; in operation 11, the NWDAF device 1 1601 may aggregate the trained ML model transmitted from the NWDAF device 2 1602 to update the ML model based on a globally trained ML model; in operation 11, the NWDAF device 1 1601 may evaluate the ML model); and
receiving, by the consumer, a notification from the machine learning training function indicating that the machine learning training job has been completed (Lee, ¶¶ [0088] and [0090] with FIG. 1; the NWDAF device 101 may generate a new model without input data related to the abnormal UE list during the observed time window and/or generate an analytics result for network data, and then may transmit the new model or the network data to the subscribed NWDAF device 101 or update the new model or the network data; ¶¶ [0162]-[0209] with FIG. 5: a notification may be received from the NWDAF device 502 that performs the MTLF using an ML model provisioning service (Nnwdaf_MLModelProvision); ML model information received through the notification may be used to output analytics from the NWDAF device 501 that performs the AnLF; the provisioning service for the ML model may be used to modify an existing ML model subscription in the NWDAF device 501; the NWDAF device 501 may simultaneously be a consumer of services provided by other NWDAF(s) and a provider of services to other NWDAF device(s) 502; in operation 2, when the NWDAF device 501 subscribes to the trained ML model(s) connected to the Analytic ID(s), the NWDAF device 502 that performs the MTLF may invoke a notification service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Notify) to transmit trained ML model information (e.g., a file address of the trained ML model) to the NWDAF device 501; the NWDAF device 502 that performs the MTLF may invoke an Nnwdaf_MLModelProvision_Notify service operation to notify an available retrained ML model when the NWDAF device 502 determines that retraining is required for a previously provided trained ML model; when a process of operation 1 is performed for subscription modification (i.e., including a subscription correlation ID), the NWDAF device 502 that performs the MTLF may invoke the notification service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Notify) to provide a new learned ML model different from that previously provided or provide a relearned ML model; <Nnwdaf_MLModelProvision Service – ML Model Provisioning Service> a service description: this service may allow a consumer to be notified when an ML model corresponding to a subscription parameter becomes available; an NWDAF may notify ML model information to a consumer instance subscribed to a specific NWDAF service; ¶ [0217] with operation 2 in FIG. 6: in operation 2, invoke an ML model information request response service operation (Nnwdaf_MLModelInfo_Request_response) to respond to the NWDAF device 601 (service consumer) with ML model information (including an ML model file address); the NWDAF device 103 that performs the MTLF may invoke an ML model information request response service operation including at least one of (i) ML model information, (ii) a validity period, and (iii) a spatial validity; at this time, the ML model information may include at least one of an ML model file address, an ML model file, a model ID, and a model version; ¶¶ [0223] with operation 4 in FIG. 7: in operation 4, the NWDAF device 702 may invoke, from an NWDAF device, a notification service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Notify); at this time, the notification service operation may include at least one of ML model information different from the ML model provided in operation 1 (e.g., including at least one of an ML model file, an ML model file address, a model version, or a model ID), a validity period, and a spatial validity; ¶¶ [0240]-[0247] with FIG. 8: during discovery of the NWDAF device 803 that performs the MTLF, the NRF device 802 may return instances of one or more candidate NWDAF devices 803 to an NF consumer, and an instance of each candidate NWDAF device 803 may include analytics filter information on an ML model of an initial model that is untrained or an ML Model that is trained for each Analytic ID; in operation 3, the NRF device 802 may invoke, from the NWDAF device 803, a registration response service operation (Nnrf_NFManagement_NFRegister_response); in operation 5, the NRF device 802 may invoke, from the NWDAF device 801, a discovery request response service operation (Nnrf_NFDiscovery_Request_response); here, the response service operation may include a list and an address of an instance ID of the NWDAF device 803; ¶¶ [0264]-[0271] with FIG. 9: during discovery of the NWDAF device 901 that supports the MTLF, the NRF device 902 may return instances of one or more candidate NWDAF devices 901 to an NF consumer, and an instance of each candidate NWDAF device 901 may include analytics filter information for an ML model that is trainable for each Analytic ID; in operation 3, the NRF device 902 may invoke an Nnrf_NFManagement_NFRegister response from the NWDAF device 901; in operation 5, the NRF device 902 may invoke, from the NWDAF device 903, a response service operation (Nnrf_NFDiscovery_Request_response); here, the response service operation may include a list and an address of an instance ID of the NWDAF device 901; ¶¶ [0272]-[0289] with operation 2 in FIG. 10: the NWDAF device 1001 may receive a notification from the NWDAF device 1002 that performs the MTLF using an ML model provisioning service (Nnwdaf_MLModelProvision) with respect to ML model information in related analytics; the ML model information received through the notification may be used by the NWDAF device 1001 to train the ML model; the provisioning service may be also used by the NWDAF device 1001 to modify an existing ML model subscription; the NWDAF device 1001 may simultaneously be a consumer of this service provided by other NWDAF(s) and a provider of this service to other NWDAF device(s) 1002; in operation 2, when the NWDAF device 1001 subscribes to the ML model of the untrained initial model or the trained ML model(s) connected to the Analytic ID(s), the NWDAF device 1002 may invoke an Nnwdaf_MLModelProvision_Notify service operation including information on the ML model of the untrained initial model or information on the trained ML model to transmit a file address of the ML model of the untrained initial model or the trained ML model; the NWDAF device 1002 that performs the MTLF may invoke a notification service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Notify) to notify an available retrained ML model when the NWDAF device 1002 determines that retraining is required for an ML model of a previously provided untrained initial model or a trained ML model; when a process of operation 1 is performed for subscription modification (i.e., including a subscription correlation ID), the NWDAF device 1002 that performs the MTLF may provide a new learned ML model different from that previously provided, or may provide a relearned ML model by invoking the Nnwdaf_MLModelProvision_Notify service operation; ¶ [0298] with operation 2 in FIG. 11: in operation 2, the NWDAF device 1102 that performs the MTLF may invoke an ML model information request response service operation (Nnwdaf_MLModelInfo_Request_response) for an ML information request service operation (Nnwdaf_MLMode1Info_Request) to respond to the NWDAF device 1101 (service consumer), including at least one of ML model information, a validity period, and a spatial validity. ML model information that may be provided by an NWDAF that performs the MTLF may include at least one of an ML model file address, an ML model file, a model ID, and a model version; ¶¶ [0299]-[0305] with FIG. 12: in operation 4, the NWDAF device 1201 may invoke an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) from the NWDAF device 1202 that performs a global update; in operation 6, the NWDAF device 1202 may invoke, from the NWDAF device 1201, a notification service operation for provisioning of the ML model; at this time, the notification service operation may include an updated model; ¶¶ [0311]-[0317] with FIG. 14: in operation 4, the NWDAF device 1401 may invoke an ML model training notification service operation (Nnwdaf_MLModelTraining_Notify) from the NWDAF device 1402 that performs a global update; the training notification service operation may include at least one of (i) an Analytic ID, (ii) a requested parameter for ML model update (e.g., a gradient), (iii) a time stamp, (iv) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and (v) a training area (for example, a list of TAs targeted for training, and the like); ¶¶ [0318]-[0324] with FIGS. 9 and 15: in operation 4, the NWDAF device 1501 may invoke an ML model training request response service operation (Nnwdaf_MLModelTraininginfo_request_response) from an NWDAF device 1502 that performs global training; the training response service operation may include at least one of an Analytic ID, a requested parameter for ML model update (e.g., a gradient), a time stamp, a training area, an ML model ID, and an ML model version; ¶¶ [0325]-[0345] with Table 1: the Nnwdaf_MLModelUpdate_Notify service operation may need to include the following input and output: (i) Input: an Analytic ID, a model update request parameter (e.g., a gradient), an update time stamp, a target model ID, and an update version; and (ii) Output: display of success or failure; when the ML model provider NWDAF provides a model to a consumer NWDAF by invoking Nnwdaf_ModelProvision_Notify, the provider NWDAF may (implicitly) subscribe to an Nnwdaf_ModelUpdate service operation of the consumer NWDAF to obtain a result of a locally updated model parameter when the consumer NWDAF is capable of training the model; ¶¶ [0346]-[0359] with FIGS. 16-17: in operation 3, the NRF device 1603 may invoke, from the NWDAF device 1 1601, a registration response service operation; in operation 5, the NRF device 1603 may invoke, from the NWDAF device 2 1602, a discovery request response service operation including an instance of the NWDAF device 11601 that provides the "MLModelProvision service"; in operation 7, the NWDAF device 1 1601 may invoke, from the NWDAF device 2 1602, a request response service operation or subscription notification service operation including a model parameter for an untrained initial version of model or a trained model; in operation 10, when a subscription to the ML model is performed, the NWDAF device 2 1602 may invoke, from the NWDAF device 1 1601, an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) to transmit information on the locally trained ML model; in operation 13, the NWDAF device 1 1601 may transmit the updated ML model to the NWDAF device 2 1602 through a notification service operation for provisioning of the ML model).
Lee further discloses an apparatus (Lee, ¶ [0379] with FIG. 1: NWDAF device 101; ¶ [0382]: data processing apparatus), comprising: at least one processor (Lee, ¶ [0380]: at least one digital signal processor (DSP), a processor, a controller, an application-specific integrated circuit (ASIC), a programmable logic element, such as a field programmable gate array (FPGA), other electronic devices, or combinations thereof; ¶ [0383]: processors suitable for processing of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer); and at least one memory including computer program code (Lee, ¶ [0383]: one or more memory devices for storing instructions and data; one or more mass storage devices for storing data), wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform the method described above (Lee, ¶¶ [0382]-[0383]: various techniques described herein may be implemented in digital electronic circuitry, computer hardware, firmware, software, or combinations thereof; a computer program may be deployed to be processed on one computer or multiple computers at one site or distributed across multiple sites and interconnected by a communication network; at least one processor for executing instructions).
Lee fails to explicitly disclose receiving, by the consumer, a notification indicating that the machine learning training job has been instantiated.
DIRAC teaches a system and a method for providing machine learning service (DIRAC, ¶ [0024]), wherein receiving, by the consumer, a notification indicating that the machine learning training job has been instantiated (DIRAC, ¶¶ [0024]-[0031]: a customizable, easy-to-use machine learning service (MLS) designed to support large numbers of users and a wide variety of algorithms and problem sizes; allow non-experts to rely on default settings or parameters for various aspects of the procedures used for building, training and using machine learning models, where the defaults are derived from the accumulated experience of other practitioners addressing similar types of machine learning problems; at the same time, expert users may customize the parameters or settings they wish to use for various types of machine learning tasks, such as input record handling, feature processing, model building, execution and evaluation; in addition to or instead of using predefined libraries implementing various types of machine learning tasks, MLS clients may be able to extend the built-in capabilities of the service, e.g., by registering their own customized functions with the service; the term "MLS control plane" may be used herein to refer to a collection of hardware and/or software entities that are responsible for implementing various types of machine learning functionality on behalf of clients of the MLS; the term "MLS data plane" may refer to the pathways and resources used for the processing, transfer, and storage of the input data used for client-requested operations, as well as the processing, transfer and storage of output data produced as a result of client-requested operations; a number of different types of entities related to machine learning tasks may be generated, modified, read, executed, and/or queried/searched via MLS programmatic interfaces; the MLS programmatic interfaces may enable users to submit respective requests for several related tasks of a given machine learning workflow, such as tasks for extracting records from data sources, generating statistics on the records, feature processing, model training, prediction, and so on; a queue of job objects may be used for storing internal representations of requested tasks; the term "task", as used herein, refers to a set of logical operations corresponding to a given request from a client, while the term "job" refers to the internal representation of a task within the MLS; the MLS may be responsible for ensuring that the dependencies of a given job have been met before the corresponding operations are initiated; the MLS may also be responsible for generating a processing plan for each job, identifying the appropriate set of resources (e.g., CPUs/cores, storage or memory) for the plan, scheduling the execution of the plan, gathering results, providing/saving the results in an appropriate destination, and at least in some cases for providing status updates or responses to the requesting clients; not all client API requests may be implemented using jobs – e.g., a relatively short or lightweight task may be performed synchronously with respect to the corresponding request, without incurring the overhead of job creation and asynchronous job scheduling; the APIs implemented by the MLS may allow clients to submit requests to create, query the attributes of, read, update/modify, search, or delete an instance of at least some of the various entity types supported; e.g., for the entity type "DataSource", respective APIs similar to "createDataSource", "describeDataSource" (to obtain the values of attributes of the data source), "updateDataSource", "searchForDataSource", and "deleteDataSource" may be supported by the MLS; a similar set of APIs may be supported for recipes, models, and so on; some entity types may also have APIs for executing or running the entities, such as "executeModel" or "executeRecipe"; some machine learning models may be created and trained, e.g., by a group of model developers or data scientists using the MLS APIs, and then published for use by another community of users; an alias may comprise an immutable name and a pointer to a model (i.e., an internal identifier generated for the model by the MLS) that has already been created and stored in an MLS artifact repository; members of a business analyst group may be allowed to run the model using its alias name, but may not be allowed to change the pointer, while model developers may be allowed to modify the pointer and/or modify the underlying model; the model developers may continue to experiment with various algorithms, parameters and/or input data sets to obtain improved versions of the underlying model, and may be able to change the pointer to point to an enhanced version to improve the quality of predictions obtained by the business analysts; as part of developing the processing plan for a job, the MLS may select a workload distribution strategy for the job; after the workload strategy is selected, the actual set of resources to be used may be identified in accordance with the strategy, and the job's operations may be scheduled on the identified resources; a pool of compute servers and/or storage servers may be pre-configured for the MLS, and the resources for a given job may be selected from such a pool; ¶¶ [0032]-[0036] with FIG. 1: the MLS may implement a set of programmatic interfaces 161 (e.g., APIs, command-line tools, web pages, or standalone GUIs) that can be used by clients 164 (e.g., hardware or software entities owned by or assigned to customers of the MLS) to submit requests 111 for a variety of machine learning tasks or operations; the administrative or control plane portion of the MLS may include MLS request handler 180, which accepts the client requests 111 and inserts corresponding job objects into MLS job queue 142, as indicated by arrow 112; i.e., job objects (e.g., model training job objects) corresponding to client requests 111 are instantiated into MLS job queue 142 in response to requests 111 submitted by clients 164; the control plane of the MLS may comprise a plurality of components (including the request handler, workload distribution strategy selectors, one or more job schedulers, metrics collectors, and modules that act as interfaces with other services) which may also be referred to collectively as the MLS manager; each job object may indicate one or more operations that are to be performed as a result of the invocation of a programmatic interface 161, and the scheduling of a given job may in some cases depend upon the successful completion of at least a subset of the operations of an earlier-generated job; job queue 142 may be managed as a first-in-first-out (FIFO) queue, with the further constraint that the dependency requirements of a given job must have been met in order for that job to be removed from the queue; asynchronously with respect to the submission of the requests 111, the next job whose dependency requirements have been met may be removed from job queue 142 as indicated by arrow 113, and a processing plan comprising a workload distribution strategy may be identified for it; the workload distribution strategy layer 175 may determine the manner in which the lower level operations of the job are to be distributed among one or more compute servers (e.g., servers selected from pool 185), and/or the manner in which the data analyzed or manipulated for the job is to be distributed among one or more storage devices or servers; results of some jobs may be stored as MLS artifacts within repository 120 as indicated by arrow 142; some relatively simple types of client requests 111 may result in the immediate generation, retrieval, storage, or modification of corresponding artifacts within MLS artifact repository 120 by the MLS request handler 180 (as indicated by arrow 141); thus, the insertion of a job object in job queue 142 may not be required for all types of client requests; clients 164maybeabletoview at least a subset of the artifacts stored in repository 120, e.g., by issuing read requests 118 via programmatic interfaces 161; a client request 111 may indicate one or more parameters that may be used by the MLS to perform the operations, such as a data source definition 150, a feature processing transformation recipe 152, or parameters 154 to be used for a particular machine learning algorithm; the output 116 of the feature processing transformations may in tum be used as input for a selected machine learning algorithm 166, which may be executed in accordance with algorithm parameters 154 using yet another set of resources from pool 185; a wide variety of machine learning algorithms may be supported natively by the MLS libraries, including for example random forest algorithms, neural network algorithms, stochastic gradient descent algorithms, and the like; the MLS may maintain knowledge base 122 containing information on best practices for various machine learning tasks; entries may be added into the best practices KB 122 by various control-plane components of the MLS, e.g., based on metrics collected from server pools 185, feedback provided by clients 164, and so on; clients 164 may be able to search for and retrieve KB entries via programmatic interfaces 161, as indicated by arrow 117, and may use the information contained in the entries to select parameters (such as specific recipes or algorithms to be used) for their request submissions; ¶¶ [0037]-[0041] with FIG. 2: the MLS utilizes storage service 202, computing service 258, and database service 255 of provider network 202; MLS gateway 222 may be established to receive client requests 210 submitted over external network 206 (such as portions of the Internet) by clients 164; MLS customers may be provided an SDK (software development kit) 204 for local installation at client computing devices, and the requests 210 may be submitted from within programs written in conformance with the SDK; a client may also or instead access MLS functions from a compute server 262 of computing service 262 that has been allocated to the client; in response to at least some client requests 210, the MLS request handler 180 may generate and store corresponding job objects within a job queue 142; the job queue 142 may itself be represented by a database object (e.g., a table) stored at database service 255; a job scheduler 272 may retrieve a job from queue 142, e.g., after checking that the job's dependency requirements have been met, and identify one or more servers 262 from computing service 258 to execute the job's computational operations; ¶¶ [0046]-[0048] with FIG. 4: the MLS control plane may be responsible for generating processing plans corresponding to each of the job objects generated in response to client requests; for each processing plan, a corresponding set of resources may then have to be identified to execute the plan, e.g., based on the workload distribution strategy selected for the plan, the available resources, and so on; MLS job queue 142 comprises five jobs, each corresponding to the invocation of a respective API by a client; corresponding to job J1, an input data cleansing plan 422 may be generated, and the plan may be executed using resource set RS1; corresponding to job J2, a statistics generation plan 424 may be generated, and subsequently executed on resource set RS2; a recipe-based feature processing plan 426 corresponding to job J3 (and APB) may be generated, and executed on resource set RS3; Job J4 may result in the generation of a model training plan 428 (which may in tum involve several iterations of training, e.g., with different sets of parameters); the model training may be performed using resource set RS4; model execution plan 430 may correspond to job JS (resulting from the client's invocation of API5), and the model may eventually be executed using resource set RS5; the same set of resources (or an overlapping set of resources) may be used for performing several or all of a client's jobs – e.g., the resource sets RS1-RS5 may not necessarily differ from one another; ¶¶ [0055]-[0061] with FIG. 6: MLS artifacts 601 may include, among others, data sources 602, statistics 603, feature processing recipes 606, model predictions 608, evaluations 610, modifiable or in-development models 630, and published models or aliases 640; the MLS may generate a respective; unique identifier for each instance of at least some of the types of artifacts shown and provide the identifiers to the clients; the identifiers may subsequently be used by clients to refer to the artifact (e.g., in subsequent API calls, in status queries, and so on); at least two types of artifacts representing machine learning models or predictors may be generated and stored; the process of developing and refining a model may take a long time, as the developer may try to improve the accuracy of the predictions using a variety of data sets and a variety of parameters; the artifacts representing models may belong to one of two categories modifiable models 630, and published models or aliases 640; the phrase "publishing a model" refers to making a particular version of a model executable by a set of users by reference to an alias name or identifier; nonexpert users 678 may be granted read and execute permissions to the aliases, while model developers 676 may also be allowed to modify models 630 (and/or the pointers of the aliases 640); after model developers 676 improve the accuracy and/or performance characteristics of a newer version of a model 630 relative to an older version for which an alias 640 has been created, they may switch the pointer of the alias so that it now points to the improved version; submit a query to learn when the underlying model was last changed, or may be notified when they request an execution of an alias that the underlying model has been changes since the last execution; the MLS may support recurring scheduling of related jobs; e.g. a client may create an artifact such as a model, and may want that same model to be re-trained and/or re-executed for different input data sets (e.g., using the same configuration of resources for each of the training or prediction iterations) at specified points in time; a respective job may be placed in the MLS job queue for each recurring training or execution iteration; in addition to the artifact types shown in FIG. 6, pipeline artifacts may be stored in the MLS artifact repository with each instance of a pipeline artifact representing a named set of recurring operations requested via such APIs; ¶¶ [0067]-[0071] with FIGS. 9a-9b: in element 901 of FIG. 9a, the MLS may receive a request from a client via a programmatic interface (such as an API, a command-line tool, a web page, or a custom GUI) to perform a particular operation on an entity belonging to a set of supported entity types of the MLS; the entity types may include, e.g., data sources, statistics, feature processing recipes, models, aliases, predictions, and/or evaluations; the operations requested may include, e.g., create, read (or describe the attributes of), modify/update attributes, execute, search, or delete operations; the request may next be validated in accordance with various rules or policies of the MLS (element 904); if the request passes the validation checks, a decision may be made as to whether a job object is to be created for the request; if an analysis of the request indicates that a job is required (as detected in element 907), a job object may be generated, indicating the nature of the lower-level operations to be performed at the MLS as well as any dependencies on other jobs, and the job object may be placed in a queue (element 913); the requesting client may be notified that the request has been accepted for execution (e.g., by indicating to the client that a job has been queued for later execution); i.e., when a job object (e.g., model training job object) is instantiated into a queue, the client requesting a machine learning task associated with the job object is notified; if the job does not have any dependencies that have yet to be met, and meets other criteria for immediate or in-line execution (as also determined in element 907), the requested operation may be performed without creating a job object (element 910) and the results may optionally be provided to the requesting client; operations corresponding to elements 901-913 may be performed for each request that is received via the MLS programmatic interface; at some point after a particular job Jk is placed in the queue, Jk may be identified (e.g., by a job scheduler component of the MLS control plane) as the next job to be implemented (element 951 of FIG. 9b); to identify the next job to be implemented, the scheduler may, e.g., start from the head of the queue (the earliest-inserted job that has not yet been executed) and search for jobs whose dependencies (if any are specified) have been met; in element 952 of FIG. 9b, one or more types of validation checks may be performed on the job Jk identified in element 951; a workload distribution strategy and processing plan may be identified for Jk – e.g., the number of processing passes or phases to be used, the degree of parallelism to be used, an iterative convergence criterion to be used for completing Jk (element 954); in accordance with the selected distribution strategy and processing plan, a set of resources may be identified for Jk (element 957); JK's operations may then be performed on the identified resources (element 960), and the client on whose behalf Jk was created may optionally be notified when the operations complete (or in the event of a failure that prevents completion of the operations); ¶¶ [0072]-[0078] with FIGS. 10a-10b: Given the asynchronous manner in which client requests are handled, clients may sometimes end up submitting the same request multiple times; such multiple submissions may occur because the client is unaware whether the previous submission was accepted or not (e.g., because the client failed to notice an indication that the previous submission was accepted, or because such an indication was lost); a duplicate request may be received because the client has assumed that since the expected results of completing the requested task have not been provided for a long time, the previous request must have failed; in order to avoid such problematic scenarios, in at least one embodiment one or more of the programmatic interfaces supported by the MLS may be designed to be idempotent, such that the re-submission of a duplicate request by the same client does not have negative consequences; as shown in element 1001, a request to create a new instance of an entity type ET1 may be received from a client C1 at the MLS via a programmatic interface such as a particular APL; the MLS may generate a representation IPR1 of the input parameters included in the client's invocation of the programmatic interface (element 1004); the MLS may check, e.g., via a lookup in the artifact repository, whether an instance of entity type ET1, with instance identifier ID1 and client identifier Cl already exists in the repository; if no such instance is found (as detected in element 1007), a new instance of type ET1 with the identifier ID1, input parameter representation IPR1 and client identifier C1 may be inserted into the repository (element 1007); a success response to the client's request (element 1016) may be generated; if, in operations corresponding to element 1007, a pre-existing instance with the same instance identifier ID1 and client identifier C1 is found in the repository, the MLS may check whether the input parameter representation of the pre-existing instance also matches IPR1 (element 1013); if the input parameter representations also match, the MLS may assume that the client's request is a (harmless) duplicate, and no new work needs to be performed; accordingly, the MLS may also indicate success to the client (either explicitly or implicitly) if such a duplicate request is found (element 1016); thus, if the client had inadvertently resubmitted the same request, the creation of a new job object and the associated resource usage may be avoided; if the client request is found to be an exact duplicate of an earlier request using the methodology described, an indication may be provided to the client that the request, while not being designated as an error, was in fact identified as a duplicate; if the input parameter representation of the pre-existing instance does not match that of the client's request, an error message may be returned to the client (element 1019), e.g., indicating that there is a pre-existing instance of the same entity type ET1 with the same identifier; as shown in element 1051, at least some of the artifacts (such as recipes and models) generated at the MLS as a result of client requests may be classified into groups based on problem domains-e. g., some artifacts may be used for financial analysis, others for computer vision applications, others for bioinformatics, and so on; the MLS control plane may comprise a set of monitoring agents that collect performance and other metrics from the resources used for the various phases of machine learning operations (element 1054); based on the collected metrics and/or feedback, respective sets of best practices for various phases of machine learning workflows may be identified (element 1057); representations or summaries of the best practices identified may be stored in a knowledge base of the MLS; access (e.g., via a browser or a search tool) to the knowledge base may be provided to MLS users (element 1060)).
Lee and DIRAC are analogous art because they are from the same field of endeavor, a system and a method for providing machine learning service. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of DIRAC to Lee. Motivation for doing so would provide a proper feedback for a requesting client to prevent resubmitting the same request multiple times and avoid their associated resources usage (DIRAC, ¶¶ [0072] and [0075]).
Claims 6 and 14
Lee in view of DIRAC discloses all the elements as stated in Claims 5 and 13 respectively and further discloses transmitting, by the consumer, a request to the machine learning training function to read a managed object instance associated with a list of machine learning training jobs; and receiving, by the consumer, a list of machine learning training jobs identifiers from the machine learning training function (Lee, ¶¶ [0094]-[0095] with FIG. 1: the NF device 102 that needs to search for an NWDAF instance that provides support for some specific service operations for a specific type of analytics may query the NRF with respect to the NWDAF device 101 that supports a required service operation and a required Analytic ID; the NWDAF device 101 that performs the MTLF may register an ML model provisioning service and a training service (that is, Nnwdaf_MLModelProvision, Nnwdaf_MLModelInfo, Nnwdaf_MLModelUpdate, Nnwdaf_MLModelTraining, and Nnwdaf_MLModelTraininginfo) when an ML model is providable and trainable with respect to the Analytic ID; ¶¶ [0241]-[0247] with FIG. 8: in operation 4, the NWDAF device 801 may invoke, from the NRF device 802, a request service operation (Nnrf_NFDiscovery_Request) for searching for the NWDAF device 803; at this time, the request service operation may include at least one of (i) a list of supported Analytic IDs, (ii) a service supported by the NWDAF device 803 (e.g., an Nnwdaf_MLModelProvision service and an Nnwdaf_MLModelInfo service), (iii) a serving area where an ML model is provided, (iv) S-NSSAI, (v) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and (vi) a federation learning capability (aggregation capability for a result of training an ML model (e.g., an ML model update capability)); in operation 5, the NRF device 802 may invoke, from the NWDAF device 801, a discovery request response service operation (Nnrf_NFDiscovery_Request_response); here, the response service operation may include a list and an address of an instance ID of the NWDAF device 803; ¶¶ [0265]-[0271] with FIG. 9: the NWDAF device 901 may invoke, from the NRF device 902, a registration service operation (Nnrf_NFManagement_NFRegister request) for the NWDAF device 901; at this time, the registration service operation may include at least one of a list of supported Analytic IDs, per supported service (e.g., an Nnwdaf_MLModelTraining service or an Nnwdaf_MLModelTraininginfo service), a serving area where an ML model is provided, S-NSSAI, ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and a federation learning capability (ML model training capability); the NWDAF device 903 may invoke, from the NRF device 902, a request service operation (Nnrf_NFDiscovery_Request) for searching for the NWDAF device 901; at this time, the request service operation may include at least one of an Analytic ID, per supported service (e.g., an Nnwdaf_MLModelTraining service or an Nnwdaf_MLModelTraininginfo service), a serving area where an ML model is provided, S-NSSAI, ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and a federation learning capability (e.g., an ML model training capability); the NRF device 902 may invoke, from the NWDAF device 903, a response service operation (Nnrf_NFDiscovery_Request_response); here, the response service operation may include a list and an address of an instance ID of the NWDAF device 901; ¶¶ [0325]-[0345] with Table 1: an Nnwdaf_MLModelProvision service operation or an Nnwdaf_MLModelInfo service operation may include the following input and output: (i) Input: an analytic ID or configured ML model information (e.g., specific purpose or pre-configured ML model ID) (if available); and (ii) Output: (a) a description of a corresponding ML model with a model parameter with respect to each requested Analytic ID, wherein the description of the ML model with the model parameter may include at least one of an ML model type (e.g., a neural network), an ML model structure (e.g., a weight matrix describing a weight), and a connection and hierarchy for each node of a neural network; (b) a model ID and model version, wherein the model ID and model version may be local information rather than global unique information of an ML model provider NWDAF; and (c) optional output: a description of a requested parameter for model update and a description of a budget for an update reporting time, wherein the description of the requested parameter for model update may include a requested parameter (e.g., a gradient and a specific method for specifying a gradient (e.g., a top-k gradient, a threshold for sparsation of gradient, and the like); an Nnwdaf_MLModelTraining service operation or an Nnwdaf_MLModelTraininginfo service operation may need to include the following input and output: (i) Input: (a) an Analytic ID, an expiry time, and ML model information (an ML model file, a model ID, and a model version); and (b) optional input: a description of a requested parameter for model update and a description of a budget for an update reporting time, wherein the description of the requested parameter for model update may include a requested parameter (e.g., a gradient and a specific method for specifying a gradient (e.g., a top-k gradient, a threshold for sparsation of gradient, and the like), S-NSSAI, a target area of interest, an application ID, a target UE, and an ML model target period; and (ii) Output: an Analytic ID, a requested parameter for ML model update (for example, a gradient), a time stamp, an ID and a version of an ML model targeted for training, and a training area (e.g., a list of TAs targeted for training, and the like)) (DIRAC, ¶¶ [0026]-[0029]: a number of different types of entities related to machine learning tasks may be generated, modified, read, executed, and/or queried/searched via MLS programmatic interfaces; the MLS programmatic interfaces may enable users to submit respective requests for several related tasks of a given machine learning workflow, such as tasks for extracting records from data sources, generating statistics on the records, feature processing, model training, prediction, and so on; the APIs implemented by the MLS may allow clients to submit requests to create, query the attributes of, read, update/modify, search, or delete an instance of at least some of the various entity types supported; ¶ [0034] with FIG. 1: clients 164maybeabletoview at least a subset of the artifacts stored in repository 120, e.g., by issuing read requests 118 via programmatic interfaces 161; ¶¶ [0046]-[0048] and [0051] with FIGS. 4-5: the MLS control plane may be responsible for generating processing plans corresponding to each of the job objects generated in response to client requests; for each processing plan, a corresponding set of resources may then have to be identified to execute the plan, e.g., based on the workload distribution strategy selected for the plan, the available resources, and so on; Job J4 (i.e., job identifier) may result in the generation of a model training plan 428 (which may in tum involve several iterations of training, e.g., with different sets of parameters); the model training may be performed using resource set RS4; model execution plan 430 may correspond to job JS (resulting from the client's invocation of APIS), and the model may eventually be executed using resource set RSS; ¶¶ [0067]-[0069] with FIGS. 9a-9b: in element 901 of FIG. 9a, the MLS may receive a request from a client via a programmatic interface (such as an API, a command-line tool, a web page, or a custom GUI) to perform a particular operation on an entity belonging to a set of supported entity types of the MLS; the operations requested may include, e.g., create, read (or describe the attributes of), modify/update attributes, execute, search, or delete operations; at some point after a particular job Jk is placed in the queue, Jk may be identified (e.g., by a job scheduler component of the MLS control plane) as the next job to be implemented (element 951 of FIG. 9b); to identify the next job to be implemented, the scheduler may, e.g., start from the head of the queue (the earliest-inserted job that has not yet been executed) and search for jobs whose dependencies (if any are specified) have been met; ¶ [0074] with FIG. 10a: a success response to the client's request (element 1016) may be generated; ¶ [0120]: the particular operation comprises one or more of: (a) a creation of the instance, (b) a read operation to obtain respective values of one or more attributes of the instance, (c) a modification of an attribute of the instance, (d) a deletion of the instance, (e) a search operation, or (f) an execute operation).
Claims 7 and 15
Lee in view of DIRAC discloses all the elements as stated in Claims 5 and 13 respectively and further discloses transmitting, by the consumer, a request from the consumer to modify managed object instance attributes associated with a list of machine learning training jobs identifiers; and receiving, by the consumer, a notification from the machine learning training function of the list of modified machine learning training jobs identifiers (Lee, ¶¶ [0023]-[0024]: locally training the ML model, invoking, from the second NWDAF device, an ML model update notification service operation, and invoking, from the second NWDAF device that globally updates the ML model, a notification service operation for provisioning of the ML model; locally training the ML model, and invoking, from the second NWDAF device that globally updates the ML model, an ML model update notification service operation; ¶¶ [0090]-[0095] with FIG. 1: the NWDAF device 101 may generate a new model without input data related to the abnormal UE list during the observed time window and/or generate an analytics result for network data, and then may transmit the new model or the network data to the subscribed NWDAF device 101 or update the new model or the network data; the NWDAF device 101 that performs the MTLF may register an ML model provisioning service and a training service (that is, Nnwdaf_MLModelProvision, Nnwdaf_MLModelInfo, Nnwdaf_MLModelUpdate, Nnwdaf_MLModelTraining, and Nnwdaf_MLModelTraininginfo) when an ML model is providable and trainable with respect to the Analytic ID; ¶ [0111] with FIG. 3: for update of the ML model, the NWDAF device may use an Nnwdaf_MLModelUpdate, Nnwdaf_MLModelTraining, or Nnwdaf_MLModelTraininginfo service; ¶¶ [0157]-[0209] with FIG. 5: the provisioning service for the ML model may be used to modify an existing ML model subscription in the NWDAF device 501; in operation 1, the NWDAF device 501, which is a service consumer, may invoke a subscription service operation for provisioning of an ML model (Nnwdaf_MLModelProvision_Subscribe) or an unsubscription service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Unsubscribe) to subscribe, modify, or unsubscribe an ML model trained in the NWDAF device 502 that supports an MTLF connected to an Analytic ID; a parameter used by the NWDAF device 501 may include at least one of (i) an Analytic ID, (ii) S-NSSAI, (iii) a target area of interest, (iv) an application ID, (v) a target UE, (vi) an ML model target period, (vii) an expiry time, and (viii) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version; when invocation of a service operation of the NWDAF device 501 is for subscription modification or unsubscription, the NWDAF device 501 may include an identifier (subscription correlation ID) to be modified in invocation of a subscription service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Subscribe); when a process of operation 1 is performed for subscription modification (that is, including a subscription correlation ID), the NWDAF device 502 that performs the MTLF may invoke the notification service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Notify) to provide a new learned ML model different from that previously provided or provide a relearned ML model; a list of Analytic IDs used to identify analytics for which an ML model is used; when a subscription is accepted by an NWDAF, a consumer NF device receives an identifier (subscription correlation ID) from the NWDAF so as to additionally manage (modify and delete) this subscription; a modification to an ML model subscription may be performed by the NWDAF based on an operator policy and a configuration; a model for analyzing a subscription correlation ID (when modifying an ML model subscription); ¶¶ [0274]-[0289] with FIG. 10: in operation 1, the NWDAF device 1001, which is a service consumer, may invoke a subscription service operation for provisioning of an ML model (Nnwdaf_MLModelProvision_Subscribe) or an unsubscription service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Unsubscribe) to subscribe, modify, or unsubscribe an ML model of an untrained initial model or a trained ML model connected to an Analytic ID; when invocation of a service operation of the NWDAF device 1001 is for subscription modification or unsubscription, the NWDAF device 1001 may include an identifier (subscription correlation ID) to be modified in invocation of Nnwdaf_MLModelProvision_Subscribe; when a process of operation 1 is performed for subscription modification (that is, including a subscription correlation ID), the NWDAF device 1002 that performs the MTLF may provide a new learned ML model different from that previously provided, or may provide a relearned ML model by invoking the Nnwdaf_MLModelProvision_Notify service operation; ¶¶ [0299]-[0305] with FIG. 12: the NWDAF device 1201 may invoke an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) from the NWDAF device 1202 that performs a global update; the notification service operation may include at least one of (i) an Analytic ID, (ii) a requested parameter for ML model update (e.g., a gradient), (iii) a time stamp, (iv) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and (v) a training area (for example, a list of target areas (TAs) targeted for training, and the like); in operation 5, the NWDAF device 1202 may globally update the ML model, wherein globally updating the ML model may mean aggregating, by each of a plurality of NWDAF devices 1202, the locally trained ML model, and then changing the ML model by reflecting a gradient of the ML model expressed as a polynomial; ¶¶ [0306]-[0310] with FIG. 13: in operation 4, the NWDAF device 1301 may invoke an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) from an NWDAF device 1302 that performs a global update; the update notification service operation may include at least one of (i) an Analytic ID, (ii) a requested parameter for ML model update (e.g., a gradient), (iii) a time stamp, (iv) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and (v) a training area (e.g., a list of TAs targeted for training, and the like); in operation 5, the NWDAF device 1302 may globally update the ML model; ¶¶ [0311]-[0317] with FIG. 14: in operation 4, the NWDAF device 1401 may invoke an ML model training notification service operation (Nnwdaf_MLModelTraining_Notify) from the NWDAF device 1402 that performs a global update; the training notification service operation may include at least one of (i) an Analytic ID, (ii) a requested parameter for ML model update (for example, a gradient), (iii) a time stamp, (iv) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and (v) a training area (for example, a list of TAs targeted for training, and the like); in operation 5, the NWDAF device 1402 may globally update the ML model; ¶¶ [0318]-[0324] with FIG. 15: in operation 4, the NWDAF device 1501 may invoke an ML model training request response service operation (Nnwdaf_MLModelTraininginfo_request response) from an NWDAF device 1502 that performs global training; the training response service operation may include at least one of an Analytic ID, a requested parameter for ML model update (e.g., a gradient), a time stamp, a training area, an ML model ID, and an ML model version; in operation 5, the NWDAF device 1502 may globally update the ML model; ¶¶ [0325]-[0345] with Table 1: the Nnwdaf_MLModelUpdate_Notify service operation may need to include the following input and output: (i) Input: (a) an Analytic ID, a model update request parameter (e.g., a gradient), an update time stamp, a target model ID, and an update version; and (b) optional input: evaluation of an updated model if available; and (ii) Output: [0338] display of success or failure; when the ML model provider NWDAF provides a model to a consumer NWDAF by invoking Nnwdaf_ModelProvision_Notify, the provider NWDAF may (implicitly) subscribe to an Nnwdaf_ModelUpdate service operation of the consumer NWDAF to obtain a result of a locally updated model parameter when the consumer NWDAF is capable of training the model; ¶¶ [0356]-[0359] with FIG. 17: in operation 10, when a subscription to the ML model is performed, the NWDAF device 2 1602 may invoke, from the NWDAF device 1 1601, an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) to transmit information on the locally trained ML model; in operation 11, the NWDAF device 1 1601 may aggregate the trained ML model transmitted from the NWDAF device 2 1602 to update the ML model based on a globally trained ML model; in operation 12, the NWDAF device 1 1601 may transmit the updated ML model to the NWDAF device 2 1602 through a notification service operation for provisioning of the ML model; ¶ [0379] with FIG. 1: the NWDAF device 101 may update an analytics model used when generating the analytics information of the first network data (for example, optimizing or additionally training the analytics model)) (DIRAC, ¶¶ [0026]-[0030]: a number of different types of entities related to machine learning tasks may be generated, modified, read, executed, and/or queried/searched via MLS programmatic interfaces; the MLS programmatic interfaces may enable users to submit respective requests for several related tasks of a given machine learning workflow, such as tasks for extracting records from data sources, generating statistics on the records, feature processing, model training, prediction, and so on; the MLS may also be responsible for generating a processing plan for each job, identifying the appropriate set of resources (e.g., CPUs/cores, storage or memory) for the plan, scheduling the execution of the plan, gathering results, providing/saving the results in an appropriate destination, and at least in some cases for providing status updates or responses to the requesting clients; the APIs implemented by the MLS may allow clients to submit requests to create, query the attributes of, read, update/modify, search, or delete an instance of at least some of the various entity types supported; model developers may be allowed to modify the pointer and/or modify the underlying model; ¶ [0034] with FIG. 1: some relatively simple types of client requests 111 may result in the immediate generation, retrieval, storage, or modification of corresponding artifacts within MLS artifact repository 120 by the MLS request handler 180 (as indicated by arrow 141); ¶ [0059] with FIG. 6: model developers 676 may also be allowed to modify models 630 (and/or the pointers of the aliases 640); submit a query to learn when the underlying model was last changed, or may be notified when they request an execution of an alias that the underlying model has been changes since the last execution; ¶¶ [0067]-[0069] with FIG. 9a: in element 901 of FIG. 9a, the MLS may receive a request from a client via a programmatic interface (such as an API, a command-line tool, a web page, or a custom GUI) to perform a particular operation on an entity belonging to a set of supported entity types of the MLS; the operations requested may include, e.g., create, read (or describe the attributes of), modify/update attributes, execute, search, or delete operations; the requesting client may be notified that the request has been accepted for execution (e.g., by indicating to the client that a job has been queued for later execution); ¶ [0074] with FIG. 10a: a success response to the client's request (element 1016) may be generated; ¶ [0120]: the particular operation comprises one or more of: (a) a creation of the instance, (b) a read operation to obtain respective values of one or more attributes of the instance, (c) a modification of an attribute of the instance, (d) a deletion of the instance, (e) a search operation, or (f) an execute operation).
Claims 8 and 16
Lee in view of DIRAC discloses all the elements as stated in Claims 5 and 13 respectively and further discloses transmitting, by the consumer, a request from the consumer to delete managed object instance attributes associated with a list of machine learning training jobs identifiers; and receiving, by the consumer, a notification from the machine learning training function of the list of deleted machine learning training jobs identifiers (Lee, ¶¶ [0023]-[0024]: locally training the ML model, invoking, from the second NWDAF device, an ML model update notification service operation, and invoking, from the second NWDAF device that globally updates the ML model, a notification service operation for provisioning of the ML model; locally training the ML model, and invoking, from the second NWDAF device that globally updates the ML model, an ML model update notification service operation; ¶¶ [0090]-[0095] with FIG. 1: the NWDAF device 101 may generate a new model without input data related to the abnormal UE list during the observed time window and/or generate an analytics result for network data, and then may transmit the new model or the network data to the subscribed NWDAF device 101 or update the new model or the network data; the NWDAF device 101 that performs the MTLF may register an ML model provisioning service and a training service (that is, Nnwdaf_MLModelProvision, Nnwdaf_MLModelInfo, Nnwdaf_MLModelUpdate, Nnwdaf_MLModelTraining, and Nnwdaf_MLModelTraininginfo) when an ML model is providable and trainable with respect to the Analytic ID; ¶ [0111] with FIG. 3: for update of the ML model, the NWDAF device may use an Nnwdaf_MLModelUpdate, Nnwdaf_MLModelTraining, or Nnwdaf_MLModelTraininginfo service; ¶¶ [0157]-[0209] with FIG. 5: the provisioning service for the ML model may be used to modify an existing ML model subscription in the NWDAF device 501; in operation 1, the NWDAF device 501, which is a service consumer, may invoke a subscription service operation for provisioning of an ML model (Nnwdaf_MLModelProvision_Subscribe) or an unsubscription service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Unsubscribe) to subscribe, modify, or unsubscribe an ML model trained in the NWDAF device 502 that supports an MTLF connected to an Analytic ID; a parameter used by the NWDAF device 501 may include at least one of (i) an Analytic ID, (ii) S-NSSAI, (iii) a target area of interest, (iv) an application ID, (v) a target UE, (vi) an ML model target period, (vii) an expiry time, and (viii) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version; when invocation of a service operation of the NWDAF device 501 is for subscription modification or unsubscription, the NWDAF device 501 may include an identifier (subscription correlation ID) to be modified in invocation of a subscription service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Subscribe); when a process of operation 1 is performed for subscription modification (that is, including a subscription correlation ID), the NWDAF device 502 that performs the MTLF may invoke the notification service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Notify) to provide a new learned ML model different from that previously provided or provide a relearned ML model; a list of Analytic IDs used to identify analytics for which an ML model is used; when a subscription is accepted by an NWDAF, a consumer NF device receives an identifier (subscription correlation ID) from the NWDAF so as to additionally manage (modify and delete) this subscription; a modification to an ML model subscription may be performed by the NWDAF based on an operator policy and a configuration; a model for analyzing a subscription correlation ID (when modifying an ML model subscription); ¶¶ [0274]-[0289] with FIG. 10: in operation 1, the NWDAF device 1001, which is a service consumer, may invoke a subscription service operation for provisioning of an ML model (Nnwdaf_MLModelProvision_Subscribe) or an unsubscription service operation for provisioning of the ML model (Nnwdaf_MLModelProvision_Unsubscribe) to subscribe, modify, or unsubscribe an ML model of an untrained initial model or a trained ML model connected to an Analytic ID; when invocation of a service operation of the NWDAF device 1001 is for subscription modification or unsubscription, the NWDAF device 1001 may include an identifier (subscription correlation ID) to be modified in invocation of Nnwdaf_MLModelProvision_Subscribe; when a process of operation 1 is performed for subscription modification (that is, including a subscription correlation ID), the NWDAF device 1002 that performs the MTLF may provide a new learned ML model different from that previously provided, or may provide a relearned ML model by invoking the Nnwdaf_MLModelProvision_Notify service operation; ¶¶ [0299]-[0305] with FIG. 12: the NWDAF device 1201 may invoke an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) from the NWDAF device 1202 that performs a global update; the notification service operation may include at least one of (i) an Analytic ID, (ii) a requested parameter for ML model update (e.g., a gradient), (iii) a time stamp, (iv) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and (v) a training area (for example, a list of target areas (TAs) targeted for training, and the like); in operation 5, the NWDAF device 1202 may globally update the ML model, wherein globally updating the ML model may mean aggregating, by each of a plurality of NWDAF devices 1202, the locally trained ML model, and then changing the ML model by reflecting a gradient of the ML model expressed as a polynomial; ¶¶ [0306]-[0310] with FIG. 13: in operation 4, the NWDAF device 1301 may invoke an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) from an NWDAF device 1302 that performs a global update; the update notification service operation may include at least one of (i) an Analytic ID, (ii) a requested parameter for ML model update (e.g., a gradient), (iii) a time stamp, (iv) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and (v) a training area (e.g., a list of TAs targeted for training, and the like); in operation 5, the NWDAF device 1302 may globally update the ML model; ¶¶ [0311]-[0317] with FIG. 14: in operation 4, the NWDAF device 1401 may invoke an ML model training notification service operation (Nnwdaf_MLModelTraining_Notify) from the NWDAF device 1402 that performs a global update; the training notification service operation may include at least one of (i) an Analytic ID, (ii) a requested parameter for ML model update (for example, a gradient), (iii) a time stamp, (iv) ML model information including at least one of an ML model file address, an ML model file, a model ID, and a model version, and (v) a training area (for example, a list of TAs targeted for training, and the like); in operation 5, the NWDAF device 1402 may globally update the ML model; ¶¶ [0318]-[0324] with FIG. 15: in operation 4, the NWDAF device 1501 may invoke an ML model training request response service operation (Nnwdaf_MLModelTraininginfo_request response) from an NWDAF device 1502 that performs global training; the training response service operation may include at least one of an Analytic ID, a requested parameter for ML model update (e.g., a gradient), a time stamp, a training area, an ML model ID, and an ML model version; in operation 5, the NWDAF device 1502 may globally update the ML model; ¶¶ [0325]-[0345] with Table 1: the Nnwdaf_MLModelUpdate_Notify service operation may need to include the following input and output: (i) Input: (a) an Analytic ID, a model update request parameter (e.g., a gradient), an update time stamp, a target model ID, and an update version; and (b) optional input: evaluation of an updated model if available; and (ii) Output: [0338] display of success or failure; when the ML model provider NWDAF provides a model to a consumer NWDAF by invoking Nnwdaf_ModelProvision_Notify, the provider NWDAF may (implicitly) subscribe to an Nnwdaf_ModelUpdate service operation of the consumer NWDAF to obtain a result of a locally updated model parameter when the consumer NWDAF is capable of training the model; ¶¶ [0356]-[0359] with FIG. 17: in operation 10, when a subscription to the ML model is performed, the NWDAF device 2 1602 may invoke, from the NWDAF device 1 1601, an ML model update notification service operation (Nnwdaf_MLModelUpdate_Notify) to transmit information on the locally trained ML model; in operation 11, the NWDAF device 1 1601 may aggregate the trained ML model transmitted from the NWDAF device 2 1602 to update the ML model based on a globally trained ML model; in operation 12, the NWDAF device 1 1601 may transmit the updated ML model to the NWDAF device 2 1602 through a notification service operation for provisioning of the ML model; ¶ [0379] with FIG. 1: the NWDAF device 101 may update an analytics model used when generating the analytics information of the first network data (for example, optimizing or additionally training the analytics model)) (DIRAC, ¶¶ [0026]-[0030]: a number of different types of entities related to machine learning tasks may be generated, modified, read, executed, and/or queried/searched via MLS programmatic interfaces; the MLS programmatic interfaces may enable users to submit respective requests for several related tasks of a given machine learning workflow, such as tasks for extracting records from data sources, generating statistics on the records, feature processing, model training, prediction, and so on; the MLS may also be responsible for generating a processing plan for each job, identifying the appropriate set of resources (e.g., CPUs/cores, storage or memory) for the plan, scheduling the execution of the plan, gathering results, providing/saving the results in an appropriate destination, and at least in some cases for providing status updates or responses to the requesting clients; the APIs implemented by the MLS may allow clients to submit requests to create, query the attributes of, read, update/modify, search, or delete an instance of at least some of the various entity types supported; model developers may be allowed to modify the pointer and/or modify the underlying model; ¶ [0034] with FIG. 1: some relatively simple types of client requests 111 may result in the immediate generation, retrieval, storage, or modification of corresponding artifacts within MLS artifact repository 120 by the MLS request handler 180 (as indicated by arrow 141); ¶ [0059] with FIG. 6: model developers 676 may also be allowed to modify models 630 (and/or the pointers of the aliases 640); submit a query to learn when the underlying model was last changed, or may be notified when they request an execution of an alias that the underlying model has been changes since the last execution; ¶¶ [0067]-[0069] with FIG. 9a: in element 901 of FIG. 9a, the MLS may receive a request from a client via a programmatic interface (such as an API, a command-line tool, a web page, or a custom GUI) to perform a particular operation on an entity belonging to a set of supported entity types of the MLS; the operations requested may include, e.g., create, read (or describe the attributes of), modify/update attributes, execute, search, or delete operations; the requesting client may be notified that the request has been accepted for execution (e.g., by indicating to the client that a job has been queued for later execution); ¶ [0073]: although idempotency may be especially useful for programmatic interfaces that involve creation of artifacts such as data sources and models, idempotent interfaces may also be supported for other types of operations (e.g., deletes or executes); ¶ [0074] with FIG. 10a: a success response to the client's request (element 1016) may be generated; ¶ [0120]: the particular operation comprises one or more of: (a) a creation of the instance, (b) a read operation to obtain respective values of one or more attributes of the instance, (c) a modification of an attribute of the instance, (d) a deletion of the instance, (e) a search operation, or (f) an execute operation).
Claims 17-18
Lee in view of DIRAC discloses all the elements as stated in Claims 1 and 5 respectively and further discloses a non-transitory computer-readable medium comprising program instructions stored thereon for performing the method described in Claims 1 and 5 (Lee, ¶¶ [0380]-[0383]: at least some of the functions or the processes described in the example embodiments may be implemented by software, and the software may be recorded on a recording medium such as magnetic storage media, optical reading media, or digital storage media; the techniques may be implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable storage device (e.g., a computer-readable medium) for processing by, or to control an operation of, a data processing apparatus, e.g., a programmable processor, a computer, or multiple computers; examples of information carriers suitable for embodying computer program instructions and data include semiconductor memory devices, e.g., magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as compact disk read only memory (CD-ROM) or digital video disks (DVDs), magneto-optical media such as floptical disks, read-only memory (ROM), random-access memory (RAM), flash memory, erasable programmable ROM (EPROM), or electrically erasable programmable ROM (EEPROM)).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
YAO et al. (WO 2023/114017 A1, priority date: 12/13/2021) discloses in ¶¶ [0087]-[0101] with FIG. 7 that (1) the ML training can be triggered by requests from one or more MLT MnS consumers 608, or initiated by the MLT MnS producer 602 (e.g. as result of model evaluation); (2) the MLT MnS consumer 608 requests the MLT MnS producer 602 to train the ML model(s) associated with an MLentity; (3) the MLT MnS consumer 608 should specify an inference type which indicates the function or purpose of the ML entity, e.g. CoverageProblemAnalysis; (4) the MLT MnS producer 602 can perform the training according to the designated inference type; (5) the MLT MnS consumer 608 may provide the data sources 606 that contain the training data which are considered as inputs candidates for training; (6) to obtain valid training outcomes, MLT MnS consumers 608 may also designate their requirements for model performance (e.g. accuracy, etc.) in the training request; (7) the MLT MnS producer 602 provides an ML training response 706 to the MLT MnS consumer 608 indicating whether the request was accepted; (8) if not accepted, the ML training response 706 may include a reason for non-acceptance, such as insufficient training data, overcapacity, insufficient priority, and so forth; (9) if the request is accepted, the MLT MnS producer 602 decides when to start the ML training with consideration of the ML training request 704 from the MLT MnS consumer 608; (10) the MLT MnS producer 602 provides training results 708 to the 30 MLT MnS consumer 608; (11) the training result 708 may include a location of the trained ML model or entity, among other types of information. YAO further discloses in ¶¶ [0130]-[0174] with FIGS. 6, 11A-B, and 12A-B that with reference to 3GPP TS 28.105, a set of examples of information model definitions for AI/ML management that may be implemented by the MLT system 600 are presented as follows: (1) 7. 3 .2 ML TrainingRequest: (A) the IOC ML TrainingRequest represents the ML entity training request that is created by the ML training MnS consumer; (B) the ML TrainingRequest MOI is contained under one ML TrainingFunction MOI; (C) each AI/MLTrainingRequest is associated to at least one MLEntity; (D) the ML TrainingRequest may have a source to identify where it is coming from, and which may be used to prioritize the training resources for different sources; (E) the sources may be for example the network functions, operator roles, or other functional differentiations; (F) each MLTrainingRequest may indicate the expectedRunTimeContext that describes the specific conditions for which the MLEntity should be trained for; (G) in case the request is accepted, the ML training MnS producer decides when to start the ML training; (H) once the MnS producer decides to start the training based on the request, the ML training MnS producer instantiates one or more MLTrainingProcess MOI(s) that are responsible to perform the followings: (i) collects (more) data for training, if the training data are not available or the data are available but not sufficient for the training; (ii) prepares and selects the required training data, with consideration of the consumer's request provided candidate training data if any, wherein the ML training MnS producer may examine the consumer's provided candidate training data and select none, some or all of them for training, and in addition, the ML training MnS producer may select some other training data that are available in order to meet the consumer's requirements for the MLentity training; (iii) trains the MLEntity using the selected and prepared training data; (I) the ML TrainingRequest may have a requestStatus field to represent the status of the specific MLTrainingRequest: (i) the attribute values are "NOT STARTED", "TRAINING IN PROGRESS", "SUSPENDED", "FINISHED", and "CANCELLED"; (ii) when value turns to "TraininginProcess", the ML training MnS producer instantiates one or more MLTrainingProcess MOI(s) representing the training process(es) being performed per the request and notifies the MLT MnS consumer(s) who subscribed to the notification; and (J) when all of the training process associated to this request are completed, the value turns to "FINISHED"; and (2) 7.6 Common notifications: a list of notifications, defined in 3GPP TS 28.532, that an MnS consumer may receive: (a) notifyMOICreation; (b) notifyMOIDeletion; (c) notifyMOIAttributeValueChanges; and (d) notifyEvent.
Monjas Llorente et al. (US 2025/0007791 A1, priority date: 12/03/2021) discloses in ¶¶ [0001] and [0009]-[0012] that (1) generating analytics in a communication network based on machine learning (ML) models, including management of such models used in 5G core (5GC) networks; (2) the services are composed of various "service operations", which are more granular divisions of the overall service functionality; (3) the interactions between service consumers and producers can be of the type "request/response" or "subscribe/notify"; (4) in the 5G SBA, network repository functions (NRF) allow every network function to discover the services offered by other network functions, and Data Storage Functions (DSF) allow every network function to store its context; (5) the Network Data Analytics Function (NWDAF) provides network analytics information (e.g., statistical information of past events and/or predictive information) to other NFs; (6) Machine learning (ML) is a type of artificial intelligence (AI) that focuses on the use of data and algorithms to imitate the way that humans learn, gradually improving its accuracy. ML algorithms build models based on sample (or "training") data, with the models being used subsequently to make predictions or decisions; (7) ML algorithms can be used in a wide variety of applications (e.g., medicine, email filtering, speech recognition, etc.) in which it is difficult or unfeasible to develop conventional algorithms to perform the needed tasks; and (8) NWDAF is the main network function for computing analytic reports and classifies NWDAF into two sub-functions (or logical functions): (a) NWDAF Analytics Logical Function (NWDAF AnLF), which performs analytics procedures; and (b) NWDAF Model Training Logical Function (NWDAF MTLF), which performs training and retraining of ML models used by NWDAF AnLF. Monjas Llorente further discloses in ¶¶ [0013]-[0072] that (1) multiple ML models can be used to generate the analytic reports provided for a particular Analytics ID; e.g., (a) different ML models can be used to generate these analytic reports depending on location, time of day, network load, or any other information filtering parameter; and (b) these analytic reports can be generated by several cascading ML models; (2) a first network node or function (NNF) configured for ML model management in a communication network (e.g., 5GC); (3) receiving, from a second NNF of the communication network, a first message including one of the following: (a) one or more ML model identifiers corresponding to one or more ML models maintained by the first node; or (b) when the ML model identifiers are not included, an identifier of an analytic based on the one or more ML models; (4) sending, to the second NNF, a second message including one of the following: (a) a plurality of tuples corresponding to a plurality of ML models on which the analytic is based, wherein each tuple includes a different one of the ML model identifiers and one or more information elements associated with the corresponding ML model; or (b) a single tuple including the analytic identifier and one or more information elements associated with a single ML model on which the analytic is based; (5) for each tuple included in the second message, the one or more information elements include a network address, which is one of the following: a universal resource locator (URL), or a fully qualified domain name (FQDN); (6) each network address included in the second message is one of the following: an address from which the corresponding ML model can be obtained, or an address from which a manifest or a metadata file associated with the corresponding ML model can be obtained; (7) the manifest or the metadata file associated with each of the ML models includes the following: (a) ML model identifier; (b) identifier of one or more analytics that are based on the ML model; (c) updated version of the ML model, where "updated" refers to the most recent version of the ML model that can be used for inference by the AnLF, since it has been trained and validated by the MTLF; (d) creation timestamp of the updated version of the ML model; and (e) network address from which the ML model can be obtained; (8) for each tuple included in the second message, the one or more information elements also include an updated version of the corresponding ML model; (9) when the first message includes the analytic identifier, using the (received) analytic identifier to determine one or more ML model identifiers corresponding to the one or more ML models on which the analytic is based; (10) when the first message includes the one or more of ML model identifiers, the first message also indicates versions associated with corresponding ML models; (11) sending the second message includes one of the following: (a) selectively sending the second message including the single tuple, based on the updated version of the single ML model being more recent than the version indicated by the first message; or (b) selectively including each of the plurality of tuples in the second message, based on the updated version of the corresponding ML model being more recent than the version indicated by the first message; (12) the first NNF is an MTLF of an NWDAF and the second NNF is an AnLF of the NWDAF; (13) the first message is an Nnwdaf_MLModelInfo_Request message and the second message is a response to the Nnwdaf_MLModelInfo_Request message; (14) the first message is an Nnwdaf_MLModelProvision_Subscribe message and the second message is an Nnwdaf_MLModelProvision_Notify message; (15) the first message also includes one or more conditions that must be fulfilled for receiving the second message, including one or more of the following: (a) performance metrics (or identifiers thereof) for the one or more ML models on which the analytic is based; (b) respective thresholds for the performance metrics; (c) logical relations between the performance metrics and the thresholds; and (d) ML/AI framework constraints for execution of the one or more ML models; (16) determining whether to retrain the one or more ML models based on whether current values of the performance metrics for the respective models satisfy the respective thresholds and logical relations (e.g., indicated in the first message); (17) sending the second message includes one of the following: (a) selectively sending the second message including the single tuple, based on the single ML model meeting the one or more conditions included in the first message; or (b) selectively including each of the plurality of tuples in the second message, based on the corresponding ML model meeting the one or more conditions included in the first message; (18) each tuple included in the second message also includes an identifier of a performance metric indicated in the first message and a value of the identified performance metric for the corresponding ML model; (19) enable an MTLF to identify the ML models it has retrained when notifying the AnLF, thereby enabling the AnLF to swap the specific ML model it is using for inference; (20) facilitate an AnLF to express conditions upon which the MTLF should retrain a ML model; e.g., facilitate an MTLF to express training performance results so that it may decide whether it will swap ML models; and (21) at a high level, improve management of ML models used for analytics in communication networks (e.g., 5GC).
Yue et al. (US 2025/0119715 A1, priority date: 01/13/2022) discloses in ABSTRACT that (1) consumer-controllable ML model provisioning and training in a wireless communication network; (2) the process comprises two procedures, which respectively correspond to the two phases of an ML model provisioning process in, e.g., 5GC, i.e., the preparation/provisioning phase and the training execution phase; (3) for the preparation/provisioning phase, new parameters are added to the request from the ML model consumer to the ML model generator (e.g., NWDAF), so that the latter can conduct the ML model provisioning according to the consumer requirements; and (4) for the training execution phase, interactions between the ML model consumer and generator(s) are considered, and the corresponding procedure for a consumer controlling the ML model training execution phase is defined. Yue further discloses in ¶¶ [0064]-[0096] with FIG. 7 that (1) at step 0 (e.g., at some time prior to the actual procedure), the NWDAF registers a profile into a registry, e.g., an NF Repository Function (NRF); (2) at step 1, discover a server NWDAF supporting consumer control from NRF; (3) at step 2, the consumer (AF/NF/OAM) starts a subscription to the NWDAF by invoking an Nnwdaf_MLModelProvision_Subscribe service operation with consumer request; (4) at step 3, the NWDAF judges the sharable artifacts based on the consumer type (e.g., internal or external consumer, the Vendor ID, etc.), Application scenarios, Analytics ID, information of initial model provider (e.g., NWDAF, or other vendors, etc.), and local policy, etc.; (5) at steps 4, the NWDAF sends a response to the consumer (AF/NF/OAM) with information about sharable artifacts; (6) at step 5 (if not terminated at step 4b ), the NWDAF discovers one or more Client NWDAF(s) from the NRF (e.g., using Nnrf_NFDiscovery_Request); (7) at step 6, the NWDAF performs Client NWDAF Selection based on the requirements from the consumer (AF/NF/OAM) on DML/FL framework, input data source, etc.; (8) at step 7, the NWDAF estimates the time for providing the required service (e.g., trained ML model, ML model meta data, ML model weights, etc.) to the consumer based on the currently selected Client NWDAF(s), and judges whether the preparation for DML/FL is ready; and (9) at steps 8, the NWDAF responses to the consumer (AF/NF/OAM) with the information about preparation status and the estimated time for providing the required service (e.g., trained ML model, ML model meta data, ML model weights, etc.) within the time window for confirming response. Yue also discloses in ¶¶ [0097]-[] with FIG. 8 that (1) at step 1, the NWDAF (Server NWDAF) begins initial DML/FL parameter provisioning to all the selected Client NWDAF(s) with a requirement on a time window for local ML model reporting; (2) at step 2, the NWDAF (Server NWDAF) receives local ML model information from Client NWDAF(s), performs ML model aggregation on the received local ML models, and judges the training status (e.g., whether the current trained ML model could satisfy the accuracy requirement or not, whether it has converged or not, the remaining time to complete training, etc.); (3) at steps 3, the NWDAF (Server NWDAF) updates the training status Judged in step 2) to the consumer (AF /NF/OAM) periodically ( one or multiple rounds of training), or dynamically ( e.g., when a predetermined status required by the consumer is achieved (such as accuracy that can be achieved by the trained ML model), etc.), according to the prior communicated output strategy for intermediate results reporting; (4) at step 4, if it received update requirements, or if the ML model training is not complete (as judged in step 2), the NWDAF (Server NWDAF) checks the Client NWDAF(s)' status (capacity, available data, area, etc.), and judges whether Client NWDAF(s) update and re-selection is needed; (5) at step 5, if ML model training is not complete (as judged in step 2), and Client NWDAF update and reselection is needed (as judged in step 4), the NWDAF (Server NWDAF) initiates the process for Client NWDAF(s) reselection based on the update information (in step 4); (6) at step 6, if update and re-selection are performed (in step 5), the NWDAF (Server NWDAF) updates ML model aggregation, and sends the aggregated ML model information to the newly selected Client NWDAF(s) [NOTE: as indicated to the right in FIG. 8, steps 2-6 should be repeated until a relevant ML model training termination condition is satisfied (e.g., the maximum number of iterations, or the result of loss function is lower than a threshold, or termination request is received)]; (7) at step 7, the NWDAF (Server NWDAF) provides the required service to the consumer (AF/NF/OAM) within the time window for final outputs; (7a) at step 7a, the NWDAF (Server NWDAF) provides the required trained ML model (or ML model meta data, ML model weights, etc.) to the consumer (AF/NF/OAM); (7b) at step 7b, the consumer (AF/NF/OAM) sends a request to the NWDAF (Server NWDAF) for model information (e.g., Nnwdaf_MLModelInfo_Request) and/or triggers retraining; (7b) at step 7c, the NWDAF (Server NWDAF) sends a response to the consumer (AF/NF/OAM) (e.g., Nnwdaf_MLModelInfo_Request response) with the required ML model information; and (8) as indicated to the left in FIG. 8, steps 2-7 could be repeated for retraining until the service required by the consumer (AF/NF/OAM) is satisfied.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HWEI-MIN LU whose telephone number is (313)446-4913. The examiner can normally be reached Mon - Fri: 9:00 AM - 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela D. Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HWEI-MIN LU/Primary Examiner, Art Unit 2142
Prosecution Insights