DETAILED ACTION
Claims 1-20 are presented for examination.
This office action is in response to submission of application on 04/09/2024.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 08/26/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-8, 10-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Peng et al. ("DL2: A Deep Learning-driven Scheduler for Deep Learning Clusters "), as cited in the IDS dated 08/26/2024, hereafter Peng, in further view of Liu et al. ("Automated Embedding Size Search in Deep Recommender Systems "), hereafter Liu.
Regarding claim 1, Peng discloses:
A computer-implemented method comprising: configuring a reinforcement learning system in connection with a deep learning … model data ingestion pipeline of a DL-… model training cluster (Fig. 5 and page 2, right column, paragraph 1, last 4 lines “After all mini-batches in the entire dataset have been processed once, one training epoch is done. The input dataset is usually trained for multiple epochs until the model converges” teaches configuring the reinforcement learning system in Fig. 5 in connection with a deep learning model data ingestion pipeline of a model training cluster),
the reinforcement learning system comprising an environment associated with the DL… model training cluster, a reinforcement learning (RL) agent, and an action space of possible actions for the RL agent to take relative to the DL… model data ingestion pipeline (Fig. 5 and Fig. 6 teaches an environment associated with the model training cluster, a reinforcement learning agent, and an action space of possible actions for the agent relative to the data ingestion pipeline),
during data ingestion into the DL… model data ingestion pipeline and execution of a corresponding DL… model, providing live feedback detailing performance of the DL… model data ingestion pipeline to the RL agent (Fig. 5, page 8, right column, paragraph 4, last 3 lines “The agent also monitors the training status, e.g., the number of trained steps, accuracy, and training speed”, and page 6, left column, paragraph 5, lines 8-10 “We design a per-timeslot reward to collect more reward samples through the job processes, for more frequent RL model updates to expedite convergence” teaches live feedback detailing performance of the model to the agent),
wherein providing the live feedback to the RL agent further causes the RL agent to: reevaluate the environment associated with the DL… model training cluster, select one or more actions of the possible actions within the action space that improve performance of the DL… model data ingestion pipeline within the reevaluated environment (Fig. 5 and Fig. 6, page 5, right column, paragraphs 5-6 “The NN produces a policy … which is a probability distribution over the action space. a represents an action, and…is the current set of parameters in the NN... allow the NN to output an action out of the following … actions through each inference” teaches reevaluating the environment associated with the model and selecting one action from possible actions),
reallocate computational resources of the environment associated with the DL… model training cluster according to the selected one or more actions (Fig. 6, Fig. 7, and page 5, last paragraph, lines 9-11 “each inference only outputs an incremental amount of resources to be allocated to one of J jobs” teaches reallocating computational resources of the environment according to the selected actions).
While Peng teaches a reinforcement learning system in connection with a deep learning … model data ingestion pipeline of a DL…model…, they do not teach the DL model to be a recommender DL-rec model.
Liu discloses:
…a reinforcement learning system in connection with a deep learning recommender (DL-Rec) model data ingestion pipeline of a DL-rec model… (Figure 1, Figure 2, and page 2308, right column, paragraph 1, lines 4-7 “an automated reinforcement learning (RL) agent to dynamically search the embedding sizes for users and items in deep recommender systems under the streaming scenario” teaches a RL system in connection with a DL recommender model data ingestion pipeline of a DL-rec model).
Peng and Liu are analogous art because they are from the same field of endeavor, reinforcement learning and machine learning models.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Peng to include a reinforcement learning system in connection with a deep learning recommender (DL-Rec) model data ingestion pipeline of a DL-rec model, based on the teachings of Liu. One of ordinary skill in the art would have been motivated to make this modification in order to obtain more accurate selection and decreased storage space, as suggested by Liu (abstract, lines 14-15).
Regarding claim 2, Peng, in view of Liu, discloses the computer-implemented method of claim 1 (and thus the rejection of claim 1 is incorporated). Peng further discloses:
wherein the environment associated with the DL… model training cluster comprises static computational resources (page 8, section 6.2 paragraph testbed teaches static computational resources), variable RL agent-uncorrelated computational resources (page 8, section 6.1, paragraph 1, lines 1-2 “We implement DL2 as a custom scheduler on Kubernetes”), and RL agent-modified computational resources (Fig. 6 teaches agent modified resources).
Regarding claim 3, Peng, in view of Liu, discloses the computer-implemented method of claim 2 (and thus the rejection of claim 2 is incorporated). Peng further discloses:
static computational resources comprise DRAM-CPU bandwidth and CPU processing speed (page 8, right column, paragraph Testbed),
variable RL agent-uncorrelated computational resources comprise DL… model latency (page 8, right column, paragraph 3, lines 5-9 “The scheduler constantly queries cluster resources and job states (e.g., training speeds) and instructs deployment of a new job or resource adjustment in an existing job via Kubernetes API server. Mapping the cluster and job states to a scheduling decision takes less than 3ms.”),
RL agent-modified computational resources comprise current latency of the DL.. model data ingestion pipeline, a number of available CPUs, and an amount of free memory space (Fig. 3, Fig. 6, and page 3, left column, last 4 lines “We advocate dynamic adjustment of worker/PS numbers in training jobs over time, to maximally utilize available resources in the DL cluster to expedite job completion”).
Regarding claim 4, Peng, in view of Liu, discloses the computer-implemented method of claim 2 (and thus the rejection of claim 2 is incorporated). Peng further discloses:
wherein reevaluating the environment associated with the DL… model training cluster comprises determining that a change has occurred relative to one or more of the static computational resources or the RL agent-modified computational resources (page 7, right column, final paragraph, lines 1-2 “When a new PS is launched, it registers itself with the coordinator” teaches determining a change has occurred relative to the computational resources).
Regarding claim 5, Peng, in view of Liu, discloses the computer-implemented method of claim 2 (and thus the rejection of claim 2 is incorporated). Peng further discloses:
reallocating computational resources of the environment associated with the DL… model training cluster according to the selected one or more actions comprises reallocating one or more RL agent- modified computational resources (Fig. 6 teaches reallocating agent modified computational resources).
Regarding claim 6, Peng, in view of Liu, discloses the computer-implemented method of claim 1 (and thus the rejection of claim 1 is incorporated). Peng further discloses:
wherein the RL agent comprises a machine learning model with a three-layer multi-layer perceptron architecture using a ReLU activation function (Fig. 6 and page 6, left column, paragraph 3, lines 1-2 “The input state matrix s is connected to a fully connected layer with the ReLU”).
Regarding claim 7, Peng, in view of Liu, discloses the computer-implemented method of claim 1 (and thus the rejection of claim 1 is incorporated). Peng further discloses:
wherein the action space of possible actions for the RL agent to take relative to the DL… model data ingestion pipeline comprises an incremental action space that allows the RL agent to choose to "raise-by-one," "maintain," "lower-by-one," "raise-by-five," or "lower-by-five" at every step (page 5, final paragraph, lines 6-7 “a void action which indicates stopping allocating resources in the current time slot” teaches allowing the RL agent to choose to maintain at each step).
Regarding claim 8, Peng, in view of Liu, discloses the computer-implemented method of claim 1 (and thus the rejection of claim 1 is incorporated). Liu further discloses:
wherein the DL-rec model data ingestion pipeline comprises a sequence of data processing tasks that transforms a recommender dataset for training the DL-rec model (Algorithm 1, page 2311, right column, final paragraph, lines 1-7 “In this subsection, we detail our AutoML-based algorithm for optimizing the whole framework. The algorithm is presented in Algorithm 1. In the streaming recommendation, user-item transaction data are collected from a steady stream S. We iterate batches of steady data from the stream to optimize the framework.” Teaches a sequence of data processing tasks that transforms the transaction data, i.e. recommender dataset, for training the DL-rec model).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Peng to include wherein the DL-rec model data ingestion pipeline comprises a sequence of data processing tasks that transforms a recommender dataset for training the DL-rec model, based on the teachings of Liu. One of ordinary skill in the art would have been motivated to make this modification in order to obtain more accurate selection and decreased storage space, as suggested by Liu (abstract, lines 14-15).
Regarding claim 10, Peng, in view of Liu, discloses the computer-implemented method of claim 1 (and thus the rejection of claim 1 is incorporated). Peng further discloses:
wherein selecting one or more actions of the possible actions within the action space that improve performance of the DL-Rec model data ingestion pipeline within the reevaluated environment comprises selecting one or more actions according to a reward function that approaches zero as memory consumption nears 100% (Fig. 3, page 6, right column, first paragraph, lines 4-5 “
PNG
media_image1.png
116
1110
media_image1.png
Greyscale
” teaches the reward function to be discounted to near zero as memory consumption nears 100%).
Claims 11 and 20 are substantially similar to claim 1, and thus are rejected on the same basis.
Claims 12-18 are substantially similar to claims 2-8, and thus are rejected on the same basis.
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Peng et al. ("DL2: A Deep Learning-driven Scheduler for Deep Learning Clusters "), as cited in the IDS dated 08/26/2024, hereafter Peng, in further view of Liu et al. ("Automated Embedding Size Search in Deep Recommender Systems "), hereafter Liu, in further view of Jie Liu et al. ("Large Scale Caching and Streaming of Training Data for Online Deep Learning"), hereafter Jie.
Regarding claim 9, Peng, in view of Liu, discloses the computer-implemented method of claim 8 (and thus the rejection of claim 8 is incorporated). Peng discloses:
wherein the sequence of data processing tasks comprises loading samples from a base dataset …using the samples to fill a batch for DL… model training …prefetching multiple batches of samples (page 10, paragraph “Total training epoch estimation”, and Fig. 14 teaches data processing tasks loading batches of samples from the base input dataset, filling batches for training, and prefetching batches for training epoch).
Peng, in view of Liu, do not explicitly disclose:
… sequence of data processing tasks comprises loading samples from a base dataset in a disk read operation, using the samples to fill a batch for DL-Rec model training, shuffling the samples within the batch, optimizing one or more user-defined-functions, and prefetching multiple batches of samples into a GPU memory.
Jie discloses:
sequence of data processing tasks comprises loading samples from a base dataset in a disk read operation, using the samples to fill a batch for DL … model training, shuffling the samples within the batch, optimizing one or more user-defined-functions, and prefetching multiple batches of samples into a GPU memory (Figure 1 loading samples from a base dataset in a disk read operation using readers, using the samples to fill a batch for deep learning training, shuffling the samples using shuffle (), optimizing user-defined-functions, and prefetching multiple batches of samples into a GPU memory using prefetch())
Peng, Liu, and Jie are analogous art because they are from the same field of endeavor, deep learning and model training.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Peng, in view of Liu, to include sequence of data processing tasks comprises loading samples from a base dataset in a disk read operation, using the samples to fill a batch for DL … model training, shuffling the samples within the batch, optimizing one or more user-defined-functions, and prefetching multiple batches of samples into a GPU memory, based on the teachings of Jie. One of ordinary skill in the art would have been motivated to make this modification in order to enable efficient access to training samples in both offline training and continual learning, as suggested by Jie (abstract, lines 12-13).
Claim 19 is substantially similar to claim 9, and thus are rejected on the same basis.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's
disclosure.
U.S. Pub No. 20210286786 A1: Zhang et al. teaches recommendation learning and reinforcement learning.
Zou et al. (“Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems”) teaches recommendation learning and reinforcement learning.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HUMAIRA ZAHIN MAUNI whose telephone number is (703)756-5654. The examiner can normally be reached Monday - Friday, 9 am - 5 pm (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MATT ELL can be reached at (571) 270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/H.Z.M./Examiner, Art Unit 2141
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141
Prosecution Insights