mDETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The action is in response to the original filing on amendment filed 3/31/2026. Claims 1-20 are pending and have been considered below with claims 1, 9 and 14 being independent. Claims 1, 9, 10 and 14 have been amended.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Jiang et al. (“CURE: Code-Aware Neural Machine Translation for Automatic Program Repair,” hereinafter Jiang) in view of Kanade et al. (“Learning and Evaluating Contextual Embedding of Source Code,” hereinafter Kanade) in further view of Golovin et al. (“Google Vizier: A Service for Black-Box Optimization,” hereinafter Golovin) and Roitman et al. (US 2022/0358906 A1).
Regarding claim 1, Jiang teaches a system comprising: one or more processors; and a memory that stores one or more programs that are configured to be executed by the one or more processors, the one or more programs including instructions to perform acts that: (Jiang, IV. Experimental Setup, Infrastructure, pp. 8; “We train and evaluate our models on one 56-core server with one NVIDIA TITAN V and three Xp GPUs,” wherein “three Xp GPUs” corresponds to one or more processors and “one 56-core server” necessarily includes a memory that stores one or more programs)
access a plurality of pre-trained deep learning models, (Jiang, Fig. 2, elements 2 and 3, “Pre-Training” and “APR models.” Jiang, II. Background, A. Overview, pp. 3; “Different from previous work, we use subword tokenization, which produces a smaller but more accurate search space that contains more correct patches. CURE uses these tokenized methods to train a new programming language model that learns developer-like source code with correct syntax (step 2),” this step encompassing the “pre-training.” “CURE also tokenizes the buggy lines, context, and correct fixes extracted from the commit history of open-source projects, referred to as patch training data, into sequences of tokens (step 1b). We use these sequences to fine-tune an ensemble of k APR models (step 3). Each APR model combines the PL model with a context-aware neural machine translation (CoNuT) model.”)
PNG
media_image1.png
224
584
media_image1.png
Greyscale
Jiang does not explicitly teach wherein a pre-trained deep learning model of the plurality of pre-trained deep learning models is pre-trained on unsupervised training data of source code and natural language text. However, Kanade, in the area of code-understanding natural language models, teaches this limitation (Kanade, 2 Related Work, pp. 3, col. 1, paragraph 2; “We also handle natural language directly, but do not require such a separation. Natural-language tokens can be mixed with source-code tokens both within and across sentences in our encoding.” Kanade, 4.1. Training Details, pp. 6, col. 1, paragraph 2; “We pre-train CuBERT with the default configuration of the BERT Large model, one model per example length (128, 256, 512, and 1,024 subword tokens) with batch sizes of 8,192, 4,096, 2,048, and 1,024 respectively, and the default BERT learning rate of
1
×
10
-
4
,” wherein “one model per example length” indicates that each pre-trained deep learning [model] of the plurality of pre-trained deep learning models is pre-trained on unsupervised training data. Kanade, 4.4 Is Transformer All You Need?, pp. 7, col. 2, paragraph 2; “One may wonder if CuBERT’s promising results derive more from using a Transformer-based model for its classification tasks, and less from the actual, unsupervised pre-training,” thereby indicating that “CuBERT” is pre-trained on unsupervised training data.).
Jiang and Kanade are analogous to the claimed invention as both are from the same field of endeavor, that is, developing natural language processing models for neural machine translation tasks. Jiang already teaches a model pre-trained on unlabeled source code data but not natural language text. Kanade’s model is pre-trained with both. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to modify the pre-training step in Jiang to include natural language data, as done in Kanade. Doing so would yield the predictable result of a pre-trained model capable of performing a given software engineering task such as automatic program repair or automatic code generation.
wherein the plurality of pre-trained deep learning models are tailored for a particular software engineering task; (Jiang, Abstract; “First, CURE pre-trains a programming language (PL) model on a large software codebase to learn developer-like source code before the APR task.” Jiang, I. Introduction, A. Our Approach, (1) Programming language models, pp. 2, col. 1, paragraph 5; “Specifically, pre-trained language models have brought great improvement to many NLP tasks [28], [29]. They learn the probability distribution over sequences of words from a large amount of natural language text. Then one fine-tunes the pretrained language model for a specific task by adding an extra model to it (e.g., adding a classifier for classification tasks),” wherein “pre-trained language models” encompass the plurality of pre-trained deep learning models.)
The combination of Jiang and Kanade does not explicitly teach receive a user request to perform a fine-tuning task that includes fine-tuning a select one of the plurality of pre-trained deep learning models on a custom dataset. However, Golovin, in the area of cloud-based platforms for parameter tuning, teaches this limitation (Golovin, 2.4.1 Parallel Processing of Suggestion Work, pp. 1489, col. 2, paragraphs 2 and 3; “As the de facto parameter tuning engine of Google, Vizier is constantly working on generating suggestions for a large number of Studies concurrently. As such, a single machine would be insufficient for handling the workload. Our Suggestion Service is therefore partitioned across several Google datacenters, with a number of machines being used in each one…When a request is received by a Suggestion Service instance to generate suggestions, the instance first places a distributed lock on the Study,” wherein “a Suggestion Service” can receive a user request to perform a fine-tuning task that includes fine-tuning a given “Study,” which is defined as follows. Golovin, 1.2 Definitions, pp. 1488, col. 1, paragraphs 4 and 5; “A Trial is a list of parameter values,
x
, that will lead to a single evaluation of
f
(
x
)
. A trial can be “Completed”, which means that it has been evaluated and the objective value
f
(
x
)
has been assigned to it, otherwise it is “Pending”. A Study represents a single optimization run over a feasible space. Each Study contains a configuration describing the feasible space, as well as a set of Trials. It is assumed that
f
(
x
)
does not change in the course of a study,” wherein “a trial” encompasses a deep learning model and “a study” is an optimization process that evaluates a set of trials to find the best fit for a given task. Golovin, 3.3 Transfer Learning, pp. 1492, col. 2, paragraph 2; “Let
D
i
=
x
t
y
,
y
t
i
t
be the dataset for study
S
i
,” wherein said “dataset” encompasses a custom dataset.).
Golovin is analogous to the claimed invention as both are from the same field of endeavor, that is, methods for fine-tuning machine learning model parameters for use in cloud-based developer platforms. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the neural machine translation pre-training steps of Jiang and Kanade with the algorithm optimization system of Golovin. The motivation to do so is to create a secure and scalable distributed system with uses in a wide variety of machine learning applications that can handle a large number of client requests simultaneously (Golovin, 1. Introduction, pp. 1487, cols. 1 and 2; “It has many important applications, such as automated tuning of the hyperparameters of machine learning systems (e.g., learning rates, or the number of hidden layers in a deep neural network), optimization of the user interfaces of web services (e.g. optimizing colors and fonts to maximize reading speed), and optimization of physical systems (e.g., optimizing airfoils in simulation).” Golovin, 2.4.1 Parallel Processing of Suggestion Work, paragraph 1, pp. 3; “Each instance of the Suggestion Service potentially can generate suggestions for several Studies in parallel, giving us a massively scalable suggestion infrastructure.”).
Golovin further teaches build an automated fine-tuning infrastructure to perform the fine-tuning task that includes fine-tuning the select one of the plurality of pre-trained deep learning models with the custom dataset (Golovin, 2.5 The Algorithm Playground, pp. 1489-90; “Vizier’s algorithm playground provides a mechanism for advanced users to easily, quickly, and safely replace Vizier’s core optimization algorithms with arbitrary algorithms… In all cases, users of the playground benefit from all of Vizier’s infrastructure aside from the core algorithms, such as access to a persistent database of Trials, the dashboard, and visualizations,” wherein the “algorithm playground” encompasses an automated fine-tuning infrastructure and “optimization” is equivalent to fine-tuning.) without user configuration input, (Golovin, 2.1 Design Goals and Constraints, pp. 1488, col. 1, paragraph 9; “We provide a default configuration for our managed service that is good enough to ensure that most users need never concern themselves with the underlying optimization algorithms.”)
wherein the automated fine-tuning infrastructure restricts user access to parameters of the select one of the plurality of pre-trained deep learning models; and (Golovin, Abstract; “Google Vizier, a Google-internal service for performing black-box optimization that has become the de facto parameter tuning engine at Google. Google Vizier is used to optimize many of our machine learning models and other systems, and also provides core capabilities to Google’s Cloud Machine Learning HyperTune subsystem.” Golovin, 1 Introduction, pp. 1487, col. 1, paragraph 1; “The adjective ‘black–box’ means that while we can evaluate
f
(
x
)
for any
x
∈
X
, we have no access to any other information about
f
, such as gradients or the Hessian,” thereby restrict[ing] access to parameters wherein
"
f
"
is the mathematical abstraction of a machine learning model and “gradients” encompasses parameters. Golovin, 2.3 Interfaces, pp. 1489, col. 1, paragraph 1; “To configure a study, the user provides a study name, owner, optional access permissions, an optimization goal from {MAXIMIZE, MINIMIZE}, and specifies the feasible region X via a set of ParameterConfigs, each of which declares a parameter name along with its values,” wherein “optional access permission” encompasses restrict[ions] [of] user access to parameters in accordance with the description provided at paragraph [00022] of the specification of the claimed invention, “[a]s the cost of generating the pre-trained models becomes more expensive, there is an increased reluctance to share the models publicly which negatively impacts the advancement of machine learning. In order to address this concern and to allow sharing of the models, automated fine-tuning restricts access to certain internal data of the pre-trained models and the fine-tuned models while allowing the models to be used by others with access only to the results generated by the fine-tuned model.”)
perform the fine-tuning task by fine-tuning the select one of the plurality of pre-trained deep learning models with the custom dataset using the automated fine-tuning infrastructure to generate a custom version of the select one of the plurality of pre-trained deep learning models (Golovin, 3.3 Transfer Learning, pp. 1492, col. 1, paragraph 2; “When doing black-box optimization, users often run studies that are similar to studies they have run before, and we can use this fact to minimize repeated work,” thereby using transfer learning, or pre-trained deep learning models, to fine-tune a given model. Golovin, Algorithm 1; Here, “Dtraining” corresponds to the custom dataset and “return StackedRegressor” corresponds to generate a custom version of the select one of the plurality of pre-trained deep learning models).
PNG
media_image2.png
371
348
media_image2.png
Greyscale
Regarding the amendments to the claims, Golovin discloses the claimed custom dataset and performing a fine-tuning task. Roitman teaches an embedding model is generated with one or more embeddings, the word embeddings may also reflect a size of a vocabulary of a respective text corpus fed into an embedding model, because a vector may be kept for each word in the vocabulary of the text corpus that is input (P 0031) different users of text with semi-structured content will usually have a unique vocabulary set for text that is in-between the tags, so training the model for a specific customer by feeding in previous semi-structured content from that specific customer can be valuable (P 0074) and new embedding layer may be implemented that indicates to which segment of the segment pair a word or word piece belongs, for example, this segment embedding layer may have two values for a segment pair, the training for the bi-directional transformer model with the next-segment prediction may include feeding some segment pairs with a correct sequence pairing from the text corpus, while other segment pairs are input while having random sequences paired together (P 0038, 0058-0060). That is, the since segment word or word piece can be considered to have been not fined tuned prior to determining to which segment of the segment word or word piece belongs, then the new embedding layer that makes this determination is applied prior to fine-tuning. When combined with Jiang, the disclosure of Roitman directed to the new imbedding layer implemented to indicate to which segment of the segment word or word piece belongs indicates occurs prior to definition of the segments, i.e. prior to fine-tuning.
Jiang does not explicitly teach wherein performing the fine-tuning task includes: determining, based on the custom dataset for the fine-tuning and prior to fine-tuning that the fine-tuning task requires a different vocabulary from a vocabulary of the select one pre-trained deep learning model; and in response to that determination, refraining from transferring an embedding layer of the select one pretrained deep learning model for fine-tuning and instead randomly initializing a different embedding layer for the select one pre-trained deep learning model.
However, in the same field of invention, Roitman teaches an embedding model is generated with one or more embeddings, the word embeddings may also reflect a size of a vocabulary of a respective text corpus fed into an embedding model, because a vector may be kept for each word in the vocabulary of the text corpus that is input (P 0031) new embedding layer may be implemented that indicates to which segment of the segment pair a word or word piece belongs, for example, this segment embedding layer may have two values for a segment pair, the training for the bi-directional transformer model with the next-segment prediction may include feeding some segment pairs with a correct sequence pairing from the text corpus, while other segment pairs are input while having random sequences paired together (P 0038, 0058-0060) DOM node embeddings and DOM level embeddings for the text corpus are randomly initialized, the DOM node embeddings and the DOM level embeddings are based on the nodes/hierarchy tags (P 0040) a DOM (node and level) embedding is generated from a (first) text corpus (P 0043) based on certain tokens a DOM node embedding is replaced (P 0057-0060) different users of text with semi-structured content will usually have a unique vocabulary set for text that is in-between the tags, so training the model for a specific customer by feeding in previous semi-structured content from that specific customer can be valuable (P 0074) from multiple text corpuses (P 0077). Therefore, considering the teachings of Jiang, Kanade, Golovin, Roitman, one having ordinary skill in the art before the effective filing date of the invention would have been motivated to determining, based on the custom dataset for the fine-tuning and prior to fine-tuning that the fine-tuning task requires a different vocabulary from a vocabulary of the select one pre-trained deep learning model; and in response to that determination, refraining from transferring an embedding layer of the select one pretrained deep learning model for fine-tuning and instead randomly initializing a different embedding layer for the select one pre-trained deep learning model with the teachings of Jiang, Kanade, Golovin with the motivation to aid computer systems in solving ambiguous Roitman (P 0001).
Regarding claim 2, the combination of Jiang, Kanade, Golovin, Roitman teaches the system of claim 1, wherein the one or more programs include instructions to perform acts that: (and thus the rejection of claim 1 is incorporated).
Golovin further teaches build an automated deployment infrastructure to deploy the custom version of the select one of the plurality of pre-trained deep learning models (Golovin, Figure 1; The figure illustrates how “trials” or pre-trained deep learning models are deploy[ed] using the “Vizier API.” This “API” encompasses all the elements of an automated deployment infrastructure according to the description given in the specification of the claimed invention at [00025], “The deployment infrastructure includes a deployment script, the model files of the fine-tuned model, a model endpoint, and the automatic configuration of the virtual machines needed to operate the model.” “The deployment script” is code inherently necessary to executing to the model, “the model files of the fine-tuned model” are passed to the “Vizier API” from “Persistent Database,” the “model endpoint” is included in the “Suggestion Service” (Golovin, 2.4.1 Parallel Processing of Suggestion Work, pp. 1489, col. 2, paragraph 2; “Google’s load balancing infrastructure is then used to allow clients to make calls to a unified endpoint, without needing to know which instance is doing the work.”), and finally “the automatic configuration of the virtual machines needed to operate the model” is handled by the “Vizier API” wherein the “Evaluation Workers” are the “virtual machines needed to operate the model” (Golovin, 1.2 Definitions, pp. 1488, col. 1, paragraph 6; “A Worker refers to a process,” or machine, “responsible for evaluating a Pending Trial,” or model, “and calculating its objective value.”)
PNG
media_image3.png
213
315
media_image3.png
Greyscale
without user configuration input, (Golovin, 2.1 Design Goals and Constraints, pp. 1488, col. 1, paragraph 9; “We provide a default configuration for our managed service that is good enough to ensure that most users need never concern themselves with the underlying optimization algorithms.”) wherein the automated deployment infrastructure restricts access to parameters of the custom version of the select one of the plurality of pre-trained deep learning models (Golovin, Abstract; “Google Vizier, a Google-internal service for performing black-box optimization that has become the de facto parameter tuning engine at Google. Google Vizier is used to optimize many of our machine learning models and other systems, and also provides core capabilities to Google’s Cloud Machine Learning HyperTune subsystem.” Golovin, 1 Introduction, pp. 1487, col. 1, paragraph 1; “The adjective ‘black–box’ means that while we can evaluate
f
(
x
)
for any
x
∈
X
, we have no access to any other information about
f
, such as gradients or the Hessian,” thereby restrict[ing] access to parameters. Golovin, 2.3.1 Configuring a Study, pp. 1489, col. 1, paragraph 1; “To configure a study, the user provides a study name, owner, optional access permissions, an optimization goal from {MAXIMIZE, MINIMIZE}, and specifies the feasible region X via a set of ParameterConfigs, each of which declares a parameter name along with its values,” wherein “optional access permission” encompasses restrict[ions] [of] user access to parameters).
Regarding claim 3, the combination of Jiang, Kanade, Golovin, Roitman teaches the system of claim 2, wherein the one or more programs include further instructions that: (and thus the rejection of claim 2 is incorporated).
Golovin further teaches run the custom version of the select one of the plurality of the pre-trained deep learning models using the automated deployment infrastructure with custom inference data (Golovin, 3.3 Transfer Learning, pp. 1492, col. 1, paragraph 2; “Vizier supports a form of Transfer Learning which leverages data from prior studies to guide and accelerate the current study. For instance, one might tune the learning rate and regularization of a machine learning system, then use that Study as a prior to tune the same ML system on a different data set,” wherein “leverages data from prior studies” indicates the deep learning model in question is pre-trained and is being deployed again for further tuning “on a different data set,” or custom inference dataset. Note that “Vizier” itself encompasses the fine-tuning infrastructure and automated deployment infrastructure outlined previously.).
Regarding claim 4, the combination of Jiang, Kanade, Golovin, Roitman teaches the system of claim 1, wherein (and thus the rejection of claim 1 is incorporated).
Golovin further teaches each of the pre-trained deep learning models includes an environment definition file, (Golovin, 2.2 Basic User Workflow, pp. 1488, col. 2, paragraph 4; “To use Vizier, a developer may use one of our client libraries (currently implemented in C++, Python, Golang), which will generate service requests encoded as protocol buffers,” wherein “client libraries” encompasses an environment definition file per the definition provided in the specification of the claimed invention at [00052], “An environment definition file includes source code language specific packages that are needed to execute the model.”) a fine-tuning script, (Golovin, 2.4.1 Parallel Processing of Suggestion Work, pp. 1489, col. 1, paragraph 2; “As the de facto parameter tuning engine of Google, Vizier is constantly working on generating suggestions for a large number of Studies concurrently,” wherein a “tuning engine” inherently encompasses a fine-tuning script) a deployment script, (Inherent in the architecture of Golovin’s “Vizier”) and a tokenizer (Jiang, I. Introduction, A. Our Approach, pp. 3, col. 1, paragraph 1; “Thus, we use byte-pair encoding (BPE), a type of subword tokenization techniques, to tokenize compound words and rare words to further address the OOV problem.”).
Regarding claim 5, the combination of Jiang, Kanade, Golovin, Roitman teaches the system of claim 1, wherein (and thus the rejection of claim 1 is incorporated).
Jiang further teaches the plurality of pre-trained deep learning models includes an encoder neural transformer with attention, a decoder neural transformer with attention, and/or encoder-decoder neural transformer with attention (Jiang, Fig. 4; Jiang, III. Approach, D. Programming Language Model, paragraph 2, pp. 5; “We use Generative Pre-trained Transformer (GPT) for PL modeling,” the “GPT” encompassing an encoder-decoder neural transformer with attention.).
PNG
media_image4.png
270
365
media_image4.png
Greyscale
Regarding claim 6, the combination of Jiang, Kanade, Golovin, Roitman teaches the system of claim 1, wherein (and thus the rejection of claim 1 is incorporated).
Golovin further teaches the automated fine-tuning infrastructure includes one or more virtual machines, a virtual operating system, and tools and/or packages for the one or more virtual machines (Golovin, 1.2 Definitions, pp. 1488, col. 1, paragraph 6; “A Worker refers to a process responsible for evaluating a Pending Trial,” wherein “a worker” in the context of a cloud-based API such as “Vizier” encompasses one or more virtual machines.).
Regarding claim 7, the combination of Jiang, Kanade, Golovin, Roitman teaches the system of claim 1, wherein (and thus the rejection of claim 1 is incorporated).
Jiang further teaches the automated fine-tuning infrastructure includes a pre-processing engine to process the custom dataset for the select one of the pre-trained deep learning models (Jiang, III. Approach, A. Overview, paragraph 1, pp. 3; “CURE also tokenizes the buggy lines, context, and correct fixes extracted from commit history of open-source projects, referred to as patch training data, into sequences of tokens,” this process encompassing pre-processing per the definition provided in the specification of the claimed invention at [00050], “The pre-processing may include removing comments from source code files, removing indent characters, and/or generating input sequences of tokens.”).
Regarding claim 8, the combination of Jiang, Kanade, Golovin, Roitman teaches the system of claim 1, wherein (and thus the rejection of claim 1 is incorporated).
The combination further teaches the parameters include weights, biases, and/or embeddings (Jiang, II. Background, Pre-Training and Fine-Tuning, pp. 3; “Parameters are the weights between the connections of the network.”).
Regarding claim 9, the combination of Jiang, Kanade, and Golovin teaches a method performed on a computing device comprising a processor and a memory, the method comprising: (Jiang, IV. Experimental Setup, Infrastructure, pp. 8; “We train and evaluate our models on one 56-core server with one NVIDIA TITAN V and three Xp GPUs,” wherein “three Xp GPUs” corresponds to a processor and “one 56-core server” necessarily includes a memory).
offering a plurality of pre-trained deep learning models to a user for reuse, (Jiang, Fig. 2, elements 2 and 3, “Pre-Training” and “APR models.” Jiang, II. Background, A. Overview, pp. 3; “Different from previous work, we use subword tokenization, which produces a smaller but more accurate search space that contains more correct patches. CURE uses these tokenized methods to train a new programming language model that learns developer-like source code with correct syntax (step 2),” this step encompassing the “pre-training.” “CURE also tokenizes the buggy lines, context, and correct fixes extracted from the commit history of open-source projects, referred to as patch training data, into sequences of tokens (step 1b). We use these sequences to fine-tune an ensemble of k APR models (step 3). Each APR model combines the PL model with a context-aware neural machine translation (CoNuT) model.”).
Jiang does not explicitly teach wherein the plurality of pre-trained deep learning models is pre-trained with unsupervised training datasets of source code and natural language text. However, Kanade, in the area of code-understanding natural language models, teaches this limitation (Kanade, 2 Related Work, pp. 3, paragraph 2; “We also handle natural language directly, but do not require such a separation. Natural-language tokens can be mixed with source-code tokens both within and across sentences in our encoding.” Kanade, 4.4 Is Transformer All You Need?, pp. 7; “One may wonder if CuBERT’s promising results derive more from using a Transformer-based model for its classification tasks, and less from the actual, unsupervised pre-training,” thereby indicating that “CuBERT” is pre-trained on unsupervised training data.).
Jiang and Kanade are analogous to the claimed invention as both are from the same field of endeavor, that is, developing natural language processing models for neural machine translation tasks. Jiang already teaches a model pre-trained on unlabeled source code data but not natural language text. Kanade’s model is pre-trained with both. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to modify the pre-training step in Jiang to include natural language data, as done in Kanade. Doing so would yield the predictable result of a pre-trained model capable of performing a given software engineering task such as automatic program repair or automatic code generation.
for software engineering tasks; (Jiang, Abstract; “First, CURE pre-trains a programming language (PL) model on a large software codebase to learn developer-like source code before the APR task.”).
The combination of Jiang and Kanade does not explicitly teach providing a plurality of model files for each of the plurality of pre-trained deep learning models, wherein the plurality of model files is isolated from the user. However, Golovin, in the area of black-box optimization for parameter tuning, teaches these limitation (Golovin, Abstract; “Google Vizier, a Google-internal service for performing black-box optimization that has become the de facto parameter tuning engine at Google. Google Vizier is used to optimize many of our machine learning models and other systems, and also provides core capabilities to Google’s Cloud Machine Learning HyperTune subsystem.” Golovin, 1 Introduction, pp. 1487, col. 1, paragraph 1; “The adjective ‘black–box’ means that while we can evaluate
f
(
x
)
for any
x
∈
X
, we have no access to any other information about
f
, such as gradients or the Hessian,” thereby isolat[ing] from the user information such as model parameters, which are necessarily stored in model files, wherein
"
f
"
is the mathematical abstraction of a machine learning model. Golovin, 1.2 Definitions, pp. 1488, col. 1, paragraph 4; “A Trial is a list of parameter values,
x
, that will lead to a single evaluation of
f
(
x
)
,” wherein “a trial” in the context of parameter tuning encompasses a model. A model, in the context of cloud-based parameter tuning, is necessarily stored in a plurality of model files.).
Golovin is analogous to the claimed invention as both are from the same field of endeavor, that is, methods for fine-tuning machine learning model parameters for use in cloud-based developer platforms. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the neural machine translation pre-training steps of Jiang and Kanade with the black-box optimization of Golovin. The motivation to do so is to give users of a cloud-based developer platform access to a number of powerful pre-trained machine learning models without revealing sensitive parameter information (Golovin, 2.5 The Algorithm Playground, pp. 3-4; “The playground serves a dual purpose; it allows rapid prototyping of new algorithms, and it allows power-users to easily customize Vizier with advanced or exotic capabilities that are particular to their use-case. In all cases, users of the playground benefit from all of Vizier’s infrastructure aside from the core algorithms, such as access to a persistent database of Trials, the dashboard, and visualizations,” wherein the “Trials” are defined as follows. Golovin, 1.2 Definitions, pp. 2; “A Trial is a list of parameter values,
x
, that will lead to a single evaluation of
f
(
x
)
.”).
Golovin further teaches wherein a first subset of the plurality of model files is for fine-tuning (Golovin, 2.4.1 Parallel Processing of Suggestion Work, pp. 1489, col. 2, paragraph 2; “As the de facto parameter tuning engine of Google, Vizier is constantly working on generating suggestions for a large number of Studies concurrently,” wherein a “tuning engine” inherently encompasses model files for fine-tuning) and a second subset of the plurality of model files is for deployment; (Inherent in the architecture of Golovin’s “Vizier”)
receiving a user request to reuse a select one of the plurality of pre-trained deep learning models; (Golovin, 2.4.1 Parallel Processing of Suggestion Work, pp. 1489, col. 2, paragraph 3; “When a request is received by a Suggestion Service instance to generate suggestions, the instance first places a distributed lock on the Study.”)
building an automated fine-tuning infrastructure to generate a custom version of a select one of the pre-trained deep learning models (Golovin, 2.5 The Algorithm Playground, pp. 1489-90; “Vizier’s algorithm playground provides a mechanism for advanced users to easily, quickly, and safely replace Vizier’s core optimization algorithms with arbitrary algorithms… In all cases, users of the playground benefit from all of Vizier’s infrastructure aside from the core algorithms, such as access to a persistent database of Trials, the dashboard, and visualizations,” wherein the “algorithm playground” encompasses an automated fine-tuning infrastructure and “optimization” is equivalent to fine-tuning.) without user configuration input, (Golovin, 2.1 Design Goals and Constraints, pp. 1488, col. 1, paragraph 9; “We provide a default configuration for our managed service that is good enough to ensure that most users need never concern themselves with the underlying optimization algorithms.”) wherein the automated fine-tuning infrastructure is built using the first subset of the model files, (Golovin, 2.4.1 Parallel Processing of Suggestion Work, pp. 1489, col. 2, paragraph 2; “As the de facto parameter tuning engine of Google, Vizier is constantly working on generating suggestions for a large number of Studies concurrently,” wherein a “tuning engine” inherently encompasses model files for fine-tuning, or the first subset of the model files) wherein the automated fine-tuning infrastructure restricts user access to parameters of the select one of the pre-trained deep learning models; and (Golovin, Abstract; “Google Vizier, a Google-internal service for performing black-box optimization that has become the de facto parameter tuning engine at Google. Google Vizier is used to optimize many of our machine learning models and other systems, and also provides core capabilities to Google’s Cloud Machine Learning HyperTune subsystem.” Golovin, 1 Introduction, pp. 1487, col. 1, paragraph 1; “The adjective ‘black–box’ means that while we can evaluate
f
(
x
)
for any
x
∈
X
, we have no access to any other information about
f
, such as gradients or the Hessian,” thereby restrict[ing] access to parameters wherein
"
f
"
is the mathematical abstraction of a machine learning model and “gradients” encompasses parameters. Golovin, 2.3.1 Configuring a Study, pp. 1489, col. 1, paragraph 1; “To configure a study, the user provides a study name, owner, optional access permissions, an optimization goal from {MAXIMIZE, MINIMIZE}, and specifies the feasible region X via a set of ParameterConfigs, each of which declares a parameter name along with its values,” wherein “optional access permission” encompasses restrict[ions] [of] user access to parameters in accordance with the description provided at paragraph [00022] of the specification of the claimed invention, “[a]s the cost of generating the pre-trained models becomes more expensive, there is an increased reluctance to share the models publicly which negatively impacts the advancement of machine learning. In order to address this concern and to allow sharing of the models, automated fine-tuning restricts access to certain internal data of the pre-trained models and the fine-tuned models while allowing the models to be used by others with access only to the results generated by the fine-tuned model.”).
building an automated deployment infrastructure to deploy the custom version of the select one of the pre-trained deep learning models (Golovin, Figure 1; The figure illustrates how “trials” or pre-trained deep learning models are deploy[ed] using the “Vizier API.” This “API” encompasses all the elements of an automated deployment infrastructure according to the description given in the specification of the claimed invention at [00025], “The deployment infrastructure includes a deployment script, the model files of the fine-tuned model, a model endpoint, and the automatic configuration of the virtual machines needed to operate the model.” “The deployment script” is code inherently necessary to executing to the model, “the model files of the fine-tuned model” are passed to the “Vizier API” from “Persistent Database,” the “model endpoint” is included in the “Suggestion Service” (Golovin, 2.4.1 Parallel Processing of Suggestion Work, pp. 1489, col. 2, paragraph 2; “Google’s load balancing infrastructure is then used to allow clients to make calls to a unified endpoint, without needing to know which instance is doing the work.”), and finally “the automatic configuration of the virtual machines needed to operate the model” is handled by the “Vizier API” wherein the “Evaluation Workers” are the “virtual machines needed to operate the model” (Golovin, 1.2 Definitions, pp. 1488, col. 1, paragraph 6; “A Worker refers to a process,” or machine, “responsible for evaluating a Pending Trial,” or model, “and calculating its objective value.”).) without user configuration input, (Golovin, 2.1 Design Goals and Constraints, pp. 1488, col. 1, paragraph 9; “We provide a default configuration for our managed service that is good enough to ensure that most users need never concern themselves with the underlying optimization algorithms.”) wherein the automated deployment infrastructure is built using the second subset of the model files; (Inherent in the architecture of Golovin’s “Vizier”).
Regarding the amendments to the claims, Golovin discloses the claimed custom dataset and performing a fine-tuning task. Roitman teaches an embedding model is generated with one or more embeddings, the word embeddings may also reflect a size of a vocabulary of a respective text corpus fed into an embedding model, because a vector may be kept for each word in the vocabulary of the text corpus that is input (P 0031) different users of text with semi-structured content will usually have a unique vocabulary set for text that is in-between the tags, so training the model for a specific customer by feeding in previous semi-structured content from that specific customer can be valuable (P 0074) and new embedding layer may be implemented that indicates to which segment of the segment pair a word or word piece belongs, for example, this segment embedding layer may have two values for a segment pair, the training for the bi-directional transformer model with the next-segment prediction may include feeding some segment pairs with a correct sequence pairing from the text corpus, while other segment pairs are input while having random sequences paired together (P 0038, 0058-0060). That is, the since segment word or word piece can be considered to have been not fined tuned prior to determining to which segment of the segment word or word piece belongs, then the new embedding layer that makes this determination is applied prior to fine-tuning. When combined with Jiang, the disclosure of Roitman directed to the new imbedding layer implemented to indicate to which segment of the segment word or word piece belongs indicates occurs prior to definition of the segments, i.e. prior to fine-tuning.
Jiang does not disclose wherein the automated fine-tuning infrastructure performs a fine-tuning task that includes: determining, based on the custom dataset for the fine-tuning and prior to the fine-tuning, that the fine-tuning task requires a different vocabulary from a vocabulary of the select one pre-trained deep learning model; and in response to that determination, refraining from transferring an embedding layer of the select one pretrained deep learning model for fine-tuning and instead randomly initializing a different embedding layer for the select one pre-trained deep learning model. However, in the same field of invention, Roitman teaches an embedding model is generated with one or more embeddings, the word embeddings may also reflect a size of a vocabulary of a respective text corpus fed into an embedding model, because a vector may be kept for each word in the vocabulary of the text corpus that is input (P 0031) new embedding layer may be implemented that indicates to which segment of the segment pair a word or word piece belongs, for example, this segment embedding layer may have two values for a segment pair, the training for the bi-directional transformer model with the next-segment prediction may include feeding some segment pairs with a correct sequence pairing from the text corpus, while other segment pairs are input while having random sequences paired together (P 0038, 0058-0060) DOM node embeddings and DOM level embeddings for the text corpus are randomly initialized, the DOM node embeddings and the DOM level embeddings are based on the nodes/hierarchy tags (P 0040) a DOM (node and level) embedding is generated from a (first) text corpus (P 0043) based on certain tokens a DOM node embedding is replaced (P 0057-0060) different users of text with semi-structured content will usually have a unique vocabulary set for text that is in-between the tags, so training the model for a specific customer by feeding in previous semi-structured content from that specific customer can be valuable (P 0074) from multiple text corpuses (P 0077). Therefore, considering the teachings of Jiang, Kanade, Golovin, Roitman, one having ordinary skill in the art before the effective filing date of the invention would have been motivated to wherein the automated fine-tuning infrastructure performs a fine-tuning task that includes: determining, based on the custom dataset for the fine-tuning and prior to the fine-tuning, that the fine-tuning task requires a different vocabulary from a vocabulary of the select one pre-trained deep learning model; and in response to that determination, refraining from transferring an embedding layer of the select one pretrained deep learning model for fine-tuning and instead randomly initializing a different embedding layer for the select one pre-trained deep learning model with the teachings of Jiang, Kanade, Golovin with the motivation to aid computer systems in solving ambiguous Roitman (P 0001).
Regarding claim 10, the combination of Jiang, Kanade, Golovin and Roitman teaches the method of claim 9, further comprising: (and thus the rejection of claim 9 is incorporated).
Golovin further teaches performing the fine-tuning task by fine-tuning the select one of the pre-trained deep learning models using the automated fine-tuning infrastructure with the custom tuning dataset to produce the custom version of the select one of the pre-trained deep learning models (Golovin, 3.3 Transfer Learning, pp. 1492, col. 1, paragraph 2; “When doing black-box optimization, users often run studies that are similar to studies they have run before, and we can use this fact to minimize repeated work,” thereby using transfer learning, or pre-trained deep learning models,) for fine-tuning a given model. Golovin, Algorithm 1; Here, “Dtraining” corresponds to the custom tuning dataset and “return StackedRegressor” corresponds to produce the custom version of the select one of the plurality of pre-trained deep learning models).
Regarding claim 11, the combination of Jiang, Kanade, Golovin and Roitman teaches the method of claim 10, further comprising: (and thus the rejection of claim 10 is incorporated).
Golovin further teaches upon successful completion of fine-tuning the select one of the pre-trained deep learning models, deploying the custom version of the select one of the pre-trained deep learning models in the automated deployment infrastructure (Golovin, 2.6 Benchmarking Suite, pp. 1490, col. 1, paragraph 5; “Vizier has an integrated framework that allows us to efficiently benchmark our algorithms on a variety of objective functions,” wherein “to efficiently benchmark” encompasses evaluating and thus deploying the custom version of the select one of the pre-trained deep learning models tuned by the user in “Vizier,” the automated deployment infrastructure.).
Regarding claim 12, the combination of Jiang, Kanade, Golovin and Roitman teaches the method of claim 11, further comprising: (and thus the rejection of claim 11 is incorporated).
Golovin further teaches generating a model endpoint to receive requests to execute the custom version of the select one of the pre-trained deep learning models (Golovin, 2.4.1 Parallel Processing of Suggestion Work, pp. 1489, col. 2, paragraph 2; “Each instance of the Suggestion Service potentially can generate suggestions for several Studies in parallel, giving us a massively scalable suggestion infrastructure. Google’s load balancing infrastructure is then used to allow clients to make calls to a unified endpoint, without needing to know which instance is doing the work,” wherein “doing the work” denotes execut[ing] the custom version of the select one of the pre-trained deep learning models) with an inference dataset (Golovin, 3.3 Transfer Learning, pp. 1492, col. 2, paragraph 2; Let
D
i
=
x
t
y
,
y
t
i
t
be the dataset for study
S
i
,” wherein said “dataset” encompasses an inference dataset.).
Claim 13 is a method claim corresponding to the steps of claim 5, and is therefore rejected for the same reasons as claim 5.
Claim 14 is an article of manufacture claim corresponding to the steps of claim 9, and is therefore rejected for the same reasons as claim 9. The additional software element, a cloud platform (Golovin, Abstract; “Google Vizier is used to optimize many of our machine learning models and other systems, and also provides core capabilities to Google’s Cloud Machine Learning HyperTune subsystem.”) and hardware elements, a plurality of services, wherein each of the plurality of services includes a processor and a memory, (Jiang, IV. Experimental Setup, Infrastructure, pp. 8; “We train and evaluate our models on one 56-core server with one NVIDIA TITAN V and three Xp GPUs,” wherein “three Xp GPUs” corresponds to a processor and “one 56-core server” necessarily includes a memory) wherein the plurality of services includes a model catalog service, (Jiang, Fig. 2, element 3; “APR Models”) a data management service, (Golovin, 2.4.1 Parallel Processing of Suggestion Work, pp. 1489, col. 2, paragraph 2; “As the de facto parameter tuning engine of Google, Vizier is constantly working on generating suggestions for a large number of Studies concurrently. As such, a single machine would be insufficient for handling the workload. Our Suggestion Service is therefore partitioned across several Google datacenters, with a number of machines being used in each one…When a request is received by a Suggestion Service instance to generate suggestions, the instance first places a distributed lock on the Study,” this system encompassing a data management service per the explanation given in the specification of the claimed invention at [0028], “The data management service 106 receives registration requests for usage of a pre-trained model.”) a model fine-tuning service, (Golovin, 2.5 The Algorithm Playground, pp. 1489, col. 2, paragraph 5; “Vizier’s algorithm playground provides a mechanism for advanced users to easily, quickly, and safely replace Vizier’s core optimization algorithms with arbitrary algorithms”) and a model deployment service (Golovin, Figure 1; “Architecture of Vizier service.”).
Claim 15 is an article of manufacture claim corresponding to the steps of claim 4, and is therefore rejected for the same reasons as claim 4.
Regarding claim 16, the combination of Jiang, Kanade, Golovin and Roitman teaches the cloud platform of claim 14, wherein (and thus the rejection of claim 14 is incorporated).
Jiang further teaches the plurality of pre-trained deep learning models are configured for a software classification task, (Jiang, I. Introduction, A. Our Approach, (1) Programming language models, “Then one fine-tunes the pre trained language model for a specific task by adding an extra model to it (e.g., adding a classifier for classification tasks).”) a software translation task, (Jiang, Abstract; “We propose CURE, a new NMT-based APR technique with three major novelties,” wherein “NMT,” or neural machine translation, encompasses a software translation task.) and an autoregressive software task (Jiang, II. Background, Context-aware neural machine translation (CoNuT) architecture, pp. 3; “We use CoNuT as our NMT architecture in this paper. CoNuT consists of a buggy lines encoder, a context encoder, a merger, a decoder, an attention module, and a token generation module, where the encoders and decoder are implemented with convolutional sequence-to-sequence architecture,” wherein “sequence-to-sequence” is a popular architecture for encoder-decoder as it processes tokens in a sequential, or autoregressive, manner.).
Claim 17 is an article of manufacture claim corresponding to the steps of claim 5, and is therefore rejected for the same reasons as claim 5.
Regarding claim 18, the combination of Jiang, Kanade, Golovin and Roitman teaches the cloud platform of claim 14, wherein the processor of the model deployment service is configured to perform acts that: (and thus the rejection of claim 14 is incorporated).
Golovin further teaches run the deployment infrastructure to deploy the fine-tuned model and generate a model endpoint for the fine-tuned model (Golovin, 2.6 Benchmarking Suite, pp. 1409, col. 1, paragraph 5; “Vizier has an integrated framework that allows us to efficiently benchmark our algorithms on a variety of objective functions,” wherein “to efficiently benchmark” encompasses evaluation and thus deployment of a fine-tuned model. Golovin, 2.4.1 Parallel Processing of Suggestion Work, pp. 1489, col. 2, paragraph 2; “Google’s load balancing infrastructure is then used to allow clients to make calls to a unified endpoint, without needing to know which instance is doing the work,” thereby generat[ing] a model endpoint.).
Regarding claim 19, the combination of Jiang, Kanade, Golovin and Roitman teaches the cloud platform of claim 14, wherein (and thus the rejection of claim 14 is incorporated).
Jiang further teaches the plurality of services includes a model execution service, having a processor and a memory, wherein the processor of the execution service is configured to execute the deployed model with an inference dataset (Jiang, IV. Experimental Setup, Infrastructure, pp. 8; “We train and evaluate our models on one 56-core server with one NVIDIA TITAN V and three Xp GPUs,” wherein “three Xp GPUs” corresponds to a processor, “one 56-core server” necessarily includes a memory, and “evaluate” is equivalent to execute. Jiang, V. Evaluation and Results, paragraph 1, pp. 8; “We use two widely-used benchmarks, Defects4J (v1.4.0) [38] and QuixBugs [47] for evaluation. Following [19], we remove two Defects4J bugs, Closure 63 and Closure 93, from our evaluation as they are duplicates of other Defects4J bugs. We compile the patched projects and run the test suites to find plausible patches, i.e., patches that pass the relevant test cases,” wherein these steps encompass the functions of a model execution service as outlined in the specification of the claimed invention at [000136], “The model execution service pre-processes the inference dataset in a similar manner as noted above (block 1106) and runs the model with the processed inference dataset (block 1108).” Removing the duplicates is a pre-processing step and “compile” and “run” encompass execution.).
Regarding claim 20, the combination of Jiang, Kanade, Golovin and Roitman teaches the cloud platform of claim 19, wherein the processor of the model execution service
is configured to: (and thus the rejection of claim 19 is incorporated).
Jiang further teaches pre-process the inference dataset into a form used by the deployed model; (Jiang, V. Evaluation and Results, paragraph 1, pp. 8; “We use two widely-used benchmarks, Defects4J (v1.4.0) [38] and QuixBugs [47] for evaluation. Following [19], we remove two Defects4J bugs, Closure 63 and Closure 93, from our evaluation as they are duplicates of other Defects4J bugs.”) and execute the deployed model with the pre-processed inference dataset (Jiang, V. Evaluation and Results, paragraph 1, pp. 8; “We compile the patched projects and run the test suites to find plausible patches, i.e., patches that pass the relevant test cases.”).
Response to Arguments
Applicant's arguments filed 3/31/2026 have been fully considered but they are not persuasive.
The applicant argues:
[N]o combination of the cited art teaches or suggests any embodiment that determines, based on a custom dataset for fine-tuning and prior to the fine- tuning, that a fine-tuning task requires a different vocabulary from a vocabulary of the selected pre-trained deep learning model and that then refrains from transferring an embedding layer, thereby conditioning model adaptation on vocabulary compatibility within a controlled infrastructure that restricts parameter access. Applicant notes that the current amendments more fully clarify aspects related to the previous amendments made. Applicant also notes how the Office relied primarily on the Roitman reference to reject the previous amendments.
Regarding the amendments to the claims, Golovin discloses the claimed custom dataset and performing a fine-tuning task. Roitman teaches an embedding model is generated with one or more embeddings, the word embeddings may also reflect a size of a vocabulary of a respective text corpus fed into an embedding model, because a vector may be kept for each word in the vocabulary of the text corpus that is input (P 0031) different users of text with semi-structured content will usually have a unique vocabulary set for text that is in-between the tags, so training the model for a specific customer by feeding in previous semi-structured content from that specific customer can be valuable (P 0074) and new embedding layer may be implemented that indicates to which segment of the segment pair a word or word piece belongs, for example, this segment embedding layer may have two values for a segment pair, the training for the bi-directional transformer model with the next-segment prediction may include feeding some segment pairs with a correct sequence pairing from the text corpus, while other segment pairs are input while having random sequences paired together (P 0038, 0058-0060). That is, the since segment word or word piece can be considered to have been not fined tuned prior to determining to which segment of the segment word or word piece belongs, then the new embedding layer that makes this determination is applied prior to fine-tuning. When combined with Jiang, the disclosure of Roitman directed to the new imbedding layer implemented to indicate to which segment of the segment word or word piece belongs indicates occurs prior to definition of the segments, i.e. prior to fine-tuning.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN M HEFFINGTON whose telephone number is 571-270-1696. The examiner can normally be reached 9:30 am to 5:30 pm Eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar B Paula can be reached at 571-272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.M.H/Examiner, Art Unit 2145 6/2/2026
/CESAR B PAULA/Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights