SYSTEM AND METHOD FOR LIGHTWEIGHT SEMANTIC MASKING

Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . DETAILED ACTION This action is in response to the application filed 03 June 2026. Claims filed 01 May 2026 have been examined. Claims 1, 8, and 15 are amended. Claims 1-20 are pending and have been examined. Continued Examination Under 37 CFR 1.114 A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 03 June 2026 has been entered. Response to Arguments Applicant' s arguments, filed 03 June 2026, see pages 11-14, with respect to the rejections of Claims 1-20 under 35 USC 103 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. APPLICANT'S ARGUMENT: Applicant argues (page 13, paragraph 2) that "the semantic pruning model recited in Claim I cannot be all of the Original Model, Feature Diagnosis, and Pruned Model of Chen." Applicant argues (page 14, paragraph 1) that "Chen simply does not disclose or suggest the claim language regarding building layers of a semantic probing model. The original model in Chen is merely pruned to create a pruned model. The original model in Chen is merely pruned to create a pruned model. ... Claim 1 specifically recites that one or more hidden layers are built within the claimed semantic probing model." EXAMINER'S RESPONSE: Examiner notes that applicant's arguments are moot. Amended Claim1 is now rejected under 35 U.S.C. 103 as being unpatentable over Chen in view of Yang in view of Alain in view of Montavon. In the rejection below, Chen's ensemble of linear classifiers is shown to teach a probing model under BRI in light of the specification (for example, instant specification [0049]: "In the semantic probing process 230, the electronic device 101 targets the hidden layers 310 and generates an equal number (such as twelve) of small semantic probing models 305 to extract a set of contextual vectors 315 from the hidden layers 310"). Chen's technique of pruning is shown to teach reconstruction of the original model under BRI in light of the specification (for example, instant specification [0056]: "To generate the reconstructed model 245, the electronic device 101 copies the pre-trained model 215 except for the hidden layers 310 that are re-processed by the semantic probing process 230, and the electronic device 101 adds the selected subset of hidden layers 310 and corresponding parameters to the copied pre-trained model 215."). In particular, Chen describes the pruning method as constructing a model, which teaches or reasonably suggests the instant reconstructing under BRI (Chen, p. 3050, 3 Feature Representation Based Parameter Pruning: "We focus on pruning convolutional layers with relatively small contributions to increasing the performance of feature representations. We then construct compact networks without the pruned layers, which achieves equivalent performance as the original complex models with transfer learning" and p. 3051, 3.2 Knowledge Transfer via Distillation: "we have obtained networks with a more compact architecture by reconstructing deep networks"). The Chen/Yang/Alain/Montavon combination appears to teach the semantic probing model of amended Claim 1 under BRI in light of the specification. Amended Claim 1 recites a semantic probing model that is different from the pre-trained model, and further claims hidden, probing, output activation layers as structure of the probing model. At [0049] and Fig. 3A, 305 of the instant specification, the probing model appears to be described in such a manner as to comprise the layers as well as components also described as part of the larger computing environment and activity of the device, for example, instant specification at [0049]: "The electronic device 101 obtains training data from a semantic database 335, which contains textual utterances and semantic labels that are not part of the main training data (such as the text inputs 205) and are used for creating the semantic probing models 305." Examiner maintains that it would have been obvious to one of ordinary skill in the art to apply the teachings of Chen/Yang/Alain/Montavon combination to arrive at a semantic probing model used to build, according to the data of the probing model, a reconstructed pre-trained model. Claim Rejections - 35 USC § 103 The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. Claims 1, 4, 8, 11, 15, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Chen, et. al, "Shallowing deep networks: Layer-wise pruning based on feature representations" (hereinafter "Chen") in view of Yang, et al., "Deepening Hidden Representations from Pre-trained Language Models" (hereinafter "Yang") in view of Alain, et al., "Understanding intermediate layers using linear classifier probes" (hereinafter "Alain") in view of Montavon, et al., "Kernel Analysis of Deep Networks" (hereinafter "Montavon"). Regarding Claim 1, Chen teaches: a method comprising: performing, using ... at least one semantic probing model, semantic probing (Chen, p. 3050, Fig. 1, "Linear classifier probes on VGG-16 (first row) and ResNet-56 (second row)" and p. 3050, 3.1 Layer-Wise Pruning via Feature Diagnosis: "Linear Classifier Probe ... trains a set of linear classifiers on features extracted at different layers within the network to explore the roles and dynamics of intermediate layers. ¶ ... [I]n this work we use a single fully-connected layer as the linear classifier to evaluate the effectiveness of a layer thus finding the ones to be pruned," where Chen describes models processing datasets with respect to semantic attributes, as in p. 3053, 4.3 Result on Multi-Label Classification: "the multi-label classification model tends to have different characteristics compared to the models for single-label classification in our experiments. ... This is likely because of the high demand on model capacity to process the data: for the complex data with abundant semantic attributes from MSCOCO, only with sufficient amount of trainable parameters ... can the performance of feature representations be significantly improved") on a pre-trained machine learning model (Chen, p. 3051, 4 Experiments: "Three different models are adopted for evaluation, including VGG-16, ResNet-56 and ResNet-101" and p. 3052, 4.1 Implementation, Data Augmentation and Data Split: "for MSCOCO multi-label classification we do not use any data augmentation besides channel-wise normalization in order to utilize the pretrained weights trained on ImageNet [31] classification") ... using at least one processor of an electronic device (Chen, p. 3053, Table 1, "Experimental Results on the CIFAR10, CIFAR100, and SVHN Datasets Using VGG16 and ResNet-56," displaying "FLOP" column for floating-point operations), wherein performing the semantic probing comprises: processing ... to determine, using the at least one semantic probing model, a performance score for each of a plurality of targeted hidden layers of the pre-trained machine learning model (Chen, p, 3052, Fig. 2, "Procedure of proposed layer-wise pruning," depicting linear classifier probes performing "Feature Diagnosis" on the hidden layers of the Original Model, and p. 3050, 3.1 Layer-Wise Pruning via Feature Diagnosis: "in this work we use a single fully-connected layer as the linear classifier to evaluate the effectiveness of a layer thus finding the ones to be pruned. By comparing the performance of classifiers trained on features computed at adjacent layers, layers that have minor improvement on the feature representations are identified based on a predefined threshold," where layer-wise accuracy scores are depicted on p. 3050, Fig. 1: "Linear classifier probes on VGG-16 (first row) and ResNet-56 (second row). Blue bars are validation accuracy of the linear classifiers trained using features from intermediate layers, black dashed lines are the validation accuracy of the CNNs. The bars with transparent color represent the layers considered to have less contributions based on the predefined threshold") ...; and selecting a subset of the targeted hidden layers based on a comparison of performance scores determined using the at least one semantic probing model to a predetermined heuristic threshold value (Chen, p. 3050, 3.1 Layer-Wise Pruning via Feature Diagnosis: "By comparing the performance of classifiers trained on features computed at adjacent layers, layers that have minor improvement on the feature representations are identified based on a predefined threshold. ... ¶ With the predefined threshold being set as 1.5 percent of the performance of original model ... nearly half of the convolutional layers in ResNet-56 have limited contributions on improving the performance of feature representations"), wherein selecting the subset of the targeted hidden layers comprises disregarding at least one hidden layer of the plurality of targeted hidden layers of the pre-trained machine learning model based on a determination that the performance score for each of the at least one disregarded hidden layer is below the predetermined heuristic threshold value (Chen, p. 3050, 3 Feature Representation Based Parameter Pruning: "With the predefined threshold being set as 1.5 percent of the performance of original model (default threshold in our experiments, a discussion can be found in Section 4.5), on CIFAR-10 and SVHN nearly half of the convolutional layers in ResNet-56 have limited contributions on improving the performance of feature representations," where Chen later describes arriving at 1.5 by means of experimentation, thus heuristically, per p. 3055, 4.5 Effects of Threshold Selection: "While 1.5 percent of the performance of original model is empirically selected as the default parameter within our experiments, in this section we focus on exploring the influences of various values for this threshold. More specifically, we use ResNet-56 on single- label classification (CIFAR-10, CIFAR-100 and SVHN) and ResNet-101 on MSCOCO multi-label classification as examples and report results with four pruning thresholds in addition to the default one, i.e., 0.75, 3, 5 and 10 percent of the original performance. We report the corresponding experimental results in Table 3"); and reconstructing, using the at least one processor, the pre-trained machine learning model based on the semantic probing to generate a reconstructed machine learning model (Chen, p. 3050, 3 Feature Representation Based Parameter Pruning: "We focus on pruning convolutional layers with relatively small contributions to increasing the performance of feature representations. We then construct compact networks without the pruned layers" and p. 3051, 3.2 Knowledge Transfer via Distillation: "we have obtained networks with a more compact architecture by reconstructing deep networks"), wherein the reconstructed machine learning model includes the selected subset of the targeted hidden layers but not the at least one disregarded hidden layer based on the determination that the performance score for each of the at least one disregarded hidden layer is below the predetermined heuristic threshold value (Chen, p. 3051, 3.1 Layer-Wise Pruning via Feature Diagnosis: "we conduct the pruning process by directly removing the layers that provide minor influences on improving the feature representations" and p. 3050, 3.1 Layer-Wise Pruning via Feature Diagnosis: "By comparing the performance of classifiers trained on features computed at adjacent layers, layers that have minor improvement on the feature representations are identified based on a predefined threshold. ... ¶ With the predefined threshold being set as 1.5 percent of the performance of original model ... nearly half of the convolutional layers in ResNet-56 have limited contributions on improving the performance of feature representations"). Chen teaches semantic probing on a pre-trained machine learning model and processing to determine a performance score. Chen does not explicitly teach semantic probing on a pre-trained machine learning model using one or more textual utterances, processing the one or more textual utterances to determine ... a performance score, and extracting a set of contextual vectors from the plurality of targeted hidden layers. However, Yang teaches: semantic probing (Yang, p. 3, 3.2 Hidden Representation Extractor: "Transformer-based encoder normally has many structure-identical layers stacked together.... To make a full use of the representations from these hidden layers, we introduce an extra component attached to the original encoder, HIdden Representation Extractor (HIRE) to capture the complementary information that the output of the last layer fails to capture") on a pre-trained machine learning model (Yang, and p. 1, 1 Introduction: "Recently, pre-trained language models based on Transformer ... such as ... RoBERTa ... have been shown to be effective for learning contextualized language representation" and "Taking advantage of the robustness of RoBERTa by using it as our backbone Transformer-based encoder ..., we conduct experiments on GLUE benchmark") using one or more textual utterances (Yang, p. 4, 3.4 Output layer: "The output layer is task-specific. The following are the concerned implementation details on two tasks, classification and regression. For classification task, given the input text’s contextualized representation F , following (Devlin et al., 2019), we take the first row C ∈ R 2 d of F corresponding to the first input token (<s>) as the aggregated representation") processing the one or more textual utterances (Yang, p. 3, 3.2 Hidden Representation Extractor: "The input to the HIRE is { H 0 , … , H j , … , H l } where l represents the number of layers in the encoder") to determine ... a performance score for each of a plurality of targeted hidden layers of the pre-trained machine learning model (Yang, p. 3, 3.2 Hidden Representation Extractor: "we concatenate GRU’s [Bidirectional Gated Recurrent Unit] each layer and each direction’s final state together. In this way, we manage to summarize the hiddenstate into a fixed-sized vector. Hence, we obtain U ∈ R l + 1 × 4 d with U i the summarized vector of H i : U i = Bi-GRU H i ∈ R 4 d (2) where 0   ≤ i   ≤ l . Then the importance value i for hidden-state H i is calculated by: α i = ReLU W T U i + b ∈ R (3) , where Yang's α i corresponds to the instant per-layer performance score) extracting a set of contextual vectors from the plurality of targeted hidden layers (Yang, p. 3, 3.2 Hidden Representation Extractor: "The input to the HIRE is { H 0 , … , H j , … , H l } .... Here ... H j ∈ R n × d the hidden-state of the encoder at the output of layer j ," where Yang's model operates on contextual inputs per p. 2, 3.1 Transformer-based encoder: "Transformer-based encoder is responsible for encoding input text into contextualized representation. Let { w 1 , … , w n } represent a sequence of n words of input text, Transformer-based encoder encodes the input sequence into its universal contextualized representation R ∈ R n × d ", supported by the instant specification at [0046]: "The pre-trained model 215 includes multiple layers 220, which include multiple hidden layers like transformer layers. Each hidden layer is an encoder that outputs one or more vectors having a certain length"). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Chen regarding semantic probing on a pre-trained machine learning model and processing to determine a performance score with those of Yang regarding semantic probing on a pre-trained machine learning model using one or more textual utterances, processing the one or more textual utterances to determine a performance score, and extracting a set of contextual vectors from the plurality of targeted hidden layers. The motivation to do so would be to facilitate refining the language representation of a pre-trained transformer model to be more suitable for downstream tasks (Yang, p. 1, 1 Introduction: "Language representation is essential to the understanding of text. Recently, pre-trained language models based on Transformer ... have been shown to be effective for learning contextualized language representation. These models have since continued to achieve new state-of-the-art results on a variety of natural language processing tasks. ... ¶ However, due to the numerous layers (i.e., Transformer blocks) and considerable depth of these pretrained models, we argue that the output of the last layer may not always be the best representation of the input text during the fine-tuning for downstream tasks. ...¶ In this paper, we propose an extra network component design for Transformer-based model, which is capable of adaptively leveraging the hidden information in the Transformer’s hidden layers to refine the language representation"). The Chen/Yang combination teaches semantic probing on a pre-trained machine learning model using one or more textual utterances, processing the one or more textual utterances to determine a performance score, and extracting a set of contextual vectors from the plurality of targeted hidden layers. The Chen/Yang combination may not explicitly teach extracting a set of ... vectors from the plurality of targeted hidden layers, building, within the at least one ... probing model, ... one or more probing layers, and one or more output activation layers on top of the set of ... vectors, wherein the at least one semantic probing model is a different model from the pre-trained machine learning model; and training the semantic probing model using the set of ... vectors to predict an amount of semantic information in each ... vector. However, Alain teaches: extracting a set of ... vectors from the plurality of targeted hidden layers (Alain, p. 4, 3.2 Linear classifier probes: "At every layer we can take the features H k from that layer and try to predict the correct labels y using a linear classifier parameterized as f k : H k → 0 ,   1 D h k ↦ softmax W h k + b . where h k ∈ H are the features of hidden layer k , 0 ,   1 D is the space of categorical distributions of the D target classes, and W , b are the probe weights and biases to be learned so as to minimize the usual cross-entropy loss"); building, within the at least one semantic probing model, ... one or more probing layers, and one or more output activation layers on top of the set of ... vectors (Alain, p. 4, 3.2 Linear classifier probes: "At every layer we can take the features H k from that layer and try to predict the correct labels y using a linear classifier," where Alain's W h k + b   corresponds to the instant probing layer, and softmax corresponds to the instant output activation layer), wherein the at least one semantic probing model is a different model from the pre-trained machine learning model (Alain, p. 4, 3.2 Linear classifier probes: "We refer to those linear classifiers as 'probes' in an effort to clarify our thinking about the model. These probes do not affect the model training. They only measure the level of linear separability of the features at a given layer"); and training the semantic probing model using the set of ... vectors (Alain, p. 4, 3.2 Linear classifier probes: " h k ∈ H are the features of hidden layer k , 0 ,   1 D is the space of categorical distributions of the D target classes, and W , b are the probe weights and biases to be learned so as to minimize the usual cross-entropy loss") to predict an amount of semantic information in each ... vector (Alain, p. 4, 3.2 Linear classifier probes: "Let L k t r a i n be the empirical loss of that linear classifier f k evaluated over the training set. We can also define L k v a l i d and L k t e s t by exporting the same linear classifier on the validation and test sets. ¶ ... we study how L k decreases as k increases ... the usefulness of L k as a diagnostic tool", where Alain previously refers to layer-wise features as information and representations at p. 4,3.1 Information theory, and monotonic improvements to linear separability: "we have that every layer has less information than its parent layer. ... ¶ One of the important lessons is that neural networks are really about distilling computationally-useful representations"). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Chen/Yang combination regarding semantic probing on a pre-trained machine learning model using one or more textual utterances, processing the one or more textual utterances to determine a performance score, and extracting a set of contextual vectors from the plurality of targeted hidden layers with those of Alain regarding extracting a set of vectors from the plurality of targeted hidden layers, building, within the at least one probing model, one or more probing layers, and one or more output activation layers on top of the set of vectors, wherein the at least one semantic probing model is a different model from the pre-trained machine learning model; and training the semantic probing model using the set of vectors to predict an amount of semantic information in each vector one or more hidden layers. The motivation to do so would be to facilitate developing better intuition about models and to diagnose potential problems with models (Alain, p. 1, Abstract: "Neural network models have a reputation for being black boxes. We propose to monitor the features at every layer of a model and measure how suitable they are for classification. We use linear classifiers, which we refer to as 'probes', trained entirely independently of the model itself. ¶ This helps us better understand the roles and dynamics of the intermediate layers. We demonstrate how this can be used to develop a better intuition about models and to diagnose potential problems"). The Chen/Yang/Alain combination teaches building, within the probing model, one or more probing layers, and one or more output activation layers on top of the set of vectors. The Chen/Yang/Alain combination may not explicitly teach building, within the at least one semantic probing model, one or more hidden layers. However, Montavon teaches: building, within the at least one semantic probing model, one or more hidden layers (Montavon, p. 2569, 2.2 Application to Deep Networks: "we describe how the analysis of representations presented above can be used to measure the layer-wise forming of the representation in deep networks. Let f x = f L ∘ f 1 x be a trained deep network of L layers. Our analysis consists of defining a sequence of 'deep kernels' that subsume the mapping performed by more and more layers of the deep network and repeating for each kernel the analysis," where Montavon's kernel corresponds to the instant hidden layer, per p. 2567, 2.1 Quantifying How Good a Representation Is: "The accuracy can in turn be obtained by measuring the prediction error e d of a linear predictor on top of the d-component kernel representation. ... The curve e d gives a complete picture of how good a representation is with respect to the learning problem," as supported by the instant specification, [0049]: "the hidden layers 320 (such as multi-layer perceptron (MLP) layers) are optional and can be used to boost performance depending on resources and time constraints"). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Chen/Yang/Alain combination regarding building, within the probing model, one or more probing layers, and one or more output activation layers on top of the set of vectors with those of Montavon regarding building, within the at least one semantic probing model, one or more hidden layers. The motivation to do so would be to facilitate having a basis for building measurably simple models of the learning problem (Montavon, p. 2567, 2.1 Quantifying How Good a Representation Is: "We choose to use the kernel principal component analysis ... as a basis for building measurably simple models of the learning problem. Our method consists of projecting the input distribution on the d first components (in terms of variance) of the kernel feature space and fitting a linear model on this low-rank representation. ... The accuracy can in turn be obtained by measuring the prediction error e d of a linear predictor on top of the d-component kernel representation. ... The curve e d gives a complete picture of how good a representation is with respect to the learning problem"). Regarding Claim 8, Chen teaches: an electronic device comprising: at least one memory configured to store instructions; and at least one processing device configured when executing the instructions (Chen, p. 3051, 4 Experiments: "To demonstrate the effectiveness of our parameter pruning method, we conduct experiments on the CIFAR-10, CIFAR-100, SVHN datasets for single-label classification and the MSCOCO [30] dataset for multi-label classification" and p. 3053, Table 1, "Experimental Results on the CIFAR10, CIFAR100, and SVHN Datasets Using VGG16 and ResNet-56," displaying "FLOP" column for floating-point operations during inference, where experiments using large datasets reasonably suggests an electronic device) to perform precisely those steps recited in the rejection of Claim 1. Claim 8 is rejected under the same rationale as Claim 1. Regarding Claim 15, Chen teaches: a non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device (Chen, p. 3051, 4 Experiments: "To demonstrate the effectiveness of our parameter pruning method, we conduct experiments on the CIFAR-10, CIFAR-100, SVHN datasets for single-label classification and the MSCOCO [30] dataset for multi-label classification" and p. 3053, Table 1, "Experimental Results on the CIFAR10, CIFAR100, and SVHN Datasets Using VGG16 and ResNet-56," displaying "FLOP" column for floating-point operations during inference, where experiments using large datasets reasonably suggests a non-transitory machine-readable medium containing instructions and a processor of an electronic device) to perform precisely those steps recited in the rejection of Claim 1. Claim 15 is rejected under the same rationale as Claim 1. Regarding Claim 4, the rejection of Claim 1 is incorporated. Yang further teaches: wherein the pre-trained machine learning model comprises a contextualized representation machine learning model (Yang, p. 3, Figure 1, "Architecture of our model," depicting Transformer-based Encoder as the input model, and p. 2, 1 Introduction: "Recently, pre-trained language models based on Transformer ... such as ... RoBERTa ... have been shown to be effective for learning contextualized language representation" and "Taking advantage of the robustness of RoBERTa by using it as our backbone Transformer-based encoder"). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Chen/Yang/Alain/Montavon combination regarding semantic probing on a pre-trained machine learning model with the further teachings of regarding wherein the pre-trained machine learning model comprises a contextualized representation machine learning model. The motivation to do so would be to facilitate use of a transformer model's contextualized representation more effectively for downstream tasks (Yang, p. 1, 1 Introduction: "Language representation is essential to the understanding of text. Recently, pre-trained language models based on Transformer ... have been shown to be effective for learning contextualized language representation. These models have since continued to achieve new state-of-the-art results on a variety of natural language processing tasks. ... ¶ However, due to the numerous layers (i.e., Transformer blocks) and considerable depth of these pretrained models, we argue that the output of the last layer may not always be the best representation of the input text during the fine-tuning for downstream tasks. ...¶ In this paper, we propose an extra network component design for Transformer-based model, which is capable of adaptively leveraging the hidden information in the Transformer’s hidden layers to refine the language representation"). Claims 11 and 18 incorporate substantively all limitations of Claim 4 in device and non-transitory machine-readable medium forms, respectively, and are rejected under the same rationale. Claims 2, 9, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Chen, et. al, "Shallowing deep networks: Layer-wise pruning based on feature representations" (hereinafter "Chen") in view of Yang, et al., "Deepening Hidden Representations from Pre-trained Language Models" (hereinafter "Yang") in view of Alain, et al., "Understanding intermediate layers using linear classifier probes" (hereinafter "Alain") in view of Montavon, et al., "Kernel Analysis of Deep Networks" (hereinafter "Montavon") in further view of Goodsitt, et al. (US 2021/0097343 A1, hereinafter "Goodsitt"). Regarding Claim 2, the rejection of Claim 1 is incorporated. The Chen/Yang/Alain/Montavon combination teaches performing semantic probing on a pre-trained machine learning model. The Chen/Yang/Alain/Montavon combination does not explicitly teach imposing a time limit on the semantic probing. However, Goodsitt teaches in the context of model tuning and reuse: imposing a time limit (Goodsitt, [0054]: "Process 300 can then proceed to step 309. In step 309, computing resources 101 can generate a trained data model using the data model received from model optimizer 107 and the synthetic dataset received from dataset genera tor 103. For example , computing resources 101 can be configured to train the data model received from model optimizer 107 until some training criterion is satisfied . The training criterion can be , for example , a performance criterion (e.g., a Mean Absolute Error, Root Mean Squared Error, percent good classification, and the like), a convergence criterion (e.g., a minimum required improvement of a performance criterion over iterations or over time, a minimum required change in model parameters over iterations or over time), elapsed time or number of iterations, or the like. In some embodiments , the performance criterion can be a threshold value for a similarity metric or prediction accuracy metric as described herein"). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Chen/Yang/Alain/Montavon combination regarding reconstructing the pre-trained model based on the semantic probing with the further teachings of Goodsitt regarding imposing a time limit. The motivation to do so would be to provide a means to ensure that computing resources for training are allocated efficiently (Goodsitt, [0146]: "Termination or assignment may be based on performance of the development instance or the performance of another development instance. In this way, the serverless architecture may more efficiently allocate resources during hyperparameter tuning traditional, server-based architectures"). Claims 9 and 16 incorporate substantively all limitations of Claim 2 in device and non-transitory machine-readable medium forms, respectively, and are rejected under the same rationale. Claims 3, 10, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Chen, et. al, "Shallowing deep networks: Layer-wise pruning based on feature representations" (hereinafter "Chen") in view of Yang, et al., "Deepening Hidden Representations from Pre-trained Language Models" (hereinafter "Yang") in view of Alain, et al., "Understanding intermediate layers using linear classifier probes" (hereinafter "Alain") in view of Montavon, et al., "Kernel Analysis of Deep Networks" (hereinafter "Montavon") in view of Merchant, et al., "What Happens To BERT Embeddings During Fine-tuning?" hereinafter Merchant. Regarding Claim 3, the rejection of Claim 1 is incorporated. The Chen/Yang/Alain/Montavon combination teaches selecting a subset of hidden layers of a pre-trained machine learning model based on a performance score determined from semantic probing of the model, then reconstructing the model using the selected hidden layers. The Chen/Yang/Alain/Montavon combination does not explicitly teach the selected subset of targeted hidden layers comprises one or more original targeted hidden layers and one or more updated hidden layers; and the reconstructed model is generated using the one or more original targeted hidden layers and the one or more updated hidden layers. However, Merchant teaches: the selected subset of targeted hidden layers comprises one or more original targeted hidden layers and one or more updated hidden layers; and the reconstructed model is generated using the one or more original targeted hidden layers and the one or more updated hidden layers (Merchant, p. 7, 5.2 Layer Ablations: "Partial Freezing can be thought of as a test for how many layers need to change for a downstream task. We freeze the bottom k layers (and the embeddings) – treating them as features – but allow the rest to adapt"). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Chen/Yang/Alain/Montavon combination regarding reconstructing the pre-trained model based on the semantic probing with with those of Merchant regarding the selected subset of targeted hidden layers comprising one or more original targeted hidden layers and one or more updated hidden layers. The motivation to do so would be to take advantage of the task-specific performance improvements of fine-tuning with decreased training effort (Merchant, p. 7, 5.2 Layer Ablations, Figure 3: "Effects of freezing an increasing number of layers during fine-tuning on performance.... The graph shows that only a few unfrozen layers are needed to improve task performance, supporting the shallow processing conclusion"). Claims 10 and 17 incorporate substantively all limitations of Claim 3 in device and non-transitory machine-readable medium forms, respectively, and are rejected under the same rationale. Claims 5-7, 12-14, 19, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Chen, et. al, "Shallowing deep networks: Layer-wise pruning based on feature representations" (hereinafter "Chen") in view of Yang, et al., "Deepening Hidden Representations from Pre-trained Language Models" (hereinafter "Yang") in view of Alain, et al., "Understanding intermediate layers using linear classifier probes" (hereinafter "Alain") in view of Montavon, et al., "Kernel Analysis of Deep Networks" (hereinafter "Montavon") in further view of Zhao, et al., "Masking as an Efficient Alternative to Finetuning for Pretrained Language Models" (hereinafter "Zhao2"). Regarding Claim 5, the rejection of Claim 1 is incorporated. The Chen/Yang/Alain/Montavon combination teaches reconstructing a pre-trained machine learning model using hidden layers selected according to a performance score. The Chen/Yang/Alain/Montavon combination does not explicitly teach generating, using the at least one processor, a binary mask based on the reconstructed machine learning model, the binary mask generated for a specific task with semantic and text embedding. However, Zhao2 teaches: generating, using the at least one processor, a binary mask based on the reconstructed machine learning model (Zhao2, p. 1, Abstract: "We present an efficient method of utilizing pretrained language models, where we learn selective binary masks for pretrained weights in lieu of modifying them through finetuning"), the binary mask generated for a specific task (Zhao2, p. 1, 1 Introduction: "Instead of directly updating the pretrained parameters, we propose to select weights important to downstream NLP tasks while discarding irrelevant ones. The selection mechanism consists of a set of binary masks, one learned per downstream task through end-to-end training") with semantic and text embedding (Zhao2, p. 1, 1 Introduction: "We show that masking, when being applied to pretrained language models like BERT, RoBERTa, and DistilBERT (Sanh et al., 2019), achieves performance comparable to finetuning in tasks like part-of-speech tagging, named-entity recognition, sequence classification, and reading comprehension"). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Chen/Yang/Alain/Montavon combination regarding reconstructing a pre-trained model based on the semantic probing with with those of Zhao2 regarding generating a binary mask based on the reconstructed model for a specific task with semantic and text embedding. The motivation to do so would be improved space efficiency for the trained model (Zhao2, p. 1, 1 Introduction: "Masking is parameter-efficient: only a set of 1- bit binary masks needs to be saved per task after training, instead of all 32-bit float parameters in finetuning. This small memory footprint enables deploying pretrained language models for solving multiple tasks on edge devices. The compactness of masking also naturally allows parameter-efficient ensembles of pretrained language models"). Regarding Claim 6, the Chen/Yang/Alain/Montavon/Zhao2 combination teaches the method of Claim 5, and thus the rejection of Claim 5 is incorporated. Zhao2 further teaches: wherein generating the binary mask comprises: generating an initial binary mask based on a threshold of real number mask weights (Zhao2, p. 3, 3.2 Learning the mask: "We associate each linear layer W l ... of the l-th transformer block with a real-valued matrix M l that is randomly initialized from a uniform distribution and has the same size as W l . We then pass M l through an element-wise thresholding function ... to obtain a binary mask M b i n l for W l ... where ... τ is a global thresholding hyperparameter") for the specific task (Zhao2, p. 3, 3.2 Learning the mask: "We learn a set of binary masks for an NLP task.... We then update each M l through Eq. 2 with the task objective during training"); applying the initial binary mask on multiple parameters of the reconstructed machine learning model to generate masked parameters (Zhao2, p. 3, 3.2 Learning the mask: "In each forward pass of training, the binary mask M b i n l (derived from M l via Eq. 1) selects weights in a pretrained linear layer W l by Hadamard product: W ^ l   : =   W l   ⊙   M b i n l "); and evaluating the masked parameters to determine whether a goal of the specific task is met (Zhao2, p. 3, 3.2 Learning the mask: "In the corresponding backward pass of training, with the associated loss function L , we cannot backpropagate through the binarizer.... [W]e use ∂ L W ^ l ∂ M b i n l as a noisy estimator of ∂ L W ^ l ∂ M l to update M l , i.e.: M l   ⃪   M l - η ∂ L W ^ l ∂ M b i n l (2) where η refers to the step size. Hence, the whole structure can be trained end-to-end," where minimizing the loss function L of a specific task during training corresponds to the instant a goal of the specific task). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior teachings of the Chen/Yang/Alain/Montavon/Zhao2 combination regarding generating, for a specific task with semantic and text embedding, a binary mask based on the reconstructed model with the further teachings of Zhao2 regarding generating an initial binary mask for the specific task, applying the initial binary mask on model parameters to generate masked parameters, and evaluating the masked parameters. The motivation to do so would be to take advantage of pretrained models for multiple tasks in a space-efficient manner (Zhao2, p. 1, 1 Introduction: "This small memory footprint enables deploying pretrained language models for solving multiple tasks on edge devices. The compactness of masking also naturally allows parameter-efficient ensembles of pretrained language models"). Regarding Claim 7, the rejection of Claim 6 is incorporated. Zhao2 further teaches: updating the binary mask (Zhao2, p. 3, 3.2 Learning the mask: Eq. 2, which represents a mask update during the loss-minimization goal of learning) in response to determining that the goal of the specific task is not met (Zhao2, p. 3, 3.2 Learning the mask: "Given a pretrained language model, we do not finetune, i.e., we do not update the pretrained parameters. Instead, we select a subset of the pretrained parameters that is critical to a downstream task while discarding irrelevant ones with binary masks" and "We learn a set of binary masks for an NLP task as follows. Recall that each linear layer W l is associated with a M l to obtain a masked linear layer W ^ l through Eq. 1. ... We then update each M l through Eq. 2 with the task objective during training," where Zhao2 later indicates that minimizing task loss indicates a stopping point for training, as in p. 8, 6.3 Loss landscape: "We are interested in testing if the two minima found by finetuning and masking can be easily connected in the loss landscape"). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior teachings of the Chen/Yang/Alain/Montavon/Zhao2 combination regarding evaluating the masked parameters to determine whether a goal of the specific task is met with the further teachings of Zhao2 regarding updating the binary mask in response to determining that the goal of the specific task is not met. The motivation to do so would be to take advantage of pretrained models for multiple tasks in a space-efficient manner (Zhao2, p. 1, Abstract: "masking can be utilized as an efficient alternative to finetuning" and p. 1, 1 Introduction: "This small memory footprint enables deploying pretrained language models for solving multiple tasks on edge devices. The compactness of masking also naturally allows parameter-efficient ensembles of pretrained language models"). Claims 12-14 incorporate substantively all limitations of Claims 5-7, respectively, in device form and are rejected under the same rationale. Claims 19 and 20 incorporate substantively all limitations of Claims 5 and 6, respectively, in non-transitory machine-readable medium form and are rejected under the same rationale. Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT N DAY whose telephone number is (703)756-1519. The examiner can normally be reached M-F 9-5. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /R.N.D./Examiner, Art Unit 2122 /KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122