DETAILED ACTION
This communication is in response to the Application No. 18/105,739 filed on January 23, 2023
in which Claims 1-20 are presented for examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation "the bias feature" in line 5. There is insufficient antecedent basis for this limitation in the claim. The phrase “the bias feature” was not previously introduced.
Since independent claim 1 is rejected under 35 U.S.C. 112(b), claims 2 - 7 are also rejected
under 35 U.S.C. 112(b) because they depend on claim 1 and inherit the same indefiniteness problems.
Claim 8 recites the limitation "the bias feature" in line 8. There is insufficient antecedent basis for this limitation in the claim. The phrase “the bias feature” was not previously introduced.
Since independent claim 8 is rejected under 35 U.S.C. 112(b), claims 8 - 14 are also rejected
under 35 U.S.C. 112(b) because they depend on claim 8 and inherit the same indefiniteness problems.
Claim 15 recites the limitation "the bias feature" in line 10. There is insufficient antecedent basis for this limitation in the claim. The phrase “the bias feature” was not previously introduced.
Since independent claim 15 is rejected under 35 U.S.C. 112(b), claims 15 - 20 are also rejected
under 35 U.S.C. 112(b) because they depend on claim 8 and inherit the same indefiniteness problems.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 - 20 are rejected under 35 U.S.C. 103 as being unpatentable over Lohaus et al. (hereafter Lohaus) , a non-patent literature reference titled “Are Two Heads the Same as One? Identifying Disparate Treatment in Fair Neural Networks”, in view of Fatemi et al. (hereinafter Fatemi) (US 20230104662) and further in view of Prasanna et al. (hereinafter Prasanna) (US 20240135238).
Regarding Claim 1, Lohaus teaches obtaining, based on the inference model, a multipath inference model comprising a first inference generation path and a second inference generation path (Lohaus, Abstract, “We show that deep networks trained to satisfy demographic parity often do so through a form of race or gender awareness, and that the more we force a network to be fair, the more accurately we can recover race or gender from the internal state of the network. Based on this observation, we investigate an alternative fairness approach: we add a second classification head to the network to explicitly predict the protected attribute (such as race or gender) alongside the original task. After training the two-headed network, we enforce demographic parity by merging the two heads, creating a network with the same architecture as the original network”, thus a multipath inference model comprising a first inference generation path and a second inference generation path is disclosed)
performing a first training procedure for the second inference generation path to configure the second inference generation path to predict the bias feature, the first training procedure providing a revised second inference generation path (Lohaus, Abstract, “we add a second classification head to the network to explicitly predict the protected attribute (such as race or gender) alongside the original task.”, & Page 5 – Section 4, “In this section, we evaluate a novel post-processing approach explicitly designed to exhibit disparate treatment while using the same single network architecture used by preprocessing and regularized approaches. In short, a network with two fully-connected output heads on the last-layer representation is trained: one head f : X → R is trained to minimize a logistic loss over the target variable y, and the other head g : X → R to minimize a squared loss over the protected attribute”, thus Lohaus teaches performing a first training procedure for the second inference generation path to configure the second inference generation path to predict the bias feature (protected attributes), thereby providing a trained or revised second inference generation path)
[…] for the revised second inference generation path […] of a shared body portion of the revised second inference generation path to obtain an unbiased shared body portion, the first inference generation path comprising the unbiased shared body portion, and the latent bias being due to a bias feature (Lohaus, Page 4 – Section 2, “We provide strong evidence that deep neural networks that are enforced to satisfy demographic parity by means of a regularizer or preprocessing suffer from disparate treatment, even when not explicitly using the protected attribute at test time—and that they do so by separating last-layer representations based on protected attributes.”, & Page 5 – Section, “In short, a network with two fully-connected output heads on the last-layer representation is trained: one head f : X → R is trained to minimize a logistic loss over the target variable y, and the other head g : X → R to minimize a squared loss over the protected attribute X → R is trained to minimize a logistic loss over the target variable y, and the other head g : X → R to minimize a squared loss over the protected attribute”, thus a shared body portion of the revised second inference generation path, obtaining an unbiased shared body portion, and the first inference generation path comprising the unbiased shared body portion, and the latent bias being due to a bias feature/protected attributes are disclosed)
[…] the first inference generation path while the unbiased shared body portion […] (Lohaus, Page 5 – Section 4, “In short, a network with two fully-connected output heads on the last-layer representation is trained: one head f : X → R is trained to minimize a logistic loss over the target variable y, and the other head g : X → R to minimize a squared loss over the protected attribute”, thus first inference generation path and unbiased shared body portion are disclosed)
Lohaus does not explicitly disclose a method for managing an inference model that may exhibit latent bias, performing a second training procedure for the first inference generation path while the unbiased shared body portion is frozen using the first training data to obtain a revised first inference generation path, or and using the revised first inference generation path to provide inferences used to provide computer implemented services.
However, Fatemi teaches a method for managing an inference model that may exhibit latent bias (Fatemi, Par. [0002], “methods for improving gender fairness of pre-trained language models”, thus a method for managing an inference model that may exhibit latent bias is disclosed). Fatemi also teaches performing a second training procedure for gender-neutral training framework while parameters are frozen using first training data to obtain a revised first inference generation path (Fatemi, Par. [0019], “As the original pre-trained parameters are frozen, forgetting information from the original training data may be largely alleviated. As for the gender equality prompts, the gender-neutral training framework trains new word/token embeddings of profession names as gender equality prompts at second-phase pre-training. Since the embeddings of profession names are newly re-initialized when de-bias training starts, gender bias from previous data that is embedded in such representations is already removed before second-phase pre-training. Therefore, the gender bias mechanism does not have to train the model to find and fix bias from scratch”, & Par, [0022], “In one embodiment, after the gender-neutral data set is built, all, most, or above a predefined threshold number of model parameters with valued determined during a previous phase of the pre-training are frozen and new trainable parameters are added. Since the pre-trained parameters are frozen, the forgetting of information from the original training data may be alleviated. Because gender bias issue is most prominent on profession names, the new trainable parameters may be new word and/or token embeddings of profession names. Thus, at second-phase pre-training, only the newly added token embeddings of profession names are updated with the gender-neutral data, conditioned on the original pre-trained model”, thus performing a second training procedure for gender-neutral training framework while parameters are frozen using first training data to obtain a revised first inference generation path). Additionally, Fatemi teaches and using the revised first inference generation path to provide inferences used to provide computer implemented services (Fatemi, Par. [0026], “Gender-neutral training framework 130 may receive a dataset and/or pre-trained language model as input 140 and generate output 150 which may be a gender neutral dataset or language model 136 trained using a gender neutral dataset. Language model 136 may receive NLP input that includes a natural language question, document, etc., as input 140 and generate output 150 that may be a response to the input, including a gender neutral answer to a natural language question, a gender neutral document summary, etc. Notably, the language model 136 that is trained to generate gender neutral output 150 has a variety of applications” thus, using the revised first inference generation path to provide inferences used to provide computer implemented services is disclosed)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to apply the gender-neutral second-phase training framework of Fatemi to the multipath inference model of Lohaus in order to reduce latent bias while preserving previously learned representations, because both references explicitly recognize that bias is encoded in shared internal representations and that freezing pre-trained parameters while selectively retraining model components reduces bias without degrading model performance. Fatemi teaches that “as the original pre-trained parameters are frozen, forgetting information from the original training data may be largely alleviated,” and that retraining newly added parameters using debiased data removes bias embedded in prior representations without requiring retraining from scratch (Fatemi, Par. [0019]). Fatemi further explains that freezing previously trained parameters while updating only selected components “alleviates catastrophic forgetting” and yields a debiased model with preserved functionality (Fatemi, Par. [0022], Par. [0037]). Lohaus also teaches that bias arises from “separating last-layer representations based on protected attributes,” and that both inference paths operate on a shared internal representation (Lohaus, Pages 4–5), thereby identifying the shared body portion as the source of latent bias. In view of these teachings, a person of ordinary skill in the art would have reasonably expected that applying Fatemi’s frozen-parameter debiasing technique to Lohaus’s shared-representation, two-head architecture would reduce latent bias in the shared body portion while maintaining inference performance of the primary task. The combination applies selective retraining with frozen shared parameters to a shared-representation multipath model, thereby reducing bias encoded in the shared internal representation while preserving learned inference capability.
Further, the combination of Lohaus and Fatemi does not appear to distinctly disclose performing an untraining procedure to reduce latent bias.
However, Prasanna teaches performing untraining procedure to reduce latent bias (Prasanna, Par. [0006], “According to another embodiment, a computer-implemented method for mitigating biased training instances associated with a machine learning model without additional refitting of the machine learning model can include receiving, using a processor, a pre-trained machine learning model and calculating a fairness influence score of training instances on group fairness metrics associated with the pre-trained machine learning model. The method can include performing, using the processor, post-hoc unfairness mitigation by removing the effect of at least one of the training instances based on the fairness influence score”, & Par. [0023], “performing post-hoc unfairness mitigation by removing the effect of at least one of the training instances based on the fairness influence score. Such can be achieved by directly editing the trained model to eliminate the need of sensitive attribute labels at test time where the base model can be trained, unconstrained, on the main task, which can be updated to remove the influence of the harmful instances, thus, improving fairness”, therefore performing an untraining procedure/ removing the effect of at least one of the training instances to reduce latent bias is disclosed)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine Lohaus with Prasanna to perform “an untraining procedure for the revised second inference generation path to reduce latent bias of a shared body portion of the revised second inference generation path to obtain an unbiased shared body portion, the first inference generation path comprising the unbiased shared body portion, and the latent bias being due to a bias feature,” because Lohaus teaches that a multipath inference model with a shared body portion encodes protected attributes within its internal representations, and Prasanna teaches that bias mitigation “can be achieved by directly editing the trained model to eliminate the need of sensitive attribute labels at test time where the base model can be trained, unconstrained, on the main task, which can be updated to remove the influence of the harmful instances, thus, improving fairness” (Prasanna, Par. [0023]). Applying Prasanna’s direct model-editing technique to Lohaus’s revised second inference generation path would have resulted in training the revised second inference generation path using second training data to reduce an ability of the revised second inference generation path to predict the bias feature, thereby reducing latent bias of the shared body portion while preserving remaining inference functionality.
Regarding Claim 2, Lohaus teaches wherein the first training procedure is performed while the shared body portion is frozen (Lohaus, Page 4 – Section 3, “From the CelebA image dataset [43] and nine of its 40 binary attributes, we choose a target attribute and a protected attribute, such as SMILING and YOUNG. We use [55] as reference for restricting our study to only nine of the 40 attributes. For each distinct pair of target and protected attribute, we train ResNet50 models for twelve different values of the fairness parameter λ. For each model, we then train a linear classifier using logistic regression to predict the protected attribute from the model’s frozen last-layer representation. We refer to these classifiers as the group classifiers”, thus the first training procedure being performed while the shared body portion / last-layer representation is frozen is disclosed)
Regarding Claim 3, Fatemi teaches wherein while the shared body portion is frozen, values of weights of hidden layers of the shared body portion are not modified during the first training procedure (Fatemi, Par. [0045], “At process 404 , parameters of the language model 206 are frozen. For example, after the language model 206 is pre-trained during the first phase, the language model 136 includes parameters with different values. The parameters may be throughout different layers of language model 206 . These parameters are frozen during the second phase. Once frozen, the values of the parameters do not change”, thus values of weights of hidden layers of the shared body portion not being modified during the first training procedure is disclosed)
Regarding Claim 4, Fatemi teaches wherein the values of the weights of the hidden layers of the shared body portion being set during a previously performed training procedure completed prior to the shared body portion being frozen and the previously performed training procedure using the first training data (Fatemi, Par. [0030], “After the first phase, language model 136 may be considered pre-trained. A language model 136 that is pre-trained, such as a pre-trained BERT, may be trained on various datasets, such as BooksCorpus that includes 800M or so words and an English Wikipedia corpus that includes 2,500M words, with two unsupervised objective functions, including a masked language modeling (MLM) function and a next sentence prediction function. In the masked language modeling, 15% of all tokens in each sequence are replaced with [MASK] token at random and the model attempts to predict the masked tokens based on the context of unmasked words in the sequence. In the next sentence prediction task, the input to the language model 136 is sequences of sentences, and the language model 136 learns to predict if the current sentence is subsequent of the previous sentence in the training corpus. For fine-tuning, the language model 136 is initialized with the pre-trained parameters, and a new classification head is added to the language model 136. Then, all of the parameters are fine-tuned using labeled data from the downstream tasks.”, & Par. [0045], “For example, after the language model 206 is pre-trained during the first phase, the language model 136 includes parameters with different values. The parameters may be throughout different layers of language model 206 . These parameters are frozen during the second phase.”, thus the values of the weights of the hidden layers of the shared body portion being set during a previously performed training procedure completed prior to the shared body portion being frozen and the previously performed training procedure using the first training data is disclosed)
Regarding Claim 5, Prasanna teaches training the revised second inference generation path using second training data to reduce an ability of the revised second inference generation path to predict the bias feature to obtain a second revised second inference generation path (Prasanna, Par. [0019], “Often, pre-processing methods can be expensive, especially for high-dimensional data and large datasets. Further, pre-processing methods are performed before training any task-specific models and are not applicable when the goal is to improve a model already trained with an expensive procedure. Pre-processing methods, typically, learn transformation of the data distribution such that they do not contain information about the sensitive attributes. Task specific models can then be learned from scratch on the debiased representations”, & Par. [0023], “Such can be achieved by directly editing the trained model to eliminate the need of sensitive attribute labels at test time where the base model can be trained, unconstrained, on the main task, which can be updated to remove the influence of the harmful instances, thus, improving fairness”, therefore training the revised second inference generation path using second training data to reduce an ability of the revised second inference generation path to predict the bias feature to obtain a second revised second inference generation path is disclosed)
Prasanna does not explicitly disclose training the second revised second inference generation path using the second training data while the shared body portion is frozen to obtain a revised head portion of the second revised second inference generation path.
However, Fatemi teaches training the second revised second inference generation path using the second training data while the shared body portion is frozen to obtain a revised head portion of the second revised second inference generation path (Fatemi, Par. [0022], “In one embodiment, after the gender-neutral data set is built, all, most, or above a predefined threshold number of model parameters with valued determined during a previous phase of the pre-training are frozen and new trainable parameters are added. Since the pre-trained parameters are frozen, the forgetting of information from the original training data may be alleviated. Because gender bias issue is most prominent on profession names, the new trainable parameters may be new word and/or token embeddings of profession names. Thus, at second-phase pre-training, only the newly added token embeddings of profession names are updated with the gender-neutral data, conditioned on the original pre-trained model”, thus Fatemi teaches freezing previously trained parameters and training only newly added parameters using second training data, thereby obtaining a revised head portion while the shared body portion remains frozen)
Regarding Claim 6, Prasanna teaches testing the shared body portion that is frozen and the revised head portion for a level of the latent bias (Prasanna, Par. [0005], “The system can also include an influence mitigation component that can perform post-hoc unfairness mitigation by removing the effect of at least one training instance based on the fairness influence score to mitigate biased training instances without refitting the pre-trained machine learning model”, thus testing the shared body portion that is frozen and the revised head portion for a level of the latent bias/ fairness influence score is disclosed). Prasanna also teaches an instance of the testing where the level of latent bias falls below a threshold, concluding that the shared body portion is the unbiased shared body portion (Prasanna, Par. [0048], “Training instances with positive influence on demographic parity are plotted in the histogram plots 300 , 302 , and 304 . Further, datapoints to the right of the dotted lines 310 , 312 , 314 are the top-500 training instances and models adjusted to mitigate the training instance influence (e.g. and are as a result substantially more fair). Dropping the top k most influential instances can yield improves bias mitigation. For instance, the validation dataset 108 can be used to select the hyperparameter k”, thus an instance of the testing where the level of latent bias falls below a threshold, concluding that the shared body portion is the unbiased shared body portion is disclosed)
Regarding Claim 7, Prasanna teaches wherein the revised first inference generation path provides inferences that exhibit reduced levels of the latent bias (Prasanna, Par. [0066], “Additionally, the method can include computing the product of the inverse Hessian vector product and gradient of a loss function of the training instances (step 812 ). Further, the method can include performing post-hoc unfairness mitigation (step 814 ) to reduce or limit the effect of a training instance. The method can include determining whether the effect of a training instance has a disproportionate impact on group fairness ( 816 ), whereby the effect of the disproportionate training instance is removed (step 818 ) resulting in the edited model”, thus the revised first inference generation path providing inferences that exhibit reduced levels of the latent bias is disclosed)
Regarding Claim 8, Lohaus teaches obtaining, based on the inference model, a multipath inference model comprising a first inference generation path and a second inference generation path (Lohaus, Abstract, “We show that deep networks trained to satisfy demographic parity often do so through a form of race or gender awareness, and that the more we force a network to be fair, the more accurately we can recover race or gender from the internal state of the network. Based on this observation, we investigate an alternative fairness approach: we add a second classification head to the network to explicitly predict the protected attribute (such as race or gender) alongside the original task. After training the two-headed network, we enforce demographic parity by merging the two heads, creating a network with the same architecture as the original network”, thus a multipath inference model comprising a first inference generation path and a second inference generation path is disclosed)
performing a first training procedure for the second inference generation path to configure the second inference generation path to predict the bias feature, the first training procedure providing a revised second inference generation path (Lohaus, Abstract, “we add a second classification head to the network to explicitly predict the protected attribute (such as race or gender) alongside the original task.”, & Page 5 – Section 4, “In this section, we evaluate a novel post-processing approach explicitly designed to exhibit disparate treatment while using the same single network architecture used by preprocessing and regularized approaches. In short, a network with two fully-connected output heads on the last-layer representation is trained: one head f : X → R is trained to minimize a logistic loss over the target variable y, and the other head g : X → R to minimize a squared loss over the protected attribute”, thus Lohaus teaches performing a first training procedure for the second inference generation path to configure the second inference generation path to predict the bias feature (protected attributes), thereby providing a trained or revised second inference generation path)
[…] for the revised second inference generation path […] of a shared body portion of the revised second inference generation path to obtain an unbiased shared body portion, the first inference generation path comprising the unbiased shared body portion, and the latent bias being due to a bias feature (Lohaus, Page 4 – Section 2, “We provide strong evidence that deep neural networks that are enforced to satisfy demographic parity by means of a regularizer or preprocessing suffer from disparate treatment, even when not explicitly using the protected attribute at test time—and that they do so by separating last-layer representations based on protected attributes.”, & Page 5 – Section, “In short, a network with two fully-connected output heads on the last-layer representation is trained: one head f : X → R is trained to minimize a logistic loss over the target variable y, and the other head g : X → R to minimize a squared loss over the protected attribute X → R is trained to minimize a logistic loss over the target variable y, and the other head g : X → R to minimize a squared loss over the protected attribute”, thus a shared body portion of the revised second inference generation path, obtaining an unbiased shared body portion, and the first inference generation path comprising the unbiased shared body portion, and the latent bias being due to a bias feature/protected attributes are disclosed)
[…] the first inference generation path while the unbiased shared body portion […] (Lohaus, Page 5 – Section 4, “In short, a network with two fully-connected output heads on the last-layer representation is trained: one head f : X → R is trained to minimize a logistic loss over the target variable y, and the other head g : X → R to minimize a squared loss over the protected attribute”, thus first inference generation path and unbiased shared body portion are disclosed)
Lohaus does not explicitly disclose a method for managing an inference model that may exhibit latent bias, performing a second training procedure for the first inference generation path while the unbiased shared body portion is frozen using the first training data to obtain a revised first inference generation path, or and using the revised first inference generation path to provide inferences used to provide computer implemented services.
However, Fatemi teaches a method for managing an inference model that may exhibit latent bias (Fatemi, Par. [0002], “methods for improving gender fairness of pre-trained language models”, thus a method for managing an inference model that may exhibit latent bias is disclosed). Fatemi also teaches performing a second training procedure for gender-neutral training framework while parameters are frozen using first training data to obtain a revised first inference generation path (Fatemi, Par. [0019], “As the original pre-trained parameters are frozen, forgetting information from the original training data may be largely alleviated. As for the gender equality prompts, the gender-neutral training framework trains new word/token embeddings of profession names as gender equality prompts at second-phase pre-training. Since the embeddings of profession names are newly re-initialized when de-bias training starts, gender bias from previous data that is embedded in such representations is already removed before second-phase pre-training. Therefore, the gender bias mechanism does not have to train the model to find and fix bias from scratch”, & Par, [0022], “In one embodiment, after the gender-neutral data set is built, all, most, or above a predefined threshold number of model parameters with valued determined during a previous phase of the pre-training are frozen and new trainable parameters are added. Since the pre-trained parameters are frozen, the forgetting of information from the original training data may be alleviated. Because gender bias issue is most prominent on profession names, the new trainable parameters may be new word and/or token embeddings of profession names. Thus, at second-phase pre-training, only the newly added token embeddings of profession names are updated with the gender-neutral data, conditioned on the original pre-trained model”, thus performing a second training procedure for gender-neutral training framework while parameters are frozen using first training data to obtain a revised first inference generation path). Additionally, Fatemi teaches and using the revised first inference generation path to provide inferences used to provide computer implemented services (Fatemi, Par. [0026], “Gender-neutral training framework 130 may receive a dataset and/or pre-trained language model as input 140 and generate output 150 which may be a gender neutral dataset or language model 136 trained using a gender neutral dataset. Language model 136 may receive NLP input that includes a natural language question, document, etc., as input 140 and generate output 150 that may be a response to the input, including a gender-neutral answer to a natural language question, a gender neutral document summary, etc. Notably, the language model 136 that is trained to generate gender neutral output 150 has a variety of applications” thus, using the revised first inference generation path to provide inferences used to provide computer implemented services is disclosed)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to apply the gender-neutral second-phase training framework of Fatemi to the multipath inference model of Lohaus in order to reduce latent bias while preserving previously learned representations, because both references explicitly recognize that bias is encoded in shared internal representations and that freezing pre-trained parameters while selectively retraining model components reduces bias without degrading model performance. Fatemi teaches that “as the original pre-trained parameters are frozen, forgetting information from the original training data may be largely alleviated,” and that retraining newly added parameters using debiased data removes bias embedded in prior representations without requiring retraining from scratch (Fatemi, Par. [0019]). Fatemi further explains that freezing previously trained parameters while updating only selected components “alleviates catastrophic forgetting” and yields a debiased model with preserved functionality (Fatemi, Par. [0022], Par. [0037]). Lohaus also teaches that bias arises from “separating last-layer representations based on protected attributes,” and that both inference paths operate on a shared internal representation (Lohaus, Pages 4–5), thereby identifying the shared body portion as the source of latent bias. In view of these teachings, a person of ordinary skill in the art would have reasonably expected that applying Fatemi’s frozen-parameter debiasing technique to Lohaus’s shared-representation, two-head architecture would reduce latent bias in the shared body portion while maintaining inference performance of the primary task. The combination applies selective retraining with frozen shared parameters to a shared-representation multipath model, thereby reducing bias encoded in the shared internal representation while preserving learned inference capability.
Further, the combination of Lohaus and Fatemi does not appear to distinctly disclose performing an untraining procedure to reduce latent bias.
However, Prasanna teaches performing untraining procedure to reduce latent bias (Prasanna, Par. [0006], “According to another embodiment, a computer-implemented method for mitigating biased training instances associated with a machine learning model without additional refitting of the machine learning model can include receiving, using a processor, a pre-trained machine learning model and calculating a fairness influence score of training instances on group fairness metrics associated with the pre-trained machine learning model. The method can include performing, using the processor, post-hoc unfairness mitigation by removing the effect of at least one of the training instances based on the fairness influence score”, & Par. [0023], “performing post-hoc unfairness mitigation by removing the effect of at least one of the training instances based on the fairness influence score. Such can be achieved by directly editing the trained model to eliminate the need of sensitive attribute labels at test time where the base model can be trained, unconstrained, on the main task, which can be updated to remove the influence of the harmful instances, thus, improving fairness”, therefore performing an untraining procedure/ removing the effect of at least one of the training instances to reduce latent bias is disclosed)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine Lohaus with Prasanna to perform “an untraining procedure for the revised second inference generation path to reduce latent bias of a shared body portion of the revised second inference generation path to obtain an unbiased shared body portion, the first inference generation path comprising the unbiased shared body portion, and the latent bias being due to a bias feature,” because Lohaus teaches that a multipath inference model with a shared body portion encodes protected attributes within its internal representations, and Prasanna teaches that bias mitigation “can be achieved by directly editing the trained model to eliminate the need of sensitive attribute labels at test time where the base model can be trained, unconstrained, on the main task, which can be updated to remove the influence of the harmful instances, thus, improving fairness” (Prasanna, Par. [0023]). Applying Prasanna’s direct model-editing technique to Lohaus’s revised second inference generation path would have resulted in training the revised second inference generation path using second training data to reduce an ability of the revised second inference generation path to predict the bias feature, thereby reducing latent bias of the shared body portion while preserving remaining inference functionality.
Regarding Claim 9, Lohaus teaches wherein the first training procedure is performed while the shared body portion is frozen (Lohaus, Page 4 – Section 3, “From the CelebA image dataset [43] and nine of its 40 binary attributes, we choose a target attribute and a protected attribute, such as SMILING and YOUNG. We use [55] as reference for restricting our study to only nine of the 40 attributes. For each distinct pair of target and protected attribute, we train ResNet50 models for twelve different values of the fairness parameter λ. For each model, we then train a linear classifier using logistic regression to predict the protected attribute from the model’s frozen last-layer representation. We refer to these classifiers as the group classifiers”, thus the first training procedure being performed while the shared body portion / last-layer representation is frozen is disclosed)
Regarding Claim 10, Fatemi teaches wherein while the shared body portion is frozen, values of weights of hidden layers of the shared body portion are not modified during the first training procedure (Fatemi, Par. [0045], “At process 404 , parameters of the language model 206 are frozen. For example, after the language model 206 is pre-trained during the first phase, the language model 136 includes parameters with different values. The parameters may be throughout different layers of language model 206 . These parameters are frozen during the second phase. Once frozen, the values of the parameters do not change”, thus values of weights of hidden layers of the shared body portion not being modified during the first training procedure is disclosed)
Regarding Claim 11, Fatemi teaches wherein the values of the weights of the hidden layers of the shared body portion being set during a previously performed training procedure completed prior to the shared body portion being frozen and the previously performed training procedure using the first training data (Fatemi, Par. [0030], “After the first phase, language model 136 may be considered pre-trained. A language model 136 that is pre-trained, such as a pre-trained BERT, may be trained on various datasets, such as BooksCorpus that includes 800M or so words and an English Wikipedia corpus that includes 2,500M words, with two unsupervised objective functions, including a masked language modeling (MLM) function and a next sentence prediction function. In the masked language modeling, 15% of all tokens in each sequence are replaced with [MASK] token at random and the model attempts to predict the masked tokens based on the context of unmasked words in the sequence. In the next sentence prediction task, the input to the language model 136 is sequences of sentences, and the language model 136 learns to predict if the current sentence is subsequent of the previous sentence in the training corpus. For fine-tuning, the language model 136 is initialized with the pre-trained parameters, and a new classification head is added to the language model 136. Then, all of the parameters are fine-tuned using labeled data from the downstream tasks.”, & Par. [0045], “For example, after the language model 206 is pre-trained during the first phase, the language model 136 includes parameters with different values. The parameters may be throughout different layers of language model 206 . These parameters are frozen during the second phase.”, thus the values of the weights of the hidden layers of the shared body portion being set during a previously performed training procedure completed prior to the shared body portion being frozen and the previously performed training procedure using the first training data is disclosed)
Regarding Claim 12, Prasanna teaches training the revised second inference generation path using second training data to reduce an ability of the revised second inference generation path to predict the bias feature to obtain a second revised second inference generation path (Prasanna, Par. [0019], “Often, pre-processing methods can be expensive, especially for high-dimensional data and large datasets. Further, pre-processing methods are performed before training any task-specific models and are not applicable when the goal is to improve a model already trained with an expensive procedure. Pre-processing methods, typically, learn transformation of the data distribution such that they do not contain information about the sensitive attributes. Task specific models can then be learned from scratch on the debiased representations”, & Par. [0023], “Such can be achieved by directly editing the trained model to eliminate the need of sensitive attribute labels at test time where the base model can be trained, unconstrained, on the main task, which can be updated to remove the influence of the harmful instances, thus, improving fairness”, therefore training the revised second inference generation path using second training data to reduce an ability of the revised second inference generation path to predict the bias feature to obtain a second revised second inference generation path is disclosed)
Prasanna does not explicitly disclose training the second revised second inference generation path using the second training data while the shared body portion is frozen to obtain a revised head portion of the second revised second inference generation path.
However, Fatemi teaches training the second revised second inference generation path using the second training data while the shared body portion is frozen to obtain a revised head portion of the second revised second inference generation path (Fatemi, Par. [0022], “In one embodiment, after the gender-neutral data set is built, all, most, or above a predefined threshold number of model parameters with valued determined during a previous phase of the pre-training are frozen and new trainable parameters are added. Since the pre-trained parameters are frozen, the forgetting of information from the original training data may be alleviated. Because gender bias issue is most prominent on profession names, the new trainable parameters may be new word and/or token embeddings of profession names. Thus, at second-phase pre-training, only the newly added token embeddings of profession names are updated with the gender-neutral data, conditioned on the original pre-trained model”, thus Fatemi teaches freezing previously trained parameters and training only newly added parameters using second training data, thereby obtaining a revised head portion while the shared body portion remains frozen)
Regarding Claim 13, Prasanna teaches testing the shared body portion that is frozen and the revised head portion for a level of the latent bias (Prasanna, Par. [0005], “The system can also include an influence mitigation component that can perform post-hoc unfairness mitigation by removing the effect of at least one training instance based on the fairness influence score to mitigate biased training instances without refitting the pre-trained machine learning model”, thus testing the shared body portion that is frozen and the revised head portion for a level of the latent bias/ fairness influence score is disclosed). Prasanna also teaches an instance of the testing where the level of latent bias falls below a threshold, concluding that the shared body portion is the unbiased shared body portion (Prasanna, Par. [0048], “Training instances with positive influence on demographic parity are plotted in the histogram plots 300 , 302 , and 304 . Further, datapoints to the right of the dotted lines 310 , 312 , 314 are the top-500 training instances and models adjusted to mitigate the training instance influence (e.g. and are as a result substantially more fair). Dropping the top k most influential instances can yield improves bias mitigation. For instance, the validation dataset 108 can be used to select the hyperparameter k”, thus an instance of the testing where the level of latent bias falls below a threshold, concluding that the shared body portion is the unbiased shared body portion is disclosed)
Regarding Claim 14, Prasanna teaches wherein the revised first inference generation path provides inferences that exhibit reduced levels of the latent bias (Prasanna, Par. [0066], “Additionally, the method can include computing the product of the inverse Hessian vector product and gradient of a loss function of the training instances (step 812 ). Further, the method can include performing post-hoc unfairness mitigation (step 814 ) to reduce or limit the effect of a training instance. The method can include determining whether the effect of a training instance has a disproportionate impact on group fairness ( 816 ), whereby the effect of the disproportionate training instance is removed (step 818 ) resulting in the edited model”, thus the revised first inference generation path providing inferences that exhibit reduced levels of the latent bias is disclosed)
Regarding Claim 15, Lohaus teaches obtaining, based on the inference model, a multipath inference model comprising a first inference generation path and a second inference generation path (Lohaus, Abstract, “We show that deep networks trained to satisfy demographic parity often do so through a form of race or gender awareness, and that the more we force a network to be fair, the more accurately we can recover race or gender from the internal state of the network. Based on this observation, we investigate an alternative fairness approach: we add a second classification head to the network to explicitly predict the protected attribute (such as race or gender) alongside the original task. After training the two-headed network, we enforce demographic parity by merging the two heads, creating a network with the same architecture as the original network”, thus a multipath inference model comprising a first inference generation path and a second inference generation path is disclosed)
performing a first training procedure for the second inference generation path to configure the second inference generation path to predict the bias feature, the first training procedure providing a revised second inference generation path (Lohaus, Abstract, “we add a second classification head to the network to explicitly predict the protected attribute (such as race or gender) alongside the original task.”, & Page 5 – Section 4, “In this section, we evaluate a novel post-processing approach explicitly designed to exhibit disparate treatment while using the same single network architecture used by preprocessing and regularized approaches. In short, a network with two fully-connected output heads on the last-layer representation is trained: one head f : X → R is trained to minimize a logistic loss over the target variable y, and the other head g : X → R to minimize a squared loss over the protected attribute”, thus Lohaus teaches performing a first training procedure for the second inference generation path to configure the second inference generation path to predict the bias feature (protected attributes), thereby providing a trained or revised second inference generation path)
[…] for the revised second inference generation path […] of a shared body portion of the revised second inference generation path to obtain an unbiased shared body portion, the first inference generation path comprising the unbiased shared body portion, and the latent bias being due to a bias feature (Lohaus, Page 4 – Section 2, “We provide strong evidence that deep neural networks that are enforced to satisfy demographic parity by means of a regularizer or preprocessing suffer from disparate treatment, even when not explicitly using the protected attribute at test time—and that they do so by separating last-layer representations based on protected attributes.”, & Page 5 – Section, “In short, a network with two fully-connected output heads on the last-layer representation is trained: one head f : X → R is trained to minimize a logistic loss over the target variable y, and the other head g : X → R to minimize a squared loss over the protected attribute X → R is trained to minimize a logistic loss over the target variable y, and the other head g : X → R to minimize a squared loss over the protected attribute”, thus a shared body portion of the revised second inference generation path, obtaining an unbiased shared body portion, and the first inference generation path comprising the unbiased shared body portion, and the latent bias being due to a bias feature/protected attributes are disclosed)
[…] the first inference generation path while the unbiased shared body portion […] (Lohaus, Page 5 – Section 4, “In short, a network with two fully-connected output heads on the last-layer representation is trained: one head f : X → R is trained to minimize a logistic loss over the target variable y, and the other head g : X → R to minimize a squared loss over the protected attribute”, thus first inference generation path and unbiased shared body portion are disclosed)
Lohaus does not explicitly disclose a method for managing an inference model that may exhibit latent bias, performing a second training procedure for the first inference generation path while the unbiased shared body portion is frozen using the first training data to obtain a revised first inference generation path, or and using the revised first inference generation path to provide inferences used to provide computer implemented services.
However, Fatemi teaches a data processing system, comprising: a processor; and a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for managing an inference model (Fatemi, Claim 9, “A system for training a neural network language model to generate gender neutral output, the system comprising: a memory configured to store the neural network language model; and a processor coupled to the memory and configured to execute instructions for training the neural network language mode”, thus a data processing system, comprising: a processor; and a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for managing an inference model is disclosed). Fatemi also teaches performing a second training procedure for gender-neutral training framework while parameters are frozen using first training data to obtain a revised first inference generation path (Fatemi, Par. [0019], “As the original pre-trained parameters are frozen, forgetting information from the original training data may be largely alleviated. As for the gender equality prompts, the gender-neutral training framework trains new word/token embeddings of profession names as gender equality prompts at second-phase pre-training. Since the embeddings of profession names are newly re-initialized when de-bias training starts, gender bias from previous data that is embedded in such representations is already removed before second-phase pre-training. Therefore, the gender bias mechanism does not have to train the model to find and fix bias from scratch”, & Par, [0022], “In one embodiment, after the gender-neutral data set is built, all, most, or above a predefined threshold number of model parameters with valued determined during a previous phase of the pre-training are frozen and new trainable parameters are added. Since the pre-trained parameters are frozen, the forgetting of information from the original training data may be alleviated. Because gender bias issue is most prominent on profession names, the new trainable parameters may be new word and/or token embeddings of profession names. Thus, at second-phase pre-training, only the newly added token embeddings of profession names are updated with the gender-neutral data, conditioned on the original pre-trained model”, thus performing a second training procedure for gender-neutral training framework while parameters are frozen using first training data to obtain a revised first inference generation path). Additionally, Fatemi teaches and using the revised first inference generation path to provide inferences used to provide computer implemented services (Fatemi, Par. [0026], “Gender-neutral training framework 130 may receive a dataset and/or pre-trained language model as input 140 and generate output 150 which may be a gender neutral dataset or language model 136 trained using a gender neutral dataset. Language model 136 may receive NLP input that includes a natural language question, document, etc., as input 140 and generate output 150 that may be a response to the input, including a gender neutral answer to a natural language question, a gender neutral document summary, etc. Notably, the language model 136 that is trained to generate gender neutral output 150 has a variety of applications” thus, using the revised first inference generation path to provide inferences used to provide computer implemented services is disclosed)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to apply the gender-neutral second-phase training framework of Fatemi to the multipath inference model of Lohaus in order to reduce latent bias while preserving previously learned representations, because both references explicitly recognize that bias is encoded in shared internal representations and that freezing pre-trained parameters while selectively retraining model components reduces bias without degrading model performance. Fatemi teaches that “as the original pre-trained parameters are frozen, forgetting information from the original training data may be largely alleviated,” and that retraining newly added parameters using debiased data removes bias embedded in prior representations without requiring retraining from scratch (Fatemi, Par. [0019]). Fatemi further explains that freezing previously trained parameters while updating only selected components “alleviates catastrophic forgetting” and yields a debiased model with preserved functionality (Fatemi, Par. [0022], Par. [0037]). Lohaus also teaches that bias arises from “separating last-layer representations based on protected attributes,” and that both inference paths operate on a shared internal representation (Lohaus, Pages 4–5), thereby identifying the shared body portion as the source of latent bias. In view of these teachings, a person of ordinary skill in the art would have reasonably expected that applying Fatemi’s frozen-parameter debiasing technique to Lohaus’s shared-representation, two-head architecture would reduce latent bias in the shared body portion while maintaining inference performance of the primary task. The combination applies selective retraining with frozen shared parameters to a shared-representation multipath model, thereby reducing bias encoded in the shared internal representation while preserving learned inference capability.
Further, the combination of Lohaus and Fatemi does not appear to distinctly disclose performing an untraining procedure to reduce latent bias.
However, Prasanna teaches performing untraining procedure to reduce latent bias (Prasanna, Par. [0006], “According to another embodiment, a computer-implemented method for mitigating biased training instances associated with a machine learning model without additional refitting of the machine learning model can include receiving, using a processor, a pre-trained machine learning model and calculating a fairness influence score of training instances on group fairness metrics associated with the pre-trained machine learning model. The method can include performing, using the processor, post-hoc unfairness mitigation by removing the effect of at least one of the training instances based on the fairness influence score”, & Par. [0023], “performing post-hoc unfairness mitigation by removing the effect of at least one of the training instances based on the fairness influence score. Such can be achieved by directly editing the trained model to eliminate the need of sensitive attribute labels at test time where the base model can be trained, unconstrained, on the main task, which can be updated to remove the influence of the harmful instances, thus, improving fairness”, therefore performing an untraining procedure/ removing the effect of at least one of the training instances to reduce latent bias is disclosed)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine Lohaus with Prasanna to perform “an untraining procedure for the revised second inference generation path to reduce latent bias of a shared body portion of the revised second inference generation path to obtain an unbiased shared body portion, the first inference generation path comprising the unbiased shared body portion, and the latent bias being due to a bias feature,” because Lohaus teaches that a multipath inference model with a shared body portion encodes protected attributes within its internal representations, and Prasanna teaches that bias mitigation “can be achieved by directly editing the trained model to eliminate the need of sensitive attribute labels at test time where the base model can be trained, unconstrained, on the main task, which can be updated to remove the influence of the harmful instances, thus, improving fairness” (Prasanna, Par. [0023]). Applying Prasanna’s direct model-editing technique to Lohaus’s revised second inference generation path would have resulted in training the revised second inference generation path using second training data to reduce an ability of the revised second inference generation path to predict the bias feature, thereby reducing latent bias of the shared body portion while preserving remaining inference functionality.
Regarding Claim 16, Lohaus teaches wherein the first training procedure is performed while the shared body portion is frozen (Lohaus, Page 4 – Section 3, “From the CelebA image dataset [43] and nine of its 40 binary attributes, we choose a target attribute and a protected attribute, such as SMILING and YOUNG. We use [55] as reference for restricting our study to only nine of the 40 attributes. For each distinct pair of target and protected attribute, we train ResNet50 models for twelve different values of the fairness parameter λ. For each model, we then train a linear classifier using logistic regression to predict the protected attribute from the model’s frozen last-layer representation. We refer to these classifiers as the group classifiers”, thus the first training procedure being performed while the shared body portion / last-layer representation is frozen is disclosed)
Regarding Claim 17, Fatemi teaches wherein while the shared body portion is frozen, values of weights of hidden layers of the shared body portion are not modified during the first training procedure (Fatemi, Par. [0045], “At process 404 , parameters of the language model 206 are frozen. For example, after the language model 206 is pre-trained during the first phase, the language model 136 includes parameters with different values. The parameters may be throughout different layers of language model 206 . These parameters are frozen during the second phase. Once frozen, the values of the parameters do not change”, thus values of weights of hidden layers of the shared body portion not being modified during the first training procedure is disclosed)
Regarding Claim 18, Fatemi teaches wherein the values of the weights of the hidden layers of the shared body portion being set during a previously performed training procedure completed prior to the shared body portion being frozen and the previously performed training procedure using the first training data (Fatemi, Par. [0030], “After the first phase, language model 136 may be considered pre-trained. A language model 136 that is pre-trained, such as a pre-trained BERT, may be trained on various datasets, such as BooksCorpus that includes 800M or so words and an English Wikipedia corpus that includes 2,500M words, with two unsupervised objective functions, including a masked language modeling (MLM) function and a next sentence prediction function. In the masked language modeling, 15% of all tokens in each sequence are replaced with [MASK] token at random and the model attempts to predict the masked tokens based on the context of unmasked words in the sequence. In the next sentence prediction task, the input to the language model 136 is sequences of sentences, and the language model 136 learns to predict if the current sentence is subsequent of the previous sentence in the training corpus. For fine-tuning, the language model 136 is initialized with the pre-trained parameters, and a new classification head is added to the language model 136. Then, all of the parameters are fine-tuned using labeled data from the downstream tasks.”, & Par. [0045], “For example, after the language model 206 is pre-trained during the first phase, the language model 136 includes parameters with different values. The parameters may be throughout different layers of language model 206 . These parameters are frozen during the second phase.”, thus the values of the weights of the hidden layers of the shared body portion being set during a previously performed training procedure completed prior to the shared body portion being frozen and the previously performed training procedure using the first training data is disclosed)
Regarding Claim 19, Prasanna teaches training the revised second inference generation path using second training data to reduce an ability of the revised second inference generation path to predict the bias feature to obtain a second revised second inference generation path (Prasanna, Par. [0019], “Often, pre-processing methods can be expensive, especially for high-dimensional data and large datasets. Further, pre-processing methods are performed before training any task-specific models and are not applicable when the goal is to improve a model already trained with an expensive procedure. Pre-processing methods, typically, learn transformation of the data distribution such that they do not contain information about the sensitive attributes. Task specific models can then be learned from scratch on the debiased representations”, & Par. [0023], “Such can be achieved by directly editing the trained model to eliminate the need of sensitive attribute labels at test time where the base model can be trained, unconstrained, on the main task, which can be updated to remove the influence of the harmful instances, thus, improving fairness”, therefore training the revised second inference generation path using second training data to reduce an ability of the revised second inference generation path to predict the bias feature to obtain a second revised second inference generation path is disclosed)
Prasanna does not explicitly disclose training the second revised second inference generation path using the second training data while the shared body portion is frozen to obtain a revised head portion of the second revised second inference generation path.
However, Fatemi teaches training the second revised second inference generation path using the second training data while the shared body portion is frozen to obtain a revised head portion of the second revised second inference generation path (Fatemi, Par. [0022], “In one embodiment, after the gender-neutral data set is built, all, most, or above a predefined threshold number of model parameters with valued determined during a previous phase of the pre-training are frozen and new trainable parameters are added. Since the pre-trained parameters are frozen, the forgetting of information from the original training data may be alleviated. Because gender bias issue is most prominent on profession names, the new trainable parameters may be new word and/or token embeddings of profession names. Thus, at second-phase pre-training, only the newly added token embeddings of profession names are updated with the gender-neutral data, conditioned on the original pre-trained model”, thus Fatemi teaches freezing previously trained parameters and training only newly added parameters using second training data, thereby obtaining a revised head portion while the shared body portion remains frozen)
Regarding Claim 20, Prasanna teaches testing the shared body portion that is frozen and the revised head portion for a level of the latent bias (Prasanna, Par. [0005], “The system can also include an influence mitigation component that can perform post-hoc unfairness mitigation by removing the effect of at least one training instance based on the fairness influence score to mitigate biased training instances without refitting the pre-trained machine learning model”, thus testing the shared body portion that is frozen and the revised head portion for a level of the latent bias/ fairness influence score is disclosed). Prasanna also teaches an instance of the testing where the level of latent bias falls below a threshold, concluding that the shared body portion is the unbiased shared body portion (Prasanna, Par. [0048], “Training instances with positive influence on demographic parity are plotted in the histogram plots 300 , 302 , and 304 . Further, datapoints to the right of the dotted lines 310 , 312 , 314 are the top-500 training instances and models adjusted to mitigate the training instance influence (e.g. and are as a result substantially more fair). Dropping the top k most influential instances can yield improves bias mitigation. For instance, the validation dataset 108 can be used to select the hyperparameter k”, thus an instance of the testing where the level of latent bias falls below a threshold, concluding that the shared body portion is the unbiased shared body portion is disclosed)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. US 11256989 is pertinent because it teaches detecting and mitigating bias within trained machine-learning models by analyzing internal model behavior and applying post-training corrective techniques. The applicant’s disclosure also discusses managing latent bias in an inference model and reducing that bias through post-training procedures, making the reference relevant to the claimed invention. US 20220391683 is another pertinent reference because it teaches detecting and mitigating bias within trained machine-learning models by analyzing internal model behavior and applying post-training corrective techniques. The applicant’s disclosure also concerns identifying latent bias in an inference model and reducing that bias through post-training procedures, making the reference relevant to the claimed invention.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MAHLIET ADMASU whose telephone number is (571)272-0034. The examiner can normally be reached Mon-Fri, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at (571)270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.T.A./Examiner, Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123