DETAILED ACTION
Status of Claims
Claim(s) 1-20 are pending and are examined herein.
Claim(s) 1-20 are rejected under 35 U.S.C. § 101 and 103.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement IDS(s) submitted on August 19, 2022 is in compliance with the provisions of 37 CFR 1.97 and have been considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION. —The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 3, 6, 13, and 16 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, for pre-AIA the applicant regards as the invention.
Regarding Claim 3, Claim 3 recites the limitation “wherein when the drift is signaled, the model is retrained, and the margin and proportion are recomputed.” However, the term “proportion” lacks clear antecedent basis. Claim 1, from which claim 3 depends, recites different proportions, including “an initial proportion” and “a new proportion.” Claim 3 merely recites “the proportion” without specifying which of the previously recited proportions is being referenced.
Thus, the recitation of “the proportion” in claim 3 lacks clear antecedent basis and renders the scope of the claim indefinite. It is unclear whether claim 3 requires recomputation of the initial proportion, the new proportion, both proportions, or some other proportion. Accordingly, one of ordinary skill in the art would not be able to ascertain, with reasonable certainty, the scope of the claimed invention. For examination purposes, the Examiner interpreted the “proportion” as referencing previously recited proportions in claim 1.
Regarding Claim 6, the claim recites the limitation “comparing a sequence of differences between the current proportions and the initial proportion to determine a drift in the performance of the model.” However, the recitation of “the current proportions” lacks clear antecedent basis.
Claim 1, from which claim 6 depends, recites different proportions, including “an initial proportion” and “a new proportion.” Claim 6 introduce the term “current proportions” without specifying which of the previously recited proportions is being referenced or whether the “current proportions” is intended to reference the new proportions, multiple new proportions over time, or some other proportion not previously introduced.
Thus the recitation of “the current proportions” in claim 6 lacks clear antecedent basis in the claim and renders the scope of the claim indefinite. It is unclear whether claim 6 requires comparison of the initial proportion with the new proportion, with a sequence of new proportions, or with some other set of proportions. Accordingly, one of ordinary skill in the art would not be able to ascertain, with reasonable certainty, the scope of the claimed invention.
Regarding Claims 13 and 16, the claims recite substantially similar limitations as those of claims 3 and 6 and are rejected for similar reasons and rationale.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
When considering subject matter eligibility under 35 U.S.C. 101, it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter (Step 1). If the claim does fall within one of the statutory categories, the second step in the analysis is to determine whether the claim is directed to a judicial exception (Step 2A). The Step 2A analysis is broken into two prongs. In the first prong (Step 2A, Prong 1), it is determined whether or not the claims recite a judicial exception (e.g., mathematical concepts, mental processes, certain methods of organizing human activity). If it is determined in Step 2A, Prong 1 that the claims recite a judicial exception, the analysis proceeds to the second prong (Step 2A, Prong 2), where it is determined whether or not the claims integrate the judicial exception into a practical application. If it is determined at step 2A, Prong 2 that the claims do not integrate the judicial exception into a practical application, the analysis proceeds to determining whether the claim is a patent-eligible application of the exception (Step 2B). If an abstract idea is present in the claim, any element or combination of elements in the claim must be sufficient to ensure that the claim integrates the judicial exception into a practical application, or else amounts to significantly more than the abstract idea itself. Applicant is advised to consult MPEP 2106 for more details of the analysis.
Under Step 1 analysis,
Claims 1-10 recite a method (representing a process); and
Claims 11-20 recite a non-transitory storage medium (representing an article of manufacture).
Therefore, each set of the claims falls into one of the four statutory categories (i.e., process, machine, article of manufacture, or composition of matter).
Claims
1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more, and hence is not patent-eligible subject matter.
Regarding Claim 1,
Step 2A Prong 1: The claim recites an abstract idea enumerated in the 2019 PEG.
defining a margin based on the first reconstruction error and the second reconstruction error; (The “defining” step is an abstract idea of Mental Process. Examiner’s note: the “defining” step, as drafted, and under its broadest reasonable interpretation (BRI), covers concepts that can practically performed in the human mind with the aid of pen and paper. This limitation involves comparing the error distributions of two data samples to identify overlapping region that define the margin. This step represents statistical analysis of reconstruction error distributions identify overlapping margin. Accordingly, the defining steps falls under the mental processes—concepts performed in the human mind (including an observation, evaluation, judgment, or opinion). See MPEP § 2106.04(a)(2)(III).)
computing an initial proportion of samples from the set of normative data whose reconstruction errors fall within a range of reconstruction errors defined by the margin; computing a new proportion of unlabeled data samples that fall within the range of reconstruction errors defined by the margin; (An abstract idea of “a Mental Step” and/or “Mathematical Concept.” The “computing” steps, as drafted, and under their broadest reasonable interpretation, cover concepts that would fall under the mental process and mathematical concept. Examiner note: these steps involves mathematically calculating the proportions of reference and current data samples whose errors fall within the predefined margin. In other words, the claim step counts the current sample that fall within the defined margin and divide it over the total number of samples. (See spec [0052]-[0055]). The computing steps can be derived manually by an individual with the aid of pen and paper. See MPEP § 2106.04(a)(2)(I).)
signaling drift in the performance of the model when said new proportion differs from said initial proportion by more than a predefined tolerance threshold. (An abstract idea of a “Mental process” and “Mathematical Concept.” The “signaling” step, as drafted, and under its broadest reasonable interpretation, covers concepts that falls under the mental process and mathematical concept. The limitation merely defines the process of identifying a drift in the performance of the model (i.e., detect a change in the data distribution over time) based on comparing the results of the proportions calculation to a predefined threshold. See spec e.g., [0056]. Thus, this claim merely describes an act of evaluation (i.e., comparing the results of two calculations to a predefined threshold). This is a mental process. See MPEP § 2106.04(a)(2)(III).)
Step 2A Prong 2: Under this prong, we evaluate whether the claim recites additional elements that integrate the abstract idea into a practical application by considering the claim as a whole. The judicial exception is not integrated into a practical application.
Additional Elements Analysis:
The claim recite the limitations:
“receiving a stream of unlabeled data samples from a model; obtaining a first reconstruction error for the unlabeled data samples; obtaining a second reconstruction error for a set of normative data;”
These steps of “receiving” and “obtaining” amount to adding insignificant extra-solution activity to the judicial exception, as discussed in MPEP § 2106.05(g). These limitations merely define a data gathering and/or outputting steps in conjunction with the abstract idea. Such data gathering and/or outputting steps do not impose meaningful limit on the scope of the claim (i.e., all uses of the recited judicial exception require such data gathering or data output).
Step 2B: Under this prong, the claim must include additional elements that amount to significantly more than the judicial exception. These elements must not be well-understood, routine, or conventional in the relevant field. When viewed individually and as an ordered combination, the claim does not include any such additional elements that are sufficient to amount to significantly more (i.e., inventive concept).
Additional Elements Analysis:
As explained above, the additional elements such as “receiving” and “obtaining” steps amount to insignificant extra-solution activities to the judicial exception. These additional elements merely represents generic computer functions (i.e., data gathering and/or outputting). The courts have recognized computer functions such as “receiving or transmitting data” or “storing and retrieving information” as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. See MPEP § 2106.05(d).
Accordingly, when viewed as a whole, the claim is primarily directed to the abstract idea of analyzing reconstruction-error distribution of two datasets (unlabeled and normative) to define an overlapping margin, computing the proportion of samples within that margin for each dataset, and comparing the difference to a predefined threshold to detect a drift. The additional limitations, such as receiving/obtaining, whether considered individually or in combination with the judicial exception, are not sufficient to integrate the judicial exception into a practical application or amount to significantly more.
Therefore, claim 1 does not recite patent-eligible subject matter.
Regarding Claim 2,
Step 2A Prong 1: Claim 2, which incorporates the rejection of claim 1, doesn’t recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
wherein the model is an unsupervised event detection model operable to detect events in a domain in which mobile edge devices are deployed. (This limitation amounts to linking the use of a judicial exception to a particular technological environment or field of use, as discussed in MPEP § 2106.05(h). The claim limitation merely defines the intended use or filed of use of the claimed method. The unsupervised event detection model recited in the claim merely defines a computer component (i.e., computer instructions) configured to implement the abstract idea (i.e., detect events). The domain in which the model is deployed (i.e., mobile edge) does not meaningfully limit the abstract idea because it merely linked the use of the abstract idea to a particular technological environment. See MPEP § 2106.05(e).)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As explained above, the additional element of using unsupervised learning model for event detection on mobile edge devices amounts to generally linking the use of a judicial exception to a particular technological environment or field of use. It is noted that a claim directed to a judicial exception cannot be made eligible simply by limitation the exception to a particular technological use. See MPEP § 2106.05(h).
Therefore, claim 2 is ineligible.
Regarding Claim 3,
Step 2A Prong 1: Claim 3, which incorporates the rejection of claim 1, recites further limitation such as:
wherein when the drift is signaled, the model is retrained, and the margin and proportion are recomputed. (That is part of the abstract idea identified in claim 1. Dependent claim 3 merely specifies a remedial action following drift detection, while the detection itself remains based on comparing proportions of samples within a reconstruction-error margin as recited in claim 1. The claim is directed to monitoring and proportion comparison to determine remedial action (i.e., retrain the model), which is an abstract idea of mental process and mathematical concept. See MPEP § 2106.04(a)(2)(I) & (III).)
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
The recitation of “the model is retrained” represents a post-solution activity that occurs after drift has already been detected. This amounts to adding insignificant extra-solution activity to the judicial exception, as discussed in MPEP § 2106.05(g). The retraining element is a post-detection remedial action that is not integrated into the drift-detection analysis itself; it is the result of the drift detection.
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As explained above in step 2A, prong Two, the additional element of retraining merely defines a post-solution activity. The limitation remains insignificant extra-solution activity, even upon reconsideration. This cannot meaningfully limit the claim as it merely defines the result of the performing the judicial exception (see MPEP § 2106.05(d)). Accordingly, this limitation cannot provide an inventive concept.
Therefore, claim 3 is ineligible.
Regarding Claim 4,
Step 2A Prong 1: Claim 4, which incorporates the rejection of claim 1, doesn’t recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
wherein the stream of unlabeled data samples is generated by one or more mobile edge nodes. (This limitation amounts to generally linking the use of a judicial exception to a particular technological environment or field of use (i.e., mobile edge nodes), as discussed in MPEP § 2106.05(h).)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, the recited limitation of claim 4 merely ties the claimed method to a particular technological environment and does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception. See MPEP § 2106.04(d).
Therefore, claim 4 is ineligible.
Regarding Claim 5,
Step 2A Prong 1: Claim 5, which incorporates the rejection of claim 1, doesn’t recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
wherein prior to receiving the stream of unlabeled data samples, a model that performs the signaling of the drift is trained using a combination of anomalous data and the normative data. (This limitation amounts to merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea, as discussed in MPEP § 2106.05(f). Examiner’s note: the claim merely defines a high-level model training using predetermined data. This represents invoking computer or other machinery in their ordinary capacity merely as a tool to perform an existing process.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As noted above, the high-level model training using predetermined data amounts to invoking generic computer component (instructions) to perform an existing process. This generic recitation cannot provide an inventive concept. See MPEP § 2106.05.
Therefore, claim 5 is ineligible.
Regarding Claim 6,
Step 2A Prong 1: Claim 6, which incorporates the rejection of claim 1, recites further limitation such as:
comparing a sequence of differences between the current proportions and the initial proportion to determine a drift in the performance of the model. (This limitation is part of the abstract idea recited claim 1. The claim merely defines the comparison between the initial proportion to the current proportion to determine whether a drift occurred. This is an act of evaluation that can be manually performed by an individual. See MPEP § 2106.04(a)(2)(III).)
Step 2A Prong 2: The claim does not recite additional element that integrates the judicial exception into a practical application.
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception.
Therefore, claim 6 is ineligible.
Regarding Claim 7,
Step 2A Prong 1: Claim 7, which incorporates the rejection of claim 1, recites further limitation such as:
wherein boundaries of the margin are defined by a plot of the second reconstruction error. (This limitation is part of the abstract idea recited claim 1. The claim merely introduce a plot that is used to determine the margin. This represents the statistical analysis to determine the margin which is an abstract idea of a mental process that can be practically performed in the human mind with the aid of pen and paper. See MPEP § 2106.04(a)(2)(III).)
Step 2A Prong 2: The claim does not recite additional element that integrates the judicial exception into a practical application.
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception.
Therefore, claim 7 is ineligible.
Regarding Claim 8,
Step 2A Prong 1: Claim 8, which incorporates the rejection of claim 1, doesn’t recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
wherein the model is deployed at each of a plurality of edge nodes. ((This limitation amounts to generally linking the use of a judicial exception to a particular technological environment or field of use (i.e., mobile edge nodes), as discussed in MPEP § 2106.05(h).)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, the recited limitation of claim 8 merely ties the claimed method to a particular technological environment and does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception. See MPEP § 2106.04(d).
Therefore, claim 8 is ineligible.
Regarding Claim 9,
Step 2A Prong 1: Claim 9, which incorporates the rejection of claim 1, doesn’t recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
wherein the stream of unlabeled data samples comprises data about a movement and/or a position of a physical mobile edge device. (This limitation defines the type/source of data being used. This is part of the linkage to a particular field of use or technological environment that does not impose meaningful limits to the claim. See MPEP § 2106.05(h).)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, the recited limitation of claim 9 defines the type of data being used and does not impose meaningful limitation to the process beyond generally linking the use of the judicial exception to a particular technological environment. See MPEP § 2106.04(d).
Therefore, claim 9 is ineligible.
Regarding Claim 10,
Step 2A Prong 1: Claim 10, which incorporates the rejection of claim 1, recites further limitation such as:
wherein a size of the margin is variable based on constraints associated with an application domain where the model is deployed. (This limitation is part of the abstract idea recited claim 1. The claim limitation merely introduce the size variable based on application-domain constraints. This is part of the abstract idea of claim 1 to define the margin using variable that represents constraints associated with an application domain. See MPEP § 2106.04(a)(2)(III).)
Step 2A Prong 2: The claim does not recite additional element that integrates the judicial exception into a practical application.
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception.
Therefore, claim 10 is ineligible.
Regarding Claim 11,
The claim recites similar limitations as corresponding claim 1. Therefore, the same analysis (subject matter eligibility analysis) that was utilized for claim 1, as described above, is equally applicable to claim 11. The only difference is that claim 1 is drawn to a method, and claim 11 is drawn to a non-transitory storage medium. The recitation of “a non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations...” merely defines computer component and instructions to implement a judicial exception, and hence the claimed additional elements listed above are merely generic elements and the implementation of the elements merely amount to no more than instruction to apply the abstract idea using a generic computer component. Therefore, the additional elements do not integrate the judicial exception into a practical application. See MPEP 2106.05(f).
Therefore, claim 11 is ineligible.
Regarding Claim 12,
The claim recites similar limitations as corresponding claim 2. Therefore, the same subject matter eligibility analysis (including the abstract idea) that was utilized for claim 2, as described above, is equally applicable to claim 12.
Therefore, claim 12 is ineligible.
Regarding Claim 13,
The claim recites similar limitations as corresponding claim 3. Therefore, the same subject matter eligibility analysis (including the abstract idea) that was utilized for claim 3, as described above, is equally applicable to claim 13.
Therefore, claim 13 is ineligible.
Regarding Claim 14,
The claim recites similar limitations as corresponding claim 4. Therefore, the same subject matter eligibility analysis (including the abstract idea) that was utilized for claim 4, as described above, is equally applicable to claim 14.
Therefore, claim 14 is ineligible.
Regarding Claim 15,
The claim recites similar limitations as corresponding claim 5. Therefore, the same subject matter eligibility analysis (including the abstract idea) that was utilized for claim 5, as described above, is equally applicable to claim 15.
Therefore, claim 15 is ineligible.
Regarding Claim 16,
The claim recites similar limitations as corresponding claim 6. Therefore, the same subject matter eligibility analysis (including the abstract idea) that was utilized for claim 6, as described above, is equally applicable to claim 16.
Therefore, claim 16 is ineligible.
Regarding Claim 17,
The claim recites similar limitations as corresponding claim 7. Therefore, the same subject matter eligibility analysis (including the abstract idea) that was utilized for claim 7, as described above, is equally applicable to claim 17.
Therefore, claim 17 is ineligible.
Regarding Claim 18,
The claim recites similar limitations as corresponding claim 8. Therefore, the same subject matter eligibility analysis (including the abstract idea) that was utilized for claim 8, as described above, is equally applicable to claim 18.
Therefore, claim 18 is ineligible.
Regarding Claim 19,
The claim recites similar limitations as corresponding claim 9. Therefore, the same subject matter eligibility analysis (including the abstract idea) that was utilized for claim 9, as described above, is equally applicable to claim 19.
Therefore, claim 19 is ineligible.
Regarding Claim 20,
The claim recites similar limitations as corresponding claim 10. Therefore, the same subject matter eligibility analysis (including the abstract idea) that was utilized for claim 10, as described above, is equally applicable to claim 20.
Therefore, claim 20 is ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1, 3, 6-7, 10-11, 13, 16, 17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Mansukhani et al., (NPL: “Data Drift Detection for Image Classifiers” (2019)) in view of Sethi et al., (NPL: "On the Reliable Detection of Concept Drift from Streaming Unlabeled Data." (2017)).
Regarding Claim 1,
Mansukhani discloses the following:
A method, comprising:
receiving a stream of unlabeled data samples from a model; (Mansukhani, [Preventing Silent Model Degradation in Production] “In the context of machine learning, we consider data drift1 to be the change in model input data that leads to a degradation of model performance. In the remainder of this article, we shall cover how to detect data drift for models that ingest image data as their input in order to prevent their silent degradation in production.” [Abstract] “Model Monitoring: The Approach: Our approach does not make any assumptions about the model that has been deployed, but it requires access to the training data used to build the model and the prediction data used for scoring. The intuitive approach to detect data drift for image data is to build a machine learned representation of the training dataset and to use this representation to reconstruct data that is being presented to the model. If the reconstruction error is high then the data being presented to the model is different from what it was trained on. The sequence of operations is as follows: 1. Learn a low dimensional representation of the training dataset (encoder) 2. Reconstruct a validation dataset using the representation from Step 1 (decoder) and store reconstruction loss as baseline reconstruction loss 3.Reconstruct batch of data that is being sent for predictions using encoder and decoder in Steps 1 and 2; store reconstruction loss. If reconstruction loss of dataset being used for predictions exceeds the baseline reconstruction loss by a predefined threshold set off an alert.” [Image Data Drift Detection in Action] “Step 4:Generate the test, train and noisy MNIST data sets.”) [Examiner’s Note: The new batch of data obtained from a deployed model represents the unlabeled data.]
obtaining a first reconstruction error for the unlabeled data samples; (Mansukhani, [Image Data Drift Detection in Action] “Step 4:Generate the test, train and noisy MNIST data sets. ... Step 5:Train the convolutional autoencoder. ... The validation loss we obtained is 0.1011. ... Step 6:Get the reconstruction loss for the noisy MNIST data set. Sample of noisy input data (top row) and reconstructed images (bottom row)... The reconstruction error that we obtained on the noisy MNIST data set is 0.1024. Compared with the baseline reconstruction error on the validation dataset this is a 1.3% increase in the error. Step 7: Get the reconstruction loss for the nonMNIST dataset; compare with validation loss for MNIST dataset and noisy MNIST…. Sample of nonMNIST data (top row) and its corresponding reconstruction (bottom row)... The reconstruction error that we obtained on the nonMNIST data set is 0.1458. Compared with the baseline reconstruction error on the validation dataset, this is a 44.2% increase in the error.”)
obtaining a second reconstruction error for a set of normative data; (Mansukhani, [Image Data Drift Detection in Action] “Step 4:Generate the test, train and noisy MNIST data sets. ... Step 5:Train the convolutional autoencoder. ... The validation loss we obtained is 0.1011. ... Step 6:Get the reconstruction loss for the noisy MNIST data set. Sample of noisy input data (top row) and reconstructed images (bottom row)... The reconstruction error that we obtained on the noisy MNIST data set is 0.1024. Compared with the baseline reconstruction error on the validation dataset this is a 1.3% increase in the error. Step 7: Get the reconstruction loss for the nonMNIST dataset; compare with validation loss for MNIST dataset and noisy MNIST…. Sample of nonMNIST data (top row) and its corresponding reconstruction (bottom row)... The reconstruction error that we obtained on the nonMNIST data set is 0.1458. Compared with the baseline reconstruction error on the validation dataset, this is a 44.2% increase in the error.”)
signaling drift in the performance of the model when said new proportion differs from said initial proportion by more than a predefined tolerance threshold. (Mansukhani, [Model Monitoring:] “The sequence of operations is as follows: 1. Learn a low dimensional representation of the training dataset (encoder) 2. Reconstruct a validation dataset using the representation from Step 1 (decoder) and store reconstruction loss as baseline reconstruction loss 3.Reconstruct batch of data that is being sent for predictions using encoder and decoder in Steps 1 and 2; store reconstruction loss. If reconstruction loss of dataset being used for predictions exceeds the baseline reconstruction loss by a predefined threshold set off an alert.” [Image Data Drift Detection in Action] “The reconstruction error that we obtained on the nonMNIST data set is 0.1458. Compared with the baseline reconstruction error on the validation dataset, this is a 44.2% increase in the error. From this example, it is clear that there is a spike in the reconstruction loss when the convolution encoder is made to reconstruct a dataset that is different than the one used to train the model. The next step as a result of this detection is to either retrain the model with new data or to investigate what led to a change in the data.” [Conclusion] “In this example, we saw that a convolutional autoencoder is capable of quantifying differences in image datasets on the basis of a reconstruction error. Using this approach of comparing reconstruction errors, we can detect changes in the input that is being presented to image classifiers.”)
[Examiner’s Note: Mansukhani teaches a process of monitoring and detecting drift in a deployed model (specifically, an autoencoder for image classifiers). The disclosed approach obtains production/live data samples (i.e., unlabeled data being sent for predictions) and baseline validation data (i.e., normative data from the training phase), obtains reconstruction errors on the production data (i.e., the first reconstruction error) and obtains reconstruction errors on the validation dataset (i.e., the second reconstruction error “baseline reconstruction loss”), and compares the reconstruction loss values between both dataset samples to determine if the reconstruction error of the production data “exceed exceeds the baseline reconstruction loss by a predefined threshold” to set an alert (i.e., signaling drift in the performance of the model when difference exceeds a predefined tolerance threshold). The reference demonstrate this approach by establishing a baseline reconstruction error of 0.1011 on the validation dataset (normative data); the noisy MNIST test and nonMNIST data (unlabeled production data) have reconstruction errors of 0.1024 and 0.1458, representing 1.3% and 44.2% increases form baseline respectively, where larger increase indicates drift requiring model retraining.]
While Mansukhani detects drift and signals an alert for model retraining based on comparing the difference of reconstruction errors to a threshold, Mansukhani does not appear to explicitly teach the following:
defining a margin based on the first reconstruction error and the second reconstruction error;
computing an initial proportion of samples from the set of normative data whose reconstruction errors fall within a range of reconstruction errors defined by the margin;
computing a new proportion of unlabeled data samples that fall within the range of reconstruction errors defined by the margin; and
signaling drift based on comparing proportions.
However, Mansukhani in view of Sethi teaches the following:
receiving a stream of unlabeled data samples from a model; (Sethi, [Pp. 13-14, Section: 4)] “The MD3 algorithm, being a streaming data algorithm, needs to operate with limited memory of past information, needs to continuously process data indefinitely over a single pass, and has to provide a quick response time. The change in margin density (∆MD) is used as a metric for detecting drift in a streaming environment. The incremental classification process continuously receives unlabeled samples X and predicts their class labels Y, based on the classification model C, as shown in Figure 10.”)
defining a margin based on the first reconstruction error and the second reconstruction error; (Sethi, [P. 2, Section: 1] “The Margin Density Drift Detection (MD3) methodology, proposed in this paper, monitors the number of samples in a classifier’s region of uncertainty (its margin), to detect drifts.” [P. 9, Section: 3.2.] “3.2. Computing the Margin Density (MD) metric The Margin Density metric (MD) is a univariate measure, which can be tracked over time to detect drifts from unlabeled data. Margin density is defined as: Definition 1. Margin Density (MD): The expected number of data samples that fall within a robust classifier’s (one that distributes importance weights among its features) region of uncertainty, i.e. its margin.” See equations (7) and (8). [P. 13, Section: 4] “The change in margin density (∆MD) is used as a metric for detecting drift in a streaming environment. The incremental classification process continuously receives unlabeled samples X and predicts their class labels Y, based on the classification model C, as shown in Figure 10. At any given time t, the signal function S (Xt) computes if the sample Xt lies within the margin of C. This computation is performed using Equations 7 and 8, based on the type of model used. This signal is used to update the expected margin density. A significant change in the margin density at time t (MDt) signals a change which requires further inspection.”) [Examiner’s Note: the classifier margin / uncertainty region MD defines the range within which samples contribute to both MDRef and MDt, the margin of both metrics of the classifier on the reference data (second reconstruction error) and unlabeled data (first reconstruction error).]
computing an initial proportion of samples from the set of normative data whose reconstruction errors fall within a range of reconstruction errors defined by the margin; (Sethi, [P. 10, Section: 3.2.2.] “The margin density MD for this type of models is computed by measuring the number of samples which have high uncertainty, as given by Equation 8.” [Pp. 13-14, Section: 4] “The MD3 algorithm (Algorithm 2), begins with an initial trained classifier C, which is obtained by learning from the initial labeled training dataset, before the model is made online. From this initial training dataset, a reference distribution summarizing margin and performance characteristics of the dataset, is learned. This reference distribution comprises of the expected margin density - MDRef , the acceptable deviation of the margin density metric- σRef , expected accuracy on the training dataset - AccRef and its deviation σAcc. These values are learned from the training dataset by using the K-fold cross validation technique,”) [Examiner’s Note: the MDRef represents the initial proportion of training (normative) samples falling within the MD margin-defined range.]
computing a new proportion of unlabeled data samples that fall within the range of reconstruction errors defined by the margin; and (Sethi, [Pp. 13-14, Section: 4] “At any given time t, the signal function S (Xt) computes if the sample Xt lies within the margin of C. This computation is performed using Equations 7 and 8, based on the type of model used. This signal is used to update the expected margin density. A significant change in the margin density at time t (MDt) signals a change which requires further inspection. ... Here, the margin density at a time t, given by MDt , is computed incrementally by using a forgetting factor λ on - MDt−1, and combining it with the signal function S (Xt), which indicates if the current sample Xt falls within the margin of the classifier C.” [Algorithm 2] “Compute margin inclusion signal 5-6”.) [Examiner’s Note: the MDt represents the new proportion of incoming unlabeled samples that fall within the defined margin.]
signaling drift in the performance of the model when said new proportion differs from said initial proportion by more than a predefined tolerance threshold. (Sethi, [p. 11, Section: 3.3] “A change scenario is setup by generating an initial distribution of 500 samples, used for training a model, and then generating 500 additional samples from a changed distribution, for testing the model. The change in margin density (∆MD) is evaluated as the difference in margin densities of the training and test data: ∆MD = |MDT rain − MDT est|. By comparing ∆MD with changes in the training and testing error (∆Err), which is representative of a metric used by fully labeled drift detectors, the effectiveness of MD to detect true drifts is evaluated.” [Pp. 13-14, Section: 4] “If the performance of C, on the Ntrain labeled samples, is found to have degraded, a drift is confirmed and the model is retrained using these labeled samples collected. In the MD3 approach, there is no need for continuous monitoring using labeled samples, as the drift detection process is unsupervised. Labeling is requested only when a drift is suspected, for confirmation and retraining. .... The same sensitivity parameter is used to detect significant drop in performance, for the obtained labeled samples, from the reference accuracy values, as per (10) (Line 14).
i
f
|
M
D
t
-
M
D
R
e
f
|
>
θ
*
σ
R
e
f
⇒
D
r
i
f
t
s
u
s
p
e
c
t
e
d
(9)
i
f
(
A
c
c
R
e
f
-
A
c
c
L
a
b
e
l
e
d
S
a
m
p
l
e
s
)
>
θ
*
σ
A
c
c
⇒
D
r
i
f
t
c
o
n
f
i
r
m
e
d
(10).” [Algorithm line 7].”) [Examiner’s Note: Sethi signals/confirms drifts based on the difference between the margin density of both dataset (i.e., new proportion differs from initial proportion) compared to a threshold
θ
.]
Accordingly, at the effective filing date, it would have been prima facie obvious to one ordinarily skilled in the art of machine learning to modify the combination of Mansukhani and Sethi to incorporate the Margin Density Drift Detection methods as taught by Sethi. One would have been motivated to make such a combination in order to obtain an algorithm for reliably detecting drifts from data streams, with significantly fewer false alarms. Thereby, reduced false alarms enables the signaling of drifts only when they are most likely to affect classification performance. (Sethi [Abstract]).
Regarding Claim 3, Mansukhani in view of Sethi teaches the elements of claim 1 as outlined above, and further teaches:
wherein when the drift is signaled, the model is retrained, and the margin and proportion are recomputed. (Sethi, [Algorithm 2, Lines 13-16] “if
A
c
c
R
e
f
-
A
c
c
L
a
b
e
l
e
d
S
a
m
p
l
e
s
>
Θ
*
σ
A
c
c
then 14 // Drift Confirmed 15 Retrain C with Labeled Samples Update Reference distribution (
M
D
R
e
f
,
σ
R
e
f
,
A
c
c
R
e
f
,
σ
R
e
f
)” [Pp. 13-14, Section: 4] “If the performance of C, on the Ntrain labeled samples, is found to have degraded, a drift is confirmed and the model is retrained using these labeled samples collected. ... These values are learned from the training dataset by using the K-fold cross validation technique... The average values and standard deviation, of the test accuracy and margin density, over the K iterations is used to form the reference distribution. ... A drop in accuracy confirms that the change is indeed a result of concept drift and that model retraining is necessary, to update the classifier C. Once retraining is performed, a new reference distribution (
M
D
R
e
f
,
σ
R
e
f
,
A
c
c
R
e
f
,
σ
R
e
f
)” is learned from the LabeledSamples set, based on the K-fold cross validation technique described above.”) [Examiner’s Note: Sethi teaches retraining the model and updating the margin/proportion parameters MD when drift is confirmed.]
Regarding Claim 6,
Mansukhani in view of Sethi teaches the elements of claim 1 as outlined above, and further teaches:
further comprising comparing a sequence of differences between the current proportions and the initial proportion to determine a drift in the performance of the model. (Sethi, [Pp. 13-14, Section: 4] “This signal is used to update the expected margin density. A significant change in the margin density at time t (MDt) signals a change which requires further inspection. .... These values are then used to signal change based on the desired level of sensitivity, given by parameter θ. Change is signaled when the margin density at a time t, given by MDt , deviates by more than θ standard deviations from the reference margin density value MDRef , as given by (9) (Line 7). The same sensitivity parameter is used to detect significant drop in performance, for the obtained labeled samples, from the reference accuracy values, as per (10) (Line 14).” ) [Examiner’s Note: Under the broadest reasonable interpretation (sequential/streaming comparison), Algorithm 2 processes stream sequentially (for t = 1, 2, 3, ....), compute the difference at each time, and compares it to threshold continuously, this is “comparing sequence of differences” in the streaming data. ]
Regarding Claim 7,
Mansukhani in view of Sethi teaches the elements of claim 1 as outlined above, and further teaches:
wherein boundaries of the margin are defined by a plot of the second reconstruction error. (Sethi, [P. 10, Section: 3.2.1.] “The trained SVM model, has an expected number of samples in the margin, due to its soft constraints. The margin density here is given by the ratio of samples which fall inside the margin of the SVM, as per Equation 7. The signal function S (w,b)(x) checks if a given sample x falls within the margin of the SVM, with parameters w and b.” [p. 23, Section: 5.4.1.] “The results of the varying the θmargin are shown in Figure 16. It is seen that the choice of the parameter θmargin does not have a significant effect on the final results, as all accuracy plots follow a similar trajectory in time. The only failure case is seen in case of a θmargin=0.05 for the phishing dataset (Figure 16 c) gray). At this margin width, the samples captured are insufficient to detect drift effectively. The margin density signal is depicted in Figure 17, for the phishing and the nsl-kdd dataset.”)
Regarding Claim 10, Mansukhani in view of Sethi teaches the elements of claim 1 as outlined above, and further teaches:
wherein a size of the margin is variable based on constraints associated with an application domain where the model is deployed. (Sethi, [Abstract] “The MD3 algorithm is a distribution independent, application independent, model independent, unsupervised and incremental algorithm for reliably detecting drifts from data streams. Experimental evaluation on 6 drift induced datasets and 4 additional datasets from the cybersecurity domain demonstrates that the MD3 approach can reliably detect drifts, with significantly fewer false alarms compared to unsupervised feature based drift detectors. At the same time, it produces performance comparable to that of a fully labeled drift detector.” [Pp. 13-14, Section: 4] “By allowing users to specify the intuitive parameter of sensitivity, suggested to be picked in the range of [0,3], the entire change detection process is made flexible to be used in different streaming environments. A larger value can be set if frequent signaling is not desired, alternatively a lower value could be used for critical applications where small changes could be harmful, if undetected.” [p. 23, Section: 5.4.2.] “In this section, we evaluate the effect of varying the parameter θmargin for a MD3-RS model, which uses C4.5 decision tree models as its base classifier type. The results of the varying the θmargin are shown in Figure 16. It is seen that the choice of the parameter θmargin does not have a significant effect on the final results.”) [Examiner’s Note: Sethi teaches the sensitivity parameter and margin control size, are adjustable, and provides guidance on setting them based on application domain constraints (critical cybersecurity concerns, etc.).]
Regarding Claim 11,
The claim recites substantially similar limitation as corresponding claim 1 and is rejected for similar reasons as claim 1 using similar teachings and rationale. Claim 1 is directed to a method, and claim 11 is directed to non-transitory storage medium.
Mansukhani in view of Sethi also discloses a computer program executed on a computer to implement the drift detection.
Regarding Claim 13,
The claim recites substantially similar limitations as corresponding claim 3 and is rejected for similar reasons as claim 3 using similar teachings and rationale.
Regarding Claim 16,
The claim recites substantially similar limitations as corresponding claim 6 and is rejected for similar reasons as claim 6 using similar teachings and rationale.
Regarding Claim 17,
The claim recites substantially similar limitations as corresponding claim 7 and is rejected for similar reasons as claim 7 using similar teachings and rationale.
Regarding Claim 20,
The claim recites substantially similar limitations as corresponding claim 20 and is rejected for similar reasons as claim 20 using similar teachings and rationale.
Claim(s) 2, 4-5, 8-9, 12, 14-15, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Mansukhani in view of Sethi as described above, and further in view of Hines et al., (Pub. No.: US 20220383038 A1).
Regarding Claim 2, Mansukhani in view of Sethi teaches the elements of claim 1 as outlined above, and further teaches:
Mansukhani in view of Sethi teaches unsupervised event detection approach applicable to event detection in cybersecurity and adversarial domains (detecting attacks, spam, malware) and defines the model is being deployed in the real world operate in a dynamic environment.
Specifically, Mansukhani discloses: “In such cases, methods from statistical process control and operations research that rely primarily on numerical data are hard to adopt and necessitates a new approach to monitoring models in production. This article explores an approach that can be used to detect data drift for models that classify/score image data. ... Our approach does not make any assumptions about the model that has been deployed, but it requires access to the training data used to build the model and the prediction data used for scoring.”
Sethi also discloses: [Abstract] “Classifiers deployed in the real world operate in a dynamic environment, where the data distribution can change over time. ... The MD3 algorithm is a distribution independent, application independent, model independent, unsupervised and incremental algorithm for reliably detecting drifts from data streams.” [P. 2, Section: 1] “Streaming data applications need to be able to operate and detect drifts from unlabeled, or at most sparsely labeled data, to be of any real use.” [P. 13, Section: 4] “The MD3 algorithm, being a streaming data algorithm, needs to operate with limited memory of past information, needs to continuously process data indefinitely over a single pass, and has to provide a quick response time.” [Pp. 19-20, Section: 5.3] “Machine learning models deployed in real world applications operate in a dynamic environment where concept drift can occur at any time. ... Especially in the domain of streaming cybersecurity applications, an overly responsive system is a serious problem, as it is vulnerable to malicious manipulation of the training process. Also, labeling is a time consuming and expensive task. A system which frequently requires manual intervention is less likely to be trusted and can cause experts to disregard its warnings.”
While Mansukhani in view of Sethi does not define the domain in which the model is being deployed to as “mobile edge device.” The broader deployment of real world applications would encompass mobile edge devices. Accordingly, it would have been obvious in view of Hines.
Hereinafter, Hines, in combination with Mansukhani and Sethi, teaches:
wherein the model is an unsupervised event detection model operable to detect events in a domain in which mobile edge devices are deployed. (Hines, [0034] “The compute device 160 can be, for example, a user device that deploys a trained machine learning model (e.g., packages the trained machine learning model received from the data drift detection device 110 in a lightweight file (e.g., a 1 megabytes file, a 2 megabytes file, a 10 megabytes file, a 20 megabytes file, a 100 megabytes file, a 200 megabytes file, etc.) and executes the lightweight file based on application-specific data received by the compute device 160). The compute device 160 can include, for example, a laptop computer, a desktop computer, a mobile phone, and/or the like, of a user (e.g., an administrator of a business, an operator of a machine, a customer of a store, and/or the like). In some instances, the compute device 160 can receive and deploy the trained machine learning model from the data drift detection device 110 and execute the deployed machine learning model on user-specific data.”)
Accordingly, it would have been prima facie obvious to one having ordinary skill in the art, before the effective filing date of the claimed invention, having the combination of Mansukhani, Sethi, and Hines, to incorporate the data drift detection method/device as taught by Hines. One would have been motivated to make such a combination in order to improve the machine learning model's efficiency and usability (Hines [0028]).
Regarding Claim 4, Mansukhani in view of Sethi teaches the elements of claim 1 as outlined above, and further teaches:
While Mansukhani in view of Sethi teaches the stream of unlabeled data samples generated from a devices, Mansukhani in view of Sethi does not appear to explicitly indicate that the stream data being generated by “one or more mobile edge nodes.”
However, it would have been obvious in view of Hines. Hereinafter, Hines, in combination with Mansukhani and Sethi, teaches:
wherein the stream of unlabeled data samples is generated by one or more mobile edge nodes. (Hines, [0034]-[0035] “The compute device 160 can include, for example, a laptop computer, a desktop computer, a mobile phone, and/or the like, of a user (e.g., an administrator of a business, an operator of a machine, a customer of a store, and/or the like). In some instances, the compute device 160 can receive and deploy the trained machine learning model from the data drift detection device 110 and execute the deployed machine learning model on user-specific data. ... second set of data (e.g., production data received at the data drift detection device 110 from the database 170 and/or the compute device 160) processed by the machine learning model after training and/or deployment (during a production phase).”)
The same motivation that was utilized for combining Mansukhani, Sethi, and Hines as set forth in claim 2 is equally applicable to claim 4.
Regarding Claim 5, Mansukhani in view of Sethi teaches the elements of claim 1 as outlined above.
Mansukhani in view of Sethi does not appear to explicitly teach:
wherein prior to receiving the stream of unlabeled data samples, a model that performs the signaling of the drift is trained using a combination of anomalous data and the normative data.
However, Hines, in combination with Mansukhani and Sethi, teaches:
wherein prior to receiving the stream of unlabeled data samples, a model that performs the signaling of the drift is trained using a combination of anomalous data and the normative data. (Hines, [0044] “At 203, a machine learning model (including a set of model parameters (e.g., nodes, weights, etc.)) is trained based on the first set of representations and the first set of distributions. ... After the machine learning model is trained (and is deployed to production), at 204, a second set of representations is extracted from second set of data different from the first set of data.” [0048]-[0049] “At 701, a first data set is received from a first remote compute device (e.g., database 170 or compute device 160). ... At 704, a machine learning model (e.g., machine learning model 115) is trained based on the first plurality of representations and the first set of distributions. ... FIG. 8 shows a flowchart of a method 800 for generating and using a set of anomaly scores, according to an embodiment. In some implementations, method 800 can be performed by processor 113 of the data drift detection device 110.”)
The same motivation that was utilized for combining Mansukhani, Sethi, and Hines as set forth in claim 2 is equally applicable to claim 5.
Regarding Claim 8,
Mansukhani in view of Sethi teaches the elements of claim 1 as outlined above, and further teaches:
While Mansukhani in view of Sethi teaches the deployment of the model to real-world applications/devices, Mansukhani in view of Sethi does not appear to explicitly recite “wherein the model is deployed at each of a plurality of edge nodes.”
However, Hines, in combination with Mansukhani and Sethi, teaches:
wherein the model is deployed at each of a plurality of edge nodes. (Hines, [0034] “The compute device 160 can be, for example, a user device that deploys a trained machine learning model (e.g., packages the trained machine learning model received from the data drift detection device 110 in a lightweight file (e.g., a 1 megabytes file, a 2 megabytes file, a 10 megabytes file, a 20 megabytes file, a 100 megabytes file, a 200 megabytes file, etc.) and executes the lightweight file based on application-specific data received by the compute device 160). The compute device 160 can include, for example, a laptop computer, a desktop computer, a mobile phone, and/or the like, of a user (e.g., an administrator of a business, an operator of a machine, a customer of a store, and/or the like).”
The same motivation that was utilized for combining Mansukhani, Sethi, and Hines as set forth in claim 2 is equally applicable to claim 8.
Regarding Claim 9,
Mansukhani in view of Sethi teaches the elements of claim 1 as outlined above, and further teaches:
While Mansukhani in view of Sethi processes the stream of unlabeled data samples from streaming environment, Mansukhani in view of Sethi does not appear to explicitly define the data samples comprises “data about a movement and/or a position of a physical mobile edge device.”
However, Hines, in combination with Mansukhani and Sethi, teaches:
wherein the stream of unlabeled data samples comprises data about a movement and/or a position of a physical mobile edge device. (Hines, [0023] “The data used by the data drift detection device 110 can include, for example, unstructured data (e.g., sensor data in Internet of Things (IoT) application(s), media and entertainment content, business documents (e.g., invoices, records, etc.), publications and listings, and/or the like), semi-structured data (e.g., emails, Hyper Text Markup Language (HTML) files, not only structured query language (NoSQL) databases, and/or the like), and/or structured data (e.g., tabular data, columnar data, and/or the like). The data can include, for example, imagery data, video data, audio data, text data, natural language data, time series data, tabular data, and/or the like. In some instances, for example, the data can include surveillance camera data, email data, diet data, preference data, fitness data, medical record data, financial data, mobile location data, demographic data, behavioral data, transaction data, loyalty data, and/or the like.”)
The same motivation that was utilized for combining Mansukhani, Sethi, and Hines as set forth in claim 2 is equally applicable to claim 9.
Regarding Claim 12,
The claim recites substantially similar limitations as corresponding claim 2 and is rejected for similar reasons as claim 2 using similar teachings and rationale.
Regarding Claim 14,
The claim recites substantially similar limitations as corresponding claim 4 and is rejected for similar reasons as claim 4 using similar teachings and rationale.
Regarding Claim 15,
The claim recites substantially similar limitations as corresponding claim 5 and is rejected for similar reasons as claim 5 using similar teachings and rationale.
Regarding Claim 18,
The claim recites substantially similar limitations as corresponding claim 8 and is rejected for similar reasons as claim 8 using similar teachings and rationale.
Regarding Claim 19,
The claim recites substantially similar limitations as corresponding claim 9 and is rejected for similar reasons as claim 9 using similar teachings and rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
(Pub. No.: US 20220198279 A1) – “Rashmi Nandipura Sundareswara” relates to “Data-Driven Methodology for Automatic Detection of Data Drift.”
(Pub. No.: US 20220374684 A1) – “Sonali SYNGAL” relates to “Artificial intelligence based methods and systems for improving classification of edge cases.”
(Pub. No.: US 20240250975 A1) – “Vahid POURAHMADI” relates to “Enhanced anomaly detection for distributed networks based at least on outlier exposure.”
NPL: Xu, Yiming, "Concept drift and covariate shift detection ensemble with lagged labels." (2021).
NPL: Dos Reis, Denis Moreira, et al. "Fast unsupervised online drift detection using incremental kolmogorov-smirnov test." (2016).
NPL: Greco, Salvatore, "Drift lens: Real-time unsupervised concept drift detection by evaluating per-label embedding distributions." (2021).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SADIK ALSHAHARI whose telephone number is (703)756-4749. The examiner can normally be reached Monday - Friday, 9 a.m. 6 p.m. ET.
Examiner interviews are available via telephone, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li Zhen can be reached on (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.A.A./Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121