DETAILED ACTION
This non-final Office action is responsive to Applicant’s Election filed January 20, 2026. Applicant has elected Group I (claims 1-11) without traverse. Non-elected claims 12-20 stand as withdrawn. Claims 1-11 are examined below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-11 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Claims 1-11 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claimed invention is directed to “generating and/or providing experiment metric values for anytime valid experimentation” (Spec: ¶ 2) without significantly more.
Step
Analysis
1: Statutory Category?
No – Claims 1-11 are directed to one or more computer-readable storage media having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform the recited operations. The computer-readable storage media may include signals per se. Therefore, claims 1-11 do not fall within at least one of the four categories of patent eligible subject matter. It is noted that, while paragraph 88 of Applicant’s Specification states, “Computer storage media excludes signals per se,” subsequent paragraph 89 describes communication media and states, “Combination of any of the above should also be included within the scope of computer-readable media.” Not only do the claims recite “computer-readable storage media” as opposed to “computer storage media,” but paragraph 89 of Applicant’s Specification opens the door to the possibility of computer-readable storage media being signals per se.
** In the interest of compact prosecution, claims 1-11 will continue to be examined below. However, appropriate correction is required.
Once it is clarified that the computer-readable storage media of claims 1-11 are limited to non-transitory media, claims 1-11 will be interpreted as article of manufacture claims.
Independent claims:
Step
Analysis
2A – Prong 1: Judicial Exception Recited?
Yes – Aside from the additional elements identified in Step 2A – Prong 2 below, the claims recite:
[Claim 1] obtaining a set of parameter values associated with an experiment using asymptotic confidence sequences, the set of parameter values including a minimal detectable effect and an uncertainty interval;
determining an expected sample size for executing the experiment based on the minimal detectable effect and the uncertainty interval; and
providing the expected sample size for utilization in association with the experiment using asymptotic confidence sequences.
Aside from the additional elements, the aforementioned claim details exemplify the abstract idea(s) of a mental process (since the details include concepts performed in the human mind, including an observation, evaluation, judgment, and/or opinion). As explained in MPEP § 2106(a)(2)(C)(III), “The courts consider a mental process (thinking) that ‘can be performed in the human mind, or by a human using a pen and paper’ to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). As the Federal Circuit explained, ‘methods which can be performed mentally, or which are the equivalent of human mental work, are unpatentable abstract ideas the ‘basic tools of scientific and technological work’ that are open to all.’’ 654 F.3d at 1371, 99 USPQ2d at 1694 (citing Gottschalk v. Benson, 409 U.S. 63, 175 USPQ 673 (1972)).” The limitations reproduced above, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitations in the mind but for the recitation of generic computer components. That is, other than reciting the additional elements identified in Step 2A – Prong 2 below, nothing in the claim elements precludes the steps from practically being performed in the mind and/or by a human using a pen and paper. For example, but for the recitations of generic computer and other processing components (identified in Step 2A – Prong 2 below), the respectively recited steps/functions of the claims, as drafted and set forth above, are a process that, under its broadest reasonable interpretation, covers performance of the limitations in the mind and/or with the use of pen and paper. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind (and/or with pen and paper) but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
The claims recite various examples of statistical analysis and other mathematical relationships, thereby presenting mathematical concepts.
2A – Prong 2: Integrated into a Practical Application?
No – The judicial exception(s) is/are not integrated into a practical application.
Claim 1 recites one or more computer-readable storage media having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform the recited operations.
The claims as a whole merely describe how to generally “apply” the abstract idea(s) in a computer environment. The claimed processing elements are recited at a high level of generality and are merely invoked as a tool to perform the abstract idea(s). Simply implementing the abstract idea(s) on a general-purpose processor is not a practical application of the abstract idea(s); Applicant’s specification discloses that the invention may be implemented using general-purpose processing elements and other generic components (Spec: ¶¶ 84-91).
The use of a processor/processing elements (e.g., as recited in all of the claims) facilitates generic processor operations. The use of a memory or machine-readable media with executable instructions facilitates generic processor operations.
The additional elements are recited at a high-level of generality (i.e., as generic processing elements performing generic computer functions) such that the incorporation of the additional processing elements amounts to no more than mere instructions to apply the judicial exception(s) using generic computer components. There is no indication in the Specification that the steps/functions of the claims require any inventive programming or necessitate any specialized or other inventive computer components (i.e., the steps/functions of the claims may be implemented using capabilities of general-purpose computer components). Accordingly, the additional elements do not integrate the abstract ideas into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea(s).
The processing components presented in the claims simply utilize the capabilities of a general-purpose computer and are, thus, merely tools to implement the abstract idea(s). As seen in MPEP § 2106.05(a)(I) and § 2106.05(f)(2), the court found that accelerating a process when the increased speed solely comes from the capabilities of a general-purpose computer is not sufficient to show an improvement in computer-functionality and it amounts to a mere invocation of computers or machinery as a tool to perform an existing process (see FairWarning IP, LLC v. Iatric Sys., 839 F.3d 1089, 1095, 120 USPQ2d 1293, 1296 (Fed. Cir. 2016)).
There is no transformation or reduction of a particular article to a different state or thing recited in the claims.
Additionally, even when considering the operations of the additional elements as an ordered combination, the ordered combination does not amount to significantly more than what is present in the claims when each operation is considered separately.
2B: Claim(s) Provide(s) an Inventive Concept?
No – The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception(s). As discussed above with respect to integration of the abstract idea(s) into a practical application, the use of the additional elements to perform the steps identified in Step 2A – Prong 1 above amounts to no more than mere instructions to apply the exceptions using a generic computer component(s). Mere instructions to apply an exception using a generic computer component(s) cannot provide an inventive concept. The claims are not patent eligible.
Dependent claims:
Step
Analysis
2A – Prong 1: Judicial Exception Recited?
Yes – Aside from the additional elements identified in Step 2A – Prong 2 below, the claims recite:
[Claim 2] wherein the set of parameter values further includes an empirical mean, a null hypothesis mean, and a total number of samples.
[Claim 3] wherein the minimal detectable effect represents a smallest effect size that the experiment can detect with a certain probability and significance level.
[Claim 4] wherein the minimal detectable effect is obtained based on a user input specifying the minimal detectable effect.
[Claim 5] wherein the uncertainty interval is determined using a quantile parameter value, a standard deviation parameter value, and an optimization parameter value.
[Claim 6] wherein the optimization parameter value is determined to optimize a boundary condition.
[Claim 7] wherein the expected sample size is determined via an optimization problem using the minimal detectable effect, the uncertainty level, and a confidence sequence associated with a null hypothesis at a time.
[Claim 8] wherein the expected sample size is determined using a root finding procedure to solve for an optimization problem.
[Claim 9] wherein providing the expected sample size for utilization comprises causing display of the expected sample size.
[Claim 10] wherein providing the expected sample size for utilization comprises employing the expected sample size in conducting the experiment.
[Claim 11] wherein the asymptotic confidence sequences maintains a one minus type I guarantee during continuous monitoring of experiment outcomes.
The dependent claims further present details of the abstract ideas identified in regard to the independent claims.
Aside from the additional elements, the aforementioned claim details exemplify the abstract idea(s) of a mental process (since the details include concepts performed in the human mind, including an observation, evaluation, judgment, and/or opinion). As explained in MPEP § 2106(a)(2)(C)(III), “The courts consider a mental process (thinking) that ‘can be performed in the human mind, or by a human using a pen and paper’ to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). As the Federal Circuit explained, ‘methods which can be performed mentally, or which are the equivalent of human mental work, are unpatentable abstract ideas the ‘basic tools of scientific and technological work’ that are open to all.’’ 654 F.3d at 1371, 99 USPQ2d at 1694 (citing Gottschalk v. Benson, 409 U.S. 63, 175 USPQ 673 (1972)).” The limitations reproduced above, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitations in the mind but for the recitation of generic computer components. That is, other than reciting the additional elements identified in Step 2A – Prong 2 below, nothing in the claim elements precludes the steps from practically being performed in the mind and/or by a human using a pen and paper. For example, but for the recitations of generic computer and other processing components (identified in Step 2A – Prong 2 below), the respectively recited steps/functions of the claims, as drafted and set forth above, are a process that, under its broadest reasonable interpretation, covers performance of the limitations in the mind and/or with the use of pen and paper. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind (and/or with pen and paper) but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
The claims recite various examples of statistical analysis and other mathematical relationships, thereby presenting mathematical concepts.
2A – Prong 2: Integrated into a Practical Application?
No – The judicial exception(s) is/are not integrated into a practical application.
The dependent claims include the additional elements of their independent claims.
Claims 1-11 recite one or more computer-readable storage media having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform the recited operations.
Claim 9 recites causing display of the expected sample size via a user interface.
The claims as a whole merely describe how to generally “apply” the abstract idea(s) in a computer environment. The claimed processing elements are recited at a high level of generality and are merely invoked as a tool to perform the abstract idea(s). Simply implementing the abstract idea(s) on a general-purpose processor is not a practical application of the abstract idea(s); Applicant’s specification discloses that the invention may be implemented using general-purpose processing elements and other generic components (Spec: ¶¶ 84-91).
The use of a processor/processing elements (e.g., as recited in all of the claims) facilitates generic processor operations. The use of a memory or machine-readable media with executable instructions facilitates generic processor operations.
The additional elements are recited at a high-level of generality (i.e., as generic processing elements performing generic computer functions) such that the incorporation of the additional processing elements amounts to no more than mere instructions to apply the judicial exception(s) using generic computer components. There is no indication in the Specification that the steps/functions of the claims require any inventive programming or necessitate any specialized or other inventive computer components (i.e., the steps/functions of the claims may be implemented using capabilities of general-purpose computer components). Accordingly, the additional elements do not integrate the abstract ideas into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea(s).
The processing components presented in the claims simply utilize the capabilities of a general-purpose computer and are, thus, merely tools to implement the abstract idea(s). As seen in MPEP § 2106.05(a)(I) and § 2106.05(f)(2), the court found that accelerating a process when the increased speed solely comes from the capabilities of a general-purpose computer is not sufficient to show an improvement in computer-functionality and it amounts to a mere invocation of computers or machinery as a tool to perform an existing process (see FairWarning IP, LLC v. Iatric Sys., 839 F.3d 1089, 1095, 120 USPQ2d 1293, 1296 (Fed. Cir. 2016)).
There is no transformation or reduction of a particular article to a different state or thing recited in the claims.
Additionally, even when considering the operations of the additional elements as an ordered combination, the ordered combination does not amount to significantly more than what is present in the claims when each operation is considered separately.
2B: Claim(s) Provide(s) an Inventive Concept?
No – The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception(s). As discussed above with respect to integration of the abstract idea(s) into a practical application, the use of the additional elements to perform the steps identified in Step 2A – Prong 1 above amounts to no more than mere instructions to apply the exceptions using a generic computer component(s). Mere instructions to apply an exception using a generic computer component(s) cannot provide an inventive concept. The claims are not patent eligible.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-4 and 7-11 are rejected under 35 U.S.C. 102(a)(1)/(a)(2) as being anticipated by Pekelis et al. (US 2017/0083429).
[Claim 1] Pekelis discloses one or more computer-readable storage media having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform operations (¶ 26) comprising:
obtaining a set of parameter values associated with an experiment (abstract, ¶ 32 – testing system, experimenter) using asymptotic confidence sequences (¶ 52 – “The statistics module 521 includes two innovations in particular: sequential hypothesis testing, and control of false discovery rate for multiple hypothesis testing.”; ¶ 119 – “As the experiment progresses, more thresholds am crossed and the p-value decreases, approaching some asymptotic value at n=∞. In terms of p-values, the statement that the probability that the α-level test is ever conclusive is α is equivalent to saying that the asymptotic p-values are uniformly distributed. So, at n=∞ the p-values are as aggressive as they can be, while still controlling Type I error. However during the experiment, the p-value is an overestimate of this asymptotic p-value so we are slightly more conservative: this can be considered the mathematical penalty we must pay to be free to stop tests as early as possible.”), the set of parameter values including a minimal detectable effect (¶ 53 – “In traditional statistics used in A/B testing such as fixed-horizon statistics; it presumes that a sample size is fixed in advance. In such contexts, fixed-horizon statistics is optimal in the following sense: given abound on die proportion of false positives, the probability of detecting any trite effect with the predetermined data set is maximized, irrespective of the size of that effect.”; ¶ 55 – “How can we take advantage of real-time data? Note that there is a subtlety in the optimality property of fixed-horizon statistics described above: while the power (probability of detection) is maximized for any true effect the power can still be very low if the effect is small compared with the sample size. To have adequate power for detecting small effects, the experimenter commits to a large sample size up front and this will be wasted effort if it turns out the true effect could have reliably been detected much sooner.”; ¶¶ 68-70, 116 – Various parameters are used to determine the degree of an effect.) and an uncertainty interval (¶¶ 42-46, 49, 93-99 – Confidence intervals are evaluated for parameters that may yield true or false data.);
determining an expected sample size for executing the experiment based on the minimal detectable effect and the uncertainty interval (¶ 55 – “How can we take advantage of real-time data? Note that there is a subtlety in the optimality property of fixed-horizon statistics described above: while the power (probability of detection) is maximized for any true effect the power can still be very low if the effect is small compared with the sample size. To have adequate power for detecting small effects, the experimenter commits to a large sample size up front and this will be wasted effort if it turns out the true effect could have reliably been detected much sooner.”; ¶ 56 – “When a sample size need not be set in advance, the data should let you learn the size of the effects to consider adaptively, and so optimize the sample size at which the test terminates.”; ¶¶ 148-149 --
PNG
media_image1.png
144
274
media_image1.png
Greyscale
;
¶¶ 42-46, 49, 93-99 – Confidence intervals are evaluated for parameters that may yield true or false data.; ¶ 107 – “Recall that our overall objective comprises two parts: to adaptively learn the effect size we should anticipate, and then to stop as soon as possible to detect that effect when it holds.”; In other words, given all of the considered parameters, Pekelis’ goal is to use the smallest sample size needed to yield results within acceptable boundaries and thresholds.); and
providing the expected sample size for utilization in association with the experiment using asymptotic confidence sequences (¶ 55 – “How can we take advantage of real-time data? Note that there is a subtlety in the optimality property of fixed-horizon statistics described above: while the power (probability of detection) is maximized for any true effect the power can still be very low if the effect is small compared with the sample size. To have adequate power for detecting small effects, the experimenter commits to a large sample size up front and this will be wasted effort if it turns out the true effect could have reliably been detected much sooner.”; ¶ 56 – “When a sample size need not be set in advance, the data should let you learn the size of the effects to consider adaptively, and so optimize the sample size at which the test terminates.”; (¶ 52 – “The statistics module 521 includes two innovations in particular: sequential hypothesis testing, and control of false discovery rate for multiple hypothesis testing.”; ¶ 119 – “As the experiment progresses, more thresholds am crossed and the p-value decreases, approaching some asymptotic value at n=∞. In terms of p-values, the statement that the probability that the α-level test is ever conclusive is α is equivalent to saying that the asymptotic p-values are uniformly distributed. So, at n=∞ the p-values are as aggressive as they can be, while still controlling Type I error. However during the experiment, the p-value is an overestimate of this asymptotic p-value so we are slightly more conservative: this can be considered the mathematical penalty we must pay to be free to stop tests as early as possible.”).
[Claim 2] Pekelis discloses wherein the set of parameter values further includes an empirical mean (¶ 137 – “A conversion rate difference calculated from 10,000 visitors will likely be closer to the true difference than one calculated from 100 visitors and will eventually be arbitrarily close to the true average, as long as the underlying process causing conversion does not change.”), a null hypothesis mean (¶ 84 – “Why are FDR estimates more actionable with multiple A/B tests? Mathematically, the main observation is that FDR reverses the conditional probability that defines Type I error. In classical statistics, a p-value reports estimates of the probability of the data given there is no difference between baseline and variation. False discovery rates reverse this conditional probability: in particular, FDR is an estimate of the probability of no difference (or a false detection) between the baseline and variation, given the data you observed.”;
PNG
media_image2.png
152
266
media_image2.png
Greyscale
A null hypothesis shows no significant difference between two tests, such as seen in A/B testing and/or in a comparison of a baseline to a variation.), and a total number of samples (¶ 109 – “This means that, conditional on the value An we observe after n visitors, at a later stage m>n the expected value of An is still An; there is no average drift upwards or downwards.”; ¶ 118 – “Given thresholds, it is easy to convert these tests into p-values. A p-value is the most conservative significance level at which the test is conclusive, so we take the p-value after n visitors to be the least α such that the threshold has already been crossed.”; ¶ 58 – “In one embodiment, n is used to index the number of visitors that have arrived. Further assume the allocation is evenly split so that n/2 visitors are in the baseline and variation as mentioned above; all the discussion below is equally valid (just with more notation) even if the allocation is unequal.”; ¶ 60 – “If a sample size of n is established, then fixed-horizon statistics would tell you that the test is conclusive at a given level α (e.g., 5%) if…”).
[Claim 3] Pekelis discloses wherein the minimal detectable effect represents a smallest effect size that the experiment can detect with a certain probability and significance level (¶¶ 40-41 –
PNG
media_image3.png
454
272
media_image3.png
Greyscale
;
¶ 44 –
PNG
media_image4.png
378
258
media_image4.png
Greyscale
;
¶¶ 148-149 --
PNG
media_image1.png
144
274
media_image1.png
Greyscale
;
In other words, an experimenter or another employee can set a minimum parameter threshold (like a level of confidence or conversion rate difference) that needs to be met for certain conditions, like declaring the end of a variation test.).
[Claim 4] Pekelis discloses wherein the minimal detectable effect is obtained based on a user input specifying the minimal detectable effect (¶¶ 40-41 –
PNG
media_image3.png
454
272
media_image3.png
Greyscale
;
¶ 44 –
PNG
media_image4.png
378
258
media_image4.png
Greyscale
;
¶¶ 148-149 --
PNG
media_image1.png
144
274
media_image1.png
Greyscale
;
In other words, an experimenter or another employee can set a minimum parameter threshold (like a level of confidence or conversion rate difference) that needs to be met for certain conditions, like declaring the end of a variation test.).
[Claim 7] Pekelis discloses wherein the expected sample size is determined via an optimization problem using the minimal detectable effect, the uncertainty level, and a confidence sequence associated with a null hypothesis at a time (¶ 55 – “How can we take advantage of real-time data? Note that there is a subtlety in the optimality property of fixed-horizon statistics described above: while the power (probability of detection) is maximized for any true effect the power can still be very low if the effect is small compared with the sample size. To have adequate power for detecting small effects, the experimenter commits to a large sample size up front and this will be wasted effort if it turns out the true effect could have reliably been detected much sooner.”; ¶ 56 – “When a sample size need not be set in advance, the data should let you learn the size of the effects to consider adaptively, and so optimize the sample size at which the test terminates.”; ¶¶ 148-149 --
PNG
media_image1.png
144
274
media_image1.png
Greyscale
;
¶¶ 42-46, 49, 93-99 – Confidence intervals are evaluated for parameters that may yield true or false data.; ¶ 107 – “Recall that our overall objective comprises two parts: to adaptively learn the effect size we should anticipate, and then to stop as soon as possible to detect that effect when it holds.”; In other words, given all of the considered parameters, Pekelis’ goal is to use the smallest sample size needed to yield results within acceptable boundaries and thresholds.; ¶ 53 – “In traditional statistics used in A/B testing such as fixed-horizon statistics; it presumes that a sample size is fixed in advance. In such contexts, fixed-horizon statistics is optimal in the following sense: given abound on die proportion of false positives, the probability of detecting any trite effect with the predetermined data set is maximized, irrespective of the size of that effect.”; ¶ 55 – “How can we take advantage of real-time data? Note that there is a subtlety in the optimality property of fixed-horizon statistics described above: while the power (probability of detection) is maximized for any true effect the power can still be very low if the effect is small compared with the sample size. To have adequate power for detecting small effects, the experimenter commits to a large sample size up front and this will be wasted effort if it turns out the true effect could have reliably been detected much sooner.”; ¶¶ 68-70, 116 – Various parameters are used to determine the degree of an effect.; ¶¶ 42-46, 49, 93-99 – Confidence intervals are evaluated for parameters that may yield true or false data.).
[Claim 8] Pekelis discloses wherein the expected sample size is determined using a root finding procedure to solve for an optimization problem (¶¶ 107-110 –
PNG
media_image5.png
386
264
media_image5.png
Greyscale
;
¶¶ 68-69, 112-113 – Additional discussions about calculations that determine if consecutive values converge to zero. This is an example of a root finding procedure.).
[Claim 9] Pekelis discloses wherein providing the sample size for utilization comprises causing display of the sample size via a user interface (¶ 34 – A number of users that receive each variation of a website is displayed. This number is indicative of a sample size.; ¶ 98 – user’s dashboard). Pekelis does not explicitly disclose that the provided and displayed content is specifically content regarding an expected sample size for utilization. These differences are only found in the non-functional descriptive material and are not functionally involved in any manipulative steps of the invention nor do they alter any recited structural elements; therefore, such differences do not effectively serve to patentably distinguish the claimed invention over the prior art. Any manipulative steps of the invention would be performed the same regardless of the specific data. Further, any structural elements remain the same regardless of the specific data. Thus, this descriptive material will not distinguish the claimed invention from the prior art in terms of patentability as the claimed invention fails to present a new and unobvious functional relationship between the descriptive material and the substrate, see In re Gulack, 703 F.2d 1381, 1385, 217 USPQ 401, 404 (Fed. Cir. 1983); In re Lowry, 32 F.3d 1579, 32 USPQ2d 1031 (Fed. Cir. 1994); In re Ngai, 367 F.3d 1336, 1336, 70 USPQ2d 1862, 1863-64 (Fed. Cir. 2004). Another indication of the existence of non-functional descriptive material is that the content of the material is merely “directed towards conveying a message or meaning to a human reader independent of the supporting product.” Please see MPEP § 2111.05(I)(B).
[Claim 10] Pekelis discloses wherein providing the expected sample size for utilization comprises employing the expected sample size in conducting the experiment (¶ 3 – “To obtain valid results in a conventional variation test such as a fixed-horizon variation test, a strict set of guidelines are followed when performing the variation test. The guidelines include setting a minimum detectable effect and sample size in advance to conducting the test, refraining from viewing the results of the variation test prior to the completion of the variation test, and refraining from testing too many goals and variations in the variation test.”; ¶ 6 – The disclosed invention improves upon the traditional testing by allowing an experimenter to continuously monitor results of a variation test.; ¶ 51 – “For example, most A/B testing platforms suggest controlling Type I error rate at around 5-10%, and choosing a sample size to ensure statistical power is at least 80%.”; ¶ 53 – “In traditional statistics used in A/B testing such as fixed-horizon statistics; it presumes that a sample size is fixed in advance. In such contexts, fixed-horizon statistics is optimal in the following sense: given abound on die proportion of false positives, the probability of detecting any trite effect with the predetermined data set is maximized, irrespective of the size of that effect.”).
[Claim 11] Pekelis discloses wherein the asymptotic confidence sequences maintains a one minus type I guarantee during continuous monitoring of experiment outcomes (¶ 32 – “However, the sequential test conducted by the statistics module 121 allows an experimenter to continuously monitor the results of the sequential test. That is, an experimenter is allowed to view the results of a sequential test being performed on the web page at any time and the statistics module 121 provides results of the variation test to the experimenter that are valid at ail [sic] times regardless of when the experimenter requests to view the results.”; ¶ 61 – “Given a desired control α on Type I error probability, the constant k can be chosen to ensure that the Type I error is equal to α.”; ¶ 119 – “As the experiment progresses, more thresholds am crossed and the p-value decreases, approaching some asymptotic value at n=∞. In terms of p-values, the statement that the probability that the α-level test is ever conclusive is α is equivalent to saying that the asymptotic p-values are uniformly distributed. So, at n=∞ the p-values are as aggressive as they can be, while still controlling Type I error. However during the experiment, the p-value is an overestimate of this asymptotic p-value so we are slightly more conservative: this can be considered the mathematical penalty we must pay to be free to stop tests as early as possible.”; ¶¶ 122, 126 – “confidence interval with 1−α coverage”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over Pekelis et al. (US 2017/0083429), as applied to claim 1 above, in view of Zhao et al. (US 2016/0103758).
[Claims 5-6] Pekelis does not explicitly disclose:
[Claim 5] wherein the uncertainty interval is determined using a quantile parameter value, a standard deviation parameter value, and an optimization parameter value;
[Claim 6] wherein the optimization parameter value is determined to optimize a boundary condition.
However, like Pekelis, Zhao performs A/B testing on online products (Zhao: ¶ 11). Additionally, Zhao evaluates a confidence interval and its lower limits/boundaries in terms of acceptable values (Zhao: ¶¶ 65, 80-81). Zhao’s confidence level is based on α (Zhao: ¶ 70 – “For this testing problem, if the outcome is significant, then the testing system can conclude that with a confidence level of 1−α, the new feature brings a significant lift which is greater than Δmin .”), the value of which is related to a quantile of a standard normal distribution with respect to the probability 1-α/2 (as described in ¶ 60 of Zhao –
PNG
media_image6.png
250
296
media_image6.png
Greyscale
).
Additionally, Zhao states, “The null hypothesis may be rejected with a confidence level 1−α if the p-value is smaller than α.” (Zhao: ¶ 73) Regarding the evaluation of optimization parameter values, Zhao explains, “Once processed into corresponding analytics data, such data can be input for determining the minimum threshold value and other parameters of online product testing.” (Zhao: ¶ 54) Differences meeting a minimum threshold value demonstrate a significance in differences (such as lift) between two buckets being tested (Zhao: ¶¶ 16, 68-70). The significant minimum threshold and lift values are examples of optimized values, including an optimized boundary condition.
The Examiner submits that it would have been obvious to one of ordinary skill in the art before the effective filing date of Applicant’s invention to modify Pekelis:
[Claim 5] wherein the uncertainty interval is determined using a quantile parameter value, a standard deviation parameter value, and an optimization parameter value;
[Claim 6] wherein the optimization parameter value is determined to optimize a boundary condition
in order to allow for Pekelis’ experimenters and employees to more efficiently determine when it is worth it to make website changes based on the significance of differences in results of A/B testing. This would help Pekelis solve problems typically associated with needless costs associated with making unnecessary changes to a website. As explained by Zhao, “Product teams may also be interested in knowing if the difference between the two versions is at least a certain magnitude. Standard tests, such as standard two-sided and one-sided tests, may fall short of providing such information. For example, a very small and unimportant difference can still achieve significant a non-zero result for standard tests, ignoring the fact that the difference may be too small to claim success in real business use cases.” (Zhao: ¶ 2)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SUSANNA M DIAZ whose telephone number is (571)272-6733. The examiner can normally be reached M-F, 8 am-4:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Brian Epstein can be reached at (571) 270-5389. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SUSANNA M. DIAZ/
Primary Examiner
Art Unit 3625A