Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Status of Claims
This communication is a First Office Action on the merits in reply to application number 18/381,978 filed on 10/19/2023.
Claims 1-20 are currently pending and have been examined.
Information Disclosure Statement
The information disclosure statement (IDS) filed on 10/19/2023 is not in compliance with the provisions of 37 CFR 1.97. With the exception of Non-Patent Literature Cite No. 6, the documents in the IDS have been considered. However, with respect to NPL Cite No. 6, the IDS does not include a publication date for this document as required by 37 CFR 1.97, and therefore the document in Cite No. 6 has not been considered.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-patentable subject matter. The claims are directed to an abstract idea without significantly more.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The judicial exception is not integrated into a practical application. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. The eligibility analysis in support of these findings is provided below, in accordance with the subject matter eligibility guidance set forth in MPEP 2106.
With respect to Step 1 of the eligibility inquiry (as explained in MPEP 2106.03), it is first noted that the claimed computer implemented methods (claims 1-8 and 9-17) and apparatus (claims 18-20) are each directed to a potentially eligible category of subject matter (i.e., processes and machine). Accordingly, claims 1-20 satisfy Step 1 of the eligibility inquiry.
With respect to Step 2A Prong One of the eligibility inquiry (as explained in MPEP 2106.04), it is next noted that the claims recite an abstract idea that falls under the “Certain methods of organizing human activity” abstract idea grouping by reciting limitations describing commercial interactions (sales or marketing activity) by obtaining marketing intelligence from testing of control and treatment variants, e.g., A/B testing (See Spec., at least par. [0031]: used in various fields such as…product development, marketing), and steps that, but for the generic computer implementation, may be implemented as “Mental Processes” (e.g., observation, evaluation, judgment, or opinion), or that may be implemented as mathematical calculations under the “Mathematical Concepts.” The limitations reciting the abstract idea as set forth in independent claim 1 are identified in bold text below, whereas the additional elements are presented in plain text and are separately evaluated under Step 2A Prong Two and Step 2B:
obtaining first testing data of a plurality of metrics from a database in memory, the first testing data having been generated from previously controlled testing of different test variants, the plurality of metrics including a target metric and a plurality of surrogate metrics that is indicative of the target metric, the different test variants including a control variant and a treatment variant of a feature of a webpage or a computer application (The “obtaining” step describes activity considered sales/marketing activity because the testing data may be directly in support of marketing activity (see, e.g., Spec. at par. 31) such as gathering A/B testing results of customer opinions on computerized output of advertisements, commercials, or a retailer’s website; and furthermore this step, but for the generic computer implementation, could be implemented as mental activity such as by observation, evaluation, judgment, or opinion. In addition, the “obtaining” step may be considered insignificant extra-solution activity, which is not enough to amount to a practical application (MPEP 2106.05(g)), and such extra-solution activity has also been recognized as well-understood, routine, and conventional, and thus insufficient to add significantly more to the abstract idea. See MPEP 2106.05(d) - Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network));
determining, by processing circuitry and based on the first testing data, correlations between each of the plurality of surrogate metrics and the target metric (The “determining” step describes activity considered sales/marketing activity because the determined correlations may be directly in support of marketing activity (see, e.g., Spec. at par. 31) such as for evaluating correlations/similarities/relationships between metrics produced by A/B testing results of customer opinions on computerized output of advertisements, commercials, or a retailer’s website; and furthermore this step, but for the generic computer implementation, could be implemented as mental activity such as by observation, evaluation, judgment, or opinion);
determining, by the processing circuitry, candidate surrogate metrics from the plurality of surrogate metrics based on the determined correlations (The “determining” step describes activity considered sales/marketing activity because the determined candidate surrogate metrics may amount to implementing marketing activity (see, e.g., Spec. at par. 31) such as for determining the highest/strongest correlated surrogate metrics to a target metric relevant to customer opinions on computerized output of advertisements, commercials, or a retailer’s website; and furthermore this step, but for the generic computer implementation, could be implemented as mental activity such as by observation, evaluation, judgment, or opinion);
determining a plurality of sensitivities of the respective candidate surrogate metrics based on the first testing data, a sensitivity of one of the candidate surrogate metrics indicating a probability that a change of the feature of the webpage or the computer application from the control variant to the treatment variant induces an effect that is detected as a statistically significant change in the one of the candidate surrogate metrics (The “determining” step describes activity considered sales/marketing activity because the determined sensitivities may amount to implementing marketing activity (see, e.g., Spec. at par. 31) such as for validating a strong correlation or perhaps inferring causation between candidate surrogate metrics and a target metric of interest, which is reasonably considered as marketing intelligence such as for making adjustments to advertisements, consumer content, retail displays, or the like; and furthermore this step, but for the generic computer implementation, could be implemented as mental activity such as by observation, evaluation, judgment, or opinion); and
selecting at least one candidate surrogate metric from the candidate surrogate metrics based on the determined plurality of sensitivities, wherein the at least one candidate surrogate metric is used to determine an output of a current controlled testing of the control variant and the treatment variant of the feature of the webpage or the computer application, and the output indicates whether the treatment variant replaces the control variant of the feature of the webpage or the computer application (The “selecting” step describes activity considered sales/marketing activity because the selection of a surrogate metric based on the sensitivities may amount to implementing marketing activity (see, e.g., Spec. at par. 31) such as for selecting and using a strongly correlated candidate surrogate metric and employing this marketing intelligence for consideration of adjustments to advertisements, consumer content, retail displays, or the like; and furthermore this step, but for the generic computer implementation, could be implemented as mental activity such as by observation, evaluation, judgment, or opinion. Furthermore, although the claim does not recite or require doing anything with the output, such as displaying, transmitting, or the like, it is nevertheless noted that merely outputting the indication of whether the treatment variant replaces the control variant would at most be considered insignificant extra-solution output activity, which is not enough to amount to a practical application or to add significantly more to the abstract idea for substantially the same reasons as set forth above in the discussion of the “obtaining” step, the rationale which is adopted here as well).
Independent claims 9 and 18 recite similar limitations as those set forth in claim 1 as discussed above, and have therefore been determined to recite the same abstract idea as claim 1.
With respect to Step 2A Prong Two of the eligibility inquiry (as explained in MPEP 2106.04(d)), the judicial exception is not integrated into a practical application. Independent claims 1, 9, and 18 include additional elements directed to computer-implemented method, database in memory, processing circuitry. The additional elements have been evaluated, but fail to integrate the abstract idea into a practical application because they amount to using generic computing elements or instructions (software) to perform the abstract idea, similar to adding the words “apply it” (or an equivalent), which merely serves to link the use of the judicial exception to a particular technological environment (generic computing environment). See MPEP 2106.05(f) and 2106.05(h). Even if the obtaining and output steps are considered as additional elements, these activities at most amount to insignificant extra-solution activity accomplished via receiving/transmitting data, which is not enough to amount to a practical application. See MPEP 2106.05(g).
Accordingly, because the Step 2A Prong One and Prong Two analysis resulted in the conclusion that the claims are directed to an abstract idea, additional analysis under Step 2B of the eligibility inquiry must be conducted in order to determine whether any claim element or combination of elements amount to significantly more than the judicial exception.
With respect to Step 2B of the eligibility inquiry (as explained in MPEP 2106.05), it has been determined that the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Independent claims 1, 9, and 18 include additional elements directed to computer-implemented method, database in memory, processing circuitry. These additional elements have been evaluated, but fail to add significantly more to the claims because they amount to using generic computing elements or instructions/software to perform the abstract idea, which merely serves to tie the abstract idea to a particular technological environment (generic computing environment), similar to adding the words “apply it” (or an equivalent). Notably, the Specification describes a litany of generic computing devices suggesting that virtually any computing device under the sun could be used to implement the invention (See, e.g., Spec. at par. [0142], noting for example that “The instructions can be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, internet of things devices, and the like.”). Accordingly, the generic computer implementation merely serves to link the use of the judicial exception to a particular technological environment and therefore does not amount to significantly more than the abstract idea itself. See, e.g., Alice Corp., 134 S. Ct. 2347, 110 USPQ2d 1976; Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015).
Even if the obtaining and output steps are considered as additional elements, these activities at most amount to insignificant extra-solution activity accomplished via receiving/transmitting data, which is well-understood, routine, and conventional activity and thus insufficient to add significantly more to the claims. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network).
In addition, when taken as an ordered combination, the ordered combination adds nothing that is not already present as when the elements are taken individually. There is no indication that the combination of elements integrate the abstract idea into a practical application. Their collective functions merely provide generic computer implementation. Therefore, when viewed as a whole, these additional claim elements do not provide meaningful limitations to transform the abstract idea into a practical application of the abstract idea or that, as an ordered combination, amount to significantly more than the abstract idea itself.
Dependent claims 2-8, 10-17, and 19-20 recite the same abstract idea(s) as recited in the independent claims, and have been determined to recite further details/steps falling under the “Certain methods of organizing human activity” and/or “Mental Processes” abstract idea groupings discussed above along with the same generic computing elements recited in the independent claims which, merely serve the purpose of tying the invention to a particular technological environment and which, as discussed above, is insufficient to integrate the abstract idea into a practical application or add significantly more to the claims.
The additional inputting, receive, and displaying activities (claims 3, 8, 12, and 17) are considered insignificant extra-solution activity, which is not enough to amount to a practical application (MPEP 2106.05(g)), and such extra-solution activity has also been recognized as well-understood, routine, and conventional, and thus insufficient to add significantly more to the abstract idea. See MPEP 2106.05(d) - Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network). The graphical user interface that facilitates the displaying activity in claims 8/17 has been considered, but merely involves a generic computing element to “apply it,” which is insufficient under Step 2A2 or Step 2B. See, e.g., Affinity Labs of Texas LLC v. DirecTV LLC, 838 F.3d 1253, 1257-1258 (Fed. Cir. 2016) (mere recitation of a GUI does not make a claim patent-eligible); Intellectual Ventures I LLC v. Capital One Bank, 792 F.3d 1363, 1370 (Fed. Cir. 2015) (“the interactive interface limitation is a generic computer element”).
The ordered combination of elements in the dependent claims (including the limitations inherited from the parent claim(s)) add nothing that is not already present as when the elements are taken individually. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide generic computer implementation. Accordingly, the subject matter encompassed by the dependent claims fails to amount to a practical application or significantly more than the abstract idea itself.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 9 and 17-18 are rejected under both 35 U.S.C. §102(a)(1) and §102(a)(2) as being anticipated by Sweeney (US 2023/0195607)
Claims 9/18: As per claim 9, Sweeney teaches a computer-implemented method (pars. 4 and 25: e.g., Systems, apparatuses, and methods; aspects discussed herein may be embodied in computer-usable or readable data and/or computer-executable instructions, such as in one or more program modules, executed by one or more computers), comprising:
selecting, by processing circuitry, at least one candidate surrogate metric from a plurality of surrogate metrics based on first testing data of a plurality of metrics from a first database in memory, the first testing data having been generated from previously controlled testing of different test variants, the plurality of metrics including a target metric and the plurality of surrogate metrics that is indicative of the target metric, the previously controlled testing being performed with first users, the different test variants including a control variant and a treatment variant of a feature of a webpage or a computer application (pars. 14-16, 29, 37, 42, and 46: metrics library may comprise a database that stores historical data collected, for example, from experiments, as well as product usage [i.e., first testing data generated from previously controlled testing performed with first users]; metrics library comprising one or more of engagement metrics (e.g., page-views, click-through-rate, user-interactions with a product, user-time spent on a product…, customer conversion event, customer purchases; experimentation platform that selects a metric and/or a methodology for testing a product; experimentation platform may comprise targeting engine (e.g., routing engine) configured to divide users of the product into experiment variants (e.g., control or treatment); experimentation platform may comprise targeting engine (e.g., routing engine) configured to divide users of the product into experiment variants (e.g., control or treatment)…product may be a new website design; configuring the product according to the selected configuration may comprise a new configuration for an application and/or a new interface for the application, a new website design; computing device may generate a list of candidate proxy metrics [i.e., surrogate metrics] from one or more engagement metrics. As described herein, one or more engagement metrics may be used as a proxy metric for a performance metric [i.e., target metric]. The computing device may generate candidate proxy metrics, for example, based on historical proxy metrics, one or more engagement metrics, a combination of engagement metrics, etc. In step 505, the computing device may determine whether a correlation exists between each of the candidate proxy metrics and the performance metric received in the request to test and/or optimize the product. The correlation may be based on historical data and/or behavior learned via a correlation model);
performing, by the processing circuitry, current controlled testing of the different test variants with second users by obtaining second testing data of the plurality of metrics for the current controlled testing and storing the second testing data in a second database in the memory, and determining current testing results that are associated with the plurality of surrogate metrics (pars. 29, 34-35, and 41: experimenter 201 may run, via the module 210, an experiment to determine if a new configuration of a product 240 results in an improvement of, for example, a performance metric associated with the product 240. The improvement (or lack thereof) of the performance metric may be determined based on one or more engagement metrics that may be proxies for (i.e., may be representative of) the performance metric; experiment comprising multiple test configurations of a product 240 [i.e., different test variants with second users, second testing data], the targeting engine 220 may sort each of a plurality of users into one of the multiple test configurations. The targeting engine 220 may ensure users are sorted into test configurations randomly (e.g., each test configuration may receive a similar demographic of users); performance data received from the product 240 should be stored in a product database of the management console 210 or stored in another database located in another component of the product experimentation platform [i.e., a second database]; targeting engine 220 may store [in the second database] performance data and/or experimental data (e.g., any additional data produced by a product 240 during an experiment that is not performance data) corresponding to the experiment configuration and the users associated with each test configuration of the product; proxy metrics may be used to test the product. In this regard, the product may have one or more test configurations that are being tested; generate a performance score for each of the test configurations, using each of the one or more proxy metrics; computing device may generate a performance score by receiving user-interaction data for each of the test configurations);
determining an output of the current controlled testing based on one or more of the current testing results associated with the at least one candidate surrogate metric, the output of the current controlled testing indicating whether the treatment variant replaces the control variant of the feature (pars. 14, 28, 42-44: computing device may recommend one or more proxy metrics to represent the performance metric. In yet further embodiments, the computing device may select one or more proxy metrics to represent the performance metric; computing device may output the performance scores for each of the plurality of test configurations of the product. Outputting the performance scores may include causing the performance scores for each of the plurality of test configurations to be displayed via a GUI; experimentation platform may comprise targeting engine (e.g., routing engine) configured to divide users of the product into experiment variants (e.g., control or treatment); analyze data associated with one or more existing or concluded product experiments. Also or alternatively, the experimenter 201 may use the module 210 (or a different module 210) to perform any other functions associated with product experimentation (e.g., to configure one or more of the products; computing device may receive a selection of a configuration from the plurality of test configurations of the product. The selection may be received via the GUI (e.g. management console). In some examples, the selection may be based on a highest performance score amongst the plurality of test configurations. In step 317, the computing device may configure the product according to the selected configuration); and
in response to the output indicating that the treatment variant replaces the control variant of the feature of the webpage or the computer application, replacing the control variant of the feature of the webpage or the computer application with the treatment variant (pars. 42-44: computing device may output the performance scores for each of the plurality of test configurations of the product. Outputting the performance scores may include causing the performance scores for each of the plurality of test configurations to be displayed via a GUI (e.g., management console). In step 315, the computing device may receive a selection of a configuration from the plurality of test configurations of the product. The selection may be received via the GUI (e.g. management console). In some examples, the selection may be based on a highest performance score amongst the plurality of test configurations. In step 317, the computing device may configure the product according to the selected configuration. As noted above, configuring the product according to the selected configuration may comprise a new configuration for an application and/or a new interface for the application, a new website design).
Claim 18 is directed to an apparatus for performing substantially similar limitations as those set forth in claim 9 and discussed above. Sweeney teaches and apparatus for performing the limitations discussed above (pars. 4 and 25: e.g., Systems, apparatuses, and methods; aspects discussed herein may be embodied in computer-usable or readable data and/or computer-executable instructions, such as in one or more program modules, executed by one or more computers), and claim 18 is therefore rejected using the same reference and for substantially the same reasons as set forth above.
Claim 17: Sweeney further teaches displaying the selected at least one candidate surrogate metric and the output of the current controlled testing in a graphical user interface (GUI) (pars. 42-44: computing device may determine one or more proxy metrics to represent the performance metric. In some instances, the one or more proxy metrics may be displayed via a GUI (e.g., management console); computing device may select one or more proxy metrics to represent the performance metric; computing device may output the performance scores for each of the plurality of test configurations of the product. Outputting the performance scores may include causing the performance scores for each of the plurality of test configurations to be displayed via a GUI; computing device may receive a selection of a configuration from the plurality of test configurations of the product. The selection may be received via the GUI).
Claims 10 and 19 are rejected under 35 U.S.C. §103 as unpatentable over Sweeney (US 2023/0195607), as applied to claims 9 and 18 above, and further in view of Gui et al. (US 2016/0253683, hereinafter “Gui”).
Claims 10/19: Sweeney further teaches wherein the current controlled testing includes A/B testing of the feature of the webpage or the computer application (pars. 14, 27, 37, and 42: experimentation platform that selects a metric and/or a methodology for testing a product; product 240 may comprise one or more applications such as a web-site; experimentation platform…may receive a request to test and/or optimize a performance of a product… product may be a new configuration for an application and/or a new interface for the application. In alternative examples, the product may be a new website design), but does not explicitly teach the testing as being A/B testing.
Gui teaches controlled testing includes A/B testing (pars. 5 and 69: A/B testing is a standard way to evaluate user engagement or satisfaction with a new service, feature, or product; A/B test is performed by presenting the treatment version to the selected clusters and tracking the response of the selected clusters to the treatment version (operation 512). For example, a treatment version of an email, …webpage, feature, layout, design).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Sweeney with Gui because the references are analogous since they are each directed to computer implemented features for executing testing/experimentation of variations of products such as webpages, which is within Applicant’s field of endeavor of applying testing to variants such as multiple versions of a feature such as a webpage, and because modifying Sweeney to incorporate Gui’s A/B testing, as claimed, would provide the expected benefit of employing a standardized technique to evaluate user satisfaction with a new product/webpage (Gui at par. 5), which would further serve the motivation to improve experimentation platforms to test new products with respect to performance metrics (Sweeney at par. 5); and further obvious because the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Allowable over the prior art
Claims 1-8 and 11-16 are allowable over the prior art. The closest prior art reference of record, Sweeney (US 2023/01956070), is directed to a system for automatic identification/selection of optimization metrics and accompanying models in experimentation platforms. Sweeney teaches several limitations of independent claim 1, including, for example: obtaining first testing data of a plurality of metrics from a database in memory, the first testing data having been generated from previously controlled testing of different test variants, the plurality of metrics including a target metric and a plurality of surrogate metrics that is indicative of the target metric, the different test variants including a control variant and a treatment variant of a feature of a webpage or a computer application (Sweeney at pars. 14, 37, 42, and 46: experimentation platform that selects a metric and/or a methodology for testing a product; experimentation platform may comprise targeting engine (e.g., routing engine) configured to divide users of the product into experiment variants (e.g., control or treatment); experimentation platform may comprise targeting engine (e.g., routing engine) configured to divide users of the product into experiment variants (e.g., control or treatment); product may be a new website design; configuring the product according to the selected configuration may comprise a new configuration for an application and/or a new interface for the application, a new website design; computing device may generate a list of candidate proxy metrics [i.e., surrogate metrics] from one or more engagement metrics. As described herein, one or more engagement metrics may be used as a proxy metric for a performance metric [i.e., target metric]. The computing device may generate candidate proxy metrics, for example, based on historical proxy metrics, one or more engagement metrics, a combination of engagement metrics, etc. In step 505, the computing device may determine whether a correlation exists between each of the candidate proxy metrics and the performance metric received in the request to test and/or optimize the product. The correlation may be based on historical data and/or behavior learned via a correlation model, or a metric correlation model); determining, by processing circuitry and based on the first testing data, correlations between each of the plurality of surrogate metrics and the target metric (Sweeney at par. 46: determine whether a correlation exists between each of the candidate proxy metrics and the performance metric received in the request to test and/or optimize the product. The correlation may be based on historical data and/or behavior learned via a correlation model, or a metric correlation model. In step 507, the computing device may determine whether the correlation between the proxy metric and the performance metric satisfies a threshold); and determining, by the processing circuitry, candidate surrogate metrics from the plurality of surrogate metrics based on the determined correlations (par. 46: computing device may determine whether the correlation between the proxy metric and the performance metric satisfies a threshold. If the correlation fails to satisfy the threshold, the method proceeds to step 511, where the computing device determines whether additional proxy metrics exists. If so, the method returns to step 505. If the correlation does satisfy the threshold, the computing device may add the candidate proxy metric to the list of proxy metrics). However, Sweeney and the other prior art references of record do not teach or render obvious the combined sequence of claim limitations directed to determining a plurality of sensitivities of the respective candidate surrogate metrics based on the first testing data, a sensitivity of one of the candidate surrogate metrics indicating a probability that a change of the feature of the webpage or the computer application from the control variant to the treatment variant induces an effect that is detected as a statistically significant change in the one of the candidate surrogate metrics; and selecting at least one candidate surrogate metric from the candidate surrogate metrics based on the determined plurality of sensitivities, as recited by independent claim 1 and as similarly encompassed by claims 11 and 20, thereby rendering claims 1/11/20 and their respective dependent claims (2-8 and 12-16) as allowable over the prior art. Claims 1-8, 11-16, and 20 are not allowable, however, because these claims stand rejected under 35 USC §101 as discussed above. Furthermore, claims 11-16 and 20 are dependent claims, and therefore even if their §101 rejection is overcome, claims 11-16 and 20 would be objected to as being dependent upon rejected base claims (independent claims 9 and 18, respectively) and would be allowable only if rewritten in independent form including all of the limitations of their respective base claims and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Ogallo et al. (US 2023/0325871): discloses subgroup analysis in A/B testing, including predicting membership of individuals to a stratus of population strata using surrogate features (par. 25).
Pekelis et al. (US 2017/0083429): discloses features for performing variation testing of content, including running A/B tests for content providers (par. 59), statistical processing techniques to control the rate of false positives (pars. 40-42), and applying a policy to determine a distribution of variations of web pages to provide to users (par. 29).
Katariya et al. (US 2017/0323329): discloses A/B testing for determining an impact of digital marketing content on conversion of products or services (par. 49).
Anderson et al. (US Patent No., 9,996,513): discloses flexible analytics-driven webpage design and optimization, including A/B testing techniques to help identify changes in web pages that increase or maximize an outcome of interest (col. 1, lines 43-67).
Hugeback et al. (US Patent No. 8,234,632): discloses adaptive website optimization, including features for reducing the probability of false positives (col. 7, lines 65-67).
Xu et al. (US 2016/0253311): discloses features for conducting A/B experiments of online content (at least pars. 24-30).
Deng, A., Lu, J., Litz, J.: Trustworthy analysis of online A/B tests. In: Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, WSDM 2017, pp. 641–649 (2017): discloses techniques for conducting experiments using online A/B tests with a focus on improving the statistical analysis of A/B tests by applying a randomization mechanism to enhance trustworthiness of the results.
R. Kohavi and R. Longbotham, "Online Experiments: Lessons Learned," in Computer, vol. 40, no. 9, pp. 103-105, Sept. 2007: discloses techniques for implementing online controlled experiments (e.g., A/B testing), including analyzing past experiments and conducting new learning experiments to gain insight into metrics and learn mappings between indicators.
A. Fabijan, P. Dmitriev, H. H. Olsson and J. Bosch, "The Benefits of Controlled Experimentation at Scale," 2017 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Vienna, Austria, 2017, pp. 18-26: discloses features for evaluating proposed changes or new features quickly using controlled online experiments.
Any inquiry of a general nature or relating to the status of this application or concerning this communication or earlier communications from the Examiner should be directed to Timothy A. Padot whose telephone number is 571.270.1252. The Examiner can normally be reached on Monday-Friday, 8:30 - 5:30. If attempts to reach the examiner by telephone are unsuccessful, the Examiner’s supervisor, Brian Epstein can be reached at 571.270.5389. The fax phone number for the organization where this application or proceeding is assigned is 571- 273-8300.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center for authorized users only. Should you have questions about access to Patent Center, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/TIMOTHY PADOT/
Primary Examiner, Art Unit 3625
04/14/2026
Prosecution Insights