DETAILED ACTION
The present application (Application No. 18/220,496), filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This Office action is in reply to communications by Applicants responding to first office action on the merits, received 05 December, 2024.
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 30 June, 2025, has been entered.
Status of Claims
Claims 1, 11, 17, are amended. Claims 8, was previously canceled. Therefore, claims 1-7, 9-20, are currently pending and addressed below.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-7, 9-20, are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
Step 1: In the instant case, claims 1-7, 9-10, are directed to a method, claims 11-16, are directed to a system, and claims 17-20 are directed to an apparatus, therefore the claims are directed to statutory categories of invention.
Step 2A- Prong 1: Independent claim 1 comprises steps of: executing a first and a second testing sequence, to a first and second digital content, prompting one or more first and second responses to a first and second digital content; the first and second testing sequences executed during a predetermined duration of time; receiving, during the predetermined duration of time, a first and a second test data generated based on the one or more first and second responses; randomly selecting, a time in the predetermined duration of time and generating one or more confidence intervals for each first test data and for each second test data at the randomly selected time; determining, at the randomly selected time, a testing metric indicating an effect of the second digital content over the first digital content; randomly selecting and the determining is performed without pausing the executing of at least one of the first testing sequence and the second testing sequence; determining, a stopping time to stop execution of the first and second testing sequences, and stopping execution of the first and second testing sequences at the stopping time; wherein the testing metric is determined at any time before expiration of the predetermined duration of time to indicate the effect of the second digital content over the first digital content., and displaying a visualization of at least one of the one or more confidence intervals for each test data in the plurality of test data, prior to expiration.
As now amended the claim further comprises steps of: determining that the testing metric is outside of bounds; terminating execution of at least one of the first and second testing sequence based on an out-of-bounds testing metric, prior to expiration of the predetermined duration of time; and displaying using a graphical user interface a visualization of at least one of the one or more confidence intervals for each test data in the plurality of test data, prior to expiration.
The instant claims are directed to a method for designing lift tests (A/B test) for determining an incremental effect and for generating lift metrics, and more particularly, the claims are directed to utilizing the known statistical techniques of sequential hypothesis testing in a generic, computerized environment. The particular incremental effect is based on executing a first and a second testing sequence, to a first and second set of responders, prompting one or more first and second responses to a first and second digital content, wherein it is noted, any assessment of a lift effect of responses to a first and second digital content is totally an advertising, marketing and sales activity or behavior. This lift concept is an assessment of conversion. The assignment of first and second digital content to a first and second set of responders, and tracking their responses are very much commercial or legal interactions by people.
Accordingly, the claimed steps represent a method of organizing commercial interactions in that they relate to commercial or legal interactions including advertising, marketing or sales activities or behaviors, which falls within the “Certain Methods of Organizing Human Activity” abstract idea grouping, and “Mathematical Concepts” grouping of abstract ideas, wherein all the claim steps can be seen as being part of the abstract idea of generating and displaying lift information (lift effect metrics).
The above claimed steps are steps of collecting/tracking data (transmitting, receiving, storing, gathering), analyzing data, making determinations/correlations/comparisons, and displaying/presenting data. All these steps, but for the use of generic computer components that execute them, are generic functions performed by general-purpose computers.
In addition, it is noted, the above claimed steps represent a process that under broadest reasonable interpretation, covers performance of the limitations in the human mind or by a human using a pen and paper, but for the recitation of generic computer components. These claimed steps relate to concepts that merely involve observing, evaluating and judging data. This concept falls under the “Mental Processes” abstract idea grouping.
Claims 11 and 17 recite substantially similar subject matter and the same subsequent analysis should be applied thereto.
Step 2A- Prong 2: Additional elements include: “at least one processor; and at least one non-transitory storage media storing instructions”; “a testing module of at least one processor”; “a confidence module of the at least one processor”; “an analysis module of the at least one processor”. These additional elements are recited at a high level of generality and the steps that they execute represent generic functions which can be performed by a general-purpose computer without any novel programming or improvement in the operation of the computer itself. These additional elements are merely invoked as tools to perform an abstract idea (mere instructions to apply the exception) as discussed in MPEP 2106.05(f). The mere nominal recitation of generic computer components does not take the claim limitations out of the mental processes grouping (see 2106.04(a)(2)(III)(C)).
As mentioned above, the examiner considers the entirety of the claimed subject matter to be representative of merely applying the abstract idea of utilizing the known statistical techniques of sequential hypothesis testing in a generic, computerized environment, wherein this is the very definition of merely “apply it” in the area of patent eligibility determinations. Sequential hypothesis testing has been well known for many decades) before the filing of the instant invention, and as explained above it is the expected and predictable knowledge of a person of ordinary skill in the A/B testing arts.
In addition, making determinations based on statistical confidence levels does not represent a technological improvement, it merely further narrows the abstract idea of collecting/tracking data (transmitting, receiving, storing, gathering), analyzing data, making determinations/correlations/comparisons, and displaying/presenting data.
Accordingly, the additional elements when the claim elements are viewed individually and as a whole do not integrate the abstract idea into a practical application.
Step 2B: Based on the reasoning provided under Step 2A- Prong 2, the claims under Step 2B do not recite “significantly more” than the abstract idea. At this point, either under the “Certain Methods of Organizing Human Activity” grouping scenario where all the claim steps can be seen as being part of the abstract ideas, or under the “Mental Processes” grouping scenario, the analysis is terminated because the same analysis with respect to Step 2A Prong Two applies here in Step 2B, i.e., mere instructions to apply an exception using a generic computer component cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B. That is, these additional elements are recited at a high level of generality and the steps that they execute represent conventional functions which can be performed by a general-purpose computer without any improvement to the programming technique or improvement in the operation of the computer itself.
The dependent claims have been considered. Further additional limitations recited in the dependent claims include:
Claims 2, 18, recite confidence intervals for each the first and the second data.
Claims 3, 12, 19, recite a ratio calculation of a lower bound of each confidence interval. This claim is a mathematical calculation concept, which can be mental.
Claims 4, 13, 20, recite a superiority calculation. This is further collecting/tracking data (transmitting, receiving, storing, gathering), analyzing data, making determinations/correlations/comparisons, and displaying/presenting data. This claim is a mathematical calculation concept, which can be mental.
Claims 5, 14, recite an A/B test. This is further collecting/tracking data (transmitting, receiving, storing, gathering), analyzing data, making determinations/correlations/comparisons, and displaying/presenting data, which can be mental.
Claims 6, 15, recite further statistical calculations. This claim is a mathematical calculation concept, which can be mental.
Claim 7, recites additional elements: a website, an email, a graphic, a video, an audio, a text, and any combination; and recites further additional elements: a click, a conversion, a time duration spent on at least one of the first and second digital contents, a time between accessing at least one of the first and second digital contents, and any combination thereof. These are computer elements executed by general-purpose computers in the context of generic functions performed by general-purpose computers. The internet/network features of this claim only represent a particular technological environment, merely a particular technical field of use to which the judicial exception is linked to, and this technological environment is used to merely transmit, receive, store, gather, analyze, make determinations/correlations with, and display data.
Claim 9 recites the additional element of a graphical user interface. This element is used to display data.
Claim 10 recites terminating the executing of at least one of the first and second testing sequence based on the testing metric prior to expiration of the predetermined duration of time upon determining the testing metric being outside of bounds of one or more effect intervals, therefore merely limit the conditional lift determinations.
The additional elements are each functional generic computer components that perform the generic functions of processing, communicating and displaying, all common to electronics and computer systems. When considered as a whole, the same analysis with respect to Step 2A Prong Two and step 2B, apply to these additional elements. They cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
The dependent claims appear to merely limit the common lift and A/B testing implementations and the various considerations for conditioning these lift and A/B testing implementations, and not add significantly more than the idea (i.e. "PEG" Step 2B=No).
Mere instructions to apply an exception using a generic computer component cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-7, 9-20, are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Katariya et al. (US 2017/0323329) (hereinafter “Katariya3329”).
Regarding claim 1, Katariya3329 discloses:
(system, comprising : at least one processor; and at least one non-transitory storage media storing instructions, that when executed by the at least one processor, cause the at least one processor to perform operations including executing, using a testing module of the at least one processor). System comprising computing devices, processors, servers, memory, computer readable media, interfaces, modules and software instructions stored in memory that enable the system to execute the steps of the method over network communications and to enable interaction between participants and the system (see at least Katariya3329, fig. 1, 11, ¶45-49, 170-185).
Accordingly, software, hardware, or executable modules may be implemented as one or more instructions and/or logic embodied on some form of computer-readable storage media and/or by one or more hardware elements 1110. The computing device 1102 may be configured to implement particular instructions and/or functions corresponding to the software and/or hardware modules. Accordingly, implementation of a module that is executable by the computing device 1102 as software may be achieved at least partially in hardware, e.g., through use of computer-readable storage media and/or hardware elements 1110 of the processing system 1104. The instructions and/or functions may be executable/operable by one or more articles of manufacture (for example, one or more computing devices 1102 and/or processing systems 1104) to implement techniques, modules, and examples described herein. (see at least Katariya3329, fig. 11, ¶181).
Sequential testing module 208 implemented at least partially in hardware to perform sequential hypothesis testing to determine an effect of different options on a statistic, e.g., conversion rate. The sequential testing module 208 may collect marketing data 206 which describes interaction of a plurality of users with digital marketing content 120. From this, an effect is determined of different items of digital marketing content 120 (e.g., items “A” and “B”) on conversion of a product or service being offered by the service provider 102. Statistical significance 210 is used to define a point at which is it considered “safe” to consider the test completed. The sequential testing module 208 then evaluates this marketing data 206 to compare groups of the users that have received item “A” with a group of the users that have received item “B,” e.g., to determine a conversion rate exhibited by the different items. Statistical significance 210 is also computed to determine whether it is “safe to stop the test” at this point, e.g., in order to reject the null hypothesis. (see at least Katariya3329, fig. 2, ¶50-53). Statistic testing module 718 (see at least Katariya3329, fig. 7, ¶146). Interaction determination module 728 (see at least Katariya3329, fig. 7, ¶148).
It follows that since testing, and confidence determinations, and data analysis, and visualization, are indeed performed by sequential testing module 208 in the context of the sequential testing, then, instructions (modules as claimed) to execute these steps (a testing module of at least one processor) (a confidence module of the at least one processor) (an analysis module of the at least one processor) are implicit in the system architecture and methodology of Katariya3329.
(executing, using a testing module of at least one processor, a first testing sequence and a second testing sequence, the first testing sequence prompting one or more first responses to a first digital content from one or more users, the second testing sequence prompting one or more second responses to a second digital content from the one or more users, the second digital content to comprise a variation of the first digital content, the first and second testing sequences executed during a predetermined duration of time).
(receiving, using the testing module of the at least one processor, during the predetermined duration of time, a first test data generated based on the one or more first responses, and a second test data generated based on the one or more second responses).
Katariya3329 generally teaches: Sequential hypothesis testing techniques and A/B testing.
In particular Katariya3329 discloses:
A/B testing. (see at least Katariya3329, fig. 5, ¶3-4, 49).
Testing is used to compare different items of digital content against a current item of digital content to determine which item operates “best” as defined by a statistic. In a digital marketing scenario, this statistic includes a determination as to which item of digital content exhibits a greatest effect on conversion. Examples of conversion include interaction of a user with the content (e.g., a “click-through”), purchase of a product or service that pertains to the digital content, and so forth. (see at least Katariya3329, abstract, ¶30).
In contrast to conventional techniques that are based on a fixed horizon of samples, the disclosed sequential hypothesis testing techniques involve testing sequences of increasingly larger number of samples until a winner is determined. In particular, the winner is determined based on whether a result of a statistic (e.g., conversion rate) has reached statistical significance that defines a confidence level in the accuracy of the results. (see at least Katariya3329, ¶32).
To perform sequential hypothesis testing, the sequential testing module 208 (testing module) evaluates the marketing data 206 as it is received, e.g., in real time, to determine an effect of digital marketing content 120 on conversion (see at least Katariya3329, ¶51). The sequential testing module 208 then evaluates this marketing data 206 to compare groups of the users that have received item “A” with a group of the users that have received item “B,” e.g., to determine a conversion rate exhibited by the different items. (see at least Katariya3329, fig. 2, ¶52).
(a first testing sequence and a second testing sequence) (the second digital content to comprise a variation of the first digital content). Example implementation to perform sequential hypothesis testing for more than two options. Testing may also be performed for multiple alternatives (e.g., “B,” “C,” “D,” and so on) against a base option, e.g., “A.” This results in multiple tests of the form (A, B), (A, C), (A, D), and so forth (see at least Katariya3329, fig. 5, ¶68).
(randomly selecting, using a confidence module of the at least one processor, without pausing the executing of at least one of the first testing sequence and the second testing sequence, a time in the predetermined duration of time and dynamically generating one or more confidence intervals for each first test data and for each second test data at the randomly selected time).
(determining, using the analysis module of the at least one processor, based on the testing metric and the one or more confidence intervals, a stopping time to stop execution of the first and second testing sequences, the stopping time is associated with preventing occurrence of a type-I error associated with executing of the first and second testing sequences).
(stopping execution of the first and second testing sequences at the stopping time, wherein the testing metric is determined at any time before expiration of the predetermined duration of time to indicate the effect of the second digital content over the first digital content, wherein in response to determining the testing metric to be outside of bounds of one or more effect intervals determined based on the one or more confidence intervals generated for the first test data and the one or more confidence intervals generated for the second test data, the executing of at least one of the first and second testing sequence based on the testing metric is terminated prior to expiration of the predetermined duration of time).
In the sequential hypothesis testing, a sample size calculator may be used before a test to estimate an amount of time that is likely needed to completed the test (see at least Katariya3329, ¶14, 37) (the first and second testing sequences executed during a predetermined duration of time). In sequential hypothesis testing, the output of the sample size determination module 602 is informative and not part of the test. In order to have sufficient Power and Type I error guarantees at the same time to allow peeking in hypothesis testing, the value of “N*” is often larger than the sample-size of the fixed-horizon test. (see at least Katariya3329, ¶153).
Sequential hypothesis testing techniques and systems are described. In contrast to conventional techniques that are based on a fixed horizon of samples, the disclosed sequential hypothesis testing techniques involve testing sequences of increasingly larger number of samples until a winner is determined. In particular, the winner is determined based on whether a result of a statistic (e.g., conversion rate) has reached statistical significance that defines a confidence level in the accuracy of the results. Thus, statistical significance defines when it is safe to conclude the test, e.g., based on a level of confidence of a computed result (e.g., conversion) against defined amounts of Type I and Type II errors. This permits the sequential hypothesis testing technique to conclude as soon as statistical significance is reached and a “winner” declared, without forcing a user to wait until the horizon “N” of a number of samples is reached. (see at least Katariya3329, ¶32).
In particular, the winner is determined based on whether a result of a statistic (e.g., conversion rate) has reached statistical significance that defines a confidence level in the accuracy of the results. Thus, statistical significance defines when it is safe to conclude the test, e.g., based on a level of confidence of a computed result (e.g., conversion) against defined amounts of Type I and Type II errors. This permits the sequential hypothesis testing technique to conclude as soon as statistical significance is reached and a “winner” declared, without forcing a user to wait until the horizon “N” of a number of samples is reached. (see at least Katariya3329, ¶32).
Statistical significance 210 is used to define a point at which is it considered “safe” to consider the test completed. That is, a “safe” point of completion is safe with respect to an amount of false positives or false negatives permitted. This is performed in sequential hypothesis testing without setting the horizon “N” beforehand, which is required under the conventional fixed-horizon hypothesis testing. (see at least Katariya3329, fig. 2, ¶51). Statistical significance 210 is also computed to determine whether it is “safe to stop the test” at this point (a stopping time), e.g., in order to reject the null hypothesis (see at least Katariya3329, fig. 2, ¶52). Based on the response from these users described in the marketing data 206, a determination is made whether to reject or not reject the null hypothesis. Whether it is safe to make this determination is based on statistical significance 210, which takes into account accuracy guarantees regarding Type I and Type II errors, e.g., to ninety-five percent confidence that these errors do not occur. (see at least Katariya3329, fig. 2, ¶53).
A user viewing the user interfaces 300, 400 may also employ the value of the statistical significance as a “soft stop” as opposed to the hard stop of the decision boundary of fixed horizon hypothesis testing (see at least Katariya3329, fig. 2, ¶62).
In other words, in sequential hypothesis testing, at any time(s) (randomly selected time) during the estimated time duration of the A/B test (at any time before expiration of the predetermined duration of time), statistical significance tests of the ongoing test outcomes are performed which can be used to determine whether it is safe to stop the test (a stopping time) (terminated prior to expiration of the predetermined duration of time).
Confidence interval (see at least Katariya3329, fig. 4).
(and dynamically generating one or more confidence intervals for each first test data and for each second test data at the randomly selected time) (bounds) (see at least Katariya3329, fig. 4, ¶32, 51-53, 57).
The data that are the subject of the above tests of statistical significance, confidence limits and/or accuracy levels, to determine an effect of digital marketing content 120 on conversion, represents “a testing metric indicating an effect of the second digital content over the first digital content”. (determining, using an analysis module of the at least one processor, without pausing the executing of at least one of the first testing sequence and the second testing sequence, at the randomly selected time, a testing metric indicating an effect of the second digital content over the first digital content).
(in response to the stopping, generating, using a graphical user interface module communicatively coupled to the at least one processor, a graphical user interface including a visualization of the one or more confidence intervals for each first test data and for each second test data, and the determined testing metric prior to expiration of the predetermined duration of time). The user interfaces 300, 400 are configured to provide information to a user while the test is running and even after it has stopped (see at least Katariya3329, fig. 3-4, ¶12-13, 37, 58-63).
A user viewing the user interfaces 300, 400 may also employ the value of the statistical significance as a “soft stop” as opposed to the hard stop of the decision boundary of fixed horizon hypothesis testing (see at least Katariya3329, fig. 2, ¶62).
Regarding claims 2, 18, Katariya3329 discloses: All the limitations of the corresponding parent claims (claim 1; and claim 17; respectively) as per the above rejection statements.
Katariya3329 discloses: (wherein the testing metric being bounded by one or more effect intervals determined based on the one or more confidence intervals generated for the first test data and the one or more confidence intervals generated for the second test data; wherein each confidence interval in the one or more confidence intervals for each the first and the second data is determined based on a respective lower bound and a respective upper bound of the first and second test data). Statistical significance 210 is used in sequential hypothesis testing to define a point at which is it considered “safe” to consider the test completed (see at least Katariya3329, fig. 2, ¶32, 51-53, 57, 62). Confidence interval (see at least Katariya3329, fig. 4).
Regarding claims 3, 12, 19, Katariya3329 discloses: All the limitations of the corresponding parent claims (claims 1-2; claim 11; and claims 17-18; respectively) as per the above rejection statements.
As indicated in the rejection of the parent claims, Katariya3329 discloses: (a lower bound of each confidence interval for the second test data and an upper bound of each confidence interval for the first test data, and an upper bound of each effect interval is determined using a ratio of an upper bound of each confidence interval for the second test data and a lower bound of each confidence interval for the first test data). (see at least Katariya3329, ¶6-8, 53, 57). Confidence interval (see at least Katariya3329, fig. 4). Moreover, it is noted, upper and lower bounds are implicit in any such statistical confidence determination
Regarding claims 4, 13, 20, Katariya3329 discloses: All the limitations of the corresponding parent claims (claim 1; claim 11; and claims 17-19; respectively) as per the above rejection statements.
As indicated in the rejection of the parent claims, Katariya3329 discloses: (wherein the effect of the second digital content over the first digital content indicates at least one of: a superiority of the second digital content over the first digital content or an inferiority of the second digital content over the first digital content; the method further comprising selecting one of the first digital content and the second digital content based on the indicated effect of the second digital content over the first digital content; and terminating the executing of the first and second testing sequences at at least one of: prior to expiration of the predetermined duration of time and expiration of the predetermined duration of time). (see at least Katariya3329, fig. 2, ¶32, 51-53, 57, 62). Confidence interval (see at least Katariya3329, fig. 4).
Regarding claims 5, 14, Katariya3329 discloses: All the limitations of the corresponding parent claims (claim 1; and claim 11; respectively) as per the above rejection statements.
As indicated in the rejection of the parent claims, Katariya3329 discloses: (wherein the first testing sequence and the second testing sequence form an A/B test, wherein the second testing sequence being at least one of: different from the first testing sequence or the same as the first testing sequence). A/B testing. (see at least Katariya3329, fig. 5, ¶3-4, 49, see also ¶30, 32, 52).
Regarding claims 6, 15, Katariya3329 discloses: All the limitations of the corresponding parent claims (claim 1; and claim 11; respectively) as per the above rejection statements.
As indicated in the rejection of the parent claims, Katariya3329 discloses: (wherein each next confidence interval in the one or more confidence intervals is determined based on the first and second test data received prior to the randomly selected time).
In sequential hypothesis A/B testing (see at least Katariya3329, fig. 2, ¶32, 51-53, 57, 62, 68). Confidence interval (see at least Katariya3329, fig. 4).
Regarding claim 7, Katariya3329 discloses: All the limitations of the corresponding parent claim (claim 1) as per the above rejection statements.
As indicated in the rejection of the parent claims, Katariya3329 discloses:
Katariya3329 further discloses: (wherein the first and second digital contents include at least one of the following: a website, an email, a graphic, a video, an audio, a text, and any combination thereof) (see at least Katariya3329, ¶41),
Katariya3329 further discloses: (wherein the one or more first responses and the one or more second responses include at least one of the following: a click, a conversion, a time duration spent on at least one of the first and second digital contents, a time between accessing at least one of the first and second digital contents, and any combination thereof). Click (see at least Katariya3329, ¶30, 161). Conversion rate (see at least Katariya3329, fig. 2, ¶11, 50-57).
Regarding claims 9, 16, Katariya3329 discloses: All the limitations of the corresponding parent claims (claim 1; and claim 11; respectively) as per the above rejection statements.
Katariya3329 further discloses:
(displaying, using a graphical user interface module communicatively coupled to the at least one processor, a graphical user interface including a visualization of at least one of: the one or more confidence intervals for each first test data and for each second test data, the determined testing metric, and any combination thereof) (see at least Katariya3329, fig. 3-4).
(wherein the displaying is performed prior to expiration of the predetermined duration of time). (see at least Katariya3329, fig. 2, ¶32, 51-53, 57, 62).
Regarding claim 10, Katariya3329 discloses: All the limitations of the corresponding parent claim (claim 1) as per the above rejection statements.
As indicated in the rejection of the parent claims, Katariya3329 discloses: (terminating the executing of at least one of the first and second testing sequence based on the testing metric prior to expiration of the predetermined duration of time upon determining the testing metric being outside of bounds of one or more effect intervals determined based on the one or more confidence intervals generated for the first test data and the one or more confidence intervals generated for the second test data). (see at least Katariya3329, fig. 2, ¶32, 51-53, 57, 62).
Regarding claims 11, 17 Katariya3329 discloses:
(system, comprising : at least one processor; and at least one non-transitory storage media storing instructions, that when executed by the at least one processor, cause the at least one processor to perform operations including executing, using a testing module of the at least one processor). System comprising computing devices, processors, servers, memory, computer readable media, interfaces, modules and software instructions stored in memory that enable the system to execute the steps of the method over network communications and to enable interaction between participants and the system (see at least Katariya3329, fig. 1, 11, ¶45-49, 170-185).
Accordingly, software, hardware, or executable modules may be implemented as one or more instructions and/or logic embodied on some form of computer-readable storage media and/or by one or more hardware elements 1110. The computing device 1102 may be configured to implement particular instructions and/or functions corresponding to the software and/or hardware modules. Accordingly, implementation of a module that is executable by the computing device 1102 as software may be achieved at least partially in hardware, e.g., through use of computer-readable storage media and/or hardware elements 1110 of the processing system 1104. The instructions and/or functions may be executable/operable by one or more articles of manufacture (for example, one or more computing devices 1102 and/or processing systems 1104) to implement techniques, modules, and examples described herein. (see at least Katariya3329, fig. 11, ¶181).
Sequential testing module 208 implemented at least partially in hardware to perform sequential hypothesis testing to determine an effect of different options on a statistic, e.g., conversion rate. The sequential testing module 208 may collect marketing data 206 which describes interaction of a plurality of users with digital marketing content 120. From this, an effect is determined of different items of digital marketing content 120 (e.g., items “A” and “B”) on conversion of a product or service being offered by the service provider 102. Statistical significance 210 is used to define a point at which is it considered “safe” to consider the test completed. The sequential testing module 208 then evaluates this marketing data 206 to compare groups of the users that have received item “A” with a group of the users that have received item “B,” e.g., to determine a conversion rate exhibited by the different items. Statistical significance 210 is also computed to determine whether it is “safe to stop the test” at this point, e.g., in order to reject the null hypothesis. (see at least Katariya3329, fig. 2, ¶50-53). Statistic testing module 718 (see at least Katariya3329, fig. 7, ¶146). Interaction determination module 728 (see at least Katariya3329, fig. 7, ¶148).
It follows that since testing, and confidence determinations, and data analysis, and visualization, are indeed performed by sequential testing module 208 in the context of the sequential testing, then instructions (modules as claimed) to execute these steps (a testing module of at least one processor) (a confidence module of the at least one processor) (an analysis module of the at least one processor) are implicit in the system architecture and methodology of Katariya3329.
(executing, using a testing module of the at least one processor, during a predetermined duration of time, one or more testing sequences prompting one or more responses to a plurality of variations of digital content from one or more users).
Katariya3329 generally teaches: Sequential hypothesis testing techniques and A/B testing.
In particular Katariya3329 discloses:
A/B testing. (see at least Katariya3329, fig. 5, ¶3-4, 49). Testing is used to compare different items of digital content against a current item of digital content to determine which item operates “best” as defined by a statistic. In a digital marketing scenario, this statistic includes a determination as to which item of digital content exhibits a greatest effect on conversion. Examples of conversion include interaction of a user with the content (e.g., a “click-through”), purchase of a product or service that pertains to the digital content, and so forth. (see at least Katariya3329, abstract, ¶30).
In contrast to conventional techniques that are based on a fixed horizon of samples, the disclosed sequential hypothesis testing techniques involve testing sequences of increasingly larger number of samples until a winner is determined. In particular, the winner is determined based on whether a result of a statistic (e.g., conversion rate) has reached statistical significance that defines a confidence level in the accuracy of the results. (see at least Katariya3329, ¶32).
To perform sequential hypothesis testing, the sequential testing module 208 (testing module) evaluates the marketing data 206 as it is received, e.g., in real time, to determine an effect of digital marketing content 120 on conversion (see at least Katariya3329, ¶51). The sequential testing module 208 then evaluates this marketing data 206 to compare groups of the users that have received item “A” with a group of the users that have received item “B,” e.g., to determine a conversion rate exhibited by the different items. (see at least Katariya3329, fig. 2, ¶52).
Example implementation to perform sequential hypothesis testing for more than two options. Testing may also be performed for multiple alternatives (e.g., “B,” “C,” “D,” and so on) against a base option, e.g., “A.” This results in multiple tests of the form (A, B), (A, C), (A, D), and so forth (see at least Katariya3329, fig. 5, ¶68).
(dynamically generating, using a confidence module of the at least one processor, without pausing the executing of the one or more testing sequences, at a randomly selected time in the predetermined duration of time, one or more confidence intervals for one or more test data received in response to the executing).
(determining, using the analysis module of the at least one processor, based on the testing metric and the one or more confidence intervals, a stopping time to stop execution of the one or more testing sequences, sequences, wherein the stopping time is associated with preventing occurrence of a type-I error associated with executing of the first and second testing seqiuences, and stopping execution of the one or more testing sequences at the stopping time).
(wherein the testing metric being bounded by one or more effect intervals determined based on the one or more confidence intervals generated for the one or more test data, each confidence interval in the one or more confidence intervals for each of the one or more test data is determined based on lower and upper bounds of the one or more test data).
(wherein the testing metric is determined at any time before expiration of the predetermined duration of time to indicate the effect of the second digital content over the first digital content, wherein in response to determining the testing metric to be outside of bounds of one or more effect intervals determined based on the one or more confidence intervals generated for the first test data and the one or more confidence intervals generated for the second test data, the executing of at least one of the first and second testing sequence based on the testing metric is terminated prior to expiration of the predetermined duration of time):
In the sequential hypothesis testing, a sample size calculator may be used before a test to estimate an amount of time that is likely needed to completed the test (see at least Katariya3329, ¶14, 37) (the first and second testing sequences executed during a predetermined duration of time). In sequential hypothesis testing, the output of the sample size determination module 602 is informative and not part of the test. In order to have sufficient Power and Type I error guarantees at the same time to allow peeking in hypothesis testing, the value of “N*” is often larger than the sample-size of the fixed-horizon test. (see at least Katariya3329, ¶153).
Sequential hypothesis testing techniques and systems are described. In contrast to conventional techniques that are based on a fixed horizon of samples, the disclosed sequential hypothesis testing techniques involve testing sequences of increasingly larger number of samples until a winner is determined. In particular, the winner is determined based on whether a result of a statistic (e.g., conversion rate) has reached statistical significance that defines a confidence level in the accuracy of the results. Thus, statistical significance defines when it is safe to conclude the test, e.g., based on a level of confidence of a computed result (e.g., conversion) against defined amounts of Type I and Type II errors. This permits the sequential hypothesis testing technique to conclude as soon as statistical significance is reached and a “winner” declared, without forcing a user to wait until the horizon “N” of a number of samples is reached. (see at least Katariya3329, ¶32).
In particular, the winner is determined based on whether a result of a statistic (e.g., conversion rate) has reached statistical significance that defines a confidence level in the accuracy of the results. Thus, statistical significance defines when it is safe to conclude the test, e.g., based on a level of confidence of a computed result (e.g., conversion) against defined amounts of Type I and Type II errors. This permits the sequential hypothesis testing technique to conclude as soon as statistical significance is reached and a “winner” declared, without forcing a user to wait until the horizon “N” of a number of samples is reached. (see at least Katariya3329, ¶32).
Statistical significance 210 is used in sequential hypothesis testing to define a point at which is it considered “safe” to consider the test completed. That is, a “safe” point of completion is safe with respect to an amount of false positives or false negatives permitted. This is performed in sequential hypothesis testing without setting the horizon “N” beforehand, which is required under the conventional fixed-horizon hypothesis testing. (see at least Katariya3329, fig. 2, ¶51). Statistical significance 210 is also computed to determine whether it is “safe to stop the test” at this point (a stopping time), e.g., in order to reject the null hypothesis (see at least Katariya3329, fig. 2, ¶52). Based on the response from these users described in the marketing data 206, a determination is made whether to reject or not reject the null hypothesis. Whether it is safe to make this determination is based on statistical significance 210, which takes into account accuracy guarantees regarding Type I and Type II errors, e.g., to ninety-five percent confidence that these errors do not occur. (see at least Katariya3329, fig. 2, ¶53).
A user viewing the user interfaces 300, 400 may also employ the value of the statistical significance as a “soft stop” as opposed to the hard stop of the decision boundary of fixed horizon hypothesis testing (see at least Katariya3329, fig. 2, ¶62).
In other words, in sequential hypothesis testing, at any time(s) (randomly selected time) during the estimated time duration of the A/B test (at any time before expiration of the predetermined duration of time), statistical significance tests of the ongoing test outcomes are performed which can be used to determine whether it is safe to stop the test (a stopping time) (terminated prior to expiration of the predetermined duration of time).
Confidence interval (see at least Katariya3329, fig. 4).
(dynamically generating, using a confidence module of the at least one processor, without pausing the executing of the one or more testing sequences, at a randomly selected time in the predetermined duration of time, one or more confidence intervals for one or more test data received in response to the executing) (bounds) (see at least Katariya3329, fig. 4, ¶32, 51-53, 57).
The data that are the subject of the above tests of statistical significance, confidence limits and/or accuracy levels, to determine an effect of digital marketing content 120 on conversion, represents “a testing metric indicating an effect of the second digital content over the first digital content”. (determining, using an analysis module of the at least one processor, without pausing the executing of the one or more testing sequences, a testing metric indicating an effect of one variation of the digital content over another variation of the digital content in the plurality of variations of digital content, wherein the testing metric is determined at any time before expiration of the predetermined duration of time to indicate the effect).
(in response to the stopping, generating, using a graphical user interface module communicatively coupled to the at least one processor, a graphical user interface including a visualization of the one or more confidence intervals for each first test data and for each second test data, and the determined testing metric prior to expiration of the predetermined duration of time ). The user interfaces 300, 400 are configured to provide information to a user while the test is running and even after it has stopped (see at least Katariya3329, fig. 3-4, ¶12-13, 37, 58-63).
Response to Arguments
Applicant's arguments filed 06/30/2025 have been fully considered.
35 U.S.C. 101
Applicant's arguments regarding 35 U.S.C. 101 are not persuasive. The rejection is maintained.
Applicant argues:
Specifically, the claimed subject matter, as recited in claim 1, provides a particular technical solution to a technical problem of determining when the right time is to terminate execution of testing sequences while execution of such sequences is ongoing and without pausing the execution. The current subject matter resolves the common problems that exist in the industry, and, in particular, the "peeking" or "early stopping" problems that inflate type-I errors in naive fixed horizon methodologies.
To resolve these problems, the current subject matter, as recited in claim 1, performs analyses of data without stopping execution of the testing sequencies, uses the analysis to confidently determine the right time to terminate execution of the sequencies, and terminates execution of the same at such time. In that regard, the undersigned respectfully disagrees with the Examiner's oversimplification of the concepts described and claimed by the present application in that "all the claim steps can be seen as being part of the abstract idea of generating and displaying lift information (lift effect metrics)." (See, Final Office Action, pg. 3).
Further, determination of precise bounds and accurate estimation of the effect/lift metric allows checking the collected data at any time during execution of the testing sequence and being certain that it accurately represents effects of digital content. As a result, if at any time, the determined effect/lift metric falls outside of the bounds, the current subject matter stops execution of the testing sequence(s), to thereby, as discussed herein, greatly reduce occurrence of type-I errors that permeate conventional solutions. (See, Specification, para. [0037]).
In response:
The pending office action very clearly and explicitly stated and explained: “the claims are directed to utilizing the known statistical techniques of sequential hypothesis testing in a generic, computerized environment.” Then in Step 2A- Prong 2: the pending rejection re-emphasized this position: “As mentioned above, the examiner considers the entirety of the claimed subject matter to be representative of merely applying the abstract idea of utilizing the known statistical techniques of sequential hypothesis testing in a generic, computerized environment, wherein this is the very definition of merely “apply it” in the area of patent eligibility determinations. Had applicants given weight to these known statistical techniques of sequential hypothesis as the position of a PHOSITA, they would not say that it is “an Examiner's oversimplification of the concepts described and claimed by the present application in that "all the claim steps can be seen as being part of the abstract idea of generating and displaying lift information (lift effect metrics)."
Sequential hypothesis testing has been well known for many decades before the filing of the instant invention, and as explained above it is the expected and predictable knowledge of a person of ordinary skill in the A/B testing arts. Sequential hypothesis testing was developed during World War II by Abraham Wald, who developed the Sequential Probability Ratio Test (SPRT) for efficient quality control. Other researchers, like George Barnard and Alan Turing, also developed similar methods for optimal stopping and hypothesis testing around the same time. The method, which allows for decisions to be made at any point during data collection, was published by Wald in 1945 and later introduced to medical research by Peter Armitage and Stuart Pocock, becoming increasingly popular in the field.
Sequential hypothesis testing is an old and well-known technique widely used by qualified testers to reduce or decrease Type I error in A/B tests (although that said, even the strongest guarantees aren’t 100% — a Type I error could always appear by sheer chance alone). A person of ordinary skill in the A/B testing arts is fully aware of a proper A/B test protocol required to minimize Type I error considerations.
The old and well known and widely used sequential hypothesis testing does not suffer from the drawbacks of the “conventional solutions” mentioned in the remarks; and while (by definition) Type I errors in A/B testing may not be not entirely preventable, a person of ordinary skill in the A/B testing arts is fully aware of a proper A/B test protocol required to minimize Type I error considerations.
It is further noted that making determinations based on statistical confidence levels does not represent a technological improvement.
Applicant argues:
Without conceding to the merits of the Final Office Action's discussion related to Step 2A, Prong 1 and Step 2B, the undersigned respectfully submits that the subject matter of the amended claim 1 is clearly integrated into a practical application, similar to the concepts discussed in Example 47 of the 2024 PEG Update, thereby satisfying requirements of Step 2A, Prong 2.
In response:
Applicants additionally draw an analogy between PEG example 47, claim 3, and the instant claimed invention (pp12-14). As correctly paraphrased in the remarks, in example 47, claim 3, “Steps (d)-(f) provide for improved network security using the information from the detection to enhance security by taking proactive measures to remediate the danger by detecting the source address associated with the potentially malicious packets. Specifically, the claim reflects the improvement in step (d), dropping potentially malicious packets in step (e), and blocking future traffic from the source address in step (f). These steps reflect the improvement described in the background.”
The examiner respectfully disagrees. This instant invention’s features: “an ability to collect specific data (e.g., execute "first and second testing sequences") and assess collected data (e.g., determining "testing metric" and "confidence intervals")”, bear no analogy whatsoever to the improvement in example 47 “in the technical field of network intrusion detection: “(d), dropping potentially malicious packets in step (e), and blocking future traffic from the source address in step (f).”
35 U.S.C. 102/103
New grounds of rejection are presented. Applicant's arguments are considered moot in view of the new grounds of rejection above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARIO IOSIF whose telephone number is (571) 270-7785. The examiner can normally be reached on Mon-Wed, 9:00am-4:00pm teleworking.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Waseem Ashraf can be reached on 571-270-39485491. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Mario C. Iosif/Primary Examiner, Art Unit 3621
Prosecution Insights