Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment/Arguments
1. Applicant’s arguments filed on June 17, 2026, regarding the rejection under 35 U.S.C. 101 have been fully considered but are not persuasive.
Applicant argues that, even assuming arguendo that the claims recite a judicial exception, the claims integrate the exception into a practical application under Step 2A, Prong Two. Applicant relies on MPEP 2106.04(d)(1) for the principle that a claim may integrate a judicial exception into a practical application when the claimed invention improves the functioning of a computer or improve another technology or technical field. The Examiner agrees with this general principle. However, MPEP 2106.05(a) further explains that “the judicial exception alone cannot provide the improvement” and that the improvement must be provided by one or more additional elements, either individually or in combination with the judicial exception.
In the present claims, the additional computer elements comprise of one or more processors and one or more non-transitory computer-readable media storing instructions. The processors are generally used to obtain data, process the data, select a stopping rule, evaluate convergence criterion, and stop data collection. These elements are recited at a high level of generality and do not require a particular processor architecture, specialized hardware arrangement, network architecture, or data acquisition structure. Rather, the processor and computer-readable media provide generic computer components to carry out the recited process. MPEP 2106.05(a) explains that “the claims must recite the details regarding how a computer aids the method, to the extent to which the computer aids the method, or the significance of a computer to the performance of the method” and that “[m]erely adding generic computer components to perform the method is not sufficient.”
Applicant next characterizes amended claim 1 as reciting a “specific technical process” in which data from an online experiment is obtained, the data is processed to determine where it exhibits a particular characteristic, a stopping rule associated with that characteristic is selected, the stopping rule is applied to determine whether a convergence criterion is met, and the online experiment is stopped by stopping ongoing data collection when the convergence criterion is met, and the online experiment is stopped by stopping ongoing data collection when the convergence criterion is met. However, Applicant has not identified how this sequence changes or improves the operation of the processor, network, data acquisition mechanism, or other technological component used to carry out the process. The recited computer components remain generic components used to implement the data analysis and resulting stopping determination. Thus, characterizing the claimed sequence as a “specific technical process” does not, by itself, demonstrate an improvement to computer functionality or another technology or technical field.
Applicant further asserts that this process represents an improvement to “online experiment technology” because processing the data in real time, selecting a stopping rule based on the detected characteristic, evaluating the corresponding convergence criterion, and stopping the experiment when the convergence criterion is met allows the experiment to stop as soon as enough data has been collected to derive a result. This argument is not persuasive. The disclosure defines an “online experiment” broadly as “any experiment in which data is being collected in real time from any source,” and provides examples including ballot information, baseball statistics, and web-server requests. Accordingly, the recitation of an “online experiment” does not itself identify a particular technological system or technological process whose operation has been improved. Rather, the asserted improvement concerns determining, based on analysis of the collected data, when sufficient data has been obtained and when further collection should cease. MPEP 2106.05(a) explains that an improvement in the abstract idea itself is not an improvement in technology.
Applicant also argues that efficiently stopping the experiment avoids unnecessary expenditure of compute resources, network resources, time, and cost. However, the claims do not recite a technological modification that causes the processor, network, or data acquisition system to use those resources more efficiently. Instead, the asserted reduction in resource usage results from determining that sufficient data has been collected and then cease further data collection. Thus, fewer resources are consumed as a consequence of performing less of the underlying activity, rather than a result of the improvement in how the underlying computer or technological system operates.
Accordingly, Applicant’s asserted improvement is attributable to the determination of when data collection should stop, while the additional computer elements provide generic computer implementation of that process. The claims therefore do not recite an improvement to the functioning of a computer or to another technology or technical field sufficient to integrate the judicial exception into a practical application under Step 2A, Prong Two. Applicant states that independent claims 10 and 19 include analogous features; therefore, Applicant’s arguments regarding those claims are unpersuasive for the same reasons.
Accordingly, for all the reasons set forth above, the rejection of claims 1-20 under 35 U.S.C. 101 is maintained.
2. Applicant’s arguments filed on June 17, 2026, regarding the rejection under 35 U.S.C. 103 have been fully considered but are not persuasive.
Applicant’s argues on pages 12-14 of the Remarks that the Action improperly relies on Pyzer-Knapp’s proposed next batch of candidate configurations rather than the results obtained from the data acquisition source. Applicant explains that the data acquisition source produces results from an actual process, that those results are used to generate a next batch of configurations, and that proposed configurations and calculated cPI values are subsequently evaluated in determining whether to continue the BBO search. Applicant therefore argues that the proposed configurations are not claimed data points obtained from an online experiment. Applicant’s argument has been considered. As explain in the present rejection, Pyzer-Knapp’s proposed configurations and calculated cPI values are not relied upon as the claimed data points. Rather, Pyzer-Knapp teaches that each sample has a corresponding actual or true observation obtained from an experimental or trial process and further teaches obtaining and storing the results produced by the last process. Pyzer-Knapp also establishes a batch size before the process is performed. Under the broadest reasonable interpretation, the actual observations or result values corresponding the predetermined batch of samples correspond to the claimed pre-determined number of data points obtained from the online experiment. Thus, the present rejection relies on the experimental observations/results themselves as the claimed data points, rather than the proposed configurations identified by the Applicant.
Applicant further argues at pages 13-14 of the Remarks that Pyzer-Knapp does not process those actual results to determine whether they exhibit a data characteristic, but instead uses the results to generate a subsequent batch of configurations. This argument does not address the references in combination. As explained in the present rejection, Pyzer-Knapp provides the experimental observations and repeatedly obtains additional observations through continued experimental runs, while Lindon teaches processing experimental data during execution of an A/B test to determine whether the data exhibits a sample ratio mismatch. Lindon further teaches detecting a statistically significant departure from an expected ratio and analyzing the same ratio mismatch after each new data point is received. Under BRI, the same ratio mismatch constitutes a characteristic of the experimental data. Accordingly, the combination of Pyzer-Knapp and Lindon corresponds to processing the experimental data points to determine whether the data points exhibit at least one characteristic.
Applicant additionally argues on page 14 of the Remarks that Pyzer-Knapp does not process the experimental results while additional results are continuously being obtained in real time. As explained in the present rejection, Pyzer-Knapp teaches repeatedly obtaining experimental observations through iterative experimental runs and performing real-time intervention in the data acquisition process. Lindon further teaches performing the characteristic analysis while the A/B test is executing and analyzing the sample ratio mismatch after each new data point is received. Thus, under BRI, the combined teachings correspond to processing the pre-determined experimental data points while additional experimental data points continue to be obtained in real time.
Applicant next argues on pages 14-15 of the Remarks that neither Pyzer-Knapp nor Lindon teaches selecting, in response to the pre-determined number of data points exhibiting a characteristic, a first stopping rule from a plurality of stopping rules, wherein the stopping rules correspond to different data characteristics. As explained in the present rejection, Kharitonov is relied upon for the additional amended limitation. Kharitonov teaches several sequential testing methods that reflect the distributions of data generated during experiments, teaches that sequential tests differ by their properties and assumptions, and describes different tests applicable to different experimental data distributions. Under BRI, the different experimental data distributions correspond to different data characteristics and the respective sequential testing procedures correspond to different stopping rules. When combined with Lindon’s determination that the experimental data exhibits a particular characteristic, Kharitonov supports selecting the stopping rule corresponding to the exhibited data characteristic from the plurality of stopping rules.
Applicant at page 15 of the Remarks that Lindon’s disclosure concerning activation of a smart page for an A/B test does not disclose selecting a stopping rule based on a characteristic of experimental data and therefore does not remedy the asserted deficiency of Pyzer-Knapp. However, the present rejection does not rely on Lindon’s smart-page disclosure for this limitation. Rather, as explained in the rejection, Lindon’s disclosure of detecting a sample ratio mismatch while the A/B test is executing is relied upon for determining a characteristic exhibited by the experimental data, while Kharitonov is relied upon for the plurality of different stopping procedures corresponding to different characteristics of the experimental data. Applicant’s argument concerning the smart page disclosure therefore does not address the teachings relied upon in the present rejection.
Accordingly, Applicant’s arguments do not overcome the rejection of independent claim 1. As explained in the present rejection, Pyzer-Knapp provides the pre-determined experimental observations and stopping framework, Lindon provides the determination of a characteristic of experimental data during continued acquisition of data, and Kharitonov provides the plurality of different stopping procedures corresponding to different characteristics of the experimental data. The references are relied upon in combination, and it would have been obvious to apply Lindon’s condition-based evaluation and Kharitonov’s different sequential stopping procedures within the stopping framework of Pyzer-Knapp in order to apply a stopping procedure appropriate to the characteristics of the collected data and terminate the experiment when sufficient data has been obtained, thereby avoid unnecessary continued data collection and the associated expenditure of time and computational resources.
Applicant asserts that independent claims 10 and 19 are allowable for the same reasons asserted with respect to claim 1. Claim 10 and 19 recited corresponding limitations and are rejected based on the same respective teachings and rationale discussed in the rejection. Accordingly, Applicant’s arguments are not persuasive for claims 10 and 19 for the same reasons.
Applicant does not separately address the additional limitations of the dependent claims, but states that those claims are allowable by virtue of depending from an allowable independent claim. Since Applicant’s arguments do not overcome the respective rejections of the dependent claims.
Accordingly, the rejection of claims 1-20 under 35 U.S.C. 103 is maintained.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
101 Subject Matter Eligibility Analysis
Step 1: Claims 1-20 are within the four statutory categories (a process, machine, manufacture or composition of matter).
Step 2A Prong One, Step 2A Prong Two, and Step 2B Analysis:
Step 2A Prong One asks if the claim recites a judicial exception (abstract idea, law of nature, or natural phenomenon). If the claim recites a judicial exception, analysis proceeds to Step 2A Prong Two, which asks if the claim recites additional elements that integrate the abstract idea into a practical application. If the claim does not integrate the judicial exception, analysis proceeds to Step 2B, which asks if the claim amounts to significantly more than the judicial exception. If the claim does not amount to significantly more than the judicial exception, the claim is not eligible subject matter under 35 U.S.C. 101.
None of the claims represent an improvement to technology.
Claims 1-9, 19 and 20 are directed to storage mediums and processors which are machines. Claims 10-18 are directed to a method consisting of a series of steps, meaning that it is directed to the statutory category of process.
Regarding claim 1, the following claim elements are abstract ideas:
process, while additional data points from the online experiment are being continuously obtained in real time, the pre-determined number of data points to determine whether the predetermined number of data points obtained from the online experiment exhibit at least one characteristic (This is an abstract idea of a mental process. The limitation involves reviewing a pre-determined set of data values and determining, based on observation and evaluation, whether those values exhibit a particular characteristic. A person could review the pre-determined set of values, for example, to determine whether the values remain constant, increase or decrease over time, or otherwise exhibit an identifiable pattern, using the human mind with the aid of pen and paper. The recitation of continuously obtaining additional data points in real time constitutes mere data gathering that provides information or context for the recited analysis and is therefore insignificant extra-solution activity.);
select, in response to the pre-determined number of data points exhibiting the at least one characteristic, a first stopping rule of a plurality of stopping rules, wherein the first stopping rule corresponds to the at least one characteristic, and each stopping rule of the plurality of stopping rules corresponds to a different data characteristic of a plurality of data characteristics comprising the at least one characteristic (This is an abstract idea of a mental process. The limitation involves selecting a rule from among multiple available rules based on an identified characteristic of a pre-determined set of data, wherein different rules are associated with different characteristics. A person could review the identified characteristic, compare it to the characteristics associated with the available stopping rules, and select the rule corresponding to that characteristic using observation and judgement. Such rule-based association and selection can be practically performed in the human mind with the aid of pen and paper or basic computational tools and thus constitutes an abstract idea of a mental process. See MPEP 2106.04(a)(2)(III).);
apply the first stopping rule to determine whether a first convergence criterion is met (This is an abstract idea of a mental process. It involves applying a rule to evaluate data and determine, based on observation and judgment, whether a condition is satisfied. A person could review data, apply a rule (e.g., compare values to a threshold or access whether results have stabilized), and decide whether the convergence is met. This type of evaluation can be performed in the human mind or with simple tolls and thus constitutes an abstract idea of a mental process.); and
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
one or more processors (This a high-level recitation of generic computer components for performing the abstract idea. See MPEP 2106.05(f).);
one or more non-transitory computer readable media (This a high-level recitation of generic computer components for performing the abstract idea. See MPEP 2106.05(f).)
obtain a pre-determined number of data points from an online experiment (The step of “obtaining” data points is merely a generic data gathering operation that amounts to receiving or retrieving information, which has been recognized by the courts as well-understood, routine, and conventional activity. See MPEP 2106.05(d)(II)(i).);
stop, when the first convergence criterion is met, the online experiment by stopping on-going data point collection (This limitation is merely an instruction to apply the abstract idea. It involves carrying out the result of the evaluation (i.e., stopping based on the determination that the criterion is met) and does not impose a meaningful limitation beyond the abstract idea.).
Regarding claim 2, the rejection of claim 1 is incorporated herein. Further, claim 2 recites the following additional elements, which taken alone or in combination with other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
obtain further data points (The step of “obtaining” additional data points is merely a generic data gathering operation that amounts to receiving or retrieving information, which has been recognized by the courts as well-understood, routine, and conventional activity.).
Regarding claim 3, the rejection of claim 2 is incorporated herein. Further, claim 3 recites the following abstract ideas:
determine that no characteristic is exhibited by the pre-determined number of data points and the additional data points (This is an abstract idea of a mental process. The limitation involves reviewing the pre-determined number of data points together with the additional data points and determining, based on observation and evaluation, that the combined data does not exhibit an identified characteristic. A person could examine the collected values, compare the values for an identifiable characteristic, and conclude that the characteristic is not present using the human mind with the aid of pen and paper or basic computational tools. Thus, this limitation constitutes an abstract idea of a mental process.); and
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
stop the online experiment after a maximum run time elapses (This limitation is merely an instruction to apply the abstract idea. It involves carrying out the result of the evaluation by stopping the experiment based on a time condition and does not impose a meaningful limitation beyond the abstract idea.).
Regarding claim 4, the rejection of claim 1 is incorporated herein. Further, claim 4 recites the following abstract ideas:
select, in response to the pre-determined number of data points exhibiting a second characteristic, a second stopping rule of the plurality of stopping rules, wherein the second stopping rule corresponds to the second characteristic (This is an abstract idea of a mental process. The limitation involves reviewing the pre-determined set of data, identifying a second characteristic exhibited by that data, and selecting from the available stopping rules the rule corresponding to the second characteristic. A person could examine the data, recognize the second characteristic, compare the characteristic with the available rule-to-characteristic associations, and choose the corresponding stopping rule using observation and judgement. Such rule-based decision-making can be practically performed in the human mind with the aid of pen and paper or basic computational tools and thus constitutes an abstract idea of a mental process.) and
apply the second stopping rule to determine whether a second convergence criterion is met (This is an abstract idea of a mental process. It involves applying a rule to evaluate data and determine, based on observation and judgment, whether a condition is satisfied. A person could review data, apply a rule (e.g., compare values to a threshold or access whether results have stabilized), and decide whether the convergence criteria is met. This type of evaluation can be performed in the human mind or with simple tools and thus constitutes an abstract idea of a mental process.),
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein the online experiment is stopped when the first convergence criterion and the second convergence criterion are met (This limitation is merely an instruction to apply the abstract idea and amounts to insignificant extra-solution activity. It involves carrying out the result of prior evaluations by stopping the experiment once the conditions are satisfied and does not impose a meaningful limitation beyond the abstract idea.).
Regarding claim 5, the rejection of claim 1 is incorporated herein. Further, claim 5 recites the following abstract ideas:
the first convergence criterion comprises determining whether a particular number of the data points are above a threshold (This is an abstract idea of a mental process. It involves counting data points and determining, based on observation and judgement, whether the count exceeds a threshold. A person could review a set of values, count the number of data points, and compare that number to a predefined threshold to decide if the condition is satisfied. This type of evaluation can be performed in the human mind or with simple tools and thus constitutes an abstract idea of a mental process.).
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein: the at least one characteristic comprises the pre-determined number of data points being constant (This limitation amounts to insignificant extra-solution activity. It merely identifies a type of data condition to which the abstract idea is applied and does not impose a meaningful limitation beyond the abstract idea.),
Regarding claim 6, the rejection of claim 1 is incorporated herein. Further, claim 6 recites the following abstract ideas:
the first convergence criterion comprises determining that the pre-determined number of data points are within a threshold of a limit (This is an abstract idea of a mental process. The limitation involves evaluating the pre-determined set of data values and determining whether those values fall within a specific threshold, and determining whether the values satisfy that condition using the human mind with the aid of pen and paper or basic computational tools. Thus, this limitation constitutes an abstract idea of a mental process.).
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein: the at least one characteristic comprises the pre-determined number of data points being monotonic (This limitation amounts to insignificant extra-solution activity. It merely identifies a type of data condition to which the abstract idea is applied and does not impose a meaningful limitation beyond the abstract idea.),
Regarding claim 7, the rejection of claim 1 is incorporated herein. Further, claim 7 recites the following abstract ideas:
the first convergence criterion comprises determining that a mean of the pre-determined number of data points is within a confidence interval of a specified maximal width (This is an abstract idea of a mental process and mathematical concept. The limitation involves calculating or evaluating a mean of a pre-determined set of data points, determining a confidence interval associated with that mean, and determining whether the confidence interval satisfies a specified maximum-width condition. These operations involve mathematical relationships and statistical calculations that can be practically performed in the human mind with the aid of pen and paper or basic computational tools. Thus, the limitation constitutes an abstract idea comprising mathematical concepts and a mental process.).
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein the at least one characteristic comprises the pre-determined number of data points having a normal distribution (This limitation amounts to insignificant extra-solution activity. It merely identifies a type of data condition to which the abstract idea is applied and does not impose a meaningful limitation beyond the abstract idea.)
Regarding claim 8, the rejection of claim 1 is incorporated herein. Further, claim 8 recites the following additional elements, which taken alone or in combination with other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein: the at least one characteristic comprises the pre-determined number of data points being modal (This limitation is merely an instruction to apply the abstract idea and amounts to insignificant extra-solution activity. It involves applying a model to evaluate data and does not impose a meaningful limitation beyond the abstract idea.),
the first convergence criterion comprises applying a regression model or a clustering model (This limitation is merely an instruction to apply the abstract idea and amounts to insignificant extra-solution activity. It involves applying a model to evaluate data and does not impose a meaningful limitation beyond the abstract idea.).
Regarding claim 9, the rejection of claim 1 is incorporated herein. Further, claim 9 recites the following additional elements, which taken alone or in combination with other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein the at least one characteristic comprises the pre-determined number of data points being autocorrelated (This limitation amounts to insignificant extra-solution activity. It merely identifies a type of data condition to which the abstract idea is applied and does not impose a meaningful limitation beyond the abstract idea.).
Regarding claim 10, the following claim elements are abstract ideas:
processing…while additional data points from the online experiment are being continuously obtained in real time, the pre-determined number of data points to determine whether the predetermined number of data points obtained from the online experiment exhibit at least one characteristic (This is an abstract idea of a mental process. The limitation involves reviewing a pre-determined set of data values and determining, based on observation and evaluation, whether those values exhibit a particular characteristic. A person could review the pre-determined set of values, for example, to determine whether the values remain constant, increase or decrease over time, or otherwise exhibit an identifiable pattern, using the human mind with the aid of pen and paper. The recitation of continuously obtaining additional data points in real time constitutes mere data gathering that provides information or context for the recited analysis and is therefore insignificant extra-solution activity.);
selecting…in response to the pre-determined number of data points exhibiting the at least one characteristic, a first stopping rule of a plurality of stopping rules, wherein the first stopping rule corresponds to the at least one characteristic, and each stopping rule of the plurality of stopping rules corresponds to a different data characteristic of a plurality of data characteristics comprising the at least one characteristic (This is an abstract idea of a mental process. The limitation involves selecting a rule from among multiple available rules based on an identified characteristic of a pre-determined set of data, wherein different rules are associated with different characteristics. A person could review the identified characteristic, compare it to the characteristics associated with the available stopping rules, and select the rule corresponding to that characteristic using observation and judgement. Such rule-based association and selection can be practically performed in the human mind with the aid of pen and paper or basic computational tools and thus constitutes an abstract idea of a mental process. See MPEP 2106.04(a)(2)(III).);
applying...the first stopping rule to determine whether a first convergence criterion is met (This is an abstract idea of a mental process. It involves applying a rule to evaluate data and determine, based on observation and judgment, whether a condition is satisfied. A person could review data, apply a rule (e.g., compare values to a threshold or access whether results have stabilized), and decide whether the convergence is met. This type of evaluation can be performed in the human mind or with simple tolls and thus constitutes an abstract idea of a mental process.); and
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
by a processing device (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).),
obtaining…a pre-determined number of data points from an online experiment (The step of “obtaining” data points is merely a generic data gathering operation that amounts to receiving or retrieving information, which has been recognized by the courts as well-understood, routine, and conventional activity. See MPEP 2106.05(d)(II)(i).);
stopping, when the first convergence criterion is met, the online experiment by stopping on-going data point collection (This limitation is merely an instruction to apply the abstract idea. It involves carrying out the result of the evaluation (i.e., stopping based on the determination that the criterion is met) and does not impose a meaningful limitation beyond the abstract idea.).
Regarding claim 11, the rejection of claim 10 is incorporated herein. The claim recites similar limitations corresponding to claim 2. Therefore, the same subject matter analysis that was utilized for claim 2, as described above, is equally applicable to claim 11.
Therefore, claim 11 is ineligible.
Regarding claim 12, the rejection of claim 11 is incorporated herein. The claim recites similar limitations corresponding to claim 3. Therefore, the same subject matter analysis that was utilized for claim 3, as described above, is equally applicable to claim 12.
Therefore, claim 12 is ineligible.
Regarding claim 13, the rejection of claim 10 is incorporated herein. The claim recites similar limitations corresponding to claim 4. Therefore, the same subject matter analysis that was utilized for claim 4, as described above, is equally applicable to claim 13.
Therefore, claim 13 is ineligible.
Regarding claim 14, the rejection of claim 10 is incorporated herein. The claim recites similar limitations corresponding to claim 5. Therefore, the same subject matter analysis that was utilized for claim 5, as described above, is equally applicable to claim 14.
Therefore, claim 14 is ineligible.
Regarding claim 15, the rejection of claim 10 is incorporated herein. The claim recites similar limitations corresponding to claim 6. Therefore, the same subject matter analysis that was utilized for claim 6, as described above, is equally applicable to claim 15.
Therefore, claim 15 is ineligible.
Regarding claim 16, the rejection of claim 10 is incorporated herein. The claim recites similar limitations corresponding to claim 7. Therefore, the same subject matter analysis that was utilized for claim 7, as described above, is equally applicable to claim 16.
Therefore, claim 16 is ineligible.
Regarding claim 17, the rejection of claim 10 is incorporated herein. The claim recites similar limitations corresponding to claim 8. Therefore, the same subject matter analysis that was utilized for claim 8, as described above, is equally applicable to claim 17.
Therefore, claim 17 is ineligible.
Regarding claim 18, the rejection of claim 10 is incorporated herein. The claim recites similar limitations corresponding to claim 9. Therefore, the same subject matter analysis that was utilized for claim 9, as described above, is equally applicable to claim 18.
Therefore, claim 18 is ineligible.
Regarding claim 19, the following claim elements are abstract ideas:
process…while additional data points from the online experiment are being continuously obtained in real time, the pre-determined number of data points to determine whether the predetermined number of data points obtained from the online experiment exhibit at least one characteristic (This is an abstract idea of a mental process. The limitation involves reviewing a pre-determined set of data values and determining, based on observation and evaluation, whether those values exhibit a particular characteristic. A person could review the pre-determined set of values, for example, to determine whether the values remain constant, increase or decrease over time, or otherwise exhibit an identifiable pattern, using the human mind with the aid of pen and paper. The recitation of continuously obtaining additional data points in real time constitutes mere data gathering that provides information or context for the recited analysis and is therefore insignificant extra-solution activity.);
select…in response to the pre-determined number of data points exhibiting the at least one characteristic, a first stopping rule of a plurality of stopping rules, wherein the first stopping rule corresponds to the at least one characteristic, and each stopping rule of the plurality of stopping rules corresponds to a different data characteristic of a plurality of data characteristics comprising the at least one characteristic (This is an abstract idea of a mental process. The limitation involves selecting a rule from among multiple available rules based on an identified characteristic of a pre-determined set of data, wherein different rules are associated with different characteristics. A person could review the identified characteristic, compare it to the characteristics associated with the available stopping rules, and select the rule corresponding to that characteristic using observation and judgement. Such rule-based association and selection can be practically performed in the human mind with the aid of pen and paper or basic computational tools and thus constitutes an abstract idea of a mental process. See MPEP 2106.04(a)(2)(III).);
apply…the first stopping rule to determine whether a first convergence criterion is met (This is an abstract idea of a mental process. It involves applying a rule to evaluate data and determine, based on observation and judgment, whether a condition is satisfied. A person could review data, apply a rule (e.g., compare values to a threshold or access whether results have stabilized), and decide whether the convergence is met. This type of evaluation can be performed in the human mind or with simple tolls and thus constitutes an abstract idea of a mental process.); and
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
A non-transitory computer-readable medium (This a high-level recitation of generic computer components for performing the abstract idea. See MPEP 2106.05(f).)
by a processing device (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).)
obtain…a pre-determined number of data points from an online experiment (The step of “obtaining” data points is merely a generic data gathering operation that amounts to receiving or retrieving information, which has been recognized by the courts as well-understood, routine, and conventional activity. See MPEP 2106.05(d)(II)(i).);
stop…when the first convergence criterion is met, the online experiment by stopping on-going data point collection (This limitation is merely an instruction to apply the abstract idea. It involves carrying out the result of the evaluation (i.e., stopping based on the determination that the criterion is met) and does not impose a meaningful limitation beyond the abstract idea.).
Regarding claim 20, the rejection of claim 19 is incorporated herein. Further, claim 20 recites the following additional elements, which taken alone or in combination with other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
when the first convergence criterion is not met, to obtain further data points (This limitation amounts to insignificant extra-solution activity. It merely recites obtaining additional data as part of the abstract idea and does not impose a meaningful limitation beyond the abstract idea.).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 10-13, and 19-20 are rejected under the 35 U.S.C. 103 as being unpatentable over Pyzer-Knapp et al., (Pub. No.: US 20220128972 A1 (Filed: 2020)) in view of Lindon (Pub. No.: US 20220129765 A1 (Filed: 2020)) further in view of Kharitonov et al., (NPL: “Sequential Testing for Early Stopping of Online Experiments” (Published: 2015)).
Regarding claim 1, Pyzer-Knapp teaches the following limitations:
An apparatus, comprising: one or more processors; and one or more non-transitory computer readable media storing instructions which, when executed by the one or more processors, cause the one or more processors to (Pyzer-Knapp, paragraph [0021] “] A processor may be a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), another suitable processing component or device, or one or more combinations thereof.” [0083] “The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random-access memory (RAM), a read-only memory (ROM),”):
obtain a pre-determined number of data points from an online experiment (Pyzer-Knapp, paragraph [0017] “. A corresponding output to the new sample is obtained. This corresponding output can be an actual or true observation obtained based on using the new sample. For instance, the actual or true observation can be obtained from running a real experimental or trial manufacturing production process… Batch Bayesian optimization (BBO) finds optimal samples in batches.” [0025] “the Bayesian optimization system 104 can be automatically programmed or triggered to start with an initial set of criteria (e.g., budget, batch size, aggression of early termination)” [0026] “A decision module 112 of the Bayesian optimization system 104 generates a batch of configurations and communicates the configurations to a data acquisition source 106…The batch of configurations includes different parameter values the data acquisition source 106 can try or experiment with… to perform a process or a run (e.g., an experimental process or trial run) to produce an actual output” [0028] “The data acquisition source 106 generates a result of the configurations (e.g., as a result of performing the real process)… The result includes one or more properties of an outcome obtained from the actual run of a process…The storage module 114 of the Bayesian optimization system 104 stores the results of the configurations” - Pyzer-Knapp teaches obtaining samples in batches, wherein each sample has a corresponding actual or true observation obtained from an experimental or trial process and further teaches establishing a batch size before performing the process. Thus, the predetermined batch size before performing the process. Thus, the predetermined batch size establishes the predetermined number of samples for which each corresponding experimental observations or results are obtained. Applicant describes data points as discrete units of data obtained during an experiment. Accordingly, under BRI, Pyzer-Knapp’s actual observations or result values corresponding to the predetermined batch of samples correspond to the claimed pre-determined number of data points. Applicant further describes an online experiment as any experiment in which data is collected in real time from any source. Pyzer-Knapp teaches performing experimental or trial runs in a real process and obtaining the corresponding actual observations or results, including trial experimental runs performed in real time. Thus, Pyzer-Knap’s experimental process falls within the broadest reasonable interpretation of the claimed online experiment.);
apply the first stopping rule to determine whether a first convergence criterion is met; and stop, when the first convergence criterion is met, the online experiment by stopping on-going data point collection (Pyzer-Knapp, paragraph [0038] “The stopping criteria evaluator 306 determines whether to stop searching for a next batch of configurations, for example, by applying a function to the acquisition function of the batch Bayesian optimization engine 304… For instance, if a stopping criterion is met, the stopping criteria evaluator 306 sends a signal indicating that the search is to be stopped. Otherwise, the stopping criteria evaluator 306 can send a signal or an indication that the searching is to be continued.” [0039] “If no termination or stop signal is received, the batch Bayesian optimization engine 304 sends the next batch of configurations to the data acquisition source 308 to use in its real or actual process. The data acquisition source 308 performs its process using this next batch of configurations, returning the result to the batch Bayesian optimization engine 304.” [0040] “Finding next batch of configurations at 304, evaluating whether stopping criteria is met at 306 and sending the next batch of configurations to the data acquisition source 308, wherein the data acquisition source 308 performs its test process, and storing the result received from the data acquisition source 308 is repeated until the stopping criteria evaluator 306 determines that the stopping criteria is met or the specified budget (e.g., number of iterations) is exhausted.” – Pyzer-Knapp teaches applying a stopping criterion to determine whether the criterion has been met and generating a stop signal when it is met. When no stop signal is received, the data acquisition source continues performing the experimental process and returning additional results; when the stopping criterion is met, the data acquisition source may be controlled to stop the experiment or trial process. Under the broadest reasonable interpretation, this corresponds to applying the first stopping rule to determine whether a convergence criterion is met and, when met, stopping the online experiment by stopping ongoing collection of experimental data points.).
However, Pyzer-Knapp does not teach but Pyzer-Knapp in view of Lindon teaches the following limitation:
process, while additional data points from the online experiment are being continuously obtained in real time. the pre-determined number of data points to determine whether the predetermined number of data points obtained from the online experiment exhibit at least one characteristic (Pyzer-Knapp, [0017] “A corresponding output to the new sample is obtained. This corresponding output can be an actual or true observation obtained based on using the new sample…This process of finding or searching for a next sample, obtaining actual observation using that next sample, and updating the surrogate model can be repeated…Batch Bayesian optimization (BBO) finds optimal samples in batches.” [0040] “Finding next batch of configurations at 304, evaluating whether stopping criteria is met at 306 and sending the next batch of configurations to the data acquisition source 308, wherein the data acquisition source 308 performs its test process, and storing the result received from the data acquisition source 308 is repeated until the stopping criteria evaluator 306 determines that the stopping criteria is met…“ [0041] “The method performs real-time intervention in a batch Bayesian optimization (BBO) system, automatically determining whether to terminate a BBO search early if a predetermined threshold is reached. In an aspect, running a data acquisition source, for example, in real-time, such as a manufacturing process and/or a robot can be costly. For instance, iteratively running a data acquisition source with new or next configurations may consume power, equipment and computer resources. The method in an embodiment intervenes in real-time to stop early iterative runs of a data acquisition source…” Lindon, paragraph [0040] “Processing logic at block 404 may, while the A/B test is executing, determine (e.g., by a processing device) that a sample ratio mismatch corresponding to the second plurality of users has occurred.” [0041] “To detect the sample ratio mismatch, processing logic may employ a variety of techniques. For example, in one embodiment, processing logic may detect a statistically significant departure from a ratio corresponding to the first plurality of users and the second plurality of users…” [0042] “the sample ratio mismatch is determined before the A/B test ends executing…In one embodiment, the sample ratio mismatch may be analyzed after each new data point is received.” – Pyzer-Knapp teaches obtaining experimental observations in batches, processing the observations to update the surrogate model, and repeatedly obtaining additional observations through iterative experimental runs. Pyzer-Knapp further teaches that the data acquisition source operates and is controlled in real time. Lindon teaches processing experimental data while the test is executing to determine whether the data exhibits a statistical characteristic, specifically a sample ratio mismatch, including analysis after each new data point is received. Under BRI, the combined teachings correspond to processing the pre-determined number of data points while additional data points continue to be obtained in real time and determining whether the pre-determined number of data points exhibits at least one characteristic.);
However, Pyzer-Knapp in view of Lindon does not teach, but Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches:
select, in response to the pre-determined number of data points exhibiting the at least one characteristic, a first stopping rule of a plurality of stopping rules, wherein; the first stopping rule corresponds to the at least one characteristic, and each stopping rule of the plurality of stopping rules corresponds to a different data characteristic of a plurality of data characteristics comprising the at least one characteristic (Kharitonov, [page 474, Introduction] “We propose several sequential testing methods that reflect the distributions of the data generated in A/B and interleaving experiments, and describe how to adjust their stopping thresholds based on query log data” [page 475, section 2.2] “Overall, sequential testing is a highly developed discipline, and a variety of tests that differ by their properties and assumptions was proposed…Due to their properties, we select the O'Brien&Fleming and MaxSPRT tests as a foundation for our study.” [page 476, section 4] “In this section, we introduce the sequential testing procedures we consider in this work…O'Brien&Fleming's sequential test, modified for interleaving (OBF-I), and the MaxSPRT test. After that, we describe two tests applicable for A/B tests: the standard O'Brien&Fleming test, and our proposed MaxSPRT-AB test, tailored for A/B test experiments. For all of these tests, we also describe algorithms to train their stopping thresholds.” [page 477] “MaxSPRT-AB The MaxSPRT rule cannot be easily applied for the case of A/B experiments, since the distribution of the considered metric is unknown both under H0 and H1. To address this, we propose to estimate both the distribution of the metric under the null hypothesis and under the alternative hypothesis using data from the experiment itself.” – As discussed above, Pyzer-Knapp in view of Lindon teaches determining that the pre-determined number of experimental data points exhibits at least one characteristic. Kharitonov teaches a plurality of different stopping rules and further teaches that the applicable sequential testing procedure depends on characteristics of the experimental data, including its distribution, with different tests being tailored to different experimental data distributions. Under BRI, these different data distributions correspond to different data characteristics and that respective sequential tests correspond to different stopping rules. Thus, in combination with the detected data characteristic discussed above, Kharitonov supports selecting the stopping rule corresponding to exhibited data characteristic from a plurality of stopping rules.);
Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having Pyzer-Knapp, Lindon, and Kharitonov before them, to apply Lindon’s conditioned-based evaluation and Kharitonov’s different sequential stopping procedures within the stopping framework of Pyzer-Knapp. One would have been motivated to make such a combination to apply a stopping procedure appropriate to the characteristics of the collected data and terminate the experiment when sufficient data has been obtained, thereby avoiding unnecessary continued data collection and the associated expenditure of time and computational resources.
Regarding claim 2, Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1. Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches:
when the first convergence criterion is not met, to: obtain further data points (Pyzer-Knapp, [0040] “evaluating whether stopping criteria is met at 306 and sending the next batch of configurations to the data acquisition source 308, wherein the data acquisition source 308 performs its test process, and storing the result received from the data acquisition source 308 is repeated until the stopping criteria evaluator 306 determines that the stopping criteria is met or the specified budget (e.g., number of iterations) is exhausted.” – Pyzer-Knapp teaches that when stopping criteria are not met, the process continues by sending a next batch of configurations to the data acquisition source, performing another experimental process, and receiving and storing additional results. As discussed with claim 1, under BRI, the experimental results correspond to data points. Thus, when the first convergence criterion is not met, Pyzer-Knapp obtains additional data points by continuing the experiment and receiving further experimental results.
Regarding claim 3, Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches all the elements of claim 2, therefore is rejected for the same reasons as those presented in claim 2. Pyzer-Knapp in view of Lindon further in view of Kharitonov further teaches:
after obtaining the additional data points, to: determine that no characteristic is exhibited by the pre-determined number of data points and the additional data points; and stop the online experiment after a maximum run time elapses (Lindon, [0041] “To detect the sample ratio mismatch, processing logic may employ a variety of techniques. For example, in one embodiment, processing logic may detect a statistically significant departure from a ratio corresponding to the first plurality of users and the second plurality of users, wherein a probability that the sample ratio mismatch has occurred is greater than a predetermined error threshold” [0042] “In one embodiment, the sample ratio mismatch may be analyzed after each new data point is received.” [0043] “In one embodiment, processing logic at block 406 may, in response to the determining, end the execution of the A/B test before a previously scheduled end of the A/B test (e.g., when a sample ratio mismatch is determined before the end of the A/B testing period).” Pyzer-Knapp, paragraph [0040] “evaluating whether stopping criteria is met at 306 and sending the next batch of configurations to the data acquisition source 308, wherein the data acquisition source 308 performs its test process, and storing the result received from the data acquisition source 308 is repeated until the stopping criteria evaluator 306 determines that the stopping criteria is met or the specified budget (e.g., number of iterations) is exhausted. “– Lindon teaches repeatedly analyzing the experimental data for a characteristic, namely a statistically significant sample ratio mismatch, including performing the analysis after each additional data point is received. Under BRI, after additional data points are received, the analysis is performed on the accumulated experimental data, corresponding to the pre-determined number of data points together with additional data points. When the analysis does not detect the sample ratio mismatch, the accumulated data do not exhibit the evaluated characteristic. Lindon further establishes a previously scheduled end of the A/B test period, while Pyzer-Knapp teaches continuing the experimental process until either a stopping criterion is met or a specified maximum budget is exhausted. Thus, when the evaluated characteristic is not exhibited, the experiment may continue until its predetermined upper limit is reached, corresponding under BRI to stopping the online experiment after a maximum run time elapses.)
Regarding claim 4, Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches all the elements of claim 1, therefore it is rejected for the same reasons as those presented for claim 1. Pyzer-Knapp in view of Lindon further in view of Kharitonov further teaches:
before stopping the online experiment, to: select, in response to the pre-determined number of data points exhibiting a second characteristic, a second stopping rule of the plurality of stopping rules, wherein the second stopping rule corresponds to the second characteristic (Lindon, paragraph [0065] “processing logic at block 502, during execution of an A/B test, determines (e.g., by a processing device using a sequential frequentist test) that a sample ratio mismatch has occurred. In one embodiment, the sequential frequentist test comprises one or more Bayesian multinomial-Dirichlet families.” Kharitonov, [page 475] “Overall, sequential testing is a highly developed discipline, and a variety of tests that differ by their properties and assumptions was proposed… Due to their properties, we select the O'Brien&Fleming and MaxSPRT tests as a foundation for our study.” [page 476, section 4] “After that, we describe two tests applicable for A/B tests: the standard O'Brien&Fleming test, and our proposed MaxSPRT-AB test, tailored for A/B test experiments.” – Lindon teaches determining, during an online experiment, that the experimental data exhibits a sample ratio mismatch, which corresponds to a second characteristic of the data. As discussed with respect to claim 1, under BRI, the data evaluated during the experiment includes the pre-determined number of data points. Kharitonov teaches a plurality of different sequential stopping procedures and selecting among different tests based on their properties and applicability to the experimental data. Thus, in view of the characteristic-based rule from the plurality of stopping rules in response to the pre-determined number of data points exhibiting the second characteristic, where in the second stopping rule corresponds to the second characteristic.); and
apply the second stopping rule to determine whether a second convergence criterion is met, wherein the online experiment is stopped when the first convergence criterion and the second convergence criterion are met (Kharitonov, [page 478] “Once
L
i
achieves a pre-fixed threshold
L
-
, the experiment is stopped and
H
1
is accepted. Otherwise, the experiment is continued.” Pyzer-Knapp, paragraph [0019] “In an embodiment, a criterion for terminating a BBO search early considers two variables, a target criterion and a batch percentage criterion. The target criterion represents the statistical significance required to terminate the search...and the percentage criterion determines how much of the batch needs to fail to terminate the search” [0046] “In an embodiment, applying the function includes evaluating data associated with the next batch of candidates based on the target criterion and the batch percentage criterion. In an embodiment, determining whether a stopping criterion is met based on the next batch of candidate configurations can include, for each of the candidate configurations in the batch, computing a contextual probability of improvement (cPI) score, and determining that a percentage of the candidates configurations with the cPI score less than the target criterion is greater than the batch percentage criterion” [0047] “At 414, responsive to determining that the stopping criterion is met, a search for the next batch of candidates is stopped or terminated. In an embodiment, the industrial process can be controlled to stop running.” – Kharitonov teaches applying a sequential stopping rule by evaluating a calculated statistic against a predetermined threshold, and stopping the experiment when the threshold is reached. Under BRI, the threshold corresponds to the claimed second convergence criterion, and evaluating whether the threshold has been reached corresponds to applying the second stopping rule to determine whether the criterion is met. Pyzer-Knapp further teaches that a termination determination may depend on multiple conditions, specifically a target criterion and a batch percentage criterion, and evaluates the data using both in determining whether the stopping criterion has been met. When the resulting stopping criterion is met, the process is stopped. Under BRI, the combined teachings correspond to requiring the respective convergence conditions associated with the first and second stopping rules to be satisfied before stopping the online environment, thereby corresponding to stopping the online experiment when the first convergence criterion and the second convergence criterion are met.).
Regarding claim 10, Pyzer-Knapp teaches the following limitations:
A computer-implemented method, comprising: obtaining, by a processing device, a pre-determined number of data points from an online experiment (Pyzer-Knapp, paragraph [0017] “. A corresponding output to the new sample is obtained. This corresponding output can be an actual or true observation obtained based on using the new sample. For instance, the actual or true observation can be obtained from running a real experimental or trial manufacturing production process… Batch Bayesian optimization (BBO) finds optimal samples in batches.” [0025] “the Bayesian optimization system 104 can be automatically programmed or triggered to start with an initial set of criteria (e.g., budget, batch size, aggression of early termination)” [0026] “A decision module 112 of the Bayesian optimization system 104 generates a batch of configurations and communicates the configurations to a data acquisition source 106…The batch of configurations includes different parameter values the data acquisition source 106 can try or experiment with… to perform a process or a run (e.g., an experimental process or trial run) to produce an actual output” [0028] “The data acquisition source 106 generates a result of the configurations (e.g., as a result of performing the real process)… The result includes one or more properties of an outcome obtained from the actual run of a process…The storage module 114 of the Bayesian optimization system 104 stores the results of the configurations” - Pyzer-Knapp teaches obtaining samples in batches, wherein each sample has a corresponding actual or true observation obtained from an experimental or trial process and further teaches establishing a batch size before performing the process. Thus, the predetermined batch size before performing the process. Thus, the predetermined batch size establishes the predetermined number of samples for which each corresponding experimental observations or results are obtained. Applicant describes data points as discrete units of data obtained during an experiment. Accordingly, under BRI, Pyzer-Knapp’s actual observations or result values corresponding to the predetermined batch of samples correspond to the claimed pre-determined number of data points. Applicant further describes an online experiment as any experiment in which data is collected in real time from any source. Pyzer-Knapp teaches performing experimental or trial runs in a real process and obtaining the corresponding actual observations or results, including trial experimental runs performed in real time. Thus, Pyzer-Knap’s experimental process falls within the broadest reasonable interpretation of the claimed online experiment.);
applying, by the processing device, the first stopping rule to determine whether a first convergence criterion is met; and stopping, when the first convergence criterion is met, the online experiment by stopping on-going data point collection (Pyzer-Knapp, paragraph [0038] “The stopping criteria evaluator 306 determines whether to stop searching for a next batch of configurations, for example, by applying a function to the acquisition function of the batch Bayesian optimization engine 304… For instance, if a stopping criterion is met, the stopping criteria evaluator 306 sends a signal indicating that the search is to be stopped. Otherwise, the stopping criteria evaluator 306 can send a signal or an indication that the searching is to be continued.” [0039] “If no termination or stop signal is received, the batch Bayesian optimization engine 304 sends the next batch of configurations to the data acquisition source 308 to use in its real or actual process. The data acquisition source 308 performs its process using this next batch of configurations, returning the result to the batch Bayesian optimization engine 304.” [0040] “Finding next batch of configurations at 304, evaluating whether stopping criteria is met at 306 and sending the next batch of configurations to the data acquisition source 308, wherein the data acquisition source 308 performs its test process, and storing the result received from the data acquisition source 308 is repeated until the stopping criteria evaluator 306 determines that the stopping criteria is met or the specified budget (e.g., number of iterations) is exhausted.” – Pyzer-Knapp teaches applying a stopping criterion to determine whether the criterion has been met and generating a stop signal when it is met. When no stop signal is received, the data acquisition source continues performing the experimental process and returning additional results; when the stopping criterion is met, the data acquisition source may be controlled to stop the experiment or trial process. Under the broadest reasonable interpretation, this corresponds to applying the first stopping rule to determine whether a convergence criterion is met and, when met, stopping the online experiment by stopping ongoing collection of experimental data points.).
However, Pyzer-Knapp does not teach but Pyzer-Knapp in view of Lindon teaches the following limitation:
processing, by the processing device and while additional data points from the online experiment are being continuously obtained in real time. the pre-determined number of data points to determine whether the predetermined number of data points obtained from the online experiment exhibit at least one characteristic (Pyzer-Knapp, [0017] “A corresponding output to the new sample is obtained. This corresponding output can be an actual or true observation obtained based on using the new sample…This process of finding or searching for a next sample, obtaining actual observation using that next sample, and updating the surrogate model can be repeated…Batch Bayesian optimization (BBO) finds optimal samples in batches.” [0040] “Finding next batch of configurations at 304, evaluating whether stopping criteria is met at 306 and sending the next batch of configurations to the data acquisition source 308, wherein the data acquisition source 308 performs its test process, and storing the result received from the data acquisition source 308 is repeated until the stopping criteria evaluator 306 determines that the stopping criteria is met…“ [0041] “The method performs real-time intervention in a batch Bayesian optimization (BBO) system, automatically determining whether to terminate a BBO search early if a predetermined threshold is reached. In an aspect, running a data acquisition source, for example, in real-time, such as a manufacturing process and/or a robot can be costly. For instance, iteratively running a data acquisition source with new or next configurations may consume power, equipment and computer resources. The method in an embodiment intervenes in real-time to stop early iterative runs of a data acquisition source…” Lindon, paragraph [0040] “Processing logic at block 404 may, while the A/B test is executing, determine (e.g., by a processing device) that a sample ratio mismatch corresponding to the second plurality of users has occurred.” [0041] “To detect the sample ratio mismatch, processing logic may employ a variety of techniques. For example, in one embodiment, processing logic may detect a statistically significant departure from a ratio corresponding to the first plurality of users and the second plurality of users…” [0042] “the sample ratio mismatch is determined before the A/B test ends executing…In one embodiment, the sample ratio mismatch may be analyzed after each new data point is received.” – Pyzer-Knapp teaches obtaining experimental observations in batches, processing the observations to update the surrogate model, and repeatedly obtaining additional observations through iterative experimental runs. Pyzer-Knapp further teaches that the data acquisition source operates and is controlled in real time. Lindon teaches processing experimental data while the test is executing to determine whether the data exhibits a statistical characteristic, specifically a sample ratio mismatch, including analysis after each new data point is received. Under BRI, the combined teachings correspond to processing the pre-determined number of data points while additional data points continue to be obtained in real time and determining whether the pre-determined number of data points exhibits at least one characteristic.);
However, Pyzer-Knapp in view of Lindon does not teach, but Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches:
selecting, by the processing device and in response to the pre-determined number of data points exhibiting the at least one characteristic, a first stopping rule of a plurality of stopping rules, wherein; the first stopping rule corresponds to the at least one characteristic, and each stopping rule of the plurality of stopping rules corresponds to a different data characteristic of a plurality of data characteristics comprising the at least one characteristic (Kharitonov, [page 474, Introduction] “We propose several sequential testing methods that reflect the distributions of the data generated in A/B and interleaving experiments, and describe how to adjust their stopping thresholds based on query log data” [page 475, section 2.2] “Overall, sequential testing is a highly developed discipline, and a variety of tests that differ by their properties and assumptions was proposed…Due to their properties, we select the O'Brien&Fleming and MaxSPRT tests as a foundation for our study.” [page 476, section 4] “In this section, we introduce the sequential testing procedures we consider in this work…O'Brien&Fleming's sequential test, modified for interleaving (OBF-I), and the MaxSPRT test. After that, we describe two tests applicable for A/B tests: the standard O'Brien&Fleming test, and our proposed MaxSPRT-AB test, tailored for A/B test experiments. For all of these tests, we also describe algorithms to train their stopping thresholds.” [page 477] “MaxSPRT-AB The MaxSPRT rule cannot be easily applied for the case of A/B experiments, since the distribution of the considered metric is unknown both under H0 and H1. To address this, we propose to estimate both the distribution of the metric under the null hypothesis and under the alternative hypothesis using data from the experiment itself.” – As discussed above, Pyzer-Knapp in view of Lindon teaches determining that the pre-determined number of experimental data points exhibits at least one characteristic. Kharitonov teaches a plurality of different stopping rules and further teaches that the applicable sequential testing procedure depends on characteristics of the experimental data, including its distribution, with different tests being tailored to different experimental data distributions. Under BRI, these different data distributions correspond to different data characteristics and that respective sequential tests correspond to different stopping rules. Thus, in combination with the detected data characteristic discussed above, Kharitonov supports selecting the stopping rule corresponding to exhibited data characteristic from a plurality of stopping rules.);
Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having Pyzer-Knapp, Lindon, and Kharitonov before them, to apply Lindon’s conditioned-based evaluation and Kharitonov’s different sequential stopping procedures within the stopping framework of Pyzer-Knapp. One would have been motivated to make such a combination to apply a stopping procedure appropriate to the characteristics of the collected data and terminate the experiment when sufficient data has been obtained, thereby avoiding unnecessary continued data collection and the associated expenditure of time and computational resources.
Regarding claim 11, Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches all the elements of claim 10, therefore is rejected for the same reasons as those presented for claim 10. The claim recites similar limitations corresponding to claim 2 and is rejected for similar reasons as claim 2 using similar teachings and rationale.
Regarding claim 12, Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches all the elements of claim 11, therefore is rejected for the same reasons as those presented for claim 11. The claim recites similar limitations corresponding to claim 3 and is rejected for similar reasons as claim 3 using similar teachings and rationale.
Regarding claim 13, Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches all the elements of claim 10, therefore is rejected for the same reasons as those presented for claim 10. The claim recites similar limitations corresponding to claim 4 and is rejected for similar reasons as claim 4 using similar teachings and rationale.
Regarding claim 19, Pyzer-Knapp teaches the following limitations:
A non-transitory computer-readable medium storing programming for execution by one or more processors, the programming comprising instructions to (Pyzer-Knapp, paragraph [0083] “The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory),” [0049] “ One or more processors 502 may execute computer instructions stored in memory 504 or received from another computer device or medium.”):
obtain, by a processing device, a pre-determined number of data points from an online experiment (Pyzer-Knapp, paragraph [0017] “. A corresponding output to the new sample is obtained. This corresponding output can be an actual or true observation obtained based on using the new sample. For instance, the actual or true observation can be obtained from running a real experimental or trial manufacturing production process… Batch Bayesian optimization (BBO) finds optimal samples in batches.” [0025] “the Bayesian optimization system 104 can be automatically programmed or triggered to start with an initial set of criteria (e.g., budget, batch size, aggression of early termination)” [0026] “A decision module 112 of the Bayesian optimization system 104 generates a batch of configurations and communicates the configurations to a data acquisition source 106…The batch of configurations includes different parameter values the data acquisition source 106 can try or experiment with… to perform a process or a run (e.g., an experimental process or trial run) to produce an actual output” [0028] “The data acquisition source 106 generates a result of the configurations (e.g., as a result of performing the real process)… The result includes one or more properties of an outcome obtained from the actual run of a process…The storage module 114 of the Bayesian optimization system 104 stores the results of the configurations” - Pyzer-Knapp teaches obtaining samples in batches, wherein each sample has a corresponding actual or true observation obtained from an experimental or trial process and further teaches establishing a batch size before performing the process. Thus, the predetermined batch size before performing the process. Thus, the predetermined batch size establishes the predetermined number of samples for which each corresponding experimental observations or results are obtained. Applicant describes data points as discrete units of data obtained during an experiment. Accordingly, under BRI, Pyzer-Knapp’s actual observations or result values corresponding to the predetermined batch of samples correspond to the claimed pre-determined number of data points. Applicant further describes an online experiment as any experiment in which data is collected in real time from any source. Pyzer-Knapp teaches performing experimental or trial runs in a real process and obtaining the corresponding actual observations or results, including trial experimental runs performed in real time. Thus, Pyzer-Knap’s experimental process falls within the broadest reasonable interpretation of the claimed online experiment.);
apply, by the processing device, the first stopping rule to determine whether a first convergence criterion is met; and stopping, when the first convergence criterion is met, the online experiment by stopping on-going data point collection (Pyzer-Knapp, paragraph [0038] “The stopping criteria evaluator 306 determines whether to stop searching for a next batch of configurations, for example, by applying a function to the acquisition function of the batch Bayesian optimization engine 304… For instance, if a stopping criterion is met, the stopping criteria evaluator 306 sends a signal indicating that the search is to be stopped. Otherwise, the stopping criteria evaluator 306 can send a signal or an indication that the searching is to be continued.” [0039] “If no termination or stop signal is received, the batch Bayesian optimization engine 304 sends the next batch of configurations to the data acquisition source 308 to use in its real or actual process. The data acquisition source 308 performs its process using this next batch of configurations, returning the result to the batch Bayesian optimization engine 304.” [0040] “Finding next batch of configurations at 304, evaluating whether stopping criteria is met at 306 and sending the next batch of configurations to the data acquisition source 308, wherein the data acquisition source 308 performs its test process, and storing the result received from the data acquisition source 308 is repeated until the stopping criteria evaluator 306 determines that the stopping criteria is met or the specified budget (e.g., number of iterations) is exhausted.” – Pyzer-Knapp teaches applying a stopping criterion to determine whether the criterion has been met and generating a stop signal when it is met. When no stop signal is received, the data acquisition source continues performing the experimental process and returning additional results; when the stopping criterion is met, the data acquisition source may be controlled to stop the experiment or trial process. Under the broadest reasonable interpretation, this corresponds to applying the first stopping rule to determine whether a convergence criterion is met and, when met, stopping the online experiment by stopping ongoing collection of experimental data points.).
However, Pyzer-Knapp does not teach but Pyzer-Knapp in view of Lindon teaches the following limitation:
process, by the processing device and while additional data points from the online experiment are being continuously obtained in real time. the pre-determined number of data points to determine whether the predetermined number of data points obtained from the online experiment exhibit at least one characteristic (Pyzer-Knapp, [0017] “A corresponding output to the new sample is obtained. This corresponding output can be an actual or true observation obtained based on using the new sample…This process of finding or searching for a next sample, obtaining actual observation using that next sample, and updating the surrogate model can be repeated…Batch Bayesian optimization (BBO) finds optimal samples in batches.” [0040] “Finding next batch of configurations at 304, evaluating whether stopping criteria is met at 306 and sending the next batch of configurations to the data acquisition source 308, wherein the data acquisition source 308 performs its test process, and storing the result received from the data acquisition source 308 is repeated until the stopping criteria evaluator 306 determines that the stopping criteria is met…“ [0041] “The method performs real-time intervention in a batch Bayesian optimization (BBO) system, automatically determining whether to terminate a BBO search early if a predetermined threshold is reached. In an aspect, running a data acquisition source, for example, in real-time, such as a manufacturing process and/or a robot can be costly. For instance, iteratively running a data acquisition source with new or next configurations may consume power, equipment and computer resources. The method in an embodiment intervenes in real-time to stop early iterative runs of a data acquisition source…” Lindon, paragraph [0040] “Processing logic at block 404 may, while the A/B test is executing, determine (e.g., by a processing device) that a sample ratio mismatch corresponding to the second plurality of users has occurred.” [0041] “To detect the sample ratio mismatch, processing logic may employ a variety of techniques. For example, in one embodiment, processing logic may detect a statistically significant departure from a ratio corresponding to the first plurality of users and the second plurality of users…” [0042] “the sample ratio mismatch is determined before the A/B test ends executing…In one embodiment, the sample ratio mismatch may be analyzed after each new data point is received.” – Pyzer-Knapp teaches obtaining experimental observations in batches, processing the observations to update the surrogate model, and repeatedly obtaining additional observations through iterative experimental runs. Pyzer-Knapp further teaches that the data acquisition source operates and is controlled in real time. Lindon teaches processing experimental data while the test is executing to determine whether the data exhibits a statistical characteristic, specifically a sample ratio mismatch, including analysis after each new data point is received. Under BRI, the combined teachings correspond to processing the pre-determined number of data points while additional data points continue to be obtained in real time and determining whether the pre-determined number of data points exhibits at least one characteristic.);
However, Pyzer-Knapp in view of Lindon does not teach, but Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches:
select, by the processing device and in response to the pre-determined number of data points exhibiting the at least one characteristic, a first stopping rule of a plurality of stopping rules, wherein: the first stopping rule corresponds to the at least one characteristic, and each stopping rule of the plurality of stopping rules corresponds to a different data characteristic of a plurality of data characteristics comprising the at least one characteristic (Kharitonov, [page 474, Introduction] “We propose several sequential testing methods that reflect the distributions of the data generated in A/B and interleaving experiments, and describe how to adjust their stopping thresholds based on query log data” [page 475, section 2.2] “Overall, sequential testing is a highly developed discipline, and a variety of tests that differ by their properties and assumptions was proposed…Due to their properties, we select the O'Brien&Fleming and MaxSPRT tests as a foundation for our study.” [page 476, section 4] “In this section, we introduce the sequential testing procedures we consider in this work…O'Brien&Fleming's sequential test, modified for interleaving (OBF-I), and the MaxSPRT test. After that, we describe two tests applicable for A/B tests: the standard O'Brien&Fleming test, and our proposed MaxSPRT-AB test, tailored for A/B test experiments. For all of these tests, we also describe algorithms to train their stopping thresholds.” [page 477] “MaxSPRT-AB The MaxSPRT rule cannot be easily applied for the case of A/B experiments, since the distribution of the considered metric is unknown both under H0 and H1. To address this, we propose to estimate both the distribution of the metric under the null hypothesis and under the alternative hypothesis using data from the experiment itself.” – As discussed above, Pyzer-Knapp in view of Lindon teaches determining that the pre-determined number of experimental data points exhibits at least one characteristic. Kharitonov teaches a plurality of different stopping rules and further teaches that the applicable sequential testing procedure depends on characteristics of the experimental data, including its distribution, with different tests being tailored to different experimental data distributions. Under BRI, these different data distributions correspond to different data characteristics and that respective sequential tests correspond to different stopping rules. Thus, in combination with the detected data characteristic discussed above, Kharitonov supports selecting the stopping rule corresponding to exhibited data characteristic from a plurality of stopping rules.);
Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having Pyzer-Knapp, Lindon, and Kharitonov before them, to apply Lindon’s conditioned-based evaluation and Kharitonov’s different sequential stopping procedures within the stopping framework of Pyzer-Knapp. One would have been motivated to make such a combination to apply a stopping procedure appropriate to the characteristics of the collected data and terminate the experiment when sufficient data has been obtained, thereby avoiding unnecessary continued data collection and the associated expenditure of time and computational resources.
Regarding claim 20, Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches all the elements of claim 19, therefore is rejected for the same reasons as those presented for claim 19. The claim recites similar limitations corresponding to claim 2 and is rejected for similar reasons as claim 2 using similar teachings and rationale.
Claims 5 and 14 are rejected under the 35 U.S.C. 103 as being unpatentable over Pyzer-Knapp et al., (Pub. No.: US 20220128972 A1 (Filed: 2020)) in view of Lindon (Pub. No.: US 20220129765 A1 (Filed: 2020)) further in view of Kharitonov et al., (NPL: “Sequential Testing for Early Stopping of Online Experiments” (Published: 2015)) further in view of Nakano et al., (Pub. No.: US 20200097522 A1 (Filed: 2017)).
Regarding claim 5, Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1. However, Pyzer-Knapp in view of Lindon further in view of Kharitonov does not teach but Pyzer-Knapp in view of Lindon further in view of Kharitonov further in view of Nakano teaches the following limitation:
wherein the at least one characteristic comprises the pre-determined number of data points being constant, and the first convergence criterion comprises determining whether a particular number of the data points are above a threshold (Nakano, paragraph [0068] “Furthermore, the “convergence of parameter estimation” as used in the present example embodiment has the same meaning as “convergence” as used in common optimization calculations and numerical calculations, and refers to a state for which a determination is made that the values of the parameter and threshold would not change even if iterative calculations were repeated any further.” [0080] “Specifically, the threshold setting unit 14 first determines whether each data point is an inlier, based on the weight w.sub.i. Since the weights take a value between zero and one, the threshold setting unit 14 determines data points having weights w.sub.i no smaller than 0.5 as inliers, for example“ [0083] “the convergence determination unit 15 may determine that convergence is reached if the threshold λ set in the latest iteration of step S4 is smaller than a predetermined minimum value.”– As discussed with respect to claim 1, Pyzer-Knapp teaches obtaining the pre-determined number of experimental data points. Nakano teaches evaluating convergence by determining when evaluated values no longer change through further iterations. Under BRI, applying this convergence determination to the pre-determined experimental data points of Pyzer-Knapp corresponds to determining that the data exhibit a constant characteristic, i.e., that the evaluated values have stabilized and no longer materially change. Nakano further teaches evaluating individual data points against a threshold condition by determining which data points have weights no smaller than 0.5, and also teaches using a predetermined threshold in determining convergence. Thus, under BRI, Nakano’s threshold-based evaluation of the data points corresponds to determining whether a particular number of data points are above a threshold.).
Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having Pyzer-Knapp, Lindon, Kharitonov and Nakano before them, to incorporate Nakano’s convergence-based determination, indicating when evaluated values no longer change, into the stopping framework of Pyzer-Knapp, as modified by Lindon and Kharitonov to evaluate characteristics of experimental data and apply an appropriate stopping rule. One would have been motivated to make such a combination in order to improve the determination of when to terminate an iterative process by identifying when the evaluated data has stabilized.
Regarding claim 14, Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches all the elements of claim 10, therefore is rejected for the same reasons as those presented for claim 10. The claim recites similar limitations corresponding to claim 5 and is rejected for similar reasons as claim 5 using similar teachings and rationale.
Claims 6 and 15 are rejected under the 35 U.S.C. 103 as being unpatentable over Pyzer-Knapp et al., (Pub. No.: US 20220128972 A1 (Filed: 2020)) in view of Lindon (Pub. No.: US 20220129765 A1 (Filed: 2020)) further in view of Kharitonov et al., (NPL: “Sequential Testing for Early Stopping of Online Experiments” (Published: 2015)) further in view of Maor et al., (Pub. No.: US 20190026351 A1 (Filed: 2016)).
Regarding claim 6, Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1. However, Pyzer-Knapp in view of Lindon does not teach but Pyzer-Knapp in view of Lindon further in view of Kharitonov further in view of Maor teaches the following limitation:
wherein the at least one characteristic comprises the pre-determined number of data points being monotonic, and the first convergence criterion comprises determining that the pre-determined number of data points are within a threshold of a limit (Moar, paragraph [0008] “ a particular parameter (e.g., blood count) that is steadily increasing or decreasing over time, or whose portion relative to another parameter is steadily increasing or decreasing, until the increase or decrease exceeds a certain threshold.” [0012] “Thus, the plurality of data points obtained and analyzed by trend detection engine 112 may include any type of a time series, i.e., any sequence of data points measured at different times. Moreover, in some examples, trend detection engine 112 may obtain a plurality of time series… determine whether the time series includes a stable trend” Pyzer-Knapp, paragraph [0015] “to automatically determine or decide whether to terminate a BBO search early if a predetermined threshold is reached, for example, if a particular criterion is met.” [0016] “A method, for example, can include determining whether the threshold is met by applying a function to a batch Bayesian optimization (BBO) acquisition score” – As discussed with respect to claim 1, Pyzer-Knapp teaches obtaining the pre-determined number of experimental data points. Maor teaches analyzing a time series comprising data points measured at different times and identifying a trend in which the values steadily increase or steadily decrease over time. Under BRI, applying Maor’s trend analysis to the pre-determined experimental data points of Pyzer-Knapp corresponds to the pre-determined number of data points being monotonic. Maor further teaches monitoring the increasing or decreasing values relative to a defined threshold, while Pyzer-Knapp teaches determining whether a predetermined threshold has been reached as a criterion for terminating the iterative process. Under BRI, the defined threshold establishes a boundary or limit for the monotonic data, and evaluating the monotonic data relative to that threshold corresponds to determining that the pre-determined number of data points are within a threshold of a limit.).
Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having a combination of Pyzer-Knapp, Lindon, Kharitonov and Maor before them, to incorporate the trend-based evaluation of Maor, which identifies parameters that are steadily increasing or decreasing over time, into the stopping rule framework of Pyzer-Knapp, as modified by Lindon and Kharitonov to evaluate characteristics of experimental data and apply an appropriate stopping rule. One would have been motivated to make such a combination in order to improve the determination of when to terminate an iterative process by considering directional behavior of the evaluated data, such as whether values are consistently increasing or decreasing, thereby providing a more reliable basis for applying the stopping rule.
Regarding claim 15, Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches all the elements of claim 10, therefore is rejected for the same reasons as those presented for claim 10. The claim recites similar limitations corresponding to claim 6 and is rejected for similar reasons as claim 6 using similar teachings and rationale.
Claims 7 and 16 are rejected under the 35 U.S.C. 103 as being unpatentable over Pyzer-Knapp et al., (Pub. No.: US 20220128972 A1 (Filed: 2020)) in view of Lindon (Pub. No.: US 20220129765 A1 (Filed: 2020)) further in view of Kharitonov et al., (NPL: “Sequential Testing for Early Stopping of Online Experiments” (Published: 2015)) further in view of Benoit et al., (Pub. No.: US 20230060325 A1 (Filed: 2021)).
Regarding claim 7, Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1. However, Pyzer-Knapp in view of Lindon further in view of Kharitonov does not teach but Pyzer-Knapp in view of Lindon further in view of Benoit teaches the following limitation:
wherein: the at least one characteristic comprises the pre-determined number of data points having a normal distribution, and the first convergence criterion comprises determining that a mean of the pre-determined number of data points is within a confidence interval of a specified maximal width (Benoit, paragraph [0015] “For each matrix element, DCL not only estimates its true value but also the uncertainty surrounding that estimate as a confidence interval. As data accumulates over time, the confidence intervals become narrower corresponding to an increase in the precision of the estimate of the Jacobian matrix.” [0023] “A drift in the mean value and/or width of the confidence intervals can indicate that the underlying physical cause and effect relationships are changing.” [0033] “The learning value may be computed through…experimental power analyses computing the reduction in uncertainty and narrowing of confidence intervals based on increasing to the current sample size.” [0035] “Within each cluster, effect measures are free of bias and/or confounding effects from external factors and follow normal distributions from which estimates of causal effects -not just associations- can be derived.” Pyzer-Knapp, paragraph [0015] “Methods and systems can be provided…to automatically determine or decide whether to terminate a BBO search early if a predetermined threshold is reached, for example, if a particular criterion is met.” – As discussed with respect to claim 1, Pyzer-Knapp teaches obtaining the pre-determined number of experimental data points. Benoit teaches experimental effect measures that follow a normal distribution. Under BRI, applying Benoit’s distributional analysis to the pre-determined experimental data points of Pyzer-Knapp corresponds to the pre-determined number of data points having normal distribution. Benoit further teaches estimating values using confidence intervals, evaluating the mean value and width of those confidence intervals, and that the intervals narrow as the amount of data increases. Under BRI, evaluating whether the confidence interval associated with the mean has narrowed to a specified width corresponds to determining that a mean of the pre-determined number of data points is within a confidence interval of a specified maximum width. Pyzer-Knapp further teaches determining whether a predetermined threshold has been reached as a criterion for terminating the iterative process.).
Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having a combination of Pyzer-Knapp, Lindon, Kharitonov and Benoit before them, to incorporate the confidence-interval-based evaluation of Benoit, which evaluates an estimated value and the width of its associated confidence interval, into the stopping framework of Pyzer-Knapp, as modified by Lindon and Kharitonov to evaluate characteristics of experimental data and apply and appropriate stopping rule. One would have been motivated to make such a combination in order to improve the determination of when to terminate an iterative process by assessing whether the estimate has stabilized within a sufficiently narrow confidence interval, thereby providing a more reliable and precise basis for applying the stopping rule.
Regarding claim 16, Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches all the elements of claim 10, therefore is rejected for the same reasons as those presented for claim 10. The claim recites similar limitations corresponding to claim 7 and is rejected for similar reasons as claim 7 using similar teachings and rationale.
Claims 8, 9, 17, and 18 are rejected under the 35 U.S.C. 103 as being unpatentable over Pyzer-Knapp et al., (Pub. No.: US 20220128972 A1 (Filed: 2020)) in view of Lindon (Pub. No.: US 20220129765 A1 (Filed: 2020)) further in view of Kharitonov et al., (NPL: “Sequential Testing for Early Stopping of Online Experiments” (Published: 2015)) further in view of Zadeh et al., (Pub. No.: US 20220121884 A1 (Filed: 2021)).
Regarding claim 8, Pyzer-Knapp in view of Lindon further in view Kharitonov teaches all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1. However, Pyzer-Knapp in view of Lindon further in view of Kharitonov does not teach but Pyzer-Knapp in view of Lindon further in view of Kharitonov further in view of Zadeh teaches the following limitation:
wherein: the at least one characteristic comprises the pre-determined number of data points being modal, and the first convergence criterion comprises applying a regression model or a clustering model (Pyzer-Knapp, [0015] “Methods and systems can be provided in one or more embodiments for real-time intervention in a batch Bayesian optimization (BBO) system, for example, running a real physical system, to automatically determine or decide whether to terminate a BBO search early if a predetermined threshold is reached, for example, if a particular criterion is met.” [0016] “A method, for example, can include determining whether the threshold is met by applying a function to a batch Bayesian optimization (BBO) acquisition score which determines the likelihood that evaluating the next batch will provide significant value.” Zadeh, paragraph [2240] “In one embodiment, we apply the multi-level thresholding. In one embodiment, we apply a model fitting method. In one embodiment, we apply the above to segment the document images, face, text, or the like. In one embodiment, we use the grey level histogram for thresholding and segmentation purpose. The histogram (and its peaks or its transition phases) is a good indicator of the multiple classes or clusters involved in the samples.” [2253] “In one embodiment, we use Vapnik's support vector machines (SVM) to classify the data or recognize the object. In one embodiment, in addition, we use kernels (e.g. using Gaussian processes or models) to be able to handle any shape of data distribution with respect to feature space, to transfer the space in such a way that the separation of classes or clusters becomes easier. In one embodiment, we use sparse kernel machines, maximum margin classifiers, multiclass SVMs, logistic regression method, multivariate linear regression, or relevance vector machines (RVM) (which is a variation of SVM with less limitations), for classification or recognition.” – As discussed with respect to claim 1, Pyzer-Knapp teaches obtaining the pre-determined number of experimental data points. Zadeh teaches analyzing samples using a histogram and identifies peaks in the histogram as indicators of classes or clusters within the samples. A histogram represents the distribution of the evaluated values, and a peak represents a value or range having a relatively high frequency of occurrence. Under BRI, such histogram peaks corresponds to modes of the data distribution. Thus, applying Zadeh’s histogram analysis to the pre-determined experimental data points of Pyzer-Knapp corresponds to the pre-determined number of data points being modal. Zadeh further teaches applying a model-fitting method and specifically identifies a logistic regression and multivariate linear regression for analyzing the data. Thus, Zadeh teaches applying a regression model. Pyzer-Knapp further teaches applying an evaluation criterion within its stopping framework to determine whether the iterative experiment should terminate. Under BRI, incorporating Zadeh’s regression-model evaluation into the stopping framework corresponds to the first convergence criterion comprising applying a regression model.).
Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having a combination of Pyzer-Knapp, Lindon, Kharitonov and Zadeh before them, to incorporate Zadeh’s regression-based analysis, including regression techniques and analysis of data distributions having peaks corresponding to modes, into the stopping framework of Pyzer-Knapp, as modified by Lindon and Kharitonov to evaluate characteristics of experimental data and applying an appropriate stopping rule. One would have been motivated to make such a combination in order to improve the determination of when to terminate an iterative process by evaluating the structure and distribution of data, including the presence of frequently occurring values or groupings, thereby providing a more reliable basis for applying the stopping rule.
Regarding claim 9, Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1. However, Pyzer-Knapp in view of Lindon further in view of Kharitonov does not teach but Pyzer-Knapp in view of Lindon further in view of Kharitonov further in view of Zadeh teaches the following limitation:
wherein the at least one characteristic comprises the pre-determined number of data points being autocorrelated (Zadeh, paragraph [2407] “In one embodiment, we use the autocorrelation matrix. In one embodiment, normalized aligned meshes or grids with the fuzzy parameters for coordinates of reference points.” – Pyzer-Knapp teaches obtaining the pre-determined number of experimental data points. Zadeh teaches evaluating data using an autocorrelation matrix. Autocorrelation measures the correlation of data values with other values in the same data sequence at different positions or lags. Accordingly, under BRI, applying Zadeh’s autocorrelation analysis to the pre-determined experimental data points of Pyzer-Knapp corresponds to determining that the pre-determined number of data points are autocorrelated.).
Regarding claim 17, Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches all the elements of claim 10, therefore is rejected for the same reasons as those presented for claim 10. The claim recites similar limitations corresponding to claim 8 and is rejected for similar reasons as claim 8 using similar teachings and rationale.
Regarding claim 18, Pyzer-Knapp in view of Lindon further in view of Kharitonov teaches all the elements of claim 10, therefore is rejected for the same reasons as those presented for claim 10. The claim recites similar limitations corresponding to claim 8 and is rejected for similar reasons as claim 9 using similar teachings and rationale.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Daravanh Phakousonh whose telephone number is (571)272-6324. The examiner can normally be reached Mon - Thurs 7 AM - 5 PM, Every other Friday 7 AM - 4PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at 571-272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Daravanh Phakousonh/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights