DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Introduction
The following is a final Office action in response to Applicant’s submission filed on 10/3/2025. Currently claims 1-20 are pending and claims 1, 11, and 20 are independent. Claims 1, 11, and 20 have been amended from the original claim set dated 8/31/2022. No claims have been added or cancelled.
Response to Amendments
Applicant’s amendments are acknowledged and necessitated the new grounds of rejection in this Office Action. In light of Applicant’s amendments, the 35 U.S.C. 101 rejections are withdrawn. Specifically, the inclusion of the limitation “responsive to determining that the measurement of the effect exceeds the threshold, automatically initiating an experiment by applying the treatment to a live version of the online system” demonstrates a controlling feature and overcomes the 101 rejection within the Step 2A (Prong 1) analysis.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-6, 11-16, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Cui et al. (US 11468348 B1) in view of Powers et al. (US 11379859 B1) further in view of Lang (US 20180053137 A1)
Regarding claims 1, 11, and 20 (Amended), Cui discloses a method comprising: accessing a machine learning model that is trained to predict behaviors of a plurality of users of an online system (Cui ABS - Methods and apparatus for identifying features that may have a high potential impact on key application metrics. These methods rely on observational data to estimate the importance of application features, and use causal inference tools such as Double Machine Learning (double ML) or Recurrent Neural Networks (RNN) to estimate the impacts of treatment features on key metrics), wherein the machine learning model is trained by: receiving historical data associated with the plurality of users of the online system, wherein the historical data is associated with a demand side and a supply side associated with the online system, and training the machine learning model based at least in part on the historical data associated with the plurality of users (Cui COL 8 ROW 6 - A first step in this method is to train an RNN model. For every user, a history of control and treatment features is gathered over a given period of time, and then the RNN model is trained using the control and treatment features to make predictions at each step); identifying a treatment for achieving a goal of the online system (Cui COL 3 ROW 62 - Treatment features 104 include one or more features of the mobile application that are of interest in regard to the target metric 106, and may be specified by the double ML development team); simulating an application of the treatment on the demand side and the supply side associated with the online system based at least in part on the historical data and a set of behaviors predicted for the plurality of users (Cui COL 8 ROW 24 - FIG. 5 is a high-level flowchart of using a sequence-to-sequence recurrent neural network (RNN) model to determine the causal impact of a treatment feature on a target metric, according to some embodiments. As indicated at 500, an RNN model is trained with feature data. As indicated at 510, a prediction is performed using all features. As indicated at 520, a prediction is performed, zeroing out the feature corresponding to the treatment. As indicated at 530, the difference between the outputs of the two predictions gives the causal impact of the treatment feature), wherein simulating the application of the treatment comprises: replaying the historical data in association with the application of the treatment, and applying the machine learning model to predict the set of behaviors for the plurality of users while replaying the historical data in association with the application of the treatment; and measuring an effect of the application of the treatment on the demand side and the supply side associated with the online system based at least in part on simulating the application of the treatment on the demand side and the supply side associated with the online system, wherein the effect is associated with the goal of the online system users (Cui COL 8 ROW 24 - FIG. 5 is a high-level flowchart of using a sequence-to-sequence recurrent neural network (RNN) model to determine the causal impact of a treatment feature on a target metric, according to some embodiments. As indicated at 500, an RNN model is trained with feature data. As indicated at 510, a prediction is performed using all features. As indicated at 520, a prediction is performed, zeroing out the feature corresponding to the treatment. As indicated at 530, the difference between the outputs of the two predictions gives the causal impact of the treatment feature).
Cui lacks behaviors of both a supply side and a demand side wherein the demand side comprised of historical order activity, and the supply side comprised of historical picker activity to fulfill orders and determining whether the measurement of the effect exceeds a threshold; and responsive to determining that the measurement of the effect exceeds the threshold, automatically initiating an experiment by applying the treatment to a live version of the online system.
Powers, from the same field of endeavor, teaches behaviors of both a supply side and a demand side wherein the demand side comprised of historical order activity, and the supply side comprised of historical picker activity to fulfill orders (Powers COL 12 ROW 33 - As used herein, the term “transaction data” refers to electronic information indicating that a transaction is occurring or has occurred via either a merchant or the promotion and marketing service. Transaction data may also include information relating to the transaction. For example, transaction data may include consumer payment or billing information, consumer shipping information, items purchased by the consumer, a merchant rewards account number associated with the consumer, the type of shipping selected by the consumer for fulfillment of the transaction, or the like) and determining whether the measurement of the effect exceeds a threshold; and responsive to determining that the measurement of the effect exceeds the threshold, automatically initiating an experiment by applying the treatment to a live version of the online system (Powers COL 41 ROW 7 - In yet further embodiments, the merchant intelligence platform may perform a complete analysis including estimation of the profitability of making a permanent change in test parameters, and automatically implement the test parameters that are predicted to provide the merchant with an optimal long-term benefit. The results of the market analysis test may be determined by means for determining the results of the market analysis, such as the merchant intelligence management circuitry 214 acting in concert with the analytics circuitry 212 as described above with respect to FIG. 2.)
It would be obvious for one of ordinary skill in the art before the effective filing date of the Applicant’s claimed invention to modify the causal analysis methodology/system of Cui by including the analytic techniques of Powers because Powers discloses “embodiments may provide for analysis of market conditions and generation of a demand model to assist merchants with evaluating the demand for particular products or services offered by the merchant. For example, embodiments may advise merchants as to whether offering a particular product or service will be likely to result in increased profits (Powers COL 7 ROW 40)”. Additionally, Cui further details that “Developers of web-based applications typically conduct randomized experiments (A/B tests) either online or offline to test the effectiveness of certain features and to optimize customer experiences or other metrics (Cui COL 2 ROW 42)” so it would be obvious to consider including the additional analytic techniques that Powers discloses because it would increase merchant profits.
Cui further lacks the treatment being an adjustment of a parameter that changes a delivery window, an impact of the treatment influencing activity of the plurality of users of both the supply side and the demand side.
Lang, from the same field of endeavor, teaches the treatment being an adjustment of a parameter that changes a delivery window, an impact of the treatment influencing activity of the plurality of users of both the supply side and the demand side (Lang ¶48 - Using the data from the simulated scenarios as a starting point, the service provider can configure multiple windows to set fees that may offset the cost of allowing the customer to refine their scheduled delivery window for a service offering. This fee structure may become part of an interface as may be seen in FIG. 3 that is published to the customer to select from at the time of scheduling the service).
It would be obvious for one of ordinary skill in the art before the effective filing date of the Applicant’s claimed invention to modify the causal analysis methodology/system of Cui by including the scheduling techniques of Lang because Lang discloses “The system and method may accumulate data to allow service providers to better understand the costs of allowing customers to book times that are convenient for themselves (Lang ¶4)”. Additionally, Cui further details that “Developers of web-based applications typically conduct randomized experiments (A/B tests) either online or offline to test the effectiveness of certain features and to optimize customer experiences or other metrics (Cui COL 2 ROW 42)” so it would be obvious to consider including the additional scheduling techniques that Lang discloses because it would improve customer experiences by enabling the delivery of goods/services to be more convenient.
Regarding claims 2 and 12, Cui in view of Powers further in view of Lang discloses determining that the effect of the application of the treatment on the demand side and the supply side associated with the online system is at least a threshold effect (Cui COL 5 ROW 26 - In some embodiments, the online tests may be used to determine one or more of the treatment features that have an impact on the target metric that is above (or, alternatively, below) a specified threshold); responsive to determining that the effect of the application of the treatment on the demand side and the supply side associated with the online system is at least the threshold effect, simulating an absence of the application of the treatment on the demand side and the supply side associated with the online system based at least in part on the historical data and an additional set of behaviors predicted for the plurality of users ((Cui COL 8 ROW 17 - To assess the effects of a specific treatment, two sets of predictions are made, once with all features unaltered and once after ‘zeroing-out’ the feature corresponding to this specific treatment. The difference in outputs (properly normalized) gives the conditional treatment effects - Cui COL 6 ROW 57 - Table 2 shows the global lift of the features, i.e. the % decrease in the target metric (e.g., mobile application visit days in the next month), if the feature is not used at all), wherein simulating the absence of the application of the treatment comprises: replaying the historical data, and applying the machine learning model to predict the set of additional behaviors for the plurality of users while replaying the historical data; measuring an additional effect of the absence of the application of the treatment on the demand side and the supply side associated with the online system based at least in part on simulating the absence of the application of the treatment on the demand side and the supply side associated with the online system, wherein the additional effect is associated with the goal of the online system; and determining a difference between the effect and the additional effect (Cui COL 8 ROW 24 - FIG. 5 is a high-level flowchart of using a sequence-to-sequence recurrent neural network (RNN) model to determine the causal impact of a treatment feature on a target metric, according to some embodiments. As indicated at 500, an RNN model is trained with feature data. As indicated at 510, a prediction is performed using all features. As indicated at 520, a prediction is performed, zeroing out the feature corresponding to the treatment. As indicated at 530, the difference between the outputs of the two predictions gives the causal impact of the treatment feature).
Regarding claims 3 and 13, Cui in view of Powers further in view of Lang discloses determining that the difference between the effect and the additional effect is at least a threshold difference (Cui COL 8 ROW 17 - To assess the effects of a specific treatment, two sets of predictions are made, once with all features unaltered and once after ‘zeroing-out’ the feature corresponding to this specific treatment. The difference in outputs (properly normalized) gives the conditional treatment effects); and responsive to determining that the difference between the effect and the additional effect is at least the threshold difference, applying the treatment to a set of users of the online system (Cui COL 5 ROW 29 - As indicated at 350, the application may be modified based on results of the tests).
Regarding claims 4 and 14, Cui in view of Powers further in view of Lang discloses determining, based at least in part on the difference between the effect and the additional effect, one or more of: a policy, a heuristic, and a constraint (Cui COL 8 ROW 46 - Treatments are assigned to subjects according to an underlying policy that depends on the subjects' features).
Regarding claims 5 and 15, Cui in view of Powers further in view of Lang discloses performing a t-test based at least in part on the effect and the additional effect (Cui COL 6 ROW 38 - Table 1 shows an example list of prioritized treatment features (features 1-10) for several user segments (states 0-4) output by a double ML model for unit increase in usage of these features with regard to a target metric (e.g., mobile application visit days in the next month)… Table 2 shows the global lift of the features, i.e. the % decrease in the target metric (e.g., mobile application visit days in the next month), if the feature is not used at all).
Regarding claims 6 and 16, Cui in view of Powers further in view of Lang discloses determining that the effect of the application of the treatment on the demand side and the supply side associated with the online system is at least a threshold effect (Cui COL 5 ROW 22 - The tests may, for example, be used to determine one or more of the treatment features that have a higher impact on the target metric than the other treatment features. In some embodiments, the online tests may be used to determine one or more of the treatment features that have an impact on the target metric that is above (or, alternatively, below) a specified threshold); and responsive to determining that the effect of the application of the treatment on the demand side and the supply side associated with the online system is at least the threshold effect, applying the treatment to a set of users of the online system (Cui COL 5 ROW 29 - As indicated at 350, the application may be modified based on results of the tests).
Claims 7-10, 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Cui et al. (US 11468348 B1) in view of Powers et al. (US 11379859 B1) further in view of Lang (US 20180053137 A1) further in view of Tendrick (US 20170249557 A1)
Regarding claims 7 and 17, Cui in view of Powers further in view of Lang discloses a method comprising: accessing a machine learning model that is trained to predict behaviors of a plurality of users of an online system (Cui ABS - Methods and apparatus for identifying features that may have a high potential impact on key application metrics. These methods rely on observational data to estimate the importance of application features, and use causal inference tools such as Double Machine Learning (double ML) or Recurrent Neural Networks (RNN) to estimate the impacts of treatment features on key metrics).
Cui in view of Powers further in view of Lang lacks the treatment affects one or more of: a size of a delivery window, an estimated delivery time, a delivery cost, a pay rate, and a probability of batching a plurality of orders.
Tendrick, from the same field of endeavor, teaches the treatment affects one or more of: a size of a delivery window, an estimated delivery time, a delivery cost, a pay rate, and a probability of batching a plurality of orders (Tendrick ¶54 - For example, a given user may always receive the same treatment each visit where a random decision is made as to which users will be offered free shipping every time the user visits the website. In another example, a user may receive the same treatment throughout a given session where a random decision is made as to which session will be offered free shipping for the duration of the session. Accordingly, a user who was offered free shipping may not receive the same offer at a subsequent visit. In a further example, a user may receive different treatment for every interaction such as each product web page being viewed may or may not include free shipping. The collection of treatments and the scheme for assigning to users, sessions, and/or interactions may be embodied as an experimental design).
It would be obvious for one of ordinary skill in the art before the effective filing date of the Applicant’s claimed invention to modify the causal analysis methodology/system of Cui by including the analytic techniques of Tendrick because Tendrick discloses “The predictive analytics may improve an experience for the customer as well as increase revenues and reduce costs for the provider (Tendrick ¶2)”. Additionally, Cui further details that “Developers of web-based applications typically conduct randomized experiments (A/B tests) either online or offline to test the effectiveness of certain features and to optimize customer experiences or other metrics (Cui COL 2 ROW 42)” so it would be obvious to consider including the additional analytic techniques that Tendrick discloses because it would improve customer experiences.
Regarding claims 8 and 18, Cui in view of Powers further in view of Lang discloses a method comprising: accessing a machine learning model that is trained to predict behaviors of a plurality of users of an online system (Cui ABS - Methods and apparatus for identifying features that may have a high potential impact on key application metrics. These methods rely on observational data to estimate the importance of application features, and use causal inference tools such as Double Machine Learning (double ML) or Recurrent Neural Networks (RNN) to estimate the impacts of treatment features on key metrics).
Cui in view of Powers further in view of Lang lacks the machine learning model predicts a likelihood that a user of the online system will perform an action selected from the group consisting of: placing an order and accepting a batch of orders for fulfillment.
Tendrick, from the same field of endeavor, teaches the machine learning model predicts (Tendrick ¶34 - As predictive models or machine learning algorithms (referred to herein collectively as “predictive models”) are used in predictive analytics, scoring engines may be utilized that represent server applications evaluating the predictive models on new observations (i.e., available information)) a likelihood that a user of the online system will perform an action selected from the group consisting of: placing an order and accepting a batch of orders for fulfillment (Tendrick ¶12 - According to the exemplary embodiments, the website may utilize predictive analytics to determine whether the offer of free delivery is necessary to increase a likelihood that the customer will purchase the product or whether the delivery cost may remain with the customer, thereby maximizing the customer experience and provider maximized revenues for each purchase session).
It would be obvious for one of ordinary skill in the art before the effective filing date of the Applicant’s claimed invention to modify the causal analysis methodology/system of Cui by including the analytic techniques of Tendrick because Tendrick discloses “The predictive analytics may improve an experience for the customer as well as increase revenues and reduce costs for the provider (Tendrick ¶2)”. Additionally, Cui further details that “Developers of web-based applications typically conduct randomized experiments (A/B tests) either online or offline to test the effectiveness of certain features and to optimize customer experiences or other metrics (Cui COL 2 ROW 42)” so it would be obvious to consider including the additional analytic techniques that Tendrick discloses because it would improve customer experiences.
Regarding claims 9 and 19, Cui in view of Powers further in view of Lang in view of Tendrick discloses a method comprising: accessing a machine learning model that is trained to predict behaviors of a plurality of users of an online system (Cui ABS - Methods and apparatus for identifying features that may have a high potential impact on key application metrics. These methods rely on observational data to estimate the importance of application features, and use causal inference tools such as Double Machine Learning (double ML) or Recurrent Neural Networks (RNN) to estimate the impacts of treatment features on key metrics).
Tendrick further teaches an input to the machine learning model comprises one or more of: a size of a delivery window, an estimated delivery time, a delivery cost, and a pay rate (Tendrick ¶63 - With the predictive model being generated and prepared for use by the scoring device 155, the broker device 150 may receive a subsequent request from the interactive application at a decision point as to whether free shipping should be offered. The predictive model associated with the analysis graph 300 may correspond to this decision point).
It would be obvious for one of ordinary skill in the art before the effective filing date of the Applicant’s claimed invention to modify the causal analysis methodology/system of Cui by including the analytic techniques of Tendrick because Tendrick discloses “The predictive analytics may improve an experience for the customer as well as increase revenues and reduce costs for the provider (Tendrick ¶2)”. Additionally, Cui further details that “Developers of web-based applications typically conduct randomized experiments (A/B tests) either online or offline to test the effectiveness of certain features and to optimize customer experiences or other metrics (Cui COL 2 ROW 42)” so it would be obvious to consider including the additional analytic techniques that Tendrick discloses because it would improve customer experiences.
Regarding claim 10, Cui in view of Powers further in view of Lang discloses a method comprising: accessing a machine learning model that is trained to predict behaviors of a plurality of users of an online system (Cui ABS - Methods and apparatus for identifying features that may have a high potential impact on key application metrics. These methods rely on observational data to estimate the importance of application features, and use causal inference tools such as Double Machine Learning (double ML) or Recurrent Neural Networks (RNN) to estimate the impacts of treatment features on key metrics).
Cui in view of Powers further in view of Lang lacks simulating the application of the treatment on the demand side and the supply side associated with the online system is further based at least in part on one or more of a policy and a constraint associated with one or more selected from the group consisting of: a set of regulations, a size of a delivery window, an estimated delivery time, a delivery cost, a pay rate, a conversion rate, a probability of batching a plurality of orders, a probability of acceptance of one or more orders for fulfillment by a user of the online system, a probability that a delivery of an order is late, and a retention rate of users of the online system.
Tendrick, from the same field of endeavor, teaches simulating the application of the treatment on the demand side and the supply side associated with the online system is further based at least in part on one or more of a policy and a constraint (Tendrick ¶34 - Decisioning engines may also be utilized in predictive analytics that are specifically intended to render business decisions using, for example, business rules) associated with one or more selected from the group consisting of: a set of regulations, a size of a delivery window, an estimated delivery time, a delivery cost, a pay rate, a conversion rate, a probability of batching a plurality of orders, a probability of acceptance of one or more orders for fulfillment by a user of the online system, a probability that a delivery of an order is late, and a retention rate of users of the online system (Tendrick ¶54 - For example, a given user may always receive the same treatment each visit where a random decision is made as to which users will be offered free shipping every time the user visits the website. In another example, a user may receive the same treatment throughout a given session where a random decision is made as to which session will be offered free shipping for the duration of the session. Accordingly, a user who was offered free shipping may not receive the same offer at a subsequent visit. In a further example, a user may receive different treatment for every interaction such as each product web page being viewed may or may not include free shipping. The collection of treatments and the scheme for assigning to users, sessions, and/or interactions may be embodied as an experimental design).
It would be obvious for one of ordinary skill in the art before the effective filing date of the Applicant’s claimed invention to modify the causal analysis methodology/system of Cui by including the analytic techniques of Tendrick because Tendrick discloses “The predictive analytics may improve an experience for the customer as well as increase revenues and reduce costs for the provider (Tendrick ¶2)”. Additionally, Cui further details that “Developers of web-based applications typically conduct randomized experiments (A/B tests) either online or offline to test the effectiveness of certain features and to optimize customer experiences or other metrics (Cui COL 2 ROW 42)” so it would be obvious to consider including the additional analytic techniques that Tendrick discloses because it would improve customer experiences.
Response to Arguments
Applicant's arguments filed 10/3/2025 have been fully considered but they are not persuasive and/or are moot in light of the new rejections addressed above.
As addressed above, and in light of Applicant’s amendments, the 35 U.S.C. 101 rejections are withdrawn. Specifically, the inclusion of the limitation “responsive to determining that the measurement of the effect exceeds the threshold, automatically initiating an experiment by applying the treatment to a live version of the online system” demonstrates a controlling feature and overcomes the 101 rejection within the Step 2A (Prong 1) analysis.
Regarding the 35 USC § 102 and 35 USC § 103 rejections on the original Office Action, Applicant amended the independent claims to further limit the claims with respect to automatically implementing the treatment as well as adjusting delivery windows. In light of this amendment, Examiner agrees that the original references did not clearly teach this, however the amendment necessitated further search and consideration. As a result of this further search and consideration, prior art was found that does teach these limitations (Powers and Lang as discussed above). As such, Applicant’s arguments (with respect to the independent claims and their respective dependent claims) are unpersuasive.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael R Koester whose telephone number is (313)446-4837. The examiner can normally be reached Monday thru Friday 8:00AM-5:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jerry O'Connor can be reached at (571) 272-6787. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL R KOESTER/Examiner, Art Unit 3624
/Jerry O'Connor/Supervisory Patent Examiner,Group Art Unit 3624