DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This office action is in response to the RCE filed on 5/13/2026.
Claims 1, 5, 8, 9, and 14 have been amended. Claims 15-20 remain withdrawn.
Claim 2 has been canceled.
Claims 1, 3-11, 13, and 14 are pending and have been examined.
Claim Interpretation
Claim 1 recites “one or more computer storage media…” Applicant’s specification recites that the computer storage media does not include transitory signals (See spec. [0126] – “Computer storage media does not comprise a propagated data signal.”). As a result, there is no need to apply a rejection under 35 USC 101 for signal per se and the claimed storage media is considered to exclude transitory signals.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 3-11, 13,14 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claims 1, 3-8 are directed to a computer storage media (see interpretation above excluding transitory signals – spec [0126]). Claims 9-11, 13,14 are directed to a method. Thus, on their face, they fall within the four statutory categories of patentable subject matter.
Step 2A prong 1:
The following limitations, when considered individually and as an ordered combination, are merely descriptive of abstract concepts:
Claim 1:
obtaining a query;
identifying a subset of campaign attributes, from among a plurality of campaign attributes comprising a goal, a product or service offering, an account, a contact, a channel, a content, and/or a post-sales activity, to determine for a campaign based on analysis of the query and operational data associated with at least one entity and obtained from a data source, the operational data generated and/or collected by the at least one entity;
identifying an order in which to determine campaign attributes of the subset of campaign attributes based on analysis of the query and the operational data, wherein identifying the subset of campaign attributes and identifying the order in which to determine campaign attributes reduces iterative generation and evaluation of multiple campaign variations;
determining, using an algorithm and using the operational data, at least one campaign attribute for the campaign in accordance with the identified order such that a first campaign attribute is determined and used to determine a subsequent second campaign attribute, wherein determining the at least one campaign attribute for the campaign comprises generating a prompt including the query and at least a portion of the operational data and providing the prompt as input to the algorithm, the portion of the operational data to include in the prompt being selected based on the query and the campaign attribute being determined; and
providing a campaign recommendation including the at least one campaign attribute for the campaign.
Claim 9:
identifying a subset of campaign attributes, from among a plurality of campaign attributes comprising a goal, a product or service offering, an account, a contract, a channel, a content, and/or post-sales activity, to determine for a campaign in accordance with a query or an event;
identifying an order in which to determine campaign attributes of the subset of campaign attributes;
determining, using an algorithm, at least one campaign attribute for the campaign based on operational data in accordance with the identified order, wherein determining the at least one campaign attribute for the campaign comprises generating a prompt including the query and the operational data and providing the prompt as input to the algorithm, the operational data including firmographics data, engagement data, product data, and/or customer data; and
providing a campaign recommendation including the at least one campaign attribute for the campaign.
The following dependent claim limitations, when considered individually and as an ordered combination, are merely further descriptive of abstract concepts:
Claim 3:
further comprising a timing for determining campaign attributes of the subset of campaign attributes.
Claims 4:
wherein the operational data includes firmographics data, engagement data, product data, and/or customer data.
Claim 5:
wherein the subset of campaign attributes is identified based on a rule-based approach, a heuristics-based approach, or a machine learning model.
Claim 6:
wherein the at least one campaign attribute is determined based on a result of a prior campaign attribute determination.
Claim 7:
further comprising selecting and obtaining the operational data relevant to determining the at least one campaign attribute.
Claims 8, 14:
wherein identifying the subset of campaign attributes comprises identifying a first campaign attribute to determine and a second campaign attribute to determine, and wherein identifying the order comprises identifying the first campaign attribute followed by the second campaign attribute.
Claim 10:
further comprising receiving the query or identifying an occurrence of the event.
Claim 11:
wherein the order in which to determine the campaign attributes is based on the query or an occurrence of the event.
Claim 13:
wherein the at least one campaign attribute comprises a goal, an offering, an account, a contact, a channel, a content, or a post-sales activity.
The claims provide entering information about an advertising campaign, identifying attributes of campaigns, identifying an order in which to determine the attributes, determining a campaign attribute, and providing the campaign attribute as a recommendation to the user. Thus, when considered individually and as an ordered combination, the claims embody certain methods of organizing human activity. Specifically, such activity is in the form of commercial interactions (in the form of advertising, marketing or sales activities or behaviors).
Step 2A prong 2: This judicial exception is not integrated into a practical application. The claims recite the following additional elements: one or more computer storage media (claim 1); artificial intelligence technology (claim 1, 9); network (claim 1); user interface with display (claim 1); campaign orchestrator (claim 1, 9); and reduces one or more of input/output operations, network throughput consumption, network latency, or packet generation costs; (claim 1);
The one or more computer storage media, artificial intelligence, user interface with display, and campaign orchestrator are recited at a high level of generality and merely act to “apply it” (the abstract idea) using generic computing components (spec [0125], [0126]) (see MPEP 2106.05(f)).
The user interface is recited generically and does not include any interface elements such that there would be an improvement to interface technology or a technical field.
The high-level recitation of artificial intelligence technology also does not go beyond the “apply it” level of implementation. Nothing in the claims recites any meaningful limitations that could be considered an improvement to artificial intelligence technology or a technical field. The limitations read as little more than do it with AI.
The campaign orchestrator appears to merely be the name given to the software module being implemented to carry out the steps (see spec [0029], Fig 2). Therefore, it also does not go beyond the “apply it” level of implementation.
Further, the inclusion of a “network” does not go beyond the “apply it” level of implementation. The network is merely the means in which the data is transferred between entities. Nothing in the claims improves upon network technology or a technical field.
The limitation regarding “and reduces one or more of input/output operations, network throughput consumption, network latency, or packet generation costs” does not go beyond the “apply it” level of implementation. The recitation of claim limitations that attempt to cover any solution to an identified problem with no restriction on how the result is accomplished and no description of the mechanism for accomplishing the result, does not integrate a judicial exception into a practical application or provide significantly more because this type of recitation is equivalent to the words "apply it". See MPEP 2106.05(f) 1. This limitation provides results based claiming with no meaningful details regarding how picking an order meaningfully accomplishes these results.
Accordingly, when considered both individually and as an ordered combination, the additional elements do not impose any meaningful limits on practicing the abstract idea.
Step 2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Similarly, as above with regard to practical application, the additional elements when considered both individually and as an ordered combination, do not provide an inventive concept as they merely provide generic computing components used as a tool to implement the abstract idea.
As a result, the claims are not patent eligible.
Claim Rejections - 35 USC § 102
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 9-11, 13, 14 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Gao et al (US 2018/0308124)
As per claim 9:
Gao teaches:
A computer-implemented method comprising (paragraph [0123], [0129]):
identifying, via a campaign orchestrator, a subset of campaign attributes, from among a plurality of campaign attributes comprising a goal, a product or service offering, an account, a contact, a channel, a content, and/or a post-sales activity, to determine for a campaign in accordance with a query or an event; (Fig. 7; paragraph [0049] Example types of features of the prediction model include targeting features, campaign attribute features, serving features, text features, image features, and/or metadata features. [0052] Example of campaign attribute features (for which a history of a campaign being active is not required) include bid price, budget, and type of charging model (e.g., CPM, CPC, or CPA). [0053] Example text features (if text is included in the campaign) include certain keywords, topic(s) referenced, number of words, number of nouns/adjectives/verbs/adverbs, and emoticons. [0054] Example image features (if an image is included in the campaign) include whether a face is detected in the image, whether an eyeball is detected, average brightness, dominant color, etc. [0071] In an embodiment, a user (e.g., a representative of a content provider) provides (e.g., through a user interface) multiple sets of attributes of a content delivery campaign. Each set of attributes corresponds to a different instance of a content delivery campaign. [0072] Each set of attributes is input into a prediction model, which outputs an estimate of a metric (of the dependent variable of the prediction model). By comparing the outputs (whether automatically or manually), the user can see which instance of the content delivery campaign will perform better. [0073] In an embodiment, a user provides two or more sets of attributes in a single file, which a campaign estimator accepts and then identifies each attribute set and inputs into a prediction model. [0074] In a related embodiment, a user is allowed to input multiple attributes of a content delivery campaign and then, after viewing actual or estimated performance of the content delivery campaign, modify one or more attributes of the content delivery campaign to see an estimate of how performance might change based on the modified attribute(s). [0116] Similarly, if one recommendation is more difficult to implement than another recommendation (e.g., increasing pCTR or changing an image may be more difficult than selecting enabling audience expansion), then the second recommendation is ranked higher than the first recommendation. [0120] Different content providers' goals may result in different recommendations. For example, increasing budget or bid price may be a recommendation for content providers whose goal is to increase impressions while improving the quality of the text and/or image in a content item may be a recommendation for content providers whose goal is to increase conversions (e.g., online purchases or filling out a form).
Examiner Note: “Operational data generally refers to any data that may be used in generating a campaign.” – spec [0041])
identifying, via the campaign orchestrator, an order in which to determine campaign attributes of the set of campaign attributes; (paragraph [0085] The attribute values may be ordered based on attribute, such that the order of the attribute values matches the order of the attributes in the GBDT, i.e., from root node to leaf node to the next root node (if a GBDT ensemble comprises multiple subtrees), etc. In the example of FIG. 4, the order of campaign attributes are budget, bid, audience expansion, and pCTR. (Attributes “Targeting US” and “FCAP rate” are attributes for which a content provider does not provide attribute values.)
determining, using artificial intelligence technology, at least one campaign attribute for the campaign based on operational data in accordance with the identified order, wherein determining the at least one campaign attribute for the campaign comprises generating a prompt including the query and the operational data and providing the prompt as input to the artificial intelligence technology, wherein the operational data includes firmographics data, engagement data, product data, and/or customer data. ([0014] Techniques are provided for using one or more machine learning techniques to generate recommendations for a content provider regarding how to improve performance of a content delivery campaign along one or more performance metrics, such as delivery, conversion rate, and audience reach. [0035] Each training instance includes multiple feature values, each corresponding to a different feature of multiple features and a label of a dependent (or “target”) variable. Each training instance corresponds to a different content delivery campaign. Thus, the multiple features correspond to (or are mapped to) attributes a content delivery campaign and the label indicates an actual performance of the content delivery campaign, such as daily budget utilization of the campaign, a computed cost-per-click of the campaign, a computed conversion rate of the campaign, or a campaign reach of the campaign. [0071] In an embodiment, a user (e.g., a representative of a content provider) provides (e.g., through a user interface) multiple sets of attributes of a content delivery campaign. Each set of attributes corresponds to a different instance of a content delivery campaign. For example, one instance of a content delivery campaign indicates that a target audience is limited to software engineers while another instance indicates that a target audience includes people with technical degrees in computer science, computer engineering, or electrical engineering. One of the sets of attributes may correspond to current attributes of the content delivery campaign, which may be actively being served by content delivery exchange 120. [0075] In a related embodiment, a user interface allows a user to input an attribute value for each of multiple attributes {a1, a2, a3, . . . } of a content delivery campaign and allows the user, for a particular one of the attributes (e.g., a1}, to select multiple attribute values. For example, a user selects US and Europe as options for the geography of a target audience. Machine learning component 300 receives the attribute values and provides, to Performance predictor 340, attribute values for the content delivery campaign where the geography is US. Performance predictor 340 generates output that represents a first prediction of performance of the content delivery campaign. Machine learning component 300 also provides, to Performance predictor 340, attribute values for the content delivery campaign where the geography is Europe. [0077] In an embodiment, a decision tree or an ensemble (or group) of decision trees is generated and used to predict a performance of a content delivery campaign and, optionally, generate one or more recommendations regarding how to improve predicted performance of the content delivery campaign. [0078] An example of a decision tree ensemble includes gradient boosting decision trees. Gradient boosting is a machine learning technique for regression and classification problems. [0082] A property of GBDTs is that the most impactful features (or the features that influence a predicted performance metric the most) are higher up the GBDT than the less impactful features. Thus, the most impactful feature may be the root node of the first subtree of a GBDT ensemble. Also, if a GBDT ensemble comprises multiple subtrees, then the subtrees have an importance or impactful ranking such that each subtree is more impactful or less impactful then another subtree. Thus, features represented by nodes in a higher ranked subtree are more impactful than features represented by nodes in a lower ranked subtree. [0085] In order to leverage a decision tree (e.g., a GBDT) to determine a predicted impact or performance of a proposed (or active) content delivery campaign, the attribute values of the campaign are used to traverse the decision tree. The attribute values may be ordered based on attribute, such that the order of the attribute values matches the order of the attributes in the GBDT, i.e., from root node to leaf node to the next root node (if a GBDT ensemble comprises multiple subtrees), etc. In the example of FIG. 4, the order of campaign attributes are budget, bid, audience expansion, and pCTR. (Attributes “Targeting US” and “FCAP rate” are attributes for which a content provider does not provide attribute values. [0101] In an embodiment, a recommendation regarding how to improve a content delivery campaign is generated using a prediction model, such as a regression model or a GBDT. (see also [0114]-[0117])
providing, via a campaign recommendation manager, a campaign recommendation including the at least one campaign attribute for the campaign. (paragraph [0110] At block 670, a recommendation is presented to a user (e.g., a representative of the content provider of the content delivery campaign). Block 670 may be performed only if the prediction determined in block 650 indicates a higher or better performance than the original prediction or indicates a similar performance but with lower cost (e.g., in dollars or in difficulty). The recommendation indicates the attribute(s) whose value(s) changed that resulted in the improved predicted performance, the original and changed value(s), and/or the predicted improvement (e.g., 22% lower cost-per-click or 52% higher budget utilization).
Gao teaches the limitations of claim 9. As per claim 10:
Gao further teaches:
further comprising receiving the query or identifying an occurrence of the event. (paragraph [0066] Performance predictor 340 may be activated to predict performance of a content delivery campaign based on user input. For example, a representative of a content provider submits data regarding a content delivery campaign and performance predictor 340 automatically predicts performance, producing an output. [0075] In a related embodiment, a user interface allows a user to input an attribute value for each of multiple attributes {a1, a2, a3, . . . } of a content delivery campaign and allows the user, for a particular one of the attributes (e.g., a1}, to select multiple attribute values.)
Gao teaches the limitations of claim 9. As per claim 11:
Gao further teaches:
wherein the order in which to determine the campaign attributes is based on the query or an occurrence of the event. (paragraph [0075] In a related embodiment, a user interface allows a user to input an attribute value for each of multiple attributes {a1, a2, a3, . . . } of a content delivery campaign and allows the user, for a particular one of the attributes (e.g., a1}, to select multiple attribute values. [0085] In order to leverage a decision tree (e.g., a GBDT) to determine a predicted impact or performance of a proposed (or active) content delivery campaign, the attribute values of the campaign are used to traverse the decision tree. The attribute values may be ordered based on attribute, such that the order of the attribute values matches the order of the attributes in the GBDT, i.e., from root node to leaf node to the next root node (if a GBDT ensemble comprises multiple subtrees), etc. In the example of FIG. 4, the order of campaign attributes are budget, bid, audience expansion, and pCTR. (Attributes “Targeting US” and “FCAP rate” are attributes for which a content provider does not provide attribute values.)
Gao teaches the limitations of claim 9. As per claim 13:
Gao further teaches:
wherein the at least one campaign attribute comprises a goal, an offering, an account, a contact, a channel, a content, or a post-sales activity (paragraph [0116] Similarly, if one recommendation is more difficult to implement than another recommendation (e.g., increasing pCTR or changing an image may be more difficult than selecting enabling audience expansion), then the second recommendation is ranked higher than the first recommendation. [0120] Different content providers' goals may result in different recommendations. For example, increasing budget or bid price may be a recommendation for content providers whose goal is to increase impressions while improving the quality of the text and/or image in a content item may be a recommendation for content providers whose goal is to increase conversions (e.g., online purchases or filling out a form).
Gao teaches the limitations of claim 9. As per claim 14:
Gao further teaches:
wherein identifying the subset of campaign attributes comprises identifying a first campaign attribute to determine and a second campaign attribute to determine, and wherein identifying the order comprises identifying the first campaign attribute followed by the second campaign attribute. (paragraph [0085] In order to leverage a decision tree (e.g., a GBDT) to determine a predicted impact or performance of a proposed (or active) content delivery campaign, the attribute values of the campaign are used to traverse the decision tree. The attribute values may be ordered based on attribute, such that the order of the attribute values matches the order of the attributes in the GBDT, i.e., from root node to leaf node to the next root node (if a GBDT ensemble comprises multiple subtrees), etc. In the example of FIG. 4, the order of campaign attributes are budget, bid, audience expansion, and pCTR. (Attributes “Targeting US” and “FCAP rate” are attributes for which a content provider does not provide attribute values.)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 4-8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gao et al (US 2018/0308124) in view of Li et al (US 2018/0005314)
As per claim 1:
Gao teaches:
One or more computer storage media having computer-executable instructions embodied thereon that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising ([0123], [0129]):
obtaining a query; (paragraph [0066] Performance predictor 340 may be activated to predict performance of a content delivery campaign based on user input. For example, a representative of a content provider submits data regarding a content delivery campaign and performance predictor 340 automatically predicts performance, producing an output. [0075] In a related embodiment, a user interface allows a user to input an attribute value for each of multiple attributes {a1, a2, a3, . . . } of a content delivery campaign and allows the user, for a particular one of the attributes (e.g., a1}, to select multiple attribute values.)
identifying, by a campaign orchestrator, a subset of campaign attributes, from among a plurality of campaign attributes comprising a goal, a product or service offering, an account, a contact, a channel, a content, and/or post-sales activity, to determine for a campaign based on analysis of the query and operational data associated with at least one entity and obtained from a data source via a network, the operation data generated and/or collected by the at least one entity; (Fig. 7; paragraph [0049] Example types of features of the prediction model include targeting features, campaign attribute features, serving features, text features, image features, and/or metadata features. [0052] Example of campaign attribute features (for which a history of a campaign being active is not required) include bid price, budget, and type of charging model (e.g., CPM, CPC, or CPA). [0053] Example text features (if text is included in the campaign) include certain keywords, topic(s) referenced, number of words, number of nouns/adjectives/verbs/adverbs, and emoticons. [0054] Example image features (if an image is included in the campaign) include whether a face is detected in the image, whether an eyeball is detected, average brightness, dominant color, etc. [0071] In an embodiment, a user (e.g., a representative of a content provider) provides (e.g., through a user interface) multiple sets of attributes of a content delivery campaign. Each set of attributes corresponds to a different instance of a content delivery campaign. [0072] Each set of attributes is input into a prediction model, which outputs an estimate of a metric (of the dependent variable of the prediction model). By comparing the outputs (whether automatically or manually), the user can see which instance of the content delivery campaign will perform better. [0073] In an embodiment, a user provides two or more sets of attributes in a single file, which a campaign estimator accepts and then identifies each attribute set and inputs into a prediction model. [0074] In a related embodiment, a user is allowed to input multiple attributes of a content delivery campaign and then, after viewing actual or estimated performance of the content delivery campaign, modify one or more attributes of the content delivery campaign to see an estimate of how performance might change based on the modified attribute(s). [0116] Similarly, if one recommendation is more difficult to implement than another recommendation (e.g., increasing pCTR or changing an image may be more difficult than selecting enabling audience expansion), then the second recommendation is ranked higher than the first recommendation. [0120] Different content providers' goals may result in different recommendations. For example, increasing budget or bid price may be a recommendation for content providers whose goal is to increase impressions while improving the quality of the text and/or image in a content item may be a recommendation for content providers whose goal is to increase conversions (e.g., online purchases or filling out a form).
Examiner Note: “Operational data generally refers to any data that may be used in generating a campaign.” – spec [0041])
identifying an order in which to determine campaign attributes of the set of campaign attributes based on analysis of the query and the operation data, wherein identifying the subset of campaign attributes and identifying the order in which to determine campaign attributes reduces iterative generation and evaluation of multiple campaign variations and reduces one or more of input/output operations, network throughput consumption, network latency, or packet generation costs;(paragraph [0082] A property of GBDTs is that the most impactful features (or the features that influence a predicted performance metric the most) are higher up the GBDT than the less impactful features. Thus, the most impactful feature may be the root node of the first subtree of a GBDT ensemble. Also, if a GBDT ensemble comprises multiple subtrees, then the subtrees have an importance or impactful ranking such that each subtree is more impactful or less impactful then another subtree. Thus, features represented by nodes in a higher ranked subtree are more impactful than features represented by nodes in a lower ranked subtree. [0085] In order to leverage a decision tree (e.g., a GBDT) to determine a predicted impact or performance of a proposed (or active) content delivery campaign, the attribute values of the campaign are used to traverse the decision tree. The attribute values may be ordered based on attribute, such that the order of the attribute values matches the order of the attributes in the GBDT, i.e., from root node to leaf node to the next root node (if a GBDT ensemble comprises multiple subtrees), etc. In the example of FIG. 4, the order of campaign attributes are budget, bid, audience expansion, and pCTR. (Attributes “Targeting US” and “FCAP rate” are attributes for which a content provider does not provide attribute values. Examiner comment: The phrase ”wherein identifying the subset of campaign attributes and identifying the order in which to determine campaign attributes reduces iterative generation and evaluation of multiple campaign variations and reduces one or more of input/output operations, network throughput consumption, network latency, or packet generation costs” is a statement of intended use of the claimed invention and must result in a structural difference between the claimed invention and the prior art in order to patentably distinguish the claimed invention from the prior art. If the prior art structure is capable of performing the intended use, then it meets the claim. Language that suggests or makes a feature or step optional but does not require that feature or step does not limit the scope of a claim under the broadest reasonable claim interpretation. The following types of claim language may raise a question as to its limiting effect: (A) statements of intended use or field of use, including statements of purpose or intended use in the preamble - See MPEP 2103 I. C. Further, see MPEP 2111.04 - …the court noted that a "‘whereby clause in a method claim is not given weight when it simply expresses the intended result of a process step positively recited.” In this particular instance, the limitation is not given weight as it simply expresses the intended result of the process step. Nothing in the claims articulate how choosing an order provides these advantages. Further, the specification ([0057]-[0062]) seems to merely select the order based on whatever a user wants it to be, a template, or determined importance.)
determining, using artificial intelligence technology and operational data, at least one campaign attribute for the campaign in accordance with the identified order {…} wherein determining the at least one campaign attribute for the campaign comprises generating a prompt including the query and at least a portion of the operational data and providing the prompt as input to the artificial intelligence technology, the portion of the operational data to include in the prompt being selected based on the query and the campaign attribute being determined; and (paragraph [0014] Techniques are provided for using one or more machine learning techniques to generate recommendations for a content provider regarding how to improve performance of a content delivery campaign along one or more performance metrics, such as delivery, conversion rate, and audience reach. [0071] In an embodiment, a user (e.g., a representative of a content provider) provides (e.g., through a user interface) multiple sets of attributes of a content delivery campaign. Each set of attributes corresponds to a different instance of a content delivery campaign. For example, one instance of a content delivery campaign indicates that a target audience is limited to software engineers while another instance indicates that a target audience includes people with technical degrees in computer science, computer engineering, or electrical engineering. One of the sets of attributes may correspond to current attributes of the content delivery campaign, which may be actively being served by content delivery exchange 120. [0077] In an embodiment, a decision tree or an ensemble (or group) of decision trees is generated and used to predict a performance of a content delivery campaign and, optionally, generate one or more recommendations regarding how to improve predicted performance of the content delivery campaign. [0078] An example of a decision tree ensemble includes gradient boosting decision trees. Gradient boosting is a machine learning technique for regression and classification problems. [0082] A property of GBDTs is that the most impactful features (or the features that influence a predicted performance metric the most) are higher up the GBDT than the less impactful features. Thus, the most impactful feature may be the root node of the first subtree of a GBDT ensemble. Also, if a GBDT ensemble comprises multiple subtrees, then the subtrees have an importance or impactful ranking such that each subtree is more impactful or less impactful then another subtree. Thus, features represented by nodes in a higher ranked subtree are more impactful than features represented by nodes in a lower ranked subtree. [0085] In order to leverage a decision tree (e.g., a GBDT) to determine a predicted impact or performance of a proposed (or active) content delivery campaign, the attribute values of the campaign are used to traverse the decision tree. The attribute values may be ordered based on attribute, such that the order of the attribute values matches the order of the attributes in the GBDT, i.e., from root node to leaf node to the next root node (if a GBDT ensemble comprises multiple subtrees), etc. In the example of FIG. 4, the order of campaign attributes are budget, bid, audience expansion, and pCTR. (Attributes “Targeting US” and “FCAP rate” are attributes for which a content provider does not provide attribute values. [0101] In an embodiment, a recommendation regarding how to improve a content delivery campaign is generated using a prediction model, such as a regression model or a GBDT. [0110] At block 670, a recommendation is presented to a user (e.g., a representative of the content provider of the content delivery campaign). Block 670 may be performed only if the prediction determined in block 650 indicates a higher or better performance than the original prediction or indicates a similar performance but with lower cost (e.g., in dollars or in difficulty). The recommendation indicates the attribute(s) whose value(s) changed that resulted in the improved predicted performance, the original and changed value(s), and/or the predicted improvement (e.g., 22% lower cost-per-click or 52% higher budget utilization). (see also [0114]-[0117])
providing, for display via a user interface, a campaign recommendation including the at least one campaign attribute for the campaign. (paragraph [0110] At block 670, a recommendation is presented to a user (e.g., a representative of the content provider of the content delivery campaign). Block 670 may be performed only if the prediction determined in block 650 indicates a higher or better performance than the original prediction or indicates a similar performance but with lower cost (e.g., in dollars or in difficulty). The recommendation indicates the attribute(s) whose value(s) changed that resulted in the improved predicted performance, the original and changed value(s), and/or the predicted improvement (e.g., 22% lower cost-per-click or 52% higher budget utilization).
Gao does not expressly teach such that a first campaign attribute is determined and used to determine a subsequent second campaign attribute.
Li teaches:
such that a first campaign attribute is determined and used to determine a subsequent second campaign attribute, ([0060] Also, the overall actual ad budget dollar amount spent for a certain 24 hour time period may be used and divided by 24 to determine the actual hourly average budget dollar amount spent during that time period.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include such that a first campaign attribute is determined and used to determine a subsequent second campaign attribute as taught by Li with the campaign recommendation system of Gao in order to optimize bid prices and budget allocation.
Gao in view of LI teaches the limitations of claims 1. As per claim 4:
Gao further teaches:
wherein the operational data includes firmographics data, engagement data, product data, and/or customer data. (paragraph [0014] Techniques are provided for using one or more machine learning techniques to generate recommendations for a content provider regarding how to improve performance of a content delivery campaign along one or more performance metrics, such as delivery, conversion rate, and audience reach. [0035] Each training instance includes multiple feature values, each corresponding to a different feature of multiple features and a label of a dependent (or “target”) variable. Each training instance corresponds to a different content delivery campaign. Thus, the multiple features correspond to (or are mapped to) attributes a content delivery campaign and the label indicates an actual performance of the content delivery campaign, such as daily budget utilization of the campaign, a computed cost-per-click of the campaign, a computed conversion rate of the campaign, or a campaign reach of the campaign. [0075] In a related embodiment, a user interface allows a user to input an attribute value for each of multiple attributes {a1, a2, a3, . . . } of a content delivery campaign and allows the user, for a particular one of the attributes (e.g., a1}, to select multiple attribute values. For example, a user selects US and Europe as options for the geography of a target audience. Machine learning component 300 receives the attribute values and provides, to Performance predictor 340, attribute values for the content delivery campaign where the geography is US. Performance predictor 340 generates output that represents a first prediction of performance of the content delivery campaign. Machine learning component 300 also provides, to Performance predictor 340, attribute values for the content delivery campaign where the geography is Europe.)
Gao in view of Li teaches the limitations of claim 1. As per claim 5:
Gao further teaches:
wherein the subset of campaign attributes is identified based on a rule-based approach, a heuristics-based approach, or a machine learning model. (paragraph [0078] An example of a decision tree ensemble includes gradient boosting decision trees. Gradient boosting is a machine learning technique for regression and classification problems.)
Gao in view of Li teaches the limitations of claim 1. As per claim 6:
Gao further teaches:
wherein the at least one campaign attribute is determined based on a result of a prior campaign attribute determination. (paragraph [0035] At block 210, a prediction model is generated based on training data that includes multiple training instances. Each training instance includes multiple feature values, each corresponding to a different feature of multiple features and a label of a dependent (or “target”) variable. Each training instance corresponds to a different content delivery campaign. Thus, the multiple features correspond to (or are mapped to) attributes a content delivery campaign and the label indicates an actual performance of the content delivery campaign, such as daily budget utilization of the campaign, a computed cost-per-click of the campaign, a computed conversion rate of the campaign, or a campaign reach of the campaign. [0049] Training database 310 contains one or more training sets, each of which includes data from each of multiple content delivery campaigns, including actual (previous) performance and one or more attributes or characteristics of the content delivery campaign that corresponds to the actual performance. Each training set may correspond to a different set of features. Example types of features of the prediction model include targeting features, campaign attribute features, serving features, text features, image features, and/or metadata features.)
Gao in view of Li teaches the limitations of claim 1. As per claim 7:
Gao further teaches:
further comprising selecting and obtaining the operational data relevant to determining the at least one campaign attribute. (paragraph [0049] Training database 310 contains one or more training sets, each of which includes data from each of multiple content delivery campaigns, including actual (previous) performance and one or more attributes or characteristics of the content delivery campaign that corresponds to the actual performance. Each training set may correspond to a different set of features. Example types of features of the prediction model include targeting features, campaign attribute features, serving features, text features, image features, and/or metadata features. [0096] Given the decision tree ensemble in FIG. 4, the feature of root node 412 is determined to be campaign budget (corresponding to blocks 520-530). If a campaign's budget is greater or equal than $960, then the right edge of root node 412 is selected to arrive at node 424 (corresponding to blocks 540-550). Because node 424 is not a leaf node (corresponding to block 560), then the attribute value that corresponds to the feature of node 424 is determined (corresponding to block 530). If the campaign's bid is less than $4.90, then the left edge of node 424 is selected to arrive at node 436 (corresponding to blocks 540-550). Thus, based on subtree 410 alone, the content delivery campaign is predicted to result (in this example where daily budget utilization is the performance metric) in at least 60% budget utilization.) (See also [0050]-[0055])
Gao in view of Li teaches the limitations of claim 1. As per claim 8:
Gao further teaches:
wherein identifying the subset of campaign attributes comprises identifying the first campaign attribute to determine and the second campaign attribute to determine, and wherein identifying the order comprises identifying the first campaign attribute followed by the second campaign attribute. (paragraph [0085] In order to leverage a decision tree (e.g., a GBDT) to determine a predicted impact or performance of a proposed (or active) content delivery campaign, the attribute values of the campaign are used to traverse the decision tree. The attribute values may be ordered based on attribute, such that the order of the attribute values matches the order of the attributes in the GBDT, i.e., from root node to leaf node to the next root node (if a GBDT ensemble comprises multiple subtrees), etc. In the example of FIG. 4, the order of campaign attributes are budget, bid, audience expansion, and pCTR. (Attributes “Targeting US” and “FCAP rate” are attributes for which a content provider does not provide attribute values.)
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gao et al (US 2018/0308124) in view of Li et al (US 2018/0005314) in view of Xu et al (US 2016/0260125)
Gao in view of Li teaches the limitations of claim 1. As per claim 3:
Gao in view of Li does not expressly teach further comprising a timing for determining campaign attributes of the set of campaign attributes.
Xu teaches:
further comprising a timing for determining campaign attributes of the subset of campaign attributes (paragraph [0005] The computer is further programmed to assign a set of weights to the semantic features when determining the score during a first time period, collect click data on the campaign while using the set of weights to run the campaign in the first time period, update the set of weights using the click data by minimizing a logistic loss function, and to assign an updated set of weights to the semantic features during a second time period.)
It would have been obvious to one of ordinary skill in the art before the effective fling date of the claimed invention to include further comprising a timing for determining campaign attributes of the set of campaign attributes as taught by Xu with AI campaign recommendation system of Gao in order to leverage information gathered to update and improve recommendations.
Response to Arguments
The examiner has considered and does not find persuasive applicant’s arguments regarding rejections under 35 USC 101. With regard to being directed to an abstract idea, the examiner respectfully disagrees. The claims very clearly recite a process for analyzing advertising campaign data to provide recommendations for an advertising campaign. The examiner finds that nothing in the claims improves the use of computing resources and network resources. The claims provide no meaningful detail with regard to the selection of the order. The specification ([0057]-[0062]) says the order can be based on whatever the user wants, a template, or which variables are deemed most important. This is in an of itself, an abstract idea. The argument appears to be that by only doing variable we are interested in, we save computing resources by not having to do all the variables. This is no way improves computers, technology, or a technical field. The same process could be performed without a computer entirely and the resources saved would then simply be human resources used for computation instead of computing resources. Automating the process simply by allowing a computer to do it instead of a human because it’s faster or more accurate is not enough to overcome 101. In OIP Technologies, Inc. v. Amazon.com, Inc. (788 F.3d 1359, 115 U.S.P.Q.2d 1090 (Fed. Cir. 2015)) on page 8 of the written opinion it states that relying on a computer to perform routine tasks more quickly or more accurately is insufficient to render a claim patent eligible. Any alleged improvement regarding restricting computation is at best an improvement to the abstract idea itself.
The order in which you decide to determine attributes is not technical in any way. It is simply part of the abstract idea that would be the same with or without the user of a computer. Again, as discussed above, the alleged improvements are only because you are doing the task on a computer and amount to saying that because we only look at the variables, we want then we aren’t making calculations for other variables we don’t want or are less important. This is in no way an improvement to computers, technology, or a technical field. This is merely part of the abstract idea and at best an improvement to the abstract idea itself.
With regard to the high-level use of artificial intelligence the examiner respectfully disagrees. The claim amount to little more than saying “do it” using artificial intelligence. The claims provide no meaningful details regarding its use and certainly nothing that could even remotely be construed as an improvement to AI technology or a technical field.
With regard to practical application and inventive concept the examiner respectfully disagrees. The examiner finds that the claims fall woefully short under 101 and that the alleged improvements are not in any way improvements to computer technology or a technical field. As discussed above, any alleged improvement is to the business concept itself and not under lying technology. The same process could be performed without a computer entirely and the resources saved would then simply be human resources used for computation instead of computing resources. As a result, such rejection has been maintained.
The examiner has considered but does not find persuasive applicant’s arguments with regard to 35 USC 103. Gao does in fact select campaign attributes that fall under at least “a goal, a product or service offering, an account, a contact, a channel, a content, and/or a post-sales activity.” Paragraph [0052] discusses the attributes included are at least bid prices, budgets, and types of charging models. The user provides these variables to the system ([0071]). The output of the model provides recommendations in accordance with the goal provided by the content providers ([0120]) including 22% lower cost per click or 52% higher budget utilization ([0110]).
With regard identifying the order based on the query and operational data the examiner disagrees. The order in the model is based upon the data entered by the user. Thus, the order having been determined form the data entered by the user must broadly be based on analysis of the query which provides the input. Again, any of the information provided be the user can also be considered operational data based on the broad definition supplied in the spec.
Gao is further not relied upon to teach the subsequent determination of the second attribute based on the first attribute. Li teaches taking a first attribute (overall budget) and using it to determine a second attribute (hourly average budget spent). As a result, such rejections have been maintained.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHRISTOPHER STROUD whose telephone number is (571)272-7930. The examiner can normally be reached Mon. - Fri. 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Waseem Ashraff can be reached at (571) 270-3948. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
CHRISTOPHER STROUD
Primary Examiner
Art Unit 3621
/CHRISTOPHER STROUD/ Primary Examiner, Art Unit 3621
Prosecution Insights