DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 9/23/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 5/8/2026 has been entered.
Claims 19-20 are presently cancelled.
Claims 21-22 are newly added.
Claims 1-18 and 20-22 are pending.
Response to Amendment
Applicant’s amendments are acknowledged.
Response to Arguments
Applicant' s arguments filed 4/22/2026 have been fully considered in view of further consideration of statutory law, Office policy, precedential common law, and the cited prior art as necessitated by the amendments to the claims, and are not persuasive for the reasons set forth below.
35 USC § 101 Rejections
First, Applicant argues that “under Step 2A, Prong 1 of the Alice framework, the claims are not directed to an abstract idea. The Applicant's claim 1 recites a specific method for optimizing feature computation in feature stores by (i) identifying when a newly requested feature overlaps with an already-computed feature…
These claim elements, read together, are directed to a specific database query optimization technique and not to any of the recognizes categories of abstract ideas including fundamental economic practice, method of organizing human activity, or mathematical concept…
This is directly analogous to Enfish, LLC v. Microsoft Corp., 822 F.3d 1327 (Fed. Cir. 2016), where the Federal Circuit held that claims to a self-referential database table were not directed to an abstract idea because they were directed to an improvement in computer functionality (database performance), and to McRO, Inc. V. Bandai Namco Games Am. Inc., 837 F.3d 1299 (Fed. Cir. 2016), where claims reciting specific rules for automating a process (lip-sync animation) were held not abstract because they improved a technological process…
The claimed selection is not a method of organizing human activity but rather a database query optimization operation performed by computer components operating on feature datasets in a feature store system, as explicitly described in the specification at paragraphs [0030]-[0033]…
The claimed operations are not mental processes but rather computer-implemented database query optimization operations that cannot practically be performed in the human mind. The claim explicitly requires "using an execution cost estimator that predicts an execution cost for each candidate execution alternative" and "selecting. based on the execution costs predicted by the execution cost estimator… Under the MPEP § 2106.04(a)(2)(III) and the 2019 Revised Patent Subject Matter Eligibility Guidance, a claim does not recite a mental process if it requires use of a computer or cannot practically be performed in the human mind. Here, predicting execution costs for PIT joins over feature datasets (which may contain thousands of records across distributed partitions) and selecting among execution alternatives based on quantitative cost metrics is not a mental process-it is a computer- implemented query optimization technique integral to database system functionality…” [Arguments, pages 8-10].
In response, Applicant’s arguments are considered but are not persuasive. Examiner respectfully maintains that the present invention recites a judicial exception without significantly more. In particular, and with respect to the assertion that the claims do not not a method of organizing human activity but rather a database query optimization operation performed by computer components operating on feature datasets in a feature store system, Examiner respectfully disagrees. Claim 1, for example, does not mention queries, databases or even data at all. Instead, Examiner observes that the Claim broadly recites comparing feature definitions, generating alternative feature definitions, and executing a PIT join of unspecified elements based on an undefined cost estimation process. Specifically, Examiner maintains that selecting an execution alternative from an execution of a PIT join using the alternative feature definition and an execution of a PIT join using the new feature definition is considered to set forth steps for following rules or instructions. Thus, Examiner maintains that the present invention recites concepts relating to certain methods of organizing human activity.
Further, Examiner maintains that the present invention recites human processes. As stated above, Claim 1 does not mention queries, databases or even data at all. Thus it is reasonable to conclude that executing a PIT join based on a cost analysis can be executed on pen and paper. Examiner observes that the recitation of generic computer components in a claim does not necessarily preclude that claim from reciting an abstract idea. With regard to independent claims 12 and 18, Examiner observes that the claimed computer components are briefly mentioned at the beginning of the claims and are not sufficiently integrated in the claims in a way that demonstrates that the PIT join execution cannot be executed on pen and paper.
Examiner observes that the patentee in Synopsys argued that the claimed methods of logic circuit design were intended to be used in conjunction with computer-based design tools, and were thus not mental processes. Synopsys, Inc. v. Mentor Graphics Corp., 839 F.3d 1138, 1147-49, 120 USPQ2d 1473, 1480-81 (Fed. Cir. 2016). The court disagreed, because it interpreted the claims as encompassing nothing other than pure mental steps (and thus falling within an abstract idea grouping) because the claims did not include any limitations requiring computer implementation.
In contrast, the patentee in Enfish argued that its claimed self-referential table for a computer database was an improvement in an existing technology and thus not directed to an abstract idea. Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1336-37, 118 USPQ2d 1684, 1689-90 (Fed. Cir. 2016). The court agreed with the patentee, based on its interpretation of the claimed "means for configuring" under 35 U.S.C. 112(f) as requiring a four-step algorithm that achieved the improvements, as opposed to merely any form of storing tabular data. See also McRO, Inc. v. Bandai Namco Games America, Inc. 837 F.3d 1299, 1314, 120 USPQ2d 1091, 1102 (Fed. Cir. 2016) (the claim’s construction incorporated rules of a particular type that improved an existing technological process).
Examiner likens the present invention to Synopsys, rather than Enfish or McRO, because the claims do not include any limitations require computer implementation. Thus, Examiner respectfully maintains that the present invention recites a judicial exception, abstract ideas. As such, Examiner remains unpersuaded.
Second, Applicant argues that “Prong two of step 2A is satisfied if the claim recites additional elements that integrate the judicial exception into a practical application… a technical improvement is reflected by the features "selecting an execution alternative. using an execution cost model that predicts an execution cost for each execution alternative" recited in the Applicant's independent claim 1. Per this recited optimization methodology, the system dynamically evaluates multiple execution strategies for computing a feature and selects the most efficient one based on predicted resource usage. Specifically, the specification describes a cost model that evaluates execution alternatives based on various factors…
The Office Action states that reciting an execution cost estimator is not sufficient to demonstrate a technological improvement and that the present claims purportedly do not detail how the execution cost estimator operates, and the claims only briefly mention that it determines an execution cost. Office Action at 3-4. The Applicant respectfully disagrees with this position. The Applicant's claim 1 recites that the execution cost estimator "predicts an execution cost for each candidate execution alternative," where the candidate execution alternatives include "(i) an execution of a first point-in-time (PIT) join using the alternative feature definition and (ii) an execution of a second PIT join using the new feature definition," and the system selects the execution alternative based on the predicted costs. This is not a bare recitation of cost estimation in the abstract, but is a concrete, functional integration of cost-based selection into a specific database query-optimization workflow that improves how feature computations are executed in feature stores by avoiding redundant computation and selecting the most efficient execution plan…
Also, additional technical improvement addressing a technical problem is reflected in the feature of "generating an alternative feature definition based on the new feature definition and the matched feature definition" recited in Applicant's claim 1. The technical problem of redundant computation, which wastes computing resources, occurs when a newly requested feature overlaps with an already-computed feature, and is described in the Applicant's specification at paragraphs [0027]-[0029]. The technical improvement addresses this technical problem by generating an alternative feature definition based on both the new feature definition and the matched feature definition…” [Arguments, pages 10-13].
In response, Applicant’s arguments are considered but are not persuasive. Examiner respectfully maintains that the present invention recites a judicial exception without significantly more. Examiners evaluate integration into a practical application by: (1) identifying whether there are any additional elements recited in the claim beyond the judicial exception(s); and (2) evaluating those additional elements individually and in combination to determine whether they integrate the exception into a practical application, using one or more of the considerations introduced in subsection I supra, and discussed in more detail in MPEP §§ 2106.04(d)(1), 2106.04(d)(2), 2106.05(a) through (c) and 2106.05(e) through (h).
With respect to the assertion that the present invention integrates the abstract idea into a practical application, Examiner disagrees and maintains that the claims fail to demonstrate that the claims represent more than a drafting effort designed to monopolize the judicial exception (See MPEP § 2106.05(e)).
Examiner likens the present claims, including the cost estimation elements, to OIP Technologies, Inc. v. Amazon.com, Inc., wherein the court determined that the additional steps to "test prices and collect data based on the customer reactions" did not meaningfully limit the abstract idea of offer-based price optimization, because the steps were well-understood, routine, conventional data-gathering activities. 788 F.3d 1359, 1363-64, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015).
Further, Examiner observes that the claims 1, 12 and 18 only recite the following additional elements –
…computer implemented… [Claim 1],
One or more physically manufactured computer-readable storage media, encoding computer-executable instructions for executing on a computer system a computer process, the computer process comprising… [Claim 12],
A system comprising: memory; one or more processor units; a feature store data preparation optimization system stored in the memory and executable by the one or more processor units, the feature store data preparation optimization system encoding computer-executable instructions on the memory for executing on the one or more processor units a computer process, the computer process comprising… [Claim 18].
These elements are recited at a high-level of generality (see MPEP § 2106.05(a)), like the following MPEP example:
iii. Gathering and analyzing information using conventional techniques and displaying the result, TLI Communications, 823 F.3d at 612-13, 118 USPQ2d at 1747-48;
Examiner thus respectfully maintains that the claims fail to demonstrate anything other than well understood, routine and conventional cost analysis or PIT join techniques. Accordingly, these additional elements do not integrate the abstract idea into a practical application. As such, Examiner remains unpersuaded.
Third, Applicant argues that “the claims clearly recite "significantly more" under Step 2B by integrating the purportedly-recited abstract idea into a practical application that improves computer functionality… claim 1 recites concrete, functional improvements to feature store database systems that constitute significantly more: (i) the claim recites generating an alternative feature definition based on the new feature definition and the matching feature definition, which improves computer performance by reducing redundant computation and minimizing unnecessary data processing; (ii) the claim recites predicting execution costs for multiple candidate execution alternatives (including PIT joins using the alternative versus original feature definitions) and selecting the lowest-cost alternative, which improves database query efficiency by enabling intelligent, cost-based execution decisions that reduce computational overhead. These features impose meaningful limits on the claim scope and do not merely recite generic computer components performing well-understood, routine, conventional activities; rather, they recite a novel workflow- matching/overlap detection, alternative feature generation, cost-based execution selection, and optimized execution-that is explicitly described in the specification as improving feature computation efficiency. This is analogous to Bascom Global Internet Servs., Inc. v. AT&T Mobility LLC, 827 F.3d 1341 (Fed. Cir. 2016), where the Federal Circuit held that even if filtering content were abstract, the claims recited significantly more by describing a specific technological implementation (filtering at the ISP level) that improved network efficiency. Here, the claims similarly recite a specific technological implementation (reuse-based feature rewriting and cost-based execution selection) that improves database performance, constituting significantly more under Step 2B and establishing patent eligibility…” [Arguments, pages 13-14].
In response, Applicant’s arguments are considered but are not persuasive. Examiner respectfully maintains that the present invention recites a judicial exception without significantly more. In particular, and with respect to Bascom, Examiner observes that the Federal Circuit held these claims eligible at Step 2B (Pathway C) because they presented a "technology-based solution" of filtering content on the Internet that overcame the disadvantages of prior art filtering systems and that amounted to significantly more than the recited abstract idea. In contrast, the present claims do not a non-conventional and non-generic arrangement of various computer components for filtering Internet content. Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, e.g., a limitation indicating that a particular function such as creating and maintaining electronic records is performed by a computer, as discussed in Alice Corp., 573 U.S. at 225-26, 110 USPQ2d at 1984 (see MPEP § 2106.05(f)). Examiner maintains that the present claims do not demonstrate anything other than “applying it” on the claimed computer. Thus, Examiner respectfully maintains that the present invention demonstrates neither an improvement to the functioning of computers, nor an improvement to any particular field or technology. As such, Examiner remains unpersuaded.
35 USC § 103 Rejections
First, Applicant argues that “The combination of Bonaci and Chen does not teach or suggest generating an alternative feature definition based on a new feature definition and a matched feature definition that at least partially includes the new feature definition…
Bonaci teaches evaluating model accuracy based on computed feature values and does not teach or suggest comparing feature definitions for semantic or structural overlap… none of these disclosures teach or suggest analyzing whether a newly requested feature definition is at least partially included in an existing feature definition previously computed and stored in a feature store. Bonaci's validation process teaches comparing model outputs against expected results, not feature definitions against one another, but Bonaci does not disclose determining that one feature definition is at least partially included in another. Accordingly, Bonaci's use of a validation example or validation set does not teach or suggest a "matched feature definition" as recited in independent claim 1…
Bonaci also does not teach or suggest generating an "alternative feature definition" based on a determination of overlap with an existing feature definition… Bonaci does not teach or suggest rewriting a feature definition to reuse a portion of an existing computed feature, does not describe defining an alternative feature definition that explicitly combines reused precomputed feature datasets with newly computed non-overlapping portions, and does not describe generating execution-aware feature definitions intended to enable reuse prior to execution…
In Bonaci's process, the underlying feature definition remains unchanged while previously computed intermediate results are reused to avoid reprocessing events. Bonaci's teaching of reusing intermediate computation state during execution of an aggregation plan therefore does not teach or suggest generating an alternative feature definition based on determining semantic overlap between feature definitions stored in a feature store, as expressly required by claim 1.
…Bonaci does not disclose comparing a newly requested feature definition against a previously computed feature definition to determine partial inclusion, nor does Bonaci disclose identifying overlapping and non-overlapping portions of feature logic between feature definitions. Instead, Bonaci's reuse mechanisms operate exclusively at the level of execution within the feature computation layer, by resuming execution of a query or aggregation plan from an eligible intermediate state identified by a resume token, as described for example in paragraph [0061]. In that process, the feature definition itself is not altered or rewritten. Bonaci therefore teaches reuse of intermediate computation state during execution of a feature computation and does not teach or suggest generation or rewriting of a feature definition itself. By contrast, claim 1 expressly recites generating an alternative feature definition in response to determining that a new feature definition is at least partially included in a matched feature definition, which requires feature-definition-level overlap analysis and deliberate construction of a new, reuse-aware feature definition…” [Arguments, pages 14-17].
In response, Applicant’s arguments are considered but are not persuasive. First, with regard to the assertion that Bonaci does not disclose determining that one feature definition at least partially includes a new feature definition, Examiner respectfully disagrees and maintains that Bonaci renders the claim element obvious. In particular Examiner observes that Bonaci utilizes a feature computation layer which generates new feature definitions through aggregations of previous feature definitions (Bonaci, ¶ 61) Examiner respectfully maintains that a new feature definition based on aggregation satisfies the claim element because an aggregated feature definition would necessarily, at least partially, include the feature definitions used to create it. With regard to the ‘semantic overlap elements’, which Applicant maintains is “expressly required by claim 1”, Examiner respectfully observes that claim 1 recites no such semantic overlap language.
Further, and with regard to the assertion that Bonaci does not disclose comparing a newly requested feature definition against a previously computed feature definition to determine partial inclusion, Examiner respectfully maintains that this comparison is rendered obvious through the training of the feature model, wherein testing the model requires comparing the new feature definitions with the features of the validation set. With regard to the argument that Bonaci's validation process teaches comparing model outputs against expected results, not feature definitions against one another, Examiner respectfully disagrees and maintains that feature computation layer 106 of Bonaci expressly generates feature definitions as an output, thus rendering the above argued claim limitation obvious. As such, Examiner remains unpersuaded.
Second, Applicant argues that “The combination of Bonaci and Chen does not teach or suggest selecting an execution alternative from among candidate execution alternatives, based on the execution costs predicted by the execution cost estimator…
Chen discloses predefined feature-set joins used to materialize wide tables, not cost-based selection among candidate PIT join execution alternatives. Chen teaches providing wide tables containing features accessible to APIs for sharing and training of machine learning models (Chen at Abstract and paragraph [0021]) and defines features as joins between different tables on the same key (e.g., address and timestamp) (id. at paragraph [0023])… These disclosures of Chen teach reuse in the sense of materialization and iterative development workflows, but Chen does not teach or suggest generating multiple candidate execution alternatives for a PIT join and selecting one of those alternatives using an execution cost estimator that predicts execution costs for each candidate. Accordingly, Chen does not teach or suggest "selecting an execution alternative from among candidate execution alternatives ... the selected execution alternative being selected based on the execution costs predicted by the execution cost estimator," as recited in independent claim 1…
Even assuming, arguendo, that the combination of the references teaches generating features by joining tables to form feature sets (Chen), reusing previously computed feature values when materializing or accessing those feature sets (Chen), computing feature values according to a user-specified feature configuration and iteratively training or retraining a predictive model using those computed feature values (Bonaci), and reusing previously computed intermediate results during execution of a query or aggregation plan to improve efficiency (Bonaci), the combination nevertheless does not teach or suggest selecting an execution alternative from among candidate execution alternatives using an execution cost estimator that predicts an execution cost for each candidate execution alternative, wherein the candidate execution alternatives include (i) execution of a first point-in-time (PIT) join using an alternative feature definition and (ii) execution of a second PIT join using the new feature definition, with the selected execution alternative being chosen based on the predicted execution costs, as recited in independent claim 1.” [Arguments, pages 17-19].
In response, Applicant’s arguments are considered but are not persuasive. Examiner respectfully disagrees and maintains that through KSR Rationale D (See MPEP 2141(III)(D)), the combination of Bonaci and Chen discloses the above argued limitation. In particular, KSR Rationale D involves applying a known technique to a known device (method, or product) ready for improvement to yield predictable results.
While Applicant appears to state that “Bonaci… was not cited as purportedly teaching the above-recited features…” [Arguments, page 18], Examiner observes that the previous Office Action did rely upon a combination of Bonaci and Chen, through KSR Rationale D, to render the above-argued claim limitation obvious.
Particularly, and as previously stated, through KSR Rationale D (See MPEP 2141(III)(D)), the combination of Bonaci and Chen discloses …selecting an execution alternative from among candidate execution alternatives, using an execution cost estimator that predicts an execution cost for each candidate execution alternative, wherein the candidate execution alternatives include (i) an execution of a PIT join using the alternative feature definition and (ii) an execution of a second PIT join using the new feature definition, the selected execution alternative being selected based on the execution costs predicted by the execution cost estimator.
First, Bonaci disclose a data aggregation technique including an expense estimation and minimization technique which selects an alternative technique for minimal cost (Bonaci, ¶ 82, According to an aspect, feature computation layer 106 is configured to simultaneously compute more than one feature, such as a large number of features. When simultaneously computing many features, it is possible to compute each feature independently and then join the computed values based on the entity and time. However, this approach is inefficient for at least two major reasons. First, computing each feature may involve retrieving and processing the same input events multiple times. Second, once the features are computed, performing an N-way join is an expensive operation. FIG. 5A illustrates an example N-way join 500a, (discloses PIT join techniques) such as a 3-way join, being performed after multiple features are individually computed. Computing two or more of the three features shown in FIG. 5A may involve retrieving and processing the same input events multiple times. After these three features are individually computed, they may be joined and output by the system), (Id., ¶ 83, Rather than employing this inefficient and expensive technique for simultaneously computing multiple features, feature computation layer 106 (discloses execution cost estimator) may instead combine all of the aggregations into a single pass over events that computes (at each point in time and for each entity) the value of all aggregations. (discloses execution alternative selected for minimal cost) The description of this flattened operation is called the aggregation plan and the process for producing it is described in more detail below. This flattened aggregation plan allows for the simultaneous computation of the aggregations necessary for all requested features with a single pass over the input, and therefore eliminates the need for the N-way join. FIG. 5B illustrates an example simultaneous feature computation 500b without an N-way join. As depicted in FIG. 5B, all of the multiple features are simultaneously computed with a single pass over the input, eliminating the need to retrieve and process the same input events multiple times), (Id., ¶ 157, In embodiments, resume tokens are utilized to continually apply the results of a query to a separate (i.e., external) data store with minimal cost. This may be achieved by first running an initial query, writing the results to the separate data store, and receiving a resume token. A query may be periodically run to update the results in the external store. Each query uses the resume token returned by the previous response. The new results may reflect only those results which have changed).
PNG
media_image1.png
452
647
media_image1.png
Greyscale
Further, Chen discloses time-sensitive PIT join techniques using new and alternative feature definitions (Chen, ¶ 23, For purposes of this document, features are joins between different tables on the same key (e.g., address, timestamp) with perhaps simple transformations applied to some fields (discloses selecting an optimal time-sensitive PIT join alternative) that will make it more suitable for machine learning training and/or service, such as the nullification of values that are obviously incorrect), (Id., ¶ 24, These two concepts allow for the separation of physical mechanisms for managing tables from the logical design of features, while allowing each layer to be iterated on separately. For example, supporting a new table format merely requires the user to implement a common interface, which could immediately be used by a feature layer without any knowledge of the physical details), (Id., ¶ 39, As described above, features are joins between tables FIG. 3 is a diagram illustrating an example of features, in accordance with an example embodiment. Features are what someone building a model would care about. They define a configuration file that references all the fields needed from each of the tables defined.), (Id., ¶ 43, When iterating over a FeatureSet for model training, two commands may be defined: [0044] 1. build_feature_set_input_table—feature-set $FEATURE_SET—path $PATH which, given a FeatureSet definition, will perform all the joins necessary for the input table and can later be iterated on. [0045] 2. build_feature_set—feature-set $FEATURE_SET—input-table $PATH will build the feature set using the input table from 1. This command would fail if the input table does not contain the necessary columns. [0046] These two commands allow a user to iterate over feature set definitions without having to recalculate all joins every time).
PNG
media_image2.png
285
493
media_image2.png
Greyscale
One of ordinary skill in the art would have recognized that applying the known technique of Bonaci would have yielded predictable results and resulted in an improved system. It would have been recognized that applying the cost estimation technique of Bonaci to the feature-definition-based PIT join teachings of Chen would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such data aggregation features into similar systems. Further, applying a cost minimization based alternative selection to Chen with feature definitions considered accordingly, would have been recognized by those of ordinary skill in the art as resulting in an improved system that would allow more cost effective data aggregation according to specific feature definitions.
Thus, Examiner respectfully maintains that Bonaci and Chen, in combination, renders obvious the above-argued limitation of the independent claims. As such, Examiner remains unpersuaded.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-18 and 21-22 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
Step 1: Claims 1-18 and 21-22 are directed to statutory categories, namely a process (claims 1-11 and 21-22), an article of manufacture (claims 12-17) and a machine (claim 18).
Step 2A, Prong 1: Claims 1, 12 and 18 in part, recite the following abstract idea:
…A … method, comprising: receiving a new feature definition specifying parameters of a feature; comparing the new feature definition with a plurality of computed feature definitions stored in a feature store; in response to determining that the new feature definition is at least partially included in a matched feature definition of the plurality of computed feature definitions, generating an alternative feature definition based on the new feature definition and the matched feature definitions; selecting an execution alternative from among candidate execution alternatives, using an execution cost estimator that predicts an execution cost for each candidate execution alternative, wherein the candidate execution alternatives include (i) an execution of a first point-in-time PIT join using the alternative feature definition and (ii) an execution of a second PIT join using the new feature definition, the selected execution alternative being selected based on the execution costs predicted by the execution cost estimator; and executing the selected execution alternative using a compute engine to generate the feature [Claim 1],
…receiving a new feature definition specifying parameters of a feature; comparing the new feature definition with a plurality of computed feature definitions stored in a feature store; in response to determining that the new feature definition is at least partially included in a matched feature definition of the plurality of computed feature definitions, generating an alternative feature definition based on the new feature definition and the matched feature definition; selecting an execution alternative from among candidate execution alternatives, using an execution cost estimator that predicts an execution cost for each candidate execution alternative, wherein the candidate execution alternatives include (i) an execution of a first point-in-time PIT join using the alternative feature definition and (ii) an execution of a second PIT join using the new feature definition, the selected execution alternative being selected based on the execution costs predicted by the execution cost estimator, wherein selecting the execution alternative further comprises evaluating, using a feature selection criterion one or more of the alternative feature definition and the new feature definition; and executing the selected execution alternative using a compute engine to generate the feature [Claim 12],
…receiving a new feature definition specifying parameters of a feature; comparing the new feature definition with a plurality of computed feature definitions stored in a feature store; in response to determining that the new feature definition is at least partially included in a matched feature definition of the plurality of computed feature definitions, generating an alternative feature definition based on the new feature definition and the matched feature definition; selecting an execution alternative from among candidate execution alternatives, using an execution cost estimator that predicts an execution cost for each candidate execution alternative, wherein the candidate execution alternatives include (i) an execution of a first point-in-time PIT join using the alternative feature definition and (ii) an execution of a second PIT join using the new feature definition, the selected execution alternative being selected based on the execution costs predicted by the execution cost estimator, wherein selecting the execution alternative further comprises evaluating, using a feature selection criterion one or more of the alternative feature definition and the new feature definition; and executing the selected execution alternative using a compute engine to generate the feature [Claim 18].
These concepts are not meaningfully different than the following concepts identified by the MPEP:
Concepts relating to certain methods of organizing human activity. The aforementioned limitations describe steps for managing personal behavior or relationships or interactions between people, including social activities, teaching, and following rules or instructions. Specifically, selecting an execution alternative from an execution of a PIT join using the alternative feature definition and an execution of a PIT join using the new feature definition is considered to describe steps for following rules or instructions.
Mental Processes. The aforementioned limitations describe steps for concepts performed in the human mind which includes an observation, evaluation, judgment, or an opinion. Specifically, selecting an execution alternative from an execution of a PIT join using the alternative feature definition and an execution of a PIT join using the new feature definition is considered to describe steps for an evaluation.
As such, claims 1, 12 and 18 recite concepts identified as abstract ideas.
The dependent claims recite limitations relative to the independent claims, including, for example:
…further comprising: receiving a plurality of candidate source data layouts that are based on current feature computation pipelines and current source data layout; determining a plurality of candidate source data layouts; and selecting a new data source layout from the plurality of candidate source data layouts that are based on current feature computation pipelines and current source data layout [Claim 2],
…wherein selecting a new data source layout further comprising evaluating the plurality of candidate source data layouts and the current source data layout based on a layout selection criterion, wherein the layout selection criterion comprises selection of a minimum cost configuration of the new data source layout [Claim 3],
…wherein the selection of the minimum cost configuration is implemented using binary integer programming [Claim 4],
…wherein selecting the execution alternative further comprises evaluating, using a feature selection criterion one or more of the alternative feature definitions and the new feature definition. [Claim 5],
…wherein the feature selection criterion comprises minimization of data to be scanned using one or more of the alternative feature definitions and the new feature definition [Claims 6 and 13],
…wherein the minimization of data to be scanned further comprises calculating a benefit based on a number of data partitions to be read by the execution of a third PIT join using the alternative feature definition and a number of data partitions to be read by the execution of the second PIT join using the new feature definition [Claims 7 and 14],
…wherein the minimization of data to be scanned further comprises calculating a benefit based on a size of data not to be read by the execution of a third PIT join using the alternative feature definition and a size of data not to be read by the execution of the second PIT join using the new feature definition [Claims 8 and 15],
…further comprising generating the plurality of candidate source data layouts further comprising: retrieving the plurality of computed feature definitions stored in a feature store; extracting data sources used to compute the plurality of computed feature definitions stored in a feature store; and partitioning each of the extracted data sources based on a predetermined granularity of time period [Claims 9 and 17],
…wherein the predetermined granularity of time period may be at least one of a month, a day, an hour, and a minute [Claim 10],
…wherein the plurality of computed feature definitions are determined using PIT joins [Claim 11],
…wherein the computer process further comprising: receiving a plurality of candidate source data layouts that are based on current feature computation pipelines and current source data layout; determining a plurality of candidate source data layouts; and selecting a new data source layout from the plurality of candidate source data layouts that are based on current feature computation pipelines and current source data layout, wherein the execution cost estimator predicts the execution cost for each candidate execution alternative that utilizes the selected new data source layout [Claim 16],
...wherein generating the alternative feature definition includes: identifying a non-overlapping portion and an overlapping portion of the new feature definition relative to the matched feature definition; defining the alternative feature definition to include instructions to: compute feature values for the non-overlapping portion of the new feature definition; and append the computed feature values to a precomputed feature dataset corresponding to the matched feature definition and generated for the overlapping portion [Claim 21],
…wherein generating the alternative feature definition includes: determining an execution plan for the alternative feature definition that reuses a precomputed feature dataset corresponding to the matched feature definition; and defining the alternative feature definition to include instructions that execute according to the execution plan [Claim 22].
The limitations of these dependent claims are merely narrowing the abstract idea identified in the independent claims, and thus, the dependent claims also recite abstract ideas.
Step 2A, Prong 2: This judicial exception is not integrated into a practical application. In particular, claims 1, 12 and 18 only recite the following additional elements –
…computer implemented… [Claim 1],
One or more physically manufactured computer-readable storage media, encoding computer-executable instructions for executing on a computer system a computer process, the computer process comprising… [Claim 12],
A system comprising: memory; one or more processor units; a feature store data preparation optimization system stored in the memory and executable by the one or more processor units, the feature store data preparation optimization system encoding computer-executable instructions on the memory for executing on the one or more processor units a computer process, the computer process comprising… [Claim 18].
The system, processor and executable instructions are recited at a high-level of generality (see MPEP § 2106.05(a)), like the following MPEP example:
iii. Gathering and analyzing information using conventional techniques and displaying the result, TLI Communications, 823 F.3d at 612-13, 118 USPQ2d at 1747-48;
Furthermore, the computer implemented element is considered to amount to no more than mere instructions to apply the exception using a generic computer component (see MPEP 2106.05(f)), like the following MPEP example:
i. A commonplace business method or mathematical algorithm being applied on a general purpose computer, Alice Corp. Pty. Ltd. V. CLS Bank Int’l, 573 U.S. 208, 223, 110 USPQ2d 1976, 1983 (2014); Gottschalk v. Benson, 409 U.S. 63, 64, 175 USPQ 673, 674 (1972); Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015);
Accordingly, these additional elements do not integrate the abstract idea into a practical application.
The remaining dependent claims do not recite any new additional elements, and thus do not integrate the abstract idea into a practical application.
Step 2B: Claims 1, 12 and 18 and their underlying limitations, steps, features and terms, considered both individually and as a whole, do not include additional elements that are sufficient to amount to significantly more than the judicial exception for the following reasons:
Independent claims 1, 12 and 18 only recite the following additional elements –
…computer implemented… [Claim 1],
One or more physically manufactured computer-readable storage media, encoding computer-executable instructions for executing on a computer system a computer process, the computer process comprising… [Claim 12],
A system comprising: memory; one or more processor units; a feature store data preparation optimization system stored in the memory and executable by the one or more processor units, the feature store data preparation optimization system encoding computer-executable instructions on the memory for executing on the one or more processor units a computer process, the computer process comprising… [Claim 18].
These elements do not amount to significantly more than the abstract idea for the reasons discussed in 2A prong 2 with regard to MPEP 2106.05(a) and MPEP 2106.05(f). By the failure of the elements to integrate the abstract idea into a practical application there, the additional elements likewise fail to amount to an inventive concept that is significantly more than an abstract idea here, in Step 2B.
As such, both individually or in combination, these limitations do not add significantly more to the judicial exception.
The remaining dependent claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the dependent claims do not recite any new additional elements other than those mentioned in the independent claims, which amount to no more than mere instructions to apply the exception using a generic computer component (see MPEP 2106.05(f)). As such, these claims are not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-18 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Bonaci et al., U.S. Publication No. 2022/0156254 [hereinafter Bonaci] in view of Chen, U.S. Publication No. 2020/0004733 [hereinafter Chen].
Regarding Claim 1, Bonaci discloses …A computer-implemented method, comprising: receiving a new feature definition specifying parameters of a feature (Bonaci, ¶ 32, In an embodiment, feature engineering system 100 may be configured to generate feature vectors and/or examples associated with a particular entity. As is discussed below in more detail, a user of system 100, such as a data scientist, may be responsible for instructing system 100 which entity or entities should be included in the feature vectors and/or examples. For example, if the user of system 100 wants to train a model to predict how much homes will sell for in Seattle, the user of system 100 may instruct system 100 to choose houses in Seattle as the entities that should be included in the feature vectors (discloses receiving a new feature definition specifying parameters of the feature) and/or examples. If the user instructed system 100 to choose, for example, houses in Los Angeles as the set of entities that should be included in the feature vectors and/or examples, the model may not be able to accurately predict selling prices for homes in Seattle), (Id., ¶ 120, Once the user has created and/or changed the feature definition and/or example selection, the feature engineering system can use this information to efficiently create the desired features and/or feature vectors and/or examples for the user. For example, the feature engineering system can use this information to create the desired features and/or feature vectors and/or examples for the user by re-using previous computations. After the desired features and/or feature vectors and/or examples have been generated, they may be exported to the user. At 704, the generated features and/or feature vectors and/or examples may be exported to the user. The user may use these exported features and/or feature vectors and/or examples to train and/or validate/evaluate the model. At 706, the user may train the model on any training examples generated by the feature engineering system. At 708, the user may validate and/or evaluate the model using any validation examples generated by the feature engineering system. If the user wants the feature engineering system to generate new or different features and/or feature vectors and/or examples, the user may easily change the dataset being used or experiment with a different dataset. For example, the user may want to try a new dataset to see if the model performs better after being trained with the new dataset. The method 700 may return to step 702, where the user may change the feature definition and/or update the example selection configuration. The user may continue to perform this iterative process until the model is generating results that satisfy the user);
comparing the new feature definition with a plurality of computed feature definitions stored in a feature store (Id., ¶ 61, In embodiments, feature computation layer 106 is configured to determine the features using the raw data and/or events stored to related event store 105. The feature computation layer 106 may be configured to determine the features by applying a variety of numerical processes to the data, such as arithmetic operations, aggregations, and various other techniques. In an embodiment, a user of the system 100 may determine useful features for a model by evaluating the features generated by feature computation layer 106 using both numerical methods and attempts to train a model using the examples generated from these features. By attempting to train the model using the generated examples, the user may see if the model trained using the features of interest has less error, such as by testing the model using a validation set, as compared to the model trained with different features), (Id., ¶ 92, In an embodiment, feature engine 103 includes a feature store 107. (discloses feature store) Feature computation layer 106 may store the determined features and/or generated feature vectors to feature store 107. Feature store 107 makes deployed features available for users. According to an aspect, feature computation layer 106 keeps feature store 107 up-to-date, such as by computing and updating values of features when new events are received and/or when a request is received from a user. Based on the features stored to feature store 107, feature computation layer 106 may avoid recomputing features using the same events. For example, if feature computation layer 106 has determined features using events up to arrival time x, feature computation layer 106 determines features using events up to arrival time x+n by only considering events that arrived after arrival time x and before arrival time x+n), (Id., ¶ 93, According to an aspect, feature computation layer 106 updates the features and/or save the new features to feature store 107. As a result, feature store 107 is configured to make up-to-date query results 113 available on-demand and computed features are readily available for quick model application. A user who wants to use a model trained on a particular exported dataset may efficiently retrieve stored pre-computed values);
in response to determining that the new feature definition is at least partially included in a matched feature definition of the plurality of computed feature definitions, generating an alternative feature definition based on the new feature definition and the matched feature definition (Id., ¶ 61, In embodiments, feature computation layer 106 is configured to determine the features using the raw data and/or events stored to related event store 105. The feature computation layer 106 may be configured to determine the features by applying a variety of numerical processes to the data, such as arithmetic operations, aggregations, and various other techniques. In an embodiment, a user of the system 100 may determine useful features for a model by evaluating the features generated by feature computation layer 106 using both numerical methods and attempts to train a model using the examples generated from these features. By attempting to train the model using the generated examples, the user may see if the model trained using the features of interest has less error, such as by testing the model using a validation set, as compared to the model trained with different features), (Id., ¶ 42, Feature engineering system 100 is configured to use the data from data sources 101,102 to efficiently provide and/or generate feature vectors, such as a predictor feature vector, for a user to use in the application stage. Applying the model may involve computing a feature vector using the same computations that were used in training of the model, but for an entity or time that may not have been part of the training or validation examples. Because feature engineering system 100 is also configured to generate feature vectors for the user to use in the training stage, the same feature (discloses generating new feature definitions) vector definitions that were used for training are automatically available during production. As discussed above, making the same feature vector definitions used for training automatically available during production allows for event-based models to be successfully used in production. For example, feature engineering system 100 may provide and/or generate predictor feature vectors for a user to use in the application stage, while the feature engineering system 100 may provide and/or generate predictor and label feature vectors for a user to use in the training and validation stage. Feature engineering system 100 may generate the feature vectors and/or validation examples in a similar manner as described above for training examples), (Id., ¶ 43, System 100 is configured to ingest event data from one or more sources 101, 102 of data. In some configurations, a data source includes historical data, e.g., from historical data source 101. In that case, the data includes data that was received and/or stored within a historic time period i.e. not real-time. The historical data is typically indicative of events that occurred within a previous time period. For example, the historic time period may be a prior year or a prior two years, e.g., relative to a current time, etc. Historical data source 101 may be stored in and/or retrieved from one or more files, one or more databases, an offline source, and the like or may be streamed from an external source. The historical data ingested by system 100 may be associated with a user of system 100, such as a data scientist, that wants to train and implement a model using features generated from the data. System 100 may ingest the data from one or more sources 101,102 and use it to compute features), (Id., ¶ 120, Once the user has created and/or changed the feature definition and/or example selection, the feature engineering system can use this information to efficiently create the desired features and/or feature vectors and/or examples for the user. For example, the feature engineering system can use this information to create the desired features and/or feature vectors and/or examples for the user by re-using previous computations. After the desired features and/or feature vectors and/or examples have been generated, they may be exported to the user. At 704, the generated features and/or feature vectors and/or examples may be exported to the user. The user may use these exported features and/or feature vectors and/or examples to train and/or validate/evaluate the model. At 706, the user may train the model on any training examples generated by the feature engineering system. At 708, the user may validate and/or evaluate the model using any validation examples generated by the feature engineering system. If the user wants the feature engineering system to generate new or different features and/or feature vectors and/or examples, the user may easily change the dataset being used or experiment with a different dataset. For example, the user may want to try a new dataset to see if the model performs better after being trained with the new dataset. The method 700 may return to step 702, where the user may change the feature definition and/or update the example selection configuration. The user may continue to perform this iterative process until the model is generating results that satisfy the user), (Id., ¶ 121, FIG. 8 shows an example network 800 for feature engineering. The network 800 includes a feature engineering system 802 and one or more clients 804. System 802 may be similar to and/or perform similar functions as those performed by system 100 and/or system 200 described above. System 802 includes an API Server 808, one or more compute nodes 814, metadata storage 810, event data storage 816, staged data storage 806, prepared data storage 812, and result data storage 818. The event data storage 816, the staged data storage 806, and/or the prepared data storage 812 may utilize an external storage system, such as Amazon S3 or any other external storage system. The compute nodes 814 may be, for example, a feature engine, such as one of the feature engines described above);
and executing the selected execution alternative using a compute engine to generate the feature (Id., ¶ 90, Feature engineering system 100 (discloses compute engine) may simplify collaboration in feature generation (discloses generating the feature) and/or selection. As discussed above, features are often defined by users, such as data scientists. A company may have multiple data scientists producing features for one or more models. The data scientists may need to use different tools to access different kinds of raw data and/or events, further complicating the process of producing features. Collaboration on features produced in ad-hoc and varied ways makes it difficult to share features between users and/or projects. In addition, the techniques for producing features may vary based on the data size and the need for producing the feature vectors “in a production environment.” This may lead to the need to implement features multiple times for different situations. However, feature engineering system 100 may address these shortcomings by ingesting and/or saving raw data and/or events from a variety of sources and making the features available to users in different locations and/or using different devices, such as via the feature studio described further herein), (Id., ¶ 91, In an embodiment, feature computation layer 106 is configured to compute feature vectors. A feature vector is a list of features of an entity. The feature computation layer 106 may be configured to compute and/or update feature vectors as events are ingested by the feature engine 103. The feature computation layer 106 may be configured to compute and/or update feature vectors in response to user queries).
While suggested in at least Fig. 2 and related text, Bonaci does not explicitly disclose …selecting an execution alternative from among candidate execution alternatives, using an execution cost estimator that predicts an execution cost for each candidate execution alternative, wherein the candidate execution alternatives include (i) an execution of a PIT join using the alternative feature definition and (ii) an execution of a second PIT join using the new feature definition, the selected execution alternative being selected based on the execution costs predicted by the execution cost estimator;
However, through KSR Rationale D (See MPEP 2141(III)(D)), the combination of Bonaci and Chen discloses …selecting an execution alternative from among candidate execution alternatives, using an execution cost estimator that predicts an execution cost for each candidate execution alternative, wherein the candidate execution alternatives include (i) an execution of a PIT join using the alternative feature definition and (ii) an execution of a second PIT join using the new feature definition, the selected execution alternative being selected based on the execution costs predicted by the execution cost estimator.
First, Bonaci disclose a data aggregation technique including an expense estimation and minimization technique which selects an alternative technique for minimal cost (Bonaci, ¶ 82, According to an aspect, feature computation layer 106 is configured to simultaneously compute more than one feature, such as a large number of features. When simultaneously computing many features, it is possible to compute each feature independently and then join the computed values based on the entity and time. However, this approach is inefficient for at least two major reasons. First, computing each feature may involve retrieving and processing the same input events multiple times. Second, once the features are computed, performing an N-way join is an expensive operation. FIG. 5A illustrates an example N-way join 500a, (discloses PIT join techniques) such as a 3-way join, being performed after multiple features are individually computed. Computing two or more of the three features shown in FIG. 5A may involve retrieving and processing the same input events multiple times. After these three features are individually computed, they may be joined and output by the system), (Id., ¶ 83, Rather than employing this inefficient and expensive technique for simultaneously computing multiple features, feature computation layer 106 (discloses execution cost estimator) may instead combine all of the aggregations into a single pass over events that computes (at each point in time and for each entity) the value of all aggregations. (discloses execution alternative selected for minimal cost) The description of this flattened operation is called the aggregation plan and the process for producing it is described in more detail below. This flattened aggregation plan allows for the simultaneous computation of the aggregations necessary for all requested features with a single pass over the input, and therefore eliminates the need for the N-way join. FIG. 5B illustrates an example simultaneous feature computation 500b without an N-way join. As depicted in FIG. 5B, all of the multiple features are simultaneously computed with a single pass over the input, eliminating the need to retrieve and process the same input events multiple times), (Id., ¶ 157, In embodiments, resume tokens are utilized to continually apply the results of a query to a separate (i.e., external) data store with minimal cost. This may be achieved by first running an initial query, writing the results to the separate data store, and receiving a resume token. A query may be periodically run to update the results in the external store. Each query uses the resume token returned by the previous response. The new results may reflect only those results which have changed).
PNG
media_image1.png
452
647
media_image1.png
Greyscale
Further, Chen discloses time-sensitive PIT join techniques using new and alternative feature definitions (Chen, ¶ 23, For purposes of this document, features are joins between different tables on the same key (e.g., address, timestamp) with perhaps simple transformations applied to some fields (discloses selecting an optimal time-sensitive PIT join alternative) that will make it more suitable for machine learning training and/or service, such as the nullification of values that are obviously incorrect), (Id., ¶ 24, These two concepts allow for the separation of physical mechanisms for managing tables from the logical design of features, while allowing each layer to be iterated on separately. For example, supporting a new table format merely requires the user to implement a common interface, which could immediately be used by a feature layer without any knowledge of the physical details), (Id., ¶ 39, As described above, features are joins between tables FIG. 3 is a diagram illustrating an example of features, in accordance with an example embodiment. Features are what someone building a model would care about. They define a configuration file that references all the fields needed from each of the tables defined.), (Id., ¶ 43, When iterating over a FeatureSet for model training, two commands may be defined: [0044] 1. build_feature_set_input_table—feature-set $FEATURE_SET—path $PATH which, given a FeatureSet definition, will perform all the joins necessary for the input table and can later be iterated on. [0045] 2. build_feature_set—feature-set $FEATURE_SET—input-table $PATH will build the feature set using the input table from 1. This command would fail if the input table does not contain the necessary columns. [0046] These two commands allow a user to iterate over feature set definitions without having to recalculate all joins every time).
PNG
media_image2.png
285
493
media_image2.png
Greyscale
One of ordinary skill in the art would have recognized that applying the known technique of Bonaci would have yielded predictable results and resulted in an improved system. It would have been recognized that applying the cost estimation technique of Bonaci to the feature-definition-based PIT join teachings of Chen would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such data aggregation features into similar systems. Further, applying a cost minimization based alternative selection to Chen with feature definitions considered accordingly, would have been recognized by those of ordinary skill in the art as resulting in an improved system that would allow more cost effective data aggregation according to specific feature definitions.
Thus, through KSR Rationale D (See MPEP 2141(III)(D)), the combination of Bonaci and Chen discloses …selecting an execution alternative from among candidate execution alternatives, using an execution cost estimator that predicts an execution cost for each candidate execution alternative, wherein the candidate execution alternatives include (i) an execution of a PIT join using the alternative feature definition and (ii) an execution of a second PIT join using the new feature definition, the selected execution alternative being selected based on the execution costs predicted by the execution cost estimator.
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have modified the feature definition generation elements of Bonaci to include the PIT join elements of Chen in the analogous art of time sensitive data stores.
The motivation for doing so would have been to implement an improved method “that produces wide tables containing features for machine learned models that allow for more efficient processing, improved sharing of information, and increased accuracy. These wide tables are made available for model training for multiple models and/or groups. These wide tables may be served on a serving database for fast access for API serving and lightweight access during interactive development. The solution decreases the time needed to add a new feature from several days to a couple of hours by enabling experimentation” (Chen, ¶ 21). Such improvements would benefit Bonaci’s method which provide an “ability to maintain feature values in real time [which] may improve the accuracy of the model. For example, the model may be able to make more accurate predictions, or a larger percentage of the predictions that the model makes may be accurate. The accuracy of the model may be improved because predictions made with more recent feature values more accurately reflect the current interests/environments, etc. that the prediction is being made about” [Chen, ¶ 21; Bonaci, ¶ 29].
Regarding Claim 2, the combination of Bonaci and Chen discloses …The computer-implemented method of claim 1…
Bonaci discloses …wherein the execution cost estimator predicts the execution cost for each candidate execution alternative… (Bonaci, ¶ 82, According to an aspect, feature computation layer 106 is configured to simultaneously compute more than one feature, such as a large number of features. When simultaneously computing many features, it is possible to compute each feature independently and then join the computed values based on the entity and time. However, this approach is inefficient for at least two major reasons. First, computing each feature may involve retrieving and processing the same input events multiple times. Second, once the features are computed, performing an N-way join is an expensive operation. FIG. 5A illustrates an example N-way join 500a, (discloses PIT join techniques) such as a 3-way join, being performed after multiple features are individually computed. Computing two or more of the three features shown in FIG. 5A may involve retrieving and processing the same input events multiple times. After these three features are individually computed, they may be joined and output by the system), (Id., ¶ 83, Rather than employing this inefficient and expensive technique for simultaneously computing multiple features, feature computation layer 106 (discloses execution cost estimator) may instead combine all of the aggregations into a single pass over events that computes (at each point in time and for each entity) the value of all aggregations. (discloses execution alternative selected for minimal cost) The description of this flattened operation is called the aggregation plan and the process for producing it is described in more detail below. This flattened aggregation plan allows for the simultaneous computation of the aggregations necessary for all requested features with a single pass over the input, and therefore eliminates the need for the N-way join. FIG. 5B illustrates an example simultaneous feature computation 500b without an N-way join. As depicted in FIG. 5B, all of the multiple features are simultaneously computed with a single pass over the input, eliminating the need to retrieve and process the same input events multiple times), (Id., ¶ 157, In embodiments, resume tokens are utilized to continually apply the results of a query to a separate (i.e., external) data store with minimal cost. This may be achieved by first running an initial query, writing the results to the separate data store, and receiving a resume token. A query may be periodically run to update the results in the external store. Each query uses the resume token returned by the previous response. The new results may reflect only those results which have changed).
While suggested in at least Fig. 2 and related text, Bonaci does not explicitly disclose …further comprising: receiving a plurality of candidate source data layouts that are based on current feature computation pipelines and current source data layout; determining a plurality of candidate source data layouts; and selecting a new data source layout from the plurality of candidate source data layouts that are based on current feature computation pipelines and current source data layout.
However, Chen discloses further comprising: receiving a plurality of candidate source data layouts that are based on current feature computation pipelines and current source data layout (Chen, ¶ 22, For purposes of this document, a table may be defined as a collection of rows held on a structured format with the same schema. It can take multiple physical representations, including, for example, a Postgres table, Parquet file, CSV file, or Big Query Table, and usually contains a key that uniquely identifies each row);
determining a plurality of candidate source data layouts (Chen, ¶ 64, In an example embodiment, in order to reduce the size of the wide table, the feature store stores a set of features pivoted by (address token, list date) for comparables. Thus, subject-comp pairs can be precomputed per distance and stored in a comparables table. In one example embodiment, the comparable data is stored in an array in a wide table. In the wide table, each row of the table is pivoted per (address token, list date) and contains all of the features (hence, why it is called wide). Specifically, it may be a column in a wide table and data can be stored in the format of [(comp_address_token, distance)], where distance is the distance between the comparable and a subject property. Alternatively, it may be stored as a separate table in a flat fashion. Each comparable has a pair of address tokens and a distance. If auxiliary information is stored about comps, instead of a flat pair, the pair may be ordered as (subject_address-token, comp_address_token, comp_information). Alternatively, it may be stored in both a wide table and as a separate table. (discloses determining a plurality of candidate source data layouts) In some example embodiments, the system may filter to the appropriate set based on heuristics, as well as perform the scoring as part of the job hierarchy);
and selecting a new data source layout from the plurality of candidate source data layouts that are based on current feature computation pipelines and current source data layout… that utilizes the selected new data source layout (Id., ¶ 38, It should be noted that the Parquet model was used in this example as the basis format to simplify interaction from the features part. Parquet was chosen as it is a good fit for model training and also has good support for parallelization, which than then be easily transformed to any serving store, such as Postgre, Cassandra, and Base), (Id., ¶ 22, For purposes of this document, a table may be defined as a collection of rows held on a structured format with the same schema. It can take multiple physical representations, including, for example, a Postgres table, Parquet file, CSV file, or Big Query Table, and usually contains a key that uniquely identifies each row).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have modified the feature definition generation elements of Bonaci to include format determination elements of Chen in the analogous art of time sensitive data stores for the same reasons as stated for claim 1.
Regarding Claim 3, the combination of Bonaci and Chen discloses …The computer-implemented method of claim 2…
Through KSR Rationale C (See MPEP 2141(III)(C)), the combination of Bonaci and Chen discloses …wherein selecting a new data source layout further comprising evaluating the plurality of candidate source data layouts and the current source data layout based on a layout selection criterion, wherein the layout selection criterion comprises selection of a minimum cost configuration of the new data source layout.
First, Bonaci discloses considerations of cost minimization with respect to data structuring (Bonaci, ¶ 157, In embodiments, resume tokens are utilized to continually apply the results of a query to a separate (i.e., external) data store with minimal cost. This may be achieved by first running an initial query, writing the results to the separate data store, and receiving a resume token. A query may be periodically run to update the results in the external store. Each query uses the resume token returned by the previous response. The new results may reflect only those results which have changed), (Id., ¶ 159, In embodiments, the system 802 may be configured to perform temporally correct joins, such as with foreign entities. A value at a point in time is temporally correct if it includes all of the events up to (and including) that point in time and none of the events after that point in time. The result of any computation may thus be a sequence of values corresponding to the temporally correct value at each point in time. By contrast, many other data processing systems instead operate on all of the data (events) in the system. This may result in the correct values at a time after all of the events. However, due to delays that occur between when events happen and when they are added to the system, this may not result in a correct value at any given point in time), (Id., ¶ 160, Being able to compute values that are correct at historic points in time, as the system 802 is able to do, is critical to creating features that may be used to train predictive models without leakage. Rather than representing the value at every point in time, the system 802 may represent only those values that are observed, such as those values that are returned as part of the results, used in additional computations, etc. The system 802 may represent the value only at the points in time when it changes. For example, the computation “sum(Event.amount)” may only change when an event occurs).
Further, Chen discloses selecting a source data layout based on optimization criterion (Chen, ¶ 22, For purposes of this document, a table may be defined as a collection of rows held on a structured format with the same schema. It can take multiple physical representations, including, for example, a Postgres table, Parquet file, CSV file, or Big Query Table, and usually contains a key that uniquely identifies each row), (Id., ¶ 64, In an example embodiment, in order to reduce the size of the wide table, the feature store stores a set of features pivoted by (address token, list date) for comparables. Thus, subject-comp pairs can be precomputed per distance and stored in a comparables table. In one example embodiment, the comparable data is stored in an array in a wide table. In the wide table, each row of the table is pivoted per (address token, list date) and contains all of the features (hence, why it is called wide). Specifically, it may be a column in a wide table and data can be stored in the format of [(comp_address_token, distance)], where distance is the distance between the comparable and a subject property. Alternatively, it may be stored as a separate table in a flat fashion. Each comparable has a pair of address tokens and a distance. If auxiliary information is stored about comps, instead of a flat pair, the pair may be ordered as (subject_address-token, comp_address_token, comp_information). Alternatively, it may be stored in both a wide table and as a separate table. (discloses determining a plurality of candidate source data layouts) In some example embodiments, the system may filter to the appropriate set based on heuristics, as well as perform the scoring as part of the job hierarchy), (Id., ¶ 38, It should be noted that the Parquet model was used in this example as the basis format to simplify interaction from the features part. Parquet was chosen as it is a good fit for model training and also has good support for parallelization, which than then be easily transformed to any serving store, such as Postgre, Cassandra, and Base).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have selected a source data layout based on cost minimization criterion as in the improvement discussed in Bonaci in the system executing the method of Chen. As in Chen, it is within the capabilities of one of ordinary skill in the art to provide temporally correct join operations to create data tables using Bonaci’s cost minimization considerations with the predicted result of providing accurate, useful and timely information to the end user.
Thus, through KSR Rationale C (See MPEP 2141(III)(C)), the combination of Bonaci and Chen discloses …wherein selecting a new data source layout further comprising evaluating the plurality of candidate source data layouts and the current source data layout based on a layout selection criterion, wherein the layout selection criterion comprises selection of a minimum cost configuration of the new data source layout.
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have modified the feature definition generation elements of Bonaci to include format determination elements of Chen in the analogous art of time sensitive data stores for the same reasons as stated for claim 1.
Regarding Claim 5, the combination of Bonaci and Chen discloses …The computer-implemented method of claim 1…
While suggested in at least Fig. 2 and related text, Bonaci does not explicitly disclose …wherein selecting the execution alternative further comprises evaluating, using a feature selection criterion one or more of the alternative feature definition and the new feature definition.
However, Chen discloses …wherein selecting the execution alternative further comprises evaluating, using a feature selection criterion one or more of the alternative feature definition and the new feature definition (Chen, ¶ 44, 1. build_feature_set_input_table—feature-set $FEATURE_SET—path $PATH which, given a FeatureSet definition, will perform all the joins necessary for the input table and can later be iterated on), (Id., ¶ 46, These two commands allow a user to iterate over feature set definitions without having to recalculate all joins every time), (Id., ¶ 60, At 516, add calculated features is performed. The calculated features are any features that potentially depend on information that is not possessed by the system until a customer visits a corresponding web site. For example, while the system may have an estimate of home square footage, the seller may provide a more accurate estimate when requesting a price estimate from the system. Calculated features run at the time that a prediction of home value is made so they can included updated information provided during the year), (Id., ¶ 63, It should be noted that the same mechanisms described above, including the feature stores and wide table, can be performed for all comparable properties as well as just the subject property. With comparables, however, it is difficult to precompute these values since there is no time yet specified in a query to guarantee against future leakage. For example, if one wanted to produce an estimate for a subject property based on comparables that occurred before the last time the subject property was sold (for example, 2012), it is difficult to precompute features for those comparables before that time is known (e.g., until the user specifies 2012 in the query, the system does not know to limit the data from comparables to 2012 or earlier). As such, in an example embodiment, features for all possible comparables for all times, based solely on distance from a subject property, for each possible subject property, can be precomputed. While this greatly improves performance at query-time, the result is more data than will fit on a single machine, and this type of computation is difficult to perform in a distributed fashion).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have modified the feature definition generation elements of Bonaci to include the selection criterion elements of Chen in the analogous art of time sensitive data stores for the same reasons as stated for claim 1.
Regarding Claim 6, the combination of Bonaci and Chen discloses …The computer-implemented method of claim 5…
Bonaci further discloses …wherein the feature selection criterion comprises minimization of data to be scanned using one or more of the alternative feature definition and the new feature definition (Bonaci, ¶ 149, In embodiments, system 802 may process all late data regardless of actual delay. Doing so in a resumable query may use any eligible intermediate state. An intermediate state is eligible if the latest event it includes is before the earliest new event. Resuming computation from such a state ensures events are processed in order, since no events later than any of the new events have yet been processed. The best eligible intermediate state may be the one that minimizes the number of events that need to be processed. The best eligible intermediate state may be determined by choosing the state with the maximum event time less than the latest new data point).
Regarding Claim 7, the combination of Bonaci and Chen discloses …The computer-implemented method of claim 6…
Bonaci further discloses …wherein the minimization of data to be scanned further comprises calculating a benefit based on a number of data partitions to be read by the execution of a third PIT join using the alternative feature definition and a number of data partitions to be read by the execution of the second PIT join using the new feature definition(Bonaci, ¶ 149, In embodiments, system 802 may process all late data regardless of actual delay. Doing so in a resumable query may use any eligible intermediate state. An intermediate state is eligible if the latest event it includes is before the earliest new event. Resuming computation from such a state ensures events are processed in order, since no events later than any of the new events have yet been processed. The best eligible intermediate state may be the one that minimizes the number of events that need to be processed. The best eligible intermediate state may be determined by choosing the state with the maximum event time less than the latest new data point), (Id., ¶ 152, Referring back to FIG. 8, the ability of the system 802 to handle late data while immediately producing results reflecting all received events and its ability to resume computations with minimal need to reprocess prior events are important for handling late data. As an example, many stream processing systems assume that late data may be bounded. Such stream processing systems may require users to configure a maximum expected delay and/or may only process events older than this maximum delay. They may discard any events that exceed the maximum lateness. All of these are undesirable features that the system 802 remedies), (Id., ¶ 161, A “temporally correct join” is a join that produces the correct value at every point in time. A lookup is one mechanism for performing a join. To be temporally correct, a lookup must use the temporally correct key to determine the foreign entity to lookup from and it must use the temporally correct value for the foreign entity. Performing a temporally correct join may require a temporal processing engine which can compute the correct values at specific points in time).
Regarding Claim 8, the combination of Bonaci and Chen discloses …The computer-implemented method of claim 6…
Bonaci further discloses …wherein the minimization of data to be scanned further comprises calculating a benefit based on a size of data not to be read by the execution of a third PIT join using the alternative feature definition and a size of data not to be read by the execution of the second PIT join using the new feature definition (Bonaci, , ¶ 149, In embodiments, system 802 may process all late data regardless of actual delay. Doing so in a resumable query may use any eligible intermediate state. An intermediate state is eligible if the latest event it includes is before the earliest new event. Resuming computation from such a state ensures events are processed in order, since no events later than any of the new events have yet been processed. The best eligible intermediate state may be the one that minimizes the number of events that need to be processed. The best eligible intermediate state may be determined by choosing the state with the maximum event time less than the latest new data point), (Id., ¶ 152, Referring back to FIG. 8, the ability of the system 802 to handle late data while immediately producing results reflecting all received events and its ability to resume computations with minimal need to reprocess prior events are important for handling late data. As an example, many stream processing systems assume that late data may be bounded. Such stream processing systems may require users to configure a maximum expected delay and/or may only process events older than this maximum delay. They may discard any events that exceed the maximum lateness. All of these are undesirable features that the system 802 remedies), (Id., ¶ 136, In embodiments, data slices may be used to select a random or pseudo-random sample of the entities. This may be used when iterating on feature engineering to reduce the total data set size being queried. This is more ideal than a solution that just takes a random sample of the events, because each of the selected entities has a complete set of events. Because each of the selected entities has a complete set of events, the feature values computed for them would be the same for the sampled data slice and on the entire data set. The selection of a random sample may use computed values. For instance, a sample of 1000 entities that are representatively distributed by age group may be requested by configuring a data slice that is sampled proportionally to the age groups in the entire data set. If a given age group represents 20% of the data, then there would be 200 entities in the produced sample), (Id., ¶ 143, Queries for the results since a previous resume token may return significantly smaller sets of results than a complete query. Rows which were previously returned may be omitted. Rows with values that have not changed since they were previously returned may also be omitted. This smaller result size may be faster to load into a storage system for serving feature values. Queries for the results since a previous page token may additionally, or alternatively, require significantly less compute time. This may be accomplished by storing intermediate states from the previous computation reflecting some or all of the events previously processed. When a query with a resume token is received, the intermediate state(s) from an earlier query may be used instead of reprocessing the corresponding events. This may allow the query to process only the new input since the previous query, rather than all of the input. In long running systems, it may quickly be the case that all previously accumulated data is significantly larger than the data arriving in any time interval, so this will often significantly speed up the queries).
Regarding Claim 9, the combination of Bonaci and Chen discloses …The computer-implemented method of claim 2…
Bonaci further discloses …further comprising generating the plurality of candidate source data layouts further comprising: retrieving the plurality of computed feature definitions stored in a feature store (Id., ¶ 92, In an embodiment, feature engine 103 includes a feature store 107. Feature computation layer 106 may store the determined features and/or generated feature vectors to feature store 107. Feature store 107 makes deployed features available for users. According to an aspect, feature computation layer 106 keeps feature store 107 up-to-date, such as by computing and updating values of features when new events are received and/or when a request is received from a user. Based on the features stored to feature store 107, feature computation layer 106 may avoid recomputing features using the same events. For example, if feature computation layer 106 has determined features using events up to arrival time x, feature computation layer 106 determines features using events up to arrival time x+n by only considering events that arrived after arrival time x and before arrival time x+n);
extracting data sources used to compute the plurality of computed feature definitions stored in a feature store (Id., ¶ 93, According to an aspect, feature computation layer 106 updates the features and/or save the new features to feature store 107. As a result, feature store 107 is configured to make up-to-date query results 113 available on-demand and computed features are readily available for quick model application. A user who wants to use a model trained on a particular exported dataset may efficiently retrieve stored pre-computed values);
and partitioning each of the extracted data sources based on a predetermined granularity of time period (Id., ¶ 86, In an embodiment, it may be desirable for feature computation layer 106 to operate on a sample of data. If feature computation layer 106 can operate on a sample of data, quick, approximate answers can be provided in response to interactive queries. To make the sampling informative, complete information for a subset of entities is included, rather than a subset of events for every entity. Without lookups, this sampling can be accomplished by taking only those events related to a subset of the entities. If the events are partitioned by entity, this could be accomplished by considering only a subset of the partitions. With lookups it is necessary to make sure that all events referenced by the sampled primary entities are available. This can be done by computing the lookup keys that the primary entity sample will need (at the selected point(s) in time) and using that set of keys as the sample of foreign entity events. While generating this sample may require filtering events from all partitions, it may be reused as features are changed so long as the definition of the lookup key does not change. In practice, the lookup key tends to change less frequently than other parts of the feature definitions, so this kind of sampling is likely to improve the performance of interactive queries), (Id., ¶ 89, The techniques discussed above allow feature engineering system 100 to maintain live feature values. Specifically, the techniques discussed above allow feature engine 103 to compute feature values using a partitioned scan over historic events. This allows exporting feature vectors and/or examples computed over the historic data in an efficient manner. Once the feature vectors and/or examples have been produced, feature engine 103 may also be configured to maintain “live” feature values which may be retrieved for a time near the current time for use when applying the model. In an embodiment, this online maintenance is achieved by storing the final accumulator values produced during the export. At any point in time the “new” events may be treated as individual rows or a batch of rows and new accumulators (and feature values) may be produced), (Id., ¶ 37, Similarly, the user may configure the selection of corresponding label times used to generate the training examples for the event-based model in a variety of different ways. In an embodiment, the user may configure the label times to be selected at fixed times. The fixed time may be, for example, today, or on the 1st of a month, or any other fixed time. (discloses fixed time granularities) In another embodiment, the user may configure the label times to be selected at fixed offset times after the prediction times. For example, as discussed above, if an event-based model is to predict whether an individual will quit a subscription service within the next month, the user may configure the label times to be selected at the points-in-time that occur one month after the respective prediction time(s). In another embodiment, the user may configure the label times to be selected when a particular event occurs. For example, as discussed above, if an event-based model is to predict, when a house is listed for sale, how much that house will eventually sell for, then the user may configure the label times to be selected at those points-in-time at which houses eventually sell. In another embodiment, the user may configure the label times to be selected at computed times. For example, if an event-based model is to predict whether scheduled flights will depart on time, then the label times may be configured to be selected at points-in-time calculated to be the scheduled departure times. The user of system 100 understands its own data and the problem that needs to be solved, so the user of system 100 may be best equipped to define the manner in which the prediction time(s) and corresponding label time(s) should be selected by system 100).
Regarding Claim 10, the combination of Bonaci and Chen discloses …The computer-implemented method of claim 9…
Bonaci further discloses …wherein the predetermined granularity of time period may be at least one of a month, a day, an hour, and a minute (Bonaci, ¶ 37, Similarly, the user may configure the selection of corresponding label times used to generate the training examples for the event-based model in a variety of different ways. In an embodiment, the user may configure the label times to be selected at fixed times. The fixed time may be, for example, today, or on the 1st of a month, or any other fixed time. (discloses fixed time granularity of one month) In another embodiment, the user may configure the label times to be selected at fixed offset times after the prediction times. For example, as discussed above, if an event-based model is to predict whether an individual will quit a subscription service within the next month, the user may configure the label times to be selected at the points-in-time that occur one month after the respective prediction time(s). In another embodiment, the user may configure the label times to be selected when a particular event occurs. For example, as discussed above, if an event-based model is to predict, when a house is listed for sale, how much that house will eventually sell for, then the user may configure the label times to be selected at those points-in-time at which houses eventually sell. In another embodiment, the user may configure the label times to be selected at computed times. For example, if an event-based model is to predict whether scheduled flights will depart on time, then the label times may be configured to be selected at points-in-time calculated to be the scheduled departure times. The user of system 100 understands its own data and the problem that needs to be solved, so the user of system 100 may be best equipped to define the manner in which the prediction time(s) and corresponding label time(s) should be selected by system 100).
Regarding Claim 11, the combination of Bonaci and Chen discloses …The computer-implemented method of claim 1…
Bonaci further discloses …wherein the plurality of computed feature definitions are determined using PIT joins (Bonaci, ¶ 72, In an embodiment, in addition to aggregations over related events, computing each feature includes zero or more lookups of values computed over other sets of events. For example, if the features are computed over events performed by user entities it may be useful to lookup properties computed from events relating to specific videos. In this case, the features computed from events related to users are “lookup” values computed from events related to videos. This “lookup” operation provides similar capabilities to a join operation), (Id., ¶ 73, If feature computation layer 106 is configured to operate over all of the input events for both the primary entity and the foreign entity, feature computation layer 106 could simultaneously compute all the necessary aggregations. While this is conceptually how temporal aggregations with lookups behave, feature computation layer 106 performs this in a partitioned and potentially distributed manner. Without lookups, temporal aggregations may be executed entirely partitioned by entity. When executing temporal joins (disclose point-in-time joins) across multiple partitions, any lookup may request data from any other entity, and therefore any other partition, thus requiring some mechanism for cross-partition communication), (Id., ¶ 66, According to an aspect, feature computation layer 106 is configured to compute features by performing aggregations across events associated with an entity. Computing features from large amounts of raw data is a technically complicated process, as it may involve computing aggregate properties across all of the raw data. In an embodiment, feature computation layer 106 is configured to compute event-based features by performing temporal aggregations across events associated with an entity. To perform temporal aggregations, feature computation layer 106 produces a feature value at every time, aggregating all of the events that happened up to that particular time. Feature computation layer 106 does not aggregate everything and produce a single value—this would prevent the feature computation layer 106 from determining how the feature value changed over time. It is important that feature vectors and/or examples reflect the real feature values that will be available when applying the model as closely as possible. For this reason, if the model is being applied to “live” feature values (computed over all the events up to that point in time), each feature vectors and/or example should also be computed over the events up to the point in time selected for that example).
Regarding Claim 12, Bonaci discloses …One or more physically manufactured computer-readable storage media, encoding computer-executable instructions for executing on a computer system a computer process, the computer process comprising: receiving a new feature definition specifiying parameters of the feature (Bonaci, ¶ 32, In an embodiment, feature engineering system 100 may be configured to generate feature vectors and/or examples associated with a particular entity. As is discussed below in more detail, a user of system 100, such as a data scientist, may be responsible for instructing system 100 which entity or entities should be included in the feature vectors and/or examples. For example, if the user of system 100 wants to train a model to predict how much homes will sell for in Seattle, the user of system 100 may instruct system 100 to choose houses in Seattle as the entities that should be included in the feature vectors (discloses receiving a new feature definition specifying parameters of the feature) and/or examples. If the user instructed system 100 to choose, for example, houses in Los Angeles as the set of entities that should be included in the feature vectors and/or examples, the model may not be able to accurately predict selling prices for homes in Seattle), (Id., ¶ 120, Once the user has created and/or changed the feature definition and/or example selection, the feature engineering system can use this information to efficiently create the desired features and/or feature vectors and/or examples for the user. For example, the feature engineering system can use this information to create the desired features and/or feature vectors and/or examples for the user by re-using previous computations. After the desired features and/or feature vectors and/or examples have been generated, they may be exported to the user. At 704, the generated features and/or feature vectors and/or examples may be exported to the user. The user may use these exported features and/or feature vectors and/or examples to train and/or validate/evaluate the model. At 706, the user may train the model on any training examples generated by the feature engineering system. At 708, the user may validate and/or evaluate the model using any validation examples generated by the feature engineering system. If the user wants the feature engineering system to generate new or different features and/or feature vectors and/or examples, the user may easily change the dataset being used or experiment with a different dataset. For example, the user may want to try a new dataset to see if the model performs better after being trained with the new dataset. The method 700 may return to step 702, where the user may change the feature definition and/or update the example selection configuration. The user may continue to perform this iterative process until the model is generating results that satisfy the user), (Id., ¶ 217, The system memory 1928 in FIG. 19 may include computer system readable media in the form of volatile memory, such as random access memory (‘RAM’) 1930 and/or cache memory 1932. Computing node 1900 may further include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only, a storage system 1934 may be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a “hard drive”). Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk, e.g., a “floppy disk,” and an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media may be provided. In such instances, each may be connected to bus 1918 by one or more data media interfaces. As will be further depicted and described below, memory 1928 may include at least one program product having a set, e.g., at least one, of program modules that are configured to carry Gut the functions of embodiments of the invention);
comparing the new feature definition with a plurality of computed feature definitions stored in a feature store (Id., ¶ 61, In embodiments, feature computation layer 106 is configured to determine the features using the raw data and/or events stored to related event store 105. The feature computation layer 106 may be configured to determine the features by applying a variety of numerical processes to the data, such as arithmetic operations, aggregations, and various other techniques. In an embodiment, a user of the system 100 may determine useful features for a model by evaluating the features generated by feature computation layer 106 using both numerical methods and attempts to train a model using the examples generated from these features. By attempting to train the model using the generated examples, the user may see if the model trained using the features of interest has less error, such as by testing the model using a validation set, as compared to the model trained with different features), (Id., ¶ 92, In an embodiment, feature engine 103 includes a feature store 107. (discloses feature store) Feature computation layer 106 may store the determined features and/or generated feature vectors to feature store 107. Feature store 107 makes deployed features available for users. According to an aspect, feature computation layer 106 keeps feature store 107 up-to-date, such as by computing and updating values of features when new events are received and/or when a request is received from a user. Based on the features stored to feature store 107, feature computation layer 106 may avoid recomputing features using the same events. For example, if feature computation layer 106 has determined features using events up to arrival time x, feature computation layer 106 determines features using events up to arrival time x+n by only considering events that arrived after arrival time x and before arrival time x+n), (Id., ¶ 93, According to an aspect, feature computation layer 106 updates the features and/or save the new features to feature store 107. As a result, feature store 107 is configured to make up-to-date query results 113 available on-demand and computed features are readily available for quick model application. A user who wants to use a model trained on a particular exported dataset may efficiently retrieve stored pre-computed values);
in response to determining that the new feature definition is at least partially included in a matched feature definition of the plurality of computed feature definitions, generating one or more alternative feature definitions based on the new feature definition and the matched feature definition (Id., ¶ 42, Feature engineering system 100 is configured to use the data from data sources 101,102 to efficiently provide and/or generate feature vectors, such as a predictor feature vector, for a user to use in the application stage. Applying the model may involve computing a feature vector using the same computations that were used in training of the model, but for an entity or time that may not have been part of the training or validation examples. Because feature engineering system 100 is also configured to generate feature vectors for the user to use in the training stage, the same feature (discloses generating new feature definitions) vector definitions that were used for training are automatically available during production. As discussed above, making the same feature vector definitions used for training automatically available during production allows for event-based models to be successfully used in production. For example, feature engineering system 100 may provide and/or generate predictor feature vectors for a user to use in the application stage, while the feature engineering system 100 may provide and/or generate predictor and label feature vectors for a user to use in the training and validation stage. Feature engineering system 100 may generate the feature vectors and/or validation examples in a similar manner as described above for training examples), (Id., ¶ 43, System 100 is configured to ingest event data from one or more sources 101, 102 of data. In some configurations, a data source includes historical data, e.g., from historical data source 101. In that case, the data includes data that was received and/or stored within a historic time period i.e. not real-time. The historical data is typically indicative of events that occurred within a previous time period. For example, the historic time period may be a prior year or a prior two years, e.g., relative to a current time, etc. Historical data source 101 may be stored in and/or retrieved from one or more files, one or more databases, an offline source, and the like or may be streamed from an external source. The historical data ingested by system 100 may be associated with a user of system 100, such as a data scientist, that wants to train and implement a model using features generated from the data. System 100 may ingest the data from one or more sources 101,102 and use it to compute features), (Id., ¶ 120, Once the user has created and/or changed the feature definition and/or example selection, the feature engineering system can use this information to efficiently create the desired features and/or feature vectors and/or examples for the user. For example, the feature engineering system can use this information to create the desired features and/or feature vectors and/or examples for the user by re-using previous computations. After the desired features and/or feature vectors and/or examples have been generated, they may be exported to the user. At 704, the generated features and/or feature vectors and/or examples may be exported to the user. The user may use these exported features and/or feature vectors and/or examples to train and/or validate/evaluate the model. At 706, the user may train the model on any training examples generated by the feature engineering system. At 708, the user may validate and/or evaluate the model using any validation examples generated by the feature engineering system. If the user wants the feature engineering system to generate new or different features and/or feature vectors and/or examples, the user may easily change the dataset being used or experiment with a different dataset. For example, the user may want to try a new dataset to see if the model performs better after being trained with the new dataset. The method 700 may return to step 702, where the user may change the feature definition and/or update the example selection configuration. The user may continue to perform this iterative process until the model is generating results that satisfy the user), (Id., ¶ 121, FIG. 8 shows an example network 800 for feature engineering. The network 800 includes a feature engineering system 802 and one or more clients 804. System 802 may be similar to and/or perform similar functions as those performed by system 100 and/or system 200 described above. System 802 includes an API Server 808, one or more compute nodes 814, metadata storage 810, event data storage 816, staged data storage 806, prepared data storage 812, and result data storage 818. The event data storage 816, the staged data storage 806, and/or the prepared data storage 812 may utilize an external storage system, such as Amazon S3 or any other external storage system. The compute nodes 814 may be, for example, a feature engine, such as one of the feature engines described above);
and executing the selected execution alternative using a compute engine to generate the feature (Id., ¶ 90, Feature engineering system 100 (discloses compute engine) may simplify collaboration in feature generation (discloses generating the feature) and/or selection. As discussed above, features are often defined by users, such as data scientists. A company may have multiple data scientists producing features for one or more models. The data scientists may need to use different tools to access different kinds of raw data and/or events, further complicating the process of producing features. Collaboration on features produced in ad-hoc and varied ways makes it difficult to share features between users and/or projects. In addition, the techniques for producing features may vary based on the data size and the need for producing the feature vectors “in a production environment.” This may lead to the need to implement features multiple times for different situations. However, feature engineering system 100 may address these shortcomings by ingesting and/or saving raw data and/or events from a variety of sources and making the features available to users in different locations and/or using different devices, such as via the feature studio described further herein), (Id., ¶ 91, In an embodiment, feature computation layer 106 is configured to compute feature vectors. A feature vector is a list of features of an entity. The feature computation layer 106 may be configured to compute and/or update feature vectors as events are ingested by the feature engine 103. The feature computation layer 106 may be configured to compute and/or update feature vectors in response to user queries).
While suggested in at least Fig. 2 and related text, Bonaci does not explicitly disclose …selecting an execution alternative from among candidate execution alternatives, using an execution cost estimator that predicts an execution cost for each candidate execution alternative, wherein the candidate execution alternatives include (i) an execution of a PIT join using the alternative feature definition and (ii) an execution of a second PIT join using the new feature definition, the selected execution alternative being selected based on the execution costs predicted by the execution cost estimator;
However, through KSR Rationale D (See MPEP 2141(III)(D)), the combination of Bonaci and Chen discloses …selecting an execution alternative from among candidate execution alternatives, using an execution cost estimator that predicts an execution cost for each candidate execution alternative, wherein the candidate execution alternatives include (i) an execution of a PIT join using the alternative feature definition and (ii) an execution of a second PIT join using the new feature definition, the selected execution alternative being selected based on the execution costs predicted by the execution cost estimator.
First, Bonaci disclose a data aggregation technique including an expense estimation and minimization technique which selects an alternative technique for minimal cost (Bonaci, ¶ 82, According to an aspect, feature computation layer 106 is configured to simultaneously compute more than one feature, such as a large number of features. When simultaneously computing many features, it is possible to compute each feature independently and then join the computed values based on the entity and time. However, this approach is inefficient for at least two major reasons. First, computing each feature may involve retrieving and processing the same input events multiple times. Second, once the features are computed, performing an N-way join is an expensive operation. FIG. 5A illustrates an example N-way join 500a, (discloses PIT join techniques) such as a 3-way join, being performed after multiple features are individually computed. Computing two or more of the three features shown in FIG. 5A may involve retrieving and processing the same input events multiple times. After these three features are individually computed, they may be joined and output by the system), (Id., ¶ 83, Rather than employing this inefficient and expensive technique for simultaneously computing multiple features, feature computation layer 106 (discloses execution cost estimator) may instead combine all of the aggregations into a single pass over events that computes (at each point in time and for each entity) the value of all aggregations. (discloses execution alternative selected for minimal cost) The description of this flattened operation is called the aggregation plan and the process for producing it is described in more detail below. This flattened aggregation plan allows for the simultaneous computation of the aggregations necessary for all requested features with a single pass over the input, and therefore eliminates the need for the N-way join. FIG. 5B illustrates an example simultaneous feature computation 500b without an N-way join. As depicted in FIG. 5B, all of the multiple features are simultaneously computed with a single pass over the input, eliminating the need to retrieve and process the same input events multiple times), (Id., ¶ 157, In embodiments, resume tokens are utilized to continually apply the results of a query to a separate (i.e., external) data store with minimal cost. This may be achieved by first running an initial query, writing the results to the separate data store, and receiving a resume token. A query may be periodically run to update the results in the external store. Each query uses the resume token returned by the previous response. The new results may reflect only those results which have changed).
PNG
media_image1.png
452
647
media_image1.png
Greyscale
Further, Chen discloses time-sensitive PIT join techniques using new and alternative feature definitions (Chen, ¶ 23, For purposes of this document, features are joins between different tables on the same key (e.g., address, timestamp) with perhaps simple transformations applied to some fields (discloses selecting an optimal time-sensitive PIT join alternative) that will make it more suitable for machine learning training and/or service, such as the nullification of values that are obviously incorrect), (Id., ¶ 24, These two concepts allow for the separation of physical mechanisms for managing tables from the logical design of features, while allowing each layer to be iterated on separately. For example, supporting a new table format merely requires the user to implement a common interface, which could immediately be used by a feature layer without any knowledge of the physical details), (Id., ¶ 39, As described above, features are joins between tables FIG. 3 is a diagram illustrating an example of features, in accordance with an example embodiment. Features are what someone building a model would care about. They define a configuration file that references all the fields needed from each of the tables defined.), (Id., ¶ 43, When iterating over a FeatureSet for model training, two commands may be defined: [0044] 1. build_feature_set_input_table—feature-set $FEATURE_SET—path $PATH which, given a FeatureSet definition, will perform all the joins necessary for the input table and can later be iterated on. [0045] 2. build_feature_set—feature-set $FEATURE_SET—input-table $PATH will build the feature set using the input table from 1. This command would fail if the input table does not contain the necessary columns. [0046] These two commands allow a user to iterate over feature set definitions without having to recalculate all joins every time).
PNG
media_image2.png
285
493
media_image2.png
Greyscale
One of ordinary skill in the art would have recognized that applying the known technique of Bonaci would have yielded predictable results and resulted in an improved system. It would have been recognized that applying the cost estimation technique of Bonaci to the feature-definition-based PIT join teachings of Chen would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such data aggregation features into similar systems. Further, applying a cost minimization based alternative selection to Chen with feature definitions considered accordingly, would have been recognized by those of ordinary skill in the art as resulting in an improved system that would allow more cost effective data aggregation according to specific feature definitions.
Thus, through KSR Rationale D (See MPEP 2141(III)(D)), the combination of Bonaci and Chen discloses …selecting an execution alternative from among candidate execution alternatives, using an execution cost estimator that predicts an execution cost for each candidate execution alternative, wherein the candidate execution alternatives include (i) an execution of a PIT join using the alternative feature definition and (ii) an execution of a second PIT join using the new feature definition, the selected execution alternative being selected based on the execution costs predicted by the execution cost estimator.
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have modified the feature definition generation elements of Bonaci to include the PIT join elements of Chen in the analogous art of time sensitive data stores for the same reasons as stated for claim 1.
Regarding Claims 13-15, these claims recite limitations substantially similar to those in claims 6-8, respectively, and are rejected for the same reasons as stated above.
Regarding Claims 16-17, these claims recite limitations substantially similar to those in claims 2 and 9, respectively, and are rejected for the same reasons as stated above.
Regarding Claim 18, Bonaci discloses … A system comprising: memory; one or more processor units; a feature store data preparation optimization system stored in the memory and executable by the one or more processor units, the feature store data preparation optimization system encoding computer-executable instructions on the memory for executing on the one or more processor units a computer process, the computer process comprising: receiving a new feature definition specifying parameters of the feature (Bonaci, ¶ 32, In an embodiment, feature engineering system 100 may be configured to generate feature vectors and/or examples associated with a particular entity. As is discussed below in more detail, a user of system 100, such as a data scientist, may be responsible for instructing system 100 which entity or entities should be included in the feature vectors and/or examples. For example, if the user of system 100 wants to train a model to predict how much homes will sell for in Seattle, the user of system 100 may instruct system 100 to choose houses in Seattle as the entities that should be included in the feature vectors (discloses receiving a new feature definition specifying parameters of the feature) and/or examples. If the user instructed system 100 to choose, for example, houses in Los Angeles as the set of entities that should be included in the feature vectors and/or examples, the model may not be able to accurately predict selling prices for homes in Seattle), (Id., ¶ 120, Once the user has created and/or changed the feature definition and/or example selection, the feature engineering system can use this information to efficiently create the desired features and/or feature vectors and/or examples for the user. For example, the feature engineering system can use this information to create the desired features and/or feature vectors and/or examples for the user by re-using previous computations. After the desired features and/or feature vectors and/or examples have been generated, they may be exported to the user. At 704, the generated features and/or feature vectors and/or examples may be exported to the user. The user may use these exported features and/or feature vectors and/or examples to train and/or validate/evaluate the model. At 706, the user may train the model on any training examples generated by the feature engineering system. At 708, the user may validate and/or evaluate the model using any validation examples generated by the feature engineering system. If the user wants the feature engineering system to generate new or different features and/or feature vectors and/or examples, the user may easily change the dataset being used or experiment with a different dataset. For example, the user may want to try a new dataset to see if the model performs better after being trained with the new dataset. The method 700 may return to step 702, where the user may change the feature definition and/or update the example selection configuration. The user may continue to perform this iterative process until the model is generating results that satisfy the user), (Id., ¶ 217, The system memory 1928 in FIG. 19 may include computer system readable media in the form of volatile memory, such as random access memory (‘RAM’) 1930 and/or cache memory 1932. Computing node 1900 may further include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only, a storage system 1934 may be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a “hard drive”). Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk, e.g., a “floppy disk,” and an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media may be provided. In such instances, each may be connected to bus 1918 by one or more data media interfaces. As will be further depicted and described below, memory 1928 may include at least one program product having a set, e.g., at least one, of program modules that are configured to carry Gut the functions of embodiments of the invention);
comparing the new feature definition with a plurality of computed feature definitions stored in a feature store (Id., ¶ 61, In embodiments, feature computation layer 106 is configured to determine the features using the raw data and/or events stored to related event store 105. The feature computation layer 106 may be configured to determine the features by applying a variety of numerical processes to the data, such as arithmetic operations, aggregations, and various other techniques. In an embodiment, a user of the system 100 may determine useful features for a model by evaluating the features generated by feature computation layer 106 using both numerical methods and attempts to train a model using the examples generated from these features. By attempting to train the model using the generated examples, the user may see if the model trained using the features of interest has less error, such as by testing the model using a validation set, as compared to the model trained with different features), (Id., ¶ 92, In an embodiment, feature engine 103 includes a feature store 107. (discloses feature store) Feature computation layer 106 may store the determined features and/or generated feature vectors to feature store 107. Feature store 107 makes deployed features available for users. According to an aspect, feature computation layer 106 keeps feature store 107 up-to-date, such as by computing and updating values of features when new events are received and/or when a request is received from a user. Based on the features stored to feature store 107, feature computation layer 106 may avoid recomputing features using the same events. For example, if feature computation layer 106 has determined features using events up to arrival time x, feature computation layer 106 determines features using events up to arrival time x+n by only considering events that arrived after arrival time x and before arrival time x+n), (Id., ¶ 93, According to an aspect, feature computation layer 106 updates the features and/or save the new features to feature store 107. As a result, feature store 107 is configured to make up-to-date query results 113 available on-demand and computed features are readily available for quick model application. A user who wants to use a model trained on a particular exported dataset may efficiently retrieve stored pre-computed values);
in response to determining that the new feature definition is at least partially included in a matched feature definition of the plurality of computed feature definitions, generating one or more alternative feature definitions based on the new feature definition and the matched feature definition (Id., ¶ 42, Feature engineering system 100 is configured to use the data from data sources 101,102 to efficiently provide and/or generate feature vectors, such as a predictor feature vector, for a user to use in the application stage. Applying the model may involve computing a feature vector using the same computations that were used in training of the model, but for an entity or time that may not have been part of the training or validation examples. Because feature engineering system 100 is also configured to generate feature vectors for the user to use in the training stage, the same feature (discloses generating new feature definitions) vector definitions that were used for training are automatically available during production. As discussed above, making the same feature vector definitions used for training automatically available during production allows for event-based models to be successfully used in production. For example, feature engineering system 100 may provide and/or generate predictor feature vectors for a user to use in the application stage, while the feature engineering system 100 may provide and/or generate predictor and label feature vectors for a user to use in the training and validation stage. Feature engineering system 100 may generate the feature vectors and/or validation examples in a similar manner as described above for training examples), (Id., ¶ 43, System 100 is configured to ingest event data from one or more sources 101, 102 of data. In some configurations, a data source includes historical data, e.g., from historical data source 101. In that case, the data includes data that was received and/or stored within a historic time period i.e. not real-time. The historical data is typically indicative of events that occurred within a previous time period. For example, the historic time period may be a prior year or a prior two years, e.g., relative to a current time, etc. Historical data source 101 may be stored in and/or retrieved from one or more files, one or more databases, an offline source, and the like or may be streamed from an external source. The historical data ingested by system 100 may be associated with a user of system 100, such as a data scientist, that wants to train and implement a model using features generated from the data. System 100 may ingest the data from one or more sources 101,102 and use it to compute features), (Id., ¶ 120, Once the user has created and/or changed the feature definition and/or example selection, the feature engineering system can use this information to efficiently create the desired features and/or feature vectors and/or examples for the user. For example, the feature engineering system can use this information to create the desired features and/or feature vectors and/or examples for the user by re-using previous computations. After the desired features and/or feature vectors and/or examples have been generated, they may be exported to the user. At 704, the generated features and/or feature vectors and/or examples may be exported to the user. The user may use these exported features and/or feature vectors and/or examples to train and/or validate/evaluate the model. At 706, the user may train the model on any training examples generated by the feature engineering system. At 708, the user may validate and/or evaluate the model using any validation examples generated by the feature engineering system. If the user wants the feature engineering system to generate new or different features and/or feature vectors and/or examples, the user may easily change the dataset being used or experiment with a different dataset. For example, the user may want to try a new dataset to see if the model performs better after being trained with the new dataset. The method 700 may return to step 702, where the user may change the feature definition and/or update the example selection configuration. The user may continue to perform this iterative process until the model is generating results that satisfy the user), (Id., ¶ 121, FIG. 8 shows an example network 800 for feature engineering. The network 800 includes a feature engineering system 802 and one or more clients 804. System 802 may be similar to and/or perform similar functions as those performed by system 100 and/or system 200 described above. System 802 includes an API Server 808, one or more compute nodes 814, metadata storage 810, event data storage 816, staged data storage 806, prepared data storage 812, and result data storage 818. The event data storage 816, the staged data storage 806, and/or the prepared data storage 812 may utilize an external storage system, such as Amazon S3 or any other external storage system. The compute nodes 814 may be, for example, a feature engine, such as one of the feature engines described above);
and executing the selected execution alternative using a compute engine to generate the feature (Id., ¶ 90, Feature engineering system 100 (discloses compute engine) may simplify collaboration in feature generation (discloses generating the feature) and/or selection. As discussed above, features are often defined by users, such as data scientists. A company may have multiple data scientists producing features for one or more models. The data scientists may need to use different tools to access different kinds of raw data and/or events, further complicating the process of producing features. Collaboration on features produced in ad-hoc and varied ways makes it difficult to share features between users and/or projects. In addition, the techniques for producing features may vary based on the data size and the need for producing the feature vectors “in a production environment.” This may lead to the need to implement features multiple times for different situations. However, feature engineering system 100 may address these shortcomings by ingesting and/or saving raw data and/or events from a variety of sources and making the features available to users in different locations and/or using different devices, such as via the feature studio described further herein), (Id., ¶ 91, In an embodiment, feature computation layer 106 is configured to compute feature vectors. A feature vector is a list of features of an entity. The feature computation layer 106 may be configured to compute and/or update feature vectors as events are ingested by the feature engine 103. The feature computation layer 106 may be configured to compute and/or update feature vectors in response to user queries).
While suggested in at least Fig. 2 and related text, Bonaci does not explicitly disclose …selecting an execution alternative from among candidate execution alternatives, using an execution cost estimator that predicts an execution cost for each candidate execution alternative, wherein the candidate execution alternatives include (i) an execution of a PIT join using the alternative feature definition and (ii) an execution of a second PIT join using the new feature definition, the selected execution alternative being selected based on the execution costs predicted by the execution cost estimator;
However, through KSR Rationale D (See MPEP 2141(III)(D)), the combination of Bonaci and Chen discloses …selecting an execution alternative from among candidate execution alternatives, using an execution cost estimator that predicts an execution cost for each candidate execution alternative, wherein the candidate execution alternatives include (i) an execution of a PIT join using the alternative feature definition and (ii) an execution of a second PIT join using the new feature definition, the selected execution alternative being selected based on the execution costs predicted by the execution cost estimator.
First, Bonaci disclose a data aggregation technique including an expense estimation and minimization technique which selects an alternative technique for minimal cost (Bonaci, ¶ 82, According to an aspect, feature computation layer 106 is configured to simultaneously compute more than one feature, such as a large number of features. When simultaneously computing many features, it is possible to compute each feature independently and then join the computed values based on the entity and time. However, this approach is inefficient for at least two major reasons. First, computing each feature may involve retrieving and processing the same input events multiple times. Second, once the features are computed, performing an N-way join is an expensive operation. FIG. 5A illustrates an example N-way join 500a, (discloses PIT join techniques) such as a 3-way join, being performed after multiple features are individually computed. Computing two or more of the three features shown in FIG. 5A may involve retrieving and processing the same input events multiple times. After these three features are individually computed, they may be joined and output by the system), (Id., ¶ 83, Rather than employing this inefficient and expensive technique for simultaneously computing multiple features, feature computation layer 106 (discloses execution cost estimator) may instead combine all of the aggregations into a single pass over events that computes (at each point in time and for each entity) the value of all aggregations. (discloses execution alternative selected for minimal cost) The description of this flattened operation is called the aggregation plan and the process for producing it is described in more detail below. This flattened aggregation plan allows for the simultaneous computation of the aggregations necessary for all requested features with a single pass over the input, and therefore eliminates the need for the N-way join. FIG. 5B illustrates an example simultaneous feature computation 500b without an N-way join. As depicted in FIG. 5B, all of the multiple features are simultaneously computed with a single pass over the input, eliminating the need to retrieve and process the same input events multiple times), (Id., ¶ 157, In embodiments, resume tokens are utilized to continually apply the results of a query to a separate (i.e., external) data store with minimal cost. This may be achieved by first running an initial query, writing the results to the separate data store, and receiving a resume token. A query may be periodically run to update the results in the external store. Each query uses the resume token returned by the previous response. The new results may reflect only those results which have changed).
PNG
media_image1.png
452
647
media_image1.png
Greyscale
Further, Chen discloses time-sensitive PIT join techniques using new and alternative feature definitions (Chen, ¶ 23, For purposes of this document, features are joins between different tables on the same key (e.g., address, timestamp) with perhaps simple transformations applied to some fields (discloses selecting an optimal time-sensitive PIT join alternative) that will make it more suitable for machine learning training and/or service, such as the nullification of values that are obviously incorrect), (Id., ¶ 24, These two concepts allow for the separation of physical mechanisms for managing tables from the logical design of features, while allowing each layer to be iterated on separately. For example, supporting a new table format merely requires the user to implement a common interface, which could immediately be used by a feature layer without any knowledge of the physical details), (Id., ¶ 39, As described above, features are joins between tables FIG. 3 is a diagram illustrating an example of features, in accordance with an example embodiment. Features are what someone building a model would care about. They define a configuration file that references all the fields needed from each of the tables defined.), (Id., ¶ 43, When iterating over a FeatureSet for model training, two commands may be defined: [0044] 1. build_feature_set_input_table—feature-set $FEATURE_SET—path $PATH which, given a FeatureSet definition, will perform all the joins necessary for the input table and can later be iterated on. [0045] 2. build_feature_set—feature-set $FEATURE_SET—input-table $PATH will build the feature set using the input table from 1. This command would fail if the input table does not contain the necessary columns. [0046] These two commands allow a user to iterate over feature set definitions without having to recalculate all joins every time).
PNG
media_image2.png
285
493
media_image2.png
Greyscale
One of ordinary skill in the art would have recognized that applying the known technique of Bonaci would have yielded predictable results and resulted in an improved system. It would have been recognized that applying the cost estimation technique of Bonaci to the feature-definition-based PIT join teachings of Chen would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such data aggregation features into similar systems. Further, applying a cost minimization based alternative selection to Chen with feature definitions considered accordingly, would have been recognized by those of ordinary skill in the art as resulting in an improved system that would allow more cost effective data aggregation according to specific feature definitions.
Thus, through KSR Rationale D (See MPEP 2141(III)(D)), the combination of Bonaci and Chen discloses …selecting an execution alternative from among candidate execution alternatives, using an execution cost estimator that predicts an execution cost for each candidate execution alternative, wherein the candidate execution alternatives include (i) an execution of a PIT join using the alternative feature definition and (ii) an execution of a second PIT join using the new feature definition, the selected execution alternative being selected based on the execution costs predicted by the execution cost estimator.
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have modified the feature definition generation elements of Bonaci to include the PIT join elements of Chen in the analogous art of time sensitive data stores for the same reasons as stated for claim 1.
Regarding Claim 22, the combination of Bonaci and Chen discloses …The method of claim 1…
Bonaci further discloses …wherein generating the alternative feature definition includes: determining an execution plan for the alternative feature definition that reuses a precomputed feature dataset corresponding to the matched feature definition (Id., ¶ 42, Feature engineering system 100 is configured to use the data from data sources 101,102 to efficiently provide and/or generate feature vectors, such as a predictor feature vector, for a user to use in the application stage. Applying the model may involve computing a feature vector using the same computations that were used in training of the model, but for an entity or time that may not have been part of the training or validation examples. Because feature engineering system 100 is also configured to generate feature vectors for the user to use in the training stage, the same feature (discloses generating new feature definitions) vector definitions that were used for training are automatically available during production. As discussed above, making the same feature vector definitions used for training automatically available during production (discloses reusing precomputed datasets) allows for event-based models to be successfully used in production. For example, feature engineering system 100 may provide and/or generate predictor feature vectors for a user to use in the application stage, while the feature engineering system 100 may provide and/or generate predictor and label feature vectors for a user to use in the training and validation stage. Feature engineering system 100 may generate the feature vectors and/or validation examples in a similar manner as described above for training examples), (Id., ¶ 43, System 100 is configured to ingest event data from one or more sources 101, 102 of data. In some configurations, a data source includes historical data, e.g., from historical data source 101. In that case, the data includes data that was received and/or stored within a historic time period i.e. not real-time. The historical data is typically indicative of events that occurred within a previous time period. For example, the historic time period may be a prior year or a prior two years, e.g., relative to a current time, etc. Historical data source 101 may be stored in and/or retrieved from one or more files, one or more databases, an offline source, and the like or may be streamed from an external source. The historical data ingested by system 100 may be associated with a user of system 100, such as a data scientist, that wants to train and implement a model using features generated from the data. System 100 may ingest the data from one or more sources 101,102 and use it to compute features)
and defining the alternative feature definition to include instructions that execute according to the execution plan (Id., ¶ 91, In an embodiment, feature computation layer 106 is configured to compute feature vectors. A feature vector is a list of features of an entity. The feature computation layer 106 may be configured to compute and/or update feature vectors as events are ingested by the feature engine 103. The feature computation layer 106 may be configured to compute and/or update feature vectors in response to user queries (discloses executing according to the execution plan)).
Claims 4 are rejected under 35 U.S.C. 103 as being unpatentable over Bonaci in view of Chen and in further view of Gao et al., U.S. Publication No. 2022/0101438 [hereinafter Gao]].
Regarding Claim 4, the combination of Bonaci and Chen discloses …The computer-implemented method of claim 3…
While suggested in at least Fig. 2 and related text of Bonaci, the combination of Bonaci and Chen does not explicitly disclose …wherein the selection of the minimum cost configuration is implemented using binary integer programming.
However, Gao discloses …wherein the selection of the minimum cost configuration is implemented using binary integer programming (Gao, 529, For Tail Risk Optimizer, mixed integer programming, binary integer programming and linear programming with rounding techniques are available. CVaR-Mean frontier can be shaped with the optimizer parallel computation ability and then illustrate the relationship between CVaR and expected return. Different CVaR-Mean frontier can be visualized according to diversification preferences (see FIG. 71) and market scenarios. The mixed integer programming can offer more accurate results according to asset price and asset tradable amount while the linear programming with rounding techniques guarantees faster performance on large scale computation (See FIG. 72)), (Id., ¶ 135, In some implementations, the database calculation engine may utilize various innovative data reduction, scaling and parallel computing techniques (e.g., techniques to use global temporary tables and sessions, data reduction techniques to drastically reduce the amount of data used for processing thus lowering processing time, and several other data parallelization techniques used for generating simulation data): [0136] Use of Multiple Batches to achieve higher degree of parallelism (DOP) [0137] Use of Global Temporary Tables (GTT) to be able to run batch in multiple sessions and limit temporary storage requirements [0138] Use of Data Reduction techniques to limit full table scans for joins between Factor Exposure and Factor Simulation table [0139] Use of Parallel Query to parallelize generation of Asset Simulation and Contribution to Value at Risk data [0140] Use of Parallel DML to parallelize inserting data related to Asset Simulation and Contribution to Value at Risk [0141] Use of DDL for faster execution of delete statements to speed up cleanup of global temporary tables), (Id., ¶ 127, In some embodiments, the MLPO may implement a database calculation engine for calculating simulation data. The database calculation engine may be a SQL-based solution that effectively utilizes different data reduction and parallel execution techniques to reduce the overall response time. Instead of using a dedicated high-performance platform (e.g., IBM Netezza Data Appliance) the database calculation engine may be used for simulation calculation providing a faster, streamlined, cost effective and scalable solution (e.g., using Oracle RDS on Cloud) that provides calculation results in substantially less amount of time. Further, the database calculation engine eliminates having to maintain a complex infrastructure and applications associated with using a dedicated high-performance platform, and having to pay for additional licensing and maintenance costs), (Id., ¶ 254, 7B illustrates embodiments of a computation engine architecture that supports and implements the business logic. In various implementations the computation engine may be characterized by the following features: … [0261] 7: Cost Optimization: Inexpensive commodity-grade virtual servers may be provisioned dynamically leveraging inexpensive Spot Instances or Reserved Instances).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have modified the feature definition generation elements of Bonaci and the PIT join elements of Chen to include the binary integer programming elements of Gao in the analogous art of machine learning portfolio simulating and optimizing apparatuses, methods and systems.
The motivation for doing so would have been to “offer more accurate results according to asset price and asset tradable amount while the linear programming with rounding techniques guarantees faster performance on large scale computation (See FIG. 72)” and wherein a “wide array format may facilitate improved performance when calculating portfolio level return metrics”(Gao, ¶¶ 529, 488), wherein such improvements would have benefited Chen’s method which “produces wide tables containing features for machine learned models that allow for more efficient processing, improved sharing of information, and increased accuracy. These wide tables are made available for model training for multiple models and/or groups. These wide tables may be served on a serving database for fast access for API serving and lightweight access during interactive development. The solution decreases the time needed to add a new feature from several days to a couple of hours by enabling experimentation” (Chen, ¶ 21). Such improvements would have further benefitted Bonaci’s method which provide an “ability to maintain feature values in real time [which] may improve the accuracy of the model. For example, the model may be able to make more accurate predictions, or a larger percentage of the predictions that the model makes may be accurate. The accuracy of the model may be improved because predictions made with more recent feature values more accurately reflect the current interests/environments, etc. that the prediction is being made about” [Gao, ¶¶ 529, 488; Chen, ¶ 21; Bonaci, ¶ 29].
Claims 21 is rejected under 35 U.S.C. 103 as being unpatentable over Bonaci in view of Chen and in further view of Kaczynski et al., U.S. Publication No. 2022/0414541 [hereinafter Kaczynski].
Regarding Claim 21, the combination of Bonaci and Chen discloses …The method of claim 1…
While suggested in at least Fig. 2 and related text of Bonaci, the combination of Bonaci and Chen does not explicitly disclose …wherein generating the alternative feature definition includes: identifying a non-overlapping portion and an overlapping portion of the new feature definition relative to the matched feature definition; defining the alternative feature definition to include instructions to: compute feature values for the non-overlapping portion of the new feature definition; and append the computed feature values to a precomputed feature dataset corresponding to the matched feature definition and generated for the overlapping portion.
However, Kaczynski discloses …wherein generating the alternative feature definition includes: identifying a non-overlapping portion and an overlapping portion of the new feature definition relative to the matched feature definition (Kaczynski, ¶ 174, Definitions can be used by a computing system for generating analytical sets (e.g., ones that have features or data generated from multiple feature sets) (discloses generating alternative feature definitions). FIG. 17 illustrates an example generation of an analytical data set (e.g., Analytical Set 1750). In this example, the computing system associates a first preconfigured feature set (e.g., Feature Set 1710) and a second preconfigured feature set (e.g., Feature Set 1720) associated with an entity (e.g., Entity 1740). The entity can represent, for example, a real-world object, event, person, or business. For example, as shown in FIG. 16A, the entities included clients, transactions and contracts. In one or more embodiments, a computing system generates requested data set (e.g., where the data set is not already available) by generating an analytical data set (e.g., Analytical Set 1750) for the entity comprising data pertaining to each of the first preconfigured feature set and the second preconfigured feature set. For example, calculation results from a specific period 1730 are used for each of feature set 1710 and feature set 1720, but data from other periods such as periods 1732 and 1734 are excluded. The Analytical Set 1750 can then have its own features 1752 and may be associated with the entity 1740. The entity may have other metadata 1742 associated with it such as pertaining to a key, time, or partition. For instance, a key can be an identifier used to associate an entity with particular features or feature sets (e.g., new Analytical Set 1750). Time can be an aggregation granularity (e.g., monthly, hourly, minutely). Partition could be a variable or feature which can be used for partitioning the data (e.g., date of opening the bank account). Partitioning can be useful for processing data faster (e.g., data can be distributed into partitions and data can be retrieved in partitions). Once the Analytical Set 1750 is set up (e.g., with its own features 1752), it can be used as input data to control development of an analytical model. Accordingly, embodiments can be used to generate new feature sets), (Id., ¶ 180, In an operation 1834, a computing system (e.g., feature store 1842) checks whether the requested data set is available for retrieval according to the preconfigured feature set. The computing system can generate an availability status (i.e., “No”) indicating that the requested data set is not available for retrieval according to the preconfigured feature set. For instance, there may be metadata, or no metadata associated with a preconfigured feature set pertaining to the requested data. Metadata may be data indicating information about another data set (e.g., if the data set where text messages, metadata could include they are text messages over a certain time period but may not have the content of the actual text messages). This absence of appropriate metadata can be used to determine whether the requested data set is available for retrieval. For instance, the presence or absence of needed metadata may be an implicit indication of availability status. For example, the available data may be specific to a particular time period and the requested data set is requested for a time that is different (e.g., in overlapping time period or in a time period that does not include the particular time period available). (discloses identifying overlapping portions of feature definitions with matched feature definitions in the feature store) For instance, if a computing system stores in metadata that calculations were performed for data received in January, if another computing system asks for February, a computing system will execute evaluations based on the definition for this period and information stored in metadata since the metadata indicates that only January is available. In other words, the computing system has no computer instructions available for locating the requested data set stored, or set-up to arrive, in the feature storage, and must determine or extrapolate the data needed);
PNG
media_image3.png
560
663
media_image3.png
Greyscale
defining the alternative feature definition to include instructions to: compute feature values for the non-overlapping portion of the new feature definition (Id., ¶ 180, In an operation 1834, a computing system (e.g., feature store 1842) checks whether the requested data set is available for retrieval according to the preconfigured feature set. The computing system can generate an availability status (i.e., “No”) indicating that the requested data set is not available for retrieval according to the preconfigured feature set. For instance, there may be metadata, or no metadata associated with a preconfigured feature set pertaining to the requested data. Metadata may be data indicating information about another data set (e.g., if the data set where text messages, metadata could include they are text messages over a certain time period but may not have the content of the actual text messages). This absence of appropriate metadata can be used to determine whether the requested data set is available for retrieval. For instance, the presence or absence of needed metadata may be an implicit indication of availability status. For example, the available data may be specific to a particular time period and the requested data set is requested for a time that is different (e.g., in overlapping time period or in a time period that does not include the particular time period available). For instance, if a computing system stores in metadata that calculations were performed for data received in January, if another computing system asks for February, a computing system will execute evaluations based on the definition for this period and information stored in metadata since the metadata indicates that only January is available. (discloses computing values for the non-overlapping portions of the feature definitions) In other words, the computing system has no computer instructions available for locating the requested data set stored, or set-up to arrive, in the feature storage, and must determine or extrapolate the data needed);
and append the computed feature values to a precomputed feature dataset corresponding to the matched feature definition and generated for the overlapping portion (Id., ¶ 182, When the availability status indicates the requested data set is not available, the computing system in an operation 1835 can evaluate historical data and retrieve data 1836 useful for generating the requested data. For instance, the computing system can generate the requested data set for the requested time period. Information used for generating the requested data may be stored remote from the computing system generating the requested data. For example, a computing system checking the availability of data may obtain the preconfigured feature set from a remote source and the obtained feature set may have computer instructions for retrieving the historical data), (Id., ¶ 184-186, Sometimes in the operation 1832, a computing system determines a feature is not present in a feature store (e.g., the feature store 1842 and/or user-side device 1840 receives an indication that a feature is not in a feature storage). In this case the user-side device 1840 can be used for data exploration or preparing codes to generate a feature and ingest data in an operation 1833. The feature store 1842 can be used to update metadata and definitions for registering a feature in an operation 1846. (discloses appending the computed feature values to a corresponding feature store/dataset) The feature can then be validated in an operation 1843 and data for the feature store obtained. Therefore, it is not necessary that features, or data be present in example systems to create new feature or generate new data).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have modified the feature definition generation elements of Bonaci and the PIT join elements of Chen to include the overlapping feature elements of Kaczynski in the analogous art of feature store managers.
The motivation for doing so would have been to provide an improved system wherein “Variable definitions [are] grouped into feature sets or as a feature list can be searchable by the end users of the application and can be selected for further processing in different use case scenarios” (Kaczynski, ¶ 167), wherein such improvements would have benefited Chen’s method which “produces wide tables containing features for machine learned models that allow for more efficient processing, improved sharing of information, and increased accuracy. These wide tables are made available for model training for multiple models and/or groups. These wide tables may be served on a serving database for fast access for API serving and lightweight access during interactive development. The solution decreases the time needed to add a new feature from several days to a couple of hours by enabling experimentation” (Chen, ¶ 21). Such improvements would have further benefitted Bonaci’s method which provide an “ability to maintain feature values in real time [which] may improve the accuracy of the model. For example, the model may be able to make more accurate predictions, or a larger percentage of the predictions that the model makes may be accurate. The accuracy of the model may be improved because predictions made with more recent feature values more accurately reflect the current interests/environments, etc. that the prediction is being made about” [Kaczynski, ¶ 167; Chen, ¶ 21; Bonaci, ¶ 29].
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Johri et al., U.S. Patent No. 8,965,879 discloses a unique join data caching method.
Zimmerman et al., U.S. Publication No. 2011/0022552 discloses systems and methods for implementing a machine-learning agent to retrieve information in response to a message.
Danna, U.S. Publication No. 2021/0403036 discloses systems and methods for encoding and searching scenario information.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS D BOLEN whose telephone number is (408)918-7631. The examiner can normally be reached Monday - Friday 8:00 AM - 5:00 PM PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Patty Munson can be reached at (571) 270-5396. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NICHOLAS D BOLEN/ Examiner, Art Unit 3624 /HAMZEH OBAID/Primary Examiner, Art Unit 3624
Prosecution Insights