Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In response to Applicant’s claims filed on March 16, 2026, claims 1-20 are now pending for examination in the application.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant’s arguments:
In regards to claim 1 on Page(s) 8, applicant argues “Here, the Office Action has not met this burden for establishing prima facie unpatentability of claim 1. First, the Office Action analyzes the claim language only at a "high level of generality" without fully considering the interactions between all elements of the claim. For example, in the rejection of claim 1, the Office Action reduces four of the five recited operations to the same mental process: the "mental process of observation and/or evaluation. "Office Action, at 11. This is a reduction at a "high level of generality.”
Examiner’s Reply:
The examiner respectfully disagrees and would like to point out that human mind using a computer as a tool is fully capable of generating code, signals, training and testing data. A human would be able to iteratively follow these steps along with any needed additional elements (eg using receiving signals).
Applicant’s arguments:
In regards to claim 1 on Page(s) 8, applicant argues “Second, the Office Action also does not give sufficient weight to the evidence on record demonstrating that the claimed invention provides a technical solution to a technical problem. As described in Applicant's Specification, In conventional systems, generating training and test data for ML models can be a manual process which involves significant human interactions and can rely on multiple independent jobs, standalone scripts, and manipulations on CSV datasets. Conventional system can have a manual setup where it may require many weeks to generate training and test data for each retraining cycle of the ML model. Additionally, the manual setup also significantly reduces experimentation velocity due to the involvement of human interaction during the execution of the evaluative methods of the ML model. Moreover, in conventional systems, data generation for machine-learning models has been identified as a key area involving technical debt with a repeated cost a plurality of software engineers per year. Applicant's Specification, at [0037] (emphasis added). The claimed invention provides solution(s) to these and other challenges existing in the field. As described in Applicant's Specification.”
Examiner’s Reply:
Applicant argues that the amended claims comprises statutory subject matter. Examiner respectfully disagrees. The examiner notes that the computer (being used as a generic tool) as recited in the claims is being used for training and testing data. The generation of input data for training a machine learning model does not improve the functioning of a computer. Therefore, the abstract idea recited in the claims is generally linking it to a computer environment, and does not integrate the abstract idea into a practical application.
Double Patenting
A rejection based on double patenting of the "same invention" type finds its support in the language of 35 U.S.C. 101 which states that "whoever invents or discovers any new and useful process ... may obtain a patent therefor ..." (Emphasis added). Thus, the term "same invention," in this context, means an invention drawn to identical subject matter. See Miller v. Eagle Mfg. Co., 151 U.S. 186 (1894); In re Ockert, 245 F.2d 467, 114 USPQ 330 (CCPA 1957); and In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970).
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory obviousness-type double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); and In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on a nonstatutory double patenting ground provided the conflicting application or patent either is shown to be commonly owned with this application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement.
Effective January 1, 1994, a registered attorney or agent of record may sign a terminal disclaimer. A terminal disclaimer signed by the assignee must fully comply with 37 CFR 3.73(b).
Claims 1-20 are rejected on the ground of nonstatutory obviousness-type double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 12242469. Although the conflicting claims are not identical, they are not patentably distinct from each other because claims 1-20 of Patent No. 12242469 contain every element of claims 1-20 of the instant application and as such anticipates claims 1-20 of the instant application.
The difference between the inventions as recited in claim 1 of ‘809 application and ‘469 patent is that claim 1 of ‘469 patent recites determining to omit a backfilling operation of the new signal because the first signal is directly generated from the raw data.
A person of ordinary skills would therefore be motivated to remove/modify some of the claim elements recited in the above two claims without affecting the context of the invention, i.e., training data. The dependent claims incorporate the differences indicated above and are considered obvious variants under the above rationale, and are rejected for the same reasons.
Patent No. 12242469
Appl. No. 19052809
1. A computer-implemented method for generating input data for training a machine-learning model, the method comprising:
receiving, from a user input, signal configuration information having instructions to generate a plurality of signals from raw data, wherein the user input includes custom code to be executed using an on-the-fly operation, the custom code defining a first signal and how to generate the first signal using the raw data;
receiving signal extraction information that has instructions to query a data store;
accessing, using Structured Query Language (SQL) code that is generated based on the signal extraction information, the raw data from the data store;
processing the raw data using the signal configuration information to generate the plurality of signals;
determining that the first signal is a new signal because the first signal was not previously generated in the prior iteration of the plurality of signals;
determining to omit a backfilling operation of the new signal because the first signal is directly generated from the raw data;
joining, using the SQL code, the plurality of signals with a first label source to generate training data and testing data; and
processing the training data and the testing data to generate input data, the input data being an ingestible file for a machine-learning pipeline.
1. A computer-implemented method for generating input data for training a machine-learning model, the method comprising:
receiving signal extraction information that has instructions to query a data store;
accessing, using a first Structured Query Language (SQL) code that is generated based on the signal extraction information, raw data from the data store; processing the raw data using signal configuration information to generate a plurality of signals, the signal configuration information having instructions on how to generate the plurality of signals from the raw data;
joining, using a second SQL code, the plurality of signals with a first label source to generate training data; and
processing the training data to generate input data, the input data being an ingestible file for a machine-learning pipeline.
16. A computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
receiving, from a user input, signal configuration information having instructions to generate the plurality of signals from raw data, wherein the user input includes custom code to be executed using an on-the-fly operation, the custom code defining a first signal and how to generate the first signal using the raw data;
receiving signal extraction information that has instructions to query a data store; accessing, using Structured Query Language (SQL) code that is generated based on the signal extraction information, the raw data from the data store;
processing the raw data using the signal configuration information to generate the plurality of signals;
determining that the first signal is a new signal because the first signal was not previously generated in the prior iteration of the plurality of signals;
determining to omit a backfilling operation of the new signal because the first signal is directly generated from the raw data;
joining, using the SQL code, the plurality of signals with a first label source to generate training data and testing data; and
processing the training data and the testing data to generate input data, the input data being an ingestible file for a machine-learning pipeline.
19. A computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
receiving signal extraction information that has instructions to query a data store;
accessing, using a first Structured Query Language (SQL) code that is generated based on the signal extraction information, raw data from the data store;
processing the raw data using signal configuration information to generate a plurality of signals, the signal configuration information having instructions on how to generate the plurality of signals from the raw data;
joining, using a second SQL code, the plurality of signals with a first label source to generate training data; and
processing the training data to generate input data, the input data being an ingestible file for a machine-learning pipeline.
17. One or more non-transitory computer-readable media that collectively store a machine-learned model, wherein the machine-learned model has been learned by performance of operations, the operations comprising:
receiving, from a user input, signal configuration information having instructions to generate the plurality of signals from raw data, wherein the user input includes custom code to be executed using an on-the-fly operation, the custom code defining a first signal and how to generate the first signal using the raw data;
receiving signal extraction information that has instructions to query a data store;
accessing, using Structured Query Language (SQL) code that is generated based on the signal extraction information, the raw data from the data store;
processing the raw data using the signal configuration information to generate the plurality of signals;
determining that the first signal is a new signal because the first signal was not previously generated in the prior iteration of the plurality of signals;
determining to omit a backfilling operation of the new signal because the first signal is directly generated from the raw data;
joining, using the SQL code, the plurality of signals with a first label source to generate training data and testing data; and
processing the training data and the testing data to generate input data, the input data being an ingestible file for a machine-learning pipeline.
20. One or more non-transitory computer-readable media that collectively store a machine-learned model, wherein the machine-learned model has been learned by performance of operations, the operations comprising:
receiving signal extraction information that has instructions to query a data store;
accessing, using a first Structured Query Language (SQL) code that is generated based on the signal extraction information, raw data from the data store;
processing the raw data using signal configuration information to generate a plurality of signals, the signal configuration information having instructions on how to generate the plurality of signals from the raw data;
joining, using a second SQL code, the plurality of signals with a first label source to generate training data; and
processing the training data to generate input data, the input data being an ingestible file for a machine-learning pipeline.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-patentable subject matter. The claims are directed to an abstract idea without significantly more.
Claim 1-20 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The judicial exception is not integrated into a practical application. The claims do not include additional elements that are sufficient to amount to significantly more than judicial exception. The eligibility analysis in support of these findings is provided below, on Claim Rejections - 35 USC 101 accordance with the "2019 Revised Patent Subject Matter Eligibility Guidance" (published on 1/7/2019 in Fed, Register, Vol. 84, No. 4 at pgs. 50-57, hereinafter referred to as the "2019 PEG").
Step 1. in accordance with Step 1 of the eligibility inquiry (as explained in MPEP 2106), it is first noted the claim method (claims 1-8), system(s) (claim 19), and readable media (claim 20) is/are directed to one of the eligible categories of subject matter and therefore satisfies Step 1.
Step 2A. In accordance with Step 2A, prong one of the 2019 PEG, it is noted that the independent claims recite an abstract idea falling within the Mental Processes enumerated groupings of abstract ideas set forth in the 2019 PEG. Examiner is of the position that independent claims 1, 19, and 20 are directed towards the Mental Process Grouping of Abstract Ideas.
Independent claims 1 recites the following limitations directed towards a Mental Processes:
accessing, using a first Structured Query Language (SQL) code that is generated based on the signal extraction information, raw data from the data store (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to access raw data);
processing the raw data using signal configuration information to generate a plurality of signals, the signal configuration information having instructions on how to generate the plurality of signals from the raw data (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to process raw data);
joining, using a second SQL code, the plurality of signals with a first label source to generate training data (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate training data); and
processing the training data to generate input data, the input data being an ingestible file for a machine-learning pipeline (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate input data).
Step 2A. In accordance with Step 2A, prong two of the 2019 PEG, the judicial exception is not integrated into a practical application because of the recitation in claim(s) 1, 19, and 20:
receiving signal extraction information that has instructions to query a data store;
(recites insignificant extra solution activity that amounts to mere data gathering).
Step 2B. Similar to the analysis under 2A Prong Two, the claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Because the additional elements of the independent claims amount to insignificant extra solution activity and/or mere instructions, the additional elements do not add significantly more to the judicial exception such that the independent claims as a whole would be patent eligible.
Therefore, independent claims 1, 19, and 20 are rejected under 35 U.S.C. 101.
With respect to claim(s) 2:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
wherein the signal configuration information is received from user input, and wherein the user input includes custom code to be executed using an on-the-fly operation (recites insignificant extra solution activity that amounts to mere data gathering).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 3:
Step 2A, prong one of the 2019 PEG:
wherein the custom code defines a first signal and instructions on how to generate the first signal using raw data (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate a signal).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 4:
Step 2A, prong one of the 2019 PEG:
determining that the first signal is a new signal because the first signal was not previously generated in the prior iteration of the plurality of signals (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to determine a new signal); and
determining to omit a backfilling operation of the new signal because the first signal is directly generated from the raw data (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to determine an operation).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 5:
Step 2A, prong one of the 2019 PEG:
wherein the first signal is a base signal that was not generated in a prior iteration of the plurality of signals, the base signal being derived by processing a plurality of inputs obtained from the raw data, the method further comprising: backfilling the base signal using an on-the-fly operation, wherein the on-the-fly operation prevents a full-fledged backfill operation (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to backfilling an operation).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 6:
Step 2A, prong one of the 2019 PEG:
joining, using the second SQL code, the plurality of signals and the first label source with a second label source to generate the training data and the testing data, the first label source being a different system than the second label source (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate a training data).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 7:
Step 2A, prong one of the 2019 PEG:
wherein the first label source is an annotated data that has human-generated labels, and the second label source is a data source that has feedback information based on user interaction (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate a training data).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 8:
Step 2A, prong one of the 2019 PEG:
wherein the first label source is associated with a first sample weight, and the second label source is associated with a second sample weight, and wherein the joining of the plurality of signals and the first label source with the second label source is further based on the first sample weight and the second sample weight (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate a training data).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 9:
Step 2A, prong one of the 2019 PEG:
wherein the first sample weight is based on a confidence level associated with the first label source, and the second sample weight is based on a different confidence level associated with the second label source (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by generate training data).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 10:
Step 2A, prong one of the 2019 PEG:
wherein the data store includes a first data source and a second data source, and wherein the signal extraction information includes instructions to query the first data source and the second data source, the first data source being a different type than the second data source (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by query a data source).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 11:
Step 2A, prong one of the 2019 PEG:
wherein the first data source is a column in a first dataset and the second data source is a human-generated label in a second dataset (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by query a data source).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 12:
Step 2A, prong one of the 2019 PEG:
performing, based on a custom request, a custom-split of the training data and the testing data (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by performing a split).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 13:
Step 2A, prong one of the 2019 PEG:
wherein the custom request is a date reference, and wherein the training data is associated with data prior to the date reference and the testing data is associated with data after the date reference (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by performing a split).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 14:
Step 2A, prong one of the 2019 PEG:
wherein the machine-learning model is trained on non-independent and identically distributed (IID) data requiring a custom-split of the training data and the testing data (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by training a model).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 15:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
transmitting an alert when an error occurs during the generating of the plurality of signals (recites insignificant extra solution activity that amounts to transmitting an alert).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 16:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
transmitting an alert when an error occurs during the generating of the training data or the testing data (recites insignificant extra solution activity that amounts to transmitting data).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 17:
Step 2A, prong one of the 2019 PEG:
wherein the training data is utilized during the training of the machine-learning model, and the testing data is utilized during a validation testing of the machine-learning model (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by training a model).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 18:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
wherein the plurality of signals are stored in a table pointing to a file dump containing all signals utilized by the machine-learning model, and wherein the table is configured, using a table alias generator, to point to the file dump that is current (recites insignificant extra solution activity that amounts to storing data).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 10-11, 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Guttmann (US Pub. No. 20190294999) in further view of Yang (US Pub. No. 20220092349).
With respect to claim 1,Guttmann teaches a computer-implemented method for generating input data for training a machine-learning model, the method comprising:
receiving signal extraction information that has instructions to query a data store (Paragraph 57 discloses receive and transmit information. For example, control signals may be transmitted and/or received through communication modules 230);
accessing, using a first Structured Query Language (SQL) code that is generated based on the signal extraction information, raw data from the data store (Paragraph 95 discloses datasets 610 may comprise database tables, and view 630 may comprise SQL expressions for generating a new table out of the original tables and/or generated table. In yet another example, datasets 610 may comprise data-points, and view 630 may comprise a rule for merging data-points, a rule for selecting a subset of the data-points, and so forth);
processing the raw data (Paragraph 100 discloses algorithms 640 may comprise algorithms for processing information and data from an external source. In some examples, the external data source may include a sensor (such as audio sensor, image sensor, motion sensor, positioning sensor, etc.), a user, an external device, an automatic process, external data repository, and so forth. Some examples of external data repositories may include a public database, a blockchain, a web crawler, and so forth);
joining, using a second SQL code, the plurality of signals with a first label source to generate training data (Paragraph 120 discloses trained machine learning algorithms (also referred to as trained machine learning models in the present disclosure) may be used to analyze inputs and generates outputs, for example in the cases described below. In some examples, a trained machine learning algorithm may be used as an inference model that when provided with an input generates an inferred output). Uttmann does not explicitly disclose processing the training data to generate input data, the input data being an ingestible file for a machine-learning pipeline.
However, Yang et al. discloses processing the training data to generate input data, the input data being an ingestible file for a machine-learning pipeline (Paragraph 82 discloses In at least one embodiment, at least some new training input data, testing input data, and/or ground truth data may be generated and used for testing and/or training the same, different, and/or modified MLM based at least on the evaluation data 126 (e.g., using the same or a different or modified input data pipeline 140)).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify Guttmann with Yang to include processing the training data to generate input data, the input data being an ingestible file for a machine-learning pipeline. This would have facilitated improving a machine learning pipeline using the generated training and test data.
The Guttmann reference as modified by Yang et al. teaches all the limitations of claim 1. Regarding claim 10, Guttmann discloses the method of claim 1, wherein the data store includes a first data source and a second data source, and wherein the signal extraction information includes instructions to query the first data source and the second data source, the first data source being a different type than the second data source (Paragraph 100 discloses algorithms 640 may comprise algorithms for processing information and data from an external source. In some examples, the external data source may include a sensor (such as audio sensor, image sensor, motion sensor, positioning sensor, etc.), a user, an external device, an automatic process, external data repository, and so forth. Some examples of external data repositories may include a public database, a blockchain, a web crawler, and so forth).
The Guttmann reference as modified by Yang et al. teaches all the limitations of claim 1. Regarding claim 11, Guttmann discloses the method of claim 10, wherein the first data source is a column in a first dataset and the second data source is a human-generated label in a second dataset (Paragraph 117discloses datasets 610 and/or annotations 620 and/or views 630 and/or algorithms 640 and/or tasks 650 and/or logs 660 and/or policies 670 and/or permissions 680 may be created and/or deleted and/or modified manually and/or automatically).
With respect to claim 19, Guttmann teaches a computing system, comprising:
one or more processors (Paragraph 39 discloses processors); and
one or more non-transitory computer-readable media (Paragraph 20 discloses media) that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
receiving signal extraction information that has instructions to query a data store (Paragraph 57 discloses receive and transmit information. For example, control signals may be transmitted and/or received through communication modules 230);
accessing, using a first Structured Query Language (SQL) code that is generated based on the signal extraction information, raw data from the data store (Paragraph 95 discloses datasets 610 may comprise database tables, and view 630 may comprise SQL expressions for generating a new table out of the original tables and/or generated table. In yet another example, datasets 610 may comprise data-points, and view 630 may comprise a rule for merging data-points, a rule for selecting a subset of the data-points, and so forth);
processing the raw data (Paragraph 100 discloses algorithms 640 may comprise algorithms for processing information and data from an external source. In some examples, the external data source may include a sensor (such as audio sensor, image sensor, motion sensor, positioning sensor, etc.), a user, an external device, an automatic process, external data repository, and so forth. Some examples of external data repositories may include a public database, a blockchain, a web crawler, and so forth);
joining, using a second SQL code, the plurality of signals with a first label source to generate training data (Paragraph 120 discloses trained machine learning algorithms (also referred to as trained machine learning models in the present disclosure) may be used to analyze inputs and generates outputs, for example in the cases described below. In some examples, a trained machine learning algorithm may be used as an inference model that when provided with an input generates an inferred output). Uttmann does not explicitly disclose processing the training data to generate input data, the input data being an ingestible file for a machine-learning pipeline.
However, Yang et al. discloses processing the training data to generate input data, the input data being an ingestible file for a machine-learning pipeline (Paragraph 82 discloses In at least one embodiment, at least some new training input data, testing input data, and/or ground truth data may be generated and used for testing and/or training the same, different, and/or modified MLM based at least on the evaluation data 126 (e.g., using the same or a different or modified input data pipeline 140)).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify Guttmann with Yang to include processing the training data to generate input data, the input data being an ingestible file for a machine-learning pipeline. This would have facilitated improving a machine learning pipeline using the generated training and test data.
With respect to claim 20, Guttmann teaches one or more non-transitory computer-readable media that collectively store a machine-learned model, wherein the machine-learned model has been learned by performance of operations, the operations comprising:
receiving signal extraction information that has instructions to query a data store (Paragraph 57 discloses receive and transmit information. For example, control signals may be transmitted and/or received through communication modules 230);
accessing, using a first Structured Query Language (SQL) code that is generated based on the signal extraction information, raw data from the data store (Paragraph 95 discloses datasets 610 may comprise database tables, and view 630 may comprise SQL expressions for generating a new table out of the original tables and/or generated table. In yet another example, datasets 610 may comprise data-points, and view 630 may comprise a rule for merging data-points, a rule for selecting a subset of the data-points, and so forth);
processing the raw data (Paragraph 100 discloses algorithms 640 may comprise algorithms for processing information and data from an external source. In some examples, the external data source may include a sensor (such as audio sensor, image sensor, motion sensor, positioning sensor, etc.), a user, an external device, an automatic process, external data repository, and so forth. Some examples of external data repositories may include a public database, a blockchain, a web crawler, and so forth);
joining, using a second SQL code, the plurality of signals with a first label source to generate training data (Paragraph 120 discloses trained machine learning algorithms (also referred to as trained machine learning models in the present disclosure) may be used to analyze inputs and generates outputs, for example in the cases described below. In some examples, a trained machine learning algorithm may be used as an inference model that when provided with an input generates an inferred output). Uttmann does not explicitly disclose processing the training data to generate input data, the input data being an ingestible file for a machine-learning pipeline.
However, Yang et al. discloses processing the training data to generate input data, the input data being an ingestible file for a machine-learning pipeline (Paragraph 82 discloses In at least one embodiment, at least some new training input data, testing input data, and/or ground truth data may be generated and used for testing and/or training the same, different, and/or modified MLM based at least on the evaluation data 126 (e.g., using the same or a different or modified input data pipeline 140)).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify Guttmann with Yang to include processing the training data to generate input data, the input data being an ingestible file for a machine-learning pipeline. This would have facilitated improving a machine learning pipeline using the generated training and test data.
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Guttmann (US Pub. No. 20190294999) and Yang (US Pub. No. 20220092349) in further view of Quinton et al. (US Pub. 202203627811).
The Guttmann reference as modified by Yang et al. teaches all the limitations of claim 1. Regarding claim 2, Guttmann as modified by Yang et al. does not disclose the method of claim 1, wherein the user input includes custom code to be executed using an on-the-fly operation.
However, Quinton et al. teaches the method of claim 1, wherein the signal configuration information is received from user input, and wherein the user input includes custom code to be executed using an on-the-fly operation (Paragraph 70 discloses the System 220 provides the Machine Learning System 290 with Composite Data 272 during the training process on a just-in-time basis (e.g. in real-time, on-the-fly, etc.) wherein the Data Generation Engine 230 may be configured to visit the Training Data 210 in a different order at various points of the Training Process, for generating different iterations of the Composite Training Data 260, and therefore different variations of the Composite Data 272 supplied to the Machine Learning System 290 on-the-fly).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify Guttmann and Yang with Quinton et al. to include wherein the user input includes custom code to be executed using an on-the-fly operation. This would have facilitated improving a machine learning pipeline using the generated training and test data.
Claim(s) 3-5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Guttmann (US Pub. No. 20190294999) and Yang (US Pub. No. 20220092349) and Quinton et al. (US Pub. 202203627811) in further view of Kleider et al. (US Pub. No. 20200252318).
The Guttmann reference as modified by Yang et al. and Quinton et al. teaches all the limitations of claim 2. Regarding claim 3, Guttmann as modified by Yang et al. and Quinton et al. does not disclose wherein the custom code defines a first signal and instructions on how to generate the first signal using raw data.
However, Kleider et al. teaches the method of claim 2, wherein the custom code defines a first signal and instructions on how to generate the first signal using raw data (Paragraph 38 discloses signal configuration parameter data 34 to operate as signal generators capable of producing wireless signals having both in-phase (I) and quadrature (Q) components (if required for a certain simulation)).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify Guttmann and Yang et al. and Quinton et al. with Kleider et al. to include receiving, from a user input, the signal configuration information having instructions to generate the plurality of signals from the raw data. This would have facilitated improving a machine learning pipeline using the generated training and test data.
The Guttmann. reference as modified by Yang et al. and Quinton et al. and Kleider et al. teaches all the limitations of claim 3. Regarding claim 4, Quinton et al. discloses the method of claim 3, further comprising:
determining that the first signal is a new signal because the first signal was not previously generated in the prior iteration of the plurality of signals (Paragraph 70 discloses the System 220 provides the Machine Learning System 290 with Composite Data 272 during the training process on a just-in-time basis (e.g. in real-time, on-the-fly, etc.) wherein the Data Generation Engine 230 may be configured to visit the Training Data 210 in a different order at various points of the Training Process, for generating different iterations of the Composite Training Data 260, and therefore different variations of the Composite Data 272 supplied to the Machine Learning System 290); and
determining to omit a backfilling operation of the new signal because the first signal is directly generated from the raw data (Paragraph 70 discloses the System 220 provides the Machine Learning System 290 with Composite Data 272 during the training process on a just-in-time basis (e.g. in real-time, on-the-fly, etc.) wherein the Data Generation Engine 230 may be configured to visit the Training Data 210 in a different order at various points of the Training Process, for generating different iterations of the Composite Training Data 260, and therefore different variations of the Composite Data 272 supplied to the Machine Learning System 290). The motivation to combine statement previously provided in the rejection of dependent claim 3 provided above, combining the Guttmann reference and the Quinton et al. reference is applicable to dependent claim 4.
The Guttmann reference as modified by Yang et al. and Quinton et al. and Kleider et al. teaches all the limitations of claim 4. Regarding claim 5, Quinton et al. discloses the method of claim 4, wherein the first signal is a base signal that was not generated in a prior iteration of the plurality of signals, the base signal being derived by processing a plurality of inputs obtained from the raw data, the method further comprising: backfilling the base signal using an on-the-fly operation, wherein the on-the-fly operation prevents a full-fledged backfill operation (Paragraph 70 discloses the System 220 provides the Machine Learning System 290 with Composite Data 272 during the training process on a just-in-time basis (e.g. in real-time, on-the-fly, etc.) wherein the Data Generation Engine 230 may be configured to visit the Training Data 210 in a different order at various points of the Training Process, for generating different iterations of the Composite Training Data 260, and therefore different variations of the Composite Data 272 supplied to the Machine Learning System 290). The motivation to combine statement previously provided in the rejection of dependent claim 4 provided above, combining the Guttmann reference and the Quinton et al. reference is applicable to dependent claim 5.
Claim(s) 6-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Guttmann (US Pub. No. 20190294999) and Yang (US Pub. No. 20220092349) and Quinton et al. (US Pub. 202203627811) in further view of Cooper et al. (US Pub. No. 20100177956).
The Guttmann reference as modified by Yang et al. and Quinton et al. teaches all the limitations of claim 1. Regarding claim 6, Guttmann as modified by Yang et al. and Quinton et al. does not disclose joining, using SQL code, the plurality of signals and the first label source with a second label source to generate the training data and the testing data, the first label source being a different system than the second label source.
However, Cooper et al. teaches the method of claim 1, further comprising:
joining, using the second SQL code, the plurality of signals and the first label source with a second label source to generate the training data and the testing data, the first label source being a different system than the second label source (Paragraph 84 discloses a label (tag) may be annotated using other labels and large data sets using the learned classification module 774 and secondary training data 722).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify Guttmann and Yang and Quinton et al. with Cooper et al. to include joining, using SQL code, the plurality of signals with a first label source to generate training data and testing data. This would have facilitated improving a machine learning pipeline using the generated training and test data.
The Guttmann reference as modified by Yang et al. and Quinton et al. and Cooper et al. teaches all the limitations of claim 6. Regarding claim 7, Cooper et al. discloses the method of claim 6, wherein the first label source is an annotated data that has human-generated labels, and the second label source is a data source that has feedback information based on user interaction (Paragraph 37 discloses the package 140 also includes user data 126 and annotations 146. The annotations 146 may include labels or tags associated with a training set of digital image files, and may also include user 103 specified annotations from the user's 103 personal image file collection and Paragraph 66 discloses a classification module 314 configured to annotate digital image files using tags or labels from stored reference annotations 330). The motivation to combine statement previously provided in the rejection of dependent claim 6 provided above, combining the Guttmann reference and the Cooper et al. reference is applicable to dependent claim 7.
The Guttmann reference as modified by Yang et al. and Quinton et al. and Cooper et al. teaches all the limitations of claim 7. Regarding claim 8, Cooper et al. discloses the method of claim 7, wherein the first label source is associated with a first sample weight, and the second label source is associated with a second sample weight, and wherein the joining of the plurality of signals and the first label source with the second label source is further based on the first sample weight and the second sample weight (Paragraph 80 discloses At each round t, a weak classifier h.sub.t is selected with a small error on the training samples as weighted by D.sub.t). The motivation to combine statement previously provided in the rejection of dependent claim 7 provided above, combining the Guttmann reference and the Cooper et al. reference is applicable to dependent claim 8.
The Guttmann reference as modified by Yang et al. and Quinton et al. and Cooper et al. teaches all the limitations of claim 7. Regarding claim 9, Yang et al. discloses the method of claim 8, wherein the first sample weight is based on a confidence level associated with the first label source, and the second sample weight is based on a different confidence level associated with the second label source (Paragraph 196 discloses Where the confidence score does not meet the threshold, and where the primary and secondary computer indicate different results (e.g., the conflict), the supervisory MCU may arbitrate). The motivation to combine statement previously provided in the rejection of dependent claim 7 provided above, combining the Guttmann reference and the Cooper et al. reference is applicable to dependent claim 9.
Claim(s) 12-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Guttmann (US Pub. No. 20190294999) and Yang (US Pub. No. 20220092349) and Quinton et al. (US Pub. 202203627811) in further view of Cakmak et al. (US Pub. 20190163666).
The Guttmann reference as modified by Yang et al. and Quinton et al. teaches all the limitations of claim 1. Regarding claim 12, Guttmann as modified by Yang et al. and Quinton et al. does not disclose performing, based on a custom request, a custom-split of the training data and the testing data.
However, Cakmak et al. teaches the method of claim 1, wherein generating the training data and the testing data further comprising:
performing, based on a custom request, a custom-split of the training data and the testing data (Paragraph 21 discloses split the customer activity data into a first set containing customer activity data from the year 1997 to the year 2007, and associate the first set with the label “training data”, and into a second set containing customer activity data from the year 2008 to 2017 and associate the second set with the label “test data.”).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify Guttmann and Yang and Quinton et al. with Cakmak et al. to include performing, based on a custom request, a custom-split of the training data and the testing data. This would have facilitated improving a machine learning pipeline using the generated training and test data.
The Guttmann reference as modified by Yang et al. and Quinton et al. and Cakmak et al. teaches all the limitations of claim 12. Regarding claim 13, Cakmak et al. discloses the method of claim 12, wherein the custom request is a date reference, and wherein the training data is associated with data prior to the date reference and the testing data is associated with data after the date reference (Paragraph 21 discloses split the customer activity data into a first set containing customer activity data from the year 1997 to the year 2007, and associate the first set with the label “training data”, and into a second set containing customer activity data from the year 2008 to 2017 and associate the second set with the label “test data.”). The motivation to combine statement previously provided in the rejection of dependent claim 12 provided above, combining the Guttmann reference and the Cakmak et al. reference is applicable to dependent claim 13.
The Guttmann reference as modified by Yang et al. and Quinton et al. and Cakmak et al. teaches all the limitations of claim 12. Regarding claim 14, Cakmak et al. discloses the method of claim 12, wherein the machine-learning model is trained on non- independent and identically distributed (IID) data requiring a custom-split of the training data and the testing data (Paragraph 21 discloses split the customer activity data into a first set containing customer activity data from the year 1997 to the year 2007, and associate the first set with the label “training data”, and into a second set containing customer activity data from the year 2008 to 2017 and associate the second set with the label “test data.”). The motivation to combine statement previously provided in the rejection of dependent claim 12 provided above, combining the Guttmann reference and the Cakmak et al. reference is applicable to dependent claim 14.
Claim(s) 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Guttmann (US Pub. No. 20190294999) and Yang (US Pub. No. 20220092349) and Quinton et al. (US Pub. 202203627811) in further view of Ramirez et al. (US Pub. 20220309407).
The Guttmann reference as modified by Yang et al. and Quinton et al. teaches all the limitations of claim 1. Regarding claim 15, Guttmann as modified by Yang et al. and Quinton et al. does not disclose transmitting an alert when an error occurs during the generating of the plurality of signals.
However, Ramirez et al. teaches the method of claim 1, further comprising:
transmitting an alert when an error occurs during the generating of the plurality of signals (Paragraph 134 discloses a member device may periodically report status or send alerts over text or email).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify Guttmann and Yang and Quinton et al. with Ramirez et al. to include transmitting an alert when an error occurs during the generating of the plurality of signals. This would have facilitated improving a machine learning pipeline using the generated training and test data.
The Guttmann reference as modified by Yang et al. and Quinton et al. teaches all the limitations of claim 1. Regarding claim 16, Guttmann as modified by Yang et al. does not disclose transmitting an alert when an error occurs during the generating of the training data or the testing data
However, Ramirez et al. teaches the method of claim 1, further comprising:
transmitting an alert when an error occurs during the generating of the training data or the testing data (Paragraph 134 discloses a member device may periodically report status or send alerts over text or email).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify Guttmann and Yang and Quinton et al. with Ramirez et al. to include transmitting an alert when an error occurs during the generating of the training data or the testing data. This would have facilitated improving a machine learning pipeline using the generated training and test data.
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Guttmann (US Pub. No. 20190294999) and Yang (US Pub. No. 20220092349) and Quinton et al. (US Pub. 202203627811) in further view of Mehta et al. (US Pub. 20170161614).
The Guttmann reference as modified by Yang et al. teaches all the limitations of claim 1. Regarding claim 17, Guttmann as modified by Yang et al. does not disclose the training data is utilized during the training of the machine-learning model, and the testing data is utilized during a validation testing of the machine-learning model.
However, Mehta et al. teaches the method of claim 1, wherein the training data is utilized during the training of the machine-learning model, and the testing data is utilized during a validation testing of the machine-learning model (Paragraph 134 discloses embodiments, environmental and emergency data (“the input data”) may be queued for being inputted to the self-learning scheme in the queuing module 516 for online testing. In some embodiments, the input data may be separated into training, validation and/or testing data for the self-learning scheme).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify Guttmann and Yang and Quinton et al. with Mehta et al. to include wherein the training data is utilized during the training of the machine-learning model, and the testing data is utilized during a validation testing of the machine-learning model. This would have facilitated improving a machine learning pipeline using the generated training and test data.
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Guttmann (US Pub. No. 20190294999) and Yang (US Pub. No. 20220092349) and Quinton et al. (US Pub. 202203627811) in further view of Li et al. (US Pub. 20170161614).
The Guttmann reference as modified by Yang et al. and Quinton et al. teaches all the limitations of claim 1. Regarding claim 18, Guttmann as modified by Yang et al. does not disclose wherein the plurality of signals are stored in a table pointing to a file dump containing all signals utilized by the machine-learning model, and wherein the table is configured, using a table alias generator, to point to the file dump that is current.
However, Li et al. teaches the method of claim 1, wherein the plurality of signals are stored in a table pointing to a file dump containing all signals utilized by the machine-learning model, and wherein the table is configured, using a table alias generator, to point to the file dump that is current (Paragraph 52 discloses model training service provides data to model training service 330 as data dump, thus allowing model training service 330 to fill the provided data into data training tables).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify Guttmann and Yang and Quinton et al. with Li et al. to include wherein the plurality of signals are stored in a table pointing to a file dump containing all signals utilized by the machine-learning model, and wherein the table is configured, using a table alias generator, to point to the file dump that is current. This would have facilitated improving a machine learning pipeline using the generated training and test data.
Relevant Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US PG-Pub. No. 20220398485 is directed to PREVENTING DATA VULNERABILITIES DURING MODEL TRAINING: [0010] The disclosed embodiments receive a first and second set of importance features associated with a first and second label, respectively, and identify first and second label features, respectively, used by a machine learning (ML) model to classify data with the first label, and the second set of importance features associated with a second label and identify second features used by the ML model to classify data with the second label. In one embodiment, the list of features can be determined by analyzing the features of each example classified by the ML model and selecting a top feature (i.e., the feature most contributing to the classification) or the top n features (where n is less than the total number of features in an example) for each example. In some embodiments, the disclosed embodiments can select the two top features per example. The disclosed embodiments generate a first feature dictionary based on the first set of importance features and a second feature dictionary based on the second set of importance features. The disclosed embodiments identify a subset of labeled examples in a training dataset used to train the ML model based on the first feature dictionary and second feature dictionary. The disclosed embodiments modify the subset of labeled examples based on the first feature dictionary and second feature dictionary, the modifying generating a modified training data set. The disclosed embodiments then retrain the ML model using the modified training data set.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS E ALLEN whose telephone number is (571)270-3562. The examiner can normally be reached Monday through Thursday 830-630.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Boris Gorney can be reached at (571) 270-5626. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.E.A/Examiner, Art Unit 2154
/BORIS GORNEY/Supervisory Patent Examiner, Art Unit 2154
Prosecution Insights