DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 02/05/2026 has been entered.
Response to Arguments
Applicant's arguments filed 02/05/2026 have been fully considered but they are not persuasive.
Regarding applicant’s remarks directed to the rejection of claims under 35 USC § 101, the applicant argues that the amended claims directed to a technical solution. Examiner respectfully agrees and withdraws the prior rejection of claims under 35 USC § 101.
Regarding applicant’s remarks directed to the rejection of claims under 35 USC § 102, the arguments are directed to newly amended limitations that were not previously examined by the examiner. Therefore, applicants arguments are rendered moot. The examiner refers to the rejection under 35 USC § 103 in the current office action for more details.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5, 15 and 22 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 5, 15 and 22 recite the limitation " the data-processing operations." There is insufficient antecedent basis for this limitation in the claim. Examiner suggests this should read as “the data-pipelining operations.”
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 3-8, 10-12, and 14-23 are rejected under 35 U.S.C. 103 as being unpatentable over US Pub No. US20190286538A1 Rickard et al. (“Rickard”) in view of Elshawi, Radwa, Mohamed Maher, and Sherif Sakr. "Automated machine learning: State-of-the-art and open challenges." arXiv preprint arXiv:1906.02287 (2019). (“Elshawi”)
In regards to claim 1,
Rickard teaches An apparatus, comprising: a communications interface; a memory storing instructions; and at least one processor coupled to the communications interface and to the memory, the at least one processor being configured to execute the instructions to:
(Rickard, “[0036] FIG. 1 illustrates an example pipeline monitoring system in which the techniques described herein may be practiced, according to some embodiments. In the example of FIG. 1, a pipeline monitoring system 100 is a system for monitoring a data pipeline. The various components of pipeline monitoring system 100 are implemented, at least partially, by hardware at one or more computing devices, such as one or more hardware processors executing stored program instructions stored in one or more memories for performing the functions that are described herein.”)
Rickard teaches obtain elements of process data associated with data-pipelining operations that support an execution of a plurality of [machine learning or artificial intelligence] processes during one or more first temporal intervals,
(Rickard, “[0003] In computer systems, a pipeline is a set of one or more coupled pipeline subsystems that process and/or analyze data. Each pipeline subsystem consists of computer programs or dedicated computers that receive data from a source, process or transform the data, and forward the data to another program, computer or system.”)
(Rickard, [0034], “In one embodiment, a collector is programmed or configured for retrieving an event data object and a current status data object from a pipeline subsystems [obtain elements of process data associated with data-pipelining operations; wherein the collector obtains event/status data from the pipeline subsystems (ie data-pipelining operations)]. An event data object is data related to historical events that occurred for one or more subsystems in the pipeline. A current status data object is data that describes the current condition of the one or more subsystems in the pipeline.”)
Rickard teaches the elements of process data indicating, for each [of the machine learning or artificial intelligence] processes, a success or failure of a corresponding data-pipelining operation during the one or more first temporal intervals;
Rickard teaches monitoring one or more (a plurality) of pipelines
(Rickard, “[0037] Pipeline monitoring system 100 is programmed or configured to efficiently monitor the health and status of one or more pipelines 102 [for each … processes ie pipeline].”)
(Rickard, “[0051] Collectors can collect varied types of fact data objects depending on the type of pipeline subsystem that is providing the fact data object. Examples of fact data objects may include information related to the amount of data being ingested and transmitted by the pipeline subsystem, such as the volume of data (e.g. number of rows in a table, aggregate size of data, etc.) received by the pipeline subsystem during a period of time, the success or failure of a job process [the elements of process data indicating a success or failure of a corresponding data-pipelining operation ie job process of the pipeline subsystem during the one or more first temporal intervals ie the time for all the job processes of all the pipeline subsystems of a pipeline to finish wherein a pipeline corresponds to a machine learning/artificial intelligence process]…”)
Rickard teaches based on the process data, determine, for each [of the machine learning or artificial intelligence] processes, metric values characterizing a status of the data-pipelining operations [that support the execution of the corresponding machine learning or artificial intelligence process],
Rickard teaches processing multiple data pipelines
(Rickard, “[0002] The present disclosure relates to computers that are programmed to process data pipelines.”)
(Rickard, “[0040] Collector 110 [based on the process data] may be coupled to validator 120. Although depicted as a single validator 120 in pipeline monitoring system 100, in another embodiment, multiple validators may be included. A validator may be defined as a process that is programmed or configured to apply one or more validation criteria to one or more fact data objects in order to determine a validation value that represents whether the pipeline is in a healthy state [metric values ie validation values characterizing a status of the data-pipelining operations].”)
Rickard teaches the metric values comprising, for each [of the machine learning or artificial intelligence] processes, a first value characterizing a number of failed executions of the data-pipelining operations during the one or more first temporal intervals
(Rickard, “[0076] In another embodiment, validator 120 can be used to validate that one or more pipeline subsystems have not exceeded a failure count threshold. A failure count represents the number of observed failures of one or more pipeline subsystems and can be customized based on the particular pipeline subsystem [a first value characterizing a number of failed executions of the data-pipelining operations during the one or more first temporal intervals].”)
Rickard teaches and a second value characterizing a number of successful executions of the data-pipelining operations during the one or more first temporal intervals;
(Rickard, “[0071] In another embodiment, validator 120 can be used to validate that all rows have succeeded for a JDBC table. For example, validator 120 can use validation criteria that validates that all rows for a JDBC table have succeeded. For example, in one embodiment, a table contains a log of job processing results, including the success and/or failure of the job processes [a second value characterizing a number of successful executions of the data-pipelining operations during the one or more first temporal intervals; wherein the count of the successful job processes would be the number of successful executions]. In another embodiment, a validator may apply data validation rules, via a SQL statement to determine that the resulting rows of the database satisfy the SQL statement. Validator 120 can thus generate a validation value that indicates whether a JDBC table only includes rows that succeeded.”)
Rickard teaches generate status data for each [of the machine learning or artificial intelligence] processes, and transmit one or more elements of the status data to a device via the communications interface, the status data comprising, [for each of the machine learning or artificial intelligence processes], the determined one or more metric values and a corresponding process identifier, and the status data causing the device to present, for each [of the machine learning or artificial intelligence] processes, a graphical representation of at least one of the determined first and second one or more metric values within a digital interface;
(Rickard, “[0040] Collector 110 may be coupled to validator 120. Although depicted as a single validator 120 in pipeline monitoring system 100, in another embodiment, multiple validators may be included. A validator may be defined as a process that is programmed or configured to apply one or more validation criteria to one or more fact data objects in order to determine a validation value that represents whether the pipeline is in a healthy state [generate status data… the status data comprising the determined one or more metric values].”)
(Rickard, “[0042] Notifier 130 may be coupled to one or more end devices 150 [transmit one or more elements of the status data to a device via the communications interface]. An end device 150 may be any computing device, including, but not limited to, a portable computing device, a server computing device, a desktop computing device, etc. Notifier 130 may send notifications to end device 150.”)
(Rickard, “[0043] One or more of collector 110, validator 120, and/or notifier 130 may be coupled to dashboard 140. A dashboard may be defined as a user interface that allows a user to access and/or view the results of one or more collectors, validators, and/or notifiers in a pipeline monitoring system [a graphical representation of at least one of the determined first and second one or more metric values within a digital interface].”)
(Rickard, [0065], “In one embodiment, a validation value may also include an identifier that identifies the pipeline subsystem and/or fact data object that it relates to [and a corresponding process identifier].”)
(See annotated figure 1 of Rickard with annotated figure 8 of Elshawi)
PNG
media_image1.png
790
898
media_image1.png
Greyscale
Rickard teaches and based on a subset of the elements of process data characterizing the data-pipelining operations [that support the execution of a corresponding one of the machine learning or artificial intelligence processes] during the one or more first temporal intervals, perform operations that modify an execution of at least one of the data-pipelining operations during a second temporal interval, the second temporal interval being subsequent to the one or more first temporal intervals.
(Rickard, “[0079] In one embodiment, validator 120 can be programmed or configured to automatically resolve an issue identified by a validation value if the validation value has changed over time [based on a subset of the elements of process data characterizing the data-pipelining operations during the one or more first temporal intervals ie validation values in a previous time, perform operations that modify an execution of at least one of the data-pipelining operations during a second temporal interval ie automatically resolve an issue during a second temporal interval (ie the time after the first validation values were identified), the second temporal interval being subsequent to the one or more first temporal intervals]. For example, validator 120 can initially determine a first validation value that indicates that a pipeline subsystem experienced a failure of some sort. Validator 120 can later determine a second validation value that indicates that the pipeline subsystem has experienced a success of some sort, thereby implying that prior failure has been resolved. In one embodiment, validator 120 can send a notification to notifiers 130 and/or dashboard 140 that the first validation value indicating a failure has been resolved. By automatically detecting and resolving validation values that have changed over time, the validator 120 can reduce the need for manual inspection and intervention for prior failures in the pipeline that have since been corrected.”)
However, Rickard does not explicitly teach [data-pipelining operations] that support an execution of a plurality of machine learning or artificial intelligence processes during one or more first temporal intervals,… for each of the machine learning or artificial intelligence processes
Elshawi teaches operations that support an execution of a plurality of machine learning or artificial intelligence processes during one or more first temporal intervals,… [for each of the] machine learning or artificial intelligence processes
(Elshawi, Section 6. Figure 8, “These aspects belongs to two main building blocks of the machine learning production pipeline [machine learning or artificial intelligence processes; ie machine learning production pipelines]: Pre-Modeling and Post-Modeling (Figure 8) [operations that support an execution of a plurality of machine learning or artificial intelligence processes during one or more first temporal intervals]. In general, Pre-Modeling is an important block of the machine learning pipeline that can dramatically affect the outcomes of the automated algorithm selection and hyper-parameters optimization process. The pre-modeling step includes a number of steps including data understanding, data preparation and data validation. In addition, the Post-Modeling block covers other important aspects including the management and deployment of produced machine learning model which represents a corner stone in the pipeline that requires the ability of packaging model for reproducibility.”)
Rickard is considered to be analogous to the claimed invention because they are in the same field of pipeline monitoring. Elshawi is considered to be analogous to the claimed invention because they are in the same field of automated machine learning. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Rickard to incorporate the teachings of Elshawi in order to provide a practical application of machine learning to the pipeline processing system of Rickard as doing so would enable monitoring of separate machine learning phases to identify points of failure (Rickard, “[0003] In computer systems, a pipeline is a set of one or more coupled pipeline subsystems that process and/or analyze data. Each pipeline subsystem consists of computer programs or dedicated computers that receive data from a source, process or transform the data, and forward the data to another program, computer or system. Such pipelines can be particularly fragile as any issue encountered at one pipeline subsystem, such as misformatted data, incomplete data, or limited computing resources, can cause the other pipeline subsystems to fail or suffer from degraded performance. Therefore, some form of monitoring would be useful, but existing systems have not provided effective solutions.”)
In regards to claim 3,
Rickard and Elshawi teach The apparatus of claim 1,
Rickard teaches wherein: the at least one processor is further configured to execute the instructions to: determine, based on the process data elements associated with each of the machine learning or artificial intelligence processes (wherein Elshawi teaches the machine learning process as a machine learning production pipeline; see claim 1), a first aggregate value characterizing the number of failed executions of the data-pipelining operations during the one or more first temporal intervals and a second aggregate value characterizing the number of successful executions of the operations during the one or more first temporal intervals; and compute at least one of an aggregate failure rate or an aggregate success rate based on the first and second aggregate values; and the status data comprises the first aggregate value, the second aggregate value, and the at least one of the aggregate failure rate or the aggregate success rate.
(Rickard, [0094], “A summary card may additionally include a graph that that illustrates the historical timeline of validation values for the one or more pipeline subsystems. For example, the graph may be a histogram or line graph [and compute at least one of an aggregate failure rate or an aggregate success rate based on the first and second aggregate values; wherein the line graph is computing the rate of change based on the respective past validation values] of past validation results for the particular pipeline subsystems associated with the summary card. In one embodiment, the graph may be limited to a subset of validation values or statuses [a first aggregate value ie validation value of failure count as taught in claim 1 characterizing the number of failed executions of the data-pipelining operations during the one or more first temporal intervals and a second aggregate value ie validation value as number of successes as taught in claim 1 characterizing the number of successful executions of the operations during the one or more first temporal intervals]. For example, the graph may only display critical status validation values. The summary cards thus provide a quick aggregate overview of the health of pipeline [and the status data comprises the first aggregate value, the second aggregate value, and the at least one of the aggregate failure rate or the aggregate success rate], by allowing a user computer to quickly view what kinds of issues may be affecting pipeline subsystems.”)
In regards to claim 4,
Rickard and Elshawi teach The apparatus of claim 3,
Rickard teaches wherein the at least one processor is further configured to execute the instructions to determine the first aggregate value and the second aggregate value during each of the one or more first temporal intervals.
(Rickard, “[0040] Collector 110 may be coupled to validator 120. Although depicted as a single validator 120 in pipeline monitoring system 100, in another embodiment, multiple validators may be included. A validator may be defined as a process that is programmed or configured to apply one or more validation criteria to one or more fact data objects in order to determine a validation value that represents whether the pipeline is in a healthy state [determine the first aggregate value and the second aggregate value during each of the one or more first temporal intervals].”)
In regards to claim 5,
Rickard and Elshawi teach The apparatus of claim 1,
Rickard teaches wherein the at least one processor is further configured to execute the instructions to: compute a rate of successful or failed execution of the data-processing operations associated with each of the machine learning or artificial intelligence processes (wherein Elshawi teaches the machine learning process as a machine learning production pipeline; see claim 1) based on corresponding ones of the first and second values; and generate, for each of the machine learning or artificial intelligence processes, an element of the status data that includes the corresponding process identifier, the corresponding ones of the first and second values, and corresponding ones of the success and failure rates.
(Rickard, [0094], “A summary card may additionally include a graph that that illustrates the historical timeline of validation values for the one or more pipeline subsystems. For example, the graph may be a histogram or line graph [compute a rate of successful or failed execution of the data-processing operations associated with each of the machine learning or artificial intelligence processes based on corresponding ones of the first and second values] of past validation results for the particular pipeline subsystems associated with the summary card. In one embodiment, the graph may be limited to a subset of validation values or statuses. For example, the graph may only display critical status validation values. The summary cards thus provide a quick aggregate overview of the health of pipeline [generate, for each of the machine learning or artificial intelligence processes, an element of the status data that includes the corresponding process identifier, the corresponding ones of the first and second values, and corresponding ones of the success and failure rates], by allowing a user computer to quickly view what kinds of issues may be affecting pipeline subsystems.”)
(Rickard, [0065], “In one embodiment, a validation value may also include an identifier that identifies the pipeline subsystem and/or fact data object that it relates to [corresponding process identifier].”)
In regards to claim 6,
Rickard and Elshawi teach The apparatus of claim 5,
Rickard teaches wherein the at least one processor is further configured to execute the instructions to determine, each of the machine learning or artificial intelligence processes (wherein Elshawi teaches the machine learning process as a machine learning production pipeline; see claim 1), the corresponding ones of the first and second values during each of the one or more first temporal intervals.
(Rickard, “[0040] Collector 110 may be coupled to validator 120. Although depicted as a single validator 120 in pipeline monitoring system 100, in another embodiment, multiple validators may be included. A validator may be defined as a process that is programmed or configured to apply one or more validation criteria to one or more fact data objects in order to determine a validation value that represents whether the pipeline is in a healthy state [determine the corresponding ones of the first and second values during each of the one or more first temporal intervals].”)
In regards to claim 7,
Rickard and Elshawi teach The apparatus of claim 1,
Rickard teaches wherein the at least one processor is further configured to execute the instructions to: obtain scheduling data associated with a corresponding one of the machine learning or artificial intelligence processes (wherein Elshawi teaches the machine learning process as a machine learning production pipeline; see claim 1),
(Rickard, “[0055] In an embodiment, an application collector is a type of collector that is programmed or configured to collect a fact data object from a pipeline subsystem that includes a software application or software system. For example, in one embodiment, an application collector may collect a fact data object from one or more job orchestration applications. A job orchestration application is an application that may be used to create, configure, monitor, and/or schedule jobs for running on a computer system. In one embodiment, an application collector may collect a fact data object from a pipeline subsystem via an application programming interface (API). For example, pipeline subsystem 104 may have an exposed API. Collector 110 can thus use the API of the pipeline subsystem 104 to request fact data objects from the pipeline subsystem 104 [obtain scheduling data ie fact data objects about the job process/pipeline subsystem associated with a corresponding one of the machine learning or artificial intelligence processes]. The pipeline subsystem 104, in response to the request, will send a response message containing one or more fact data objects to the collector 110.”)
Rickard teaches and generate an element of the status data that includes the corresponding process identifier, the first temporal data, and the second temporal data.
(Rickard, “[0103] A row entry may include one or more time period summary panels 824. A time period summary panel is a visualization of fact data objects [generate an element of the status data that includes the corresponding process identifier, the first temporal data, and the second temporal data], validation values, and/or notifications related to a row entry for a particular period of time, such as an hour, day, week, etc. In one embodiment, a time period summary panel 824 may include a first numerical indicator that identifies the number of validation values and/or notifications that have failed to satisfy some criteria over the given time period. For example, the first indicator may identify the number of validation values that have not received a “Pass” status for the given row entry. Thus, in time period summary panel 824, the first numerical indicator indicates that eight validation values do not have a “Pass” status for the “Server” tag on July 27. In one embodiment, a time period summary panel 824 may include a second numerical indicator that identifies the total number of validation values and/or notifications received for the particular row entry over the given time period. Thus, in time period summary panel 824, the second numerical indicator indicates that twelve validation values have been received for the “Server” tag on July 27.”)
However, Rickard does not explicitly teach the scheduling data comprising first temporal data and second temporal data, the first temporal data characterizing a scheduled initiation of a first one of the operations associated with the corresponding machine learning or artificial intelligence processes, and the second temporal data characterizing a scheduled provisioning of predicted output associated with the corresponding machine learning or artificial intelligence processes to a computing system;
Elshawi teaches the scheduling data comprising first temporal data and second temporal data, the first temporal data characterizing a scheduled initiation of a first one of the operations associated with the corresponding machine learning or artificial intelligence processes,
Examiner’s note: Examiner interprets “first temporal data characterizing a scheduled initiation of a first one of the operations” in its BRI as the time it takes for the first operation (pipeline subsystem) to execute.
(Elshawi, Section 6., “The pre-modeling step [the first temporal data characterizing a scheduled initiation of a first one ie pre-modeling of the operations associated with the corresponding machine learning or artificial intelligence processes] includes a number of steps including data understanding, data preparation and data validation.”)
Elshawi teaches and the second temporal data characterizing a scheduled provisioning of predicted output associated with the corresponding machine learning or artificial intelligence processes to a computing system;
(Elshawi, Section 6., “In addition, the Post-Modeling block covers other important aspects including the management and deployment [the second temporal data characterizing a scheduled provisioning of predicted output ie deployment of the model associated with the corresponding machine learning or artificial intelligence processes to a computing system] of produced machine learning model which represents a corner stone in the pipeline that requires the ability of packaging model for reproducibility.”)
In regards to claim 8,
Rickard and Elshawi teach The apparatus of claim 7,
Rickard teaches wherein the at least one processor is further configured to execute the instructions to: obtain third temporal data associated with the corresponding machine learning or artificial intelligence process (wherein Elshawi teaches the machine learning process as a machine learning production pipeline; see claim 1), the third temporal data characterizing a re-initiation of the first one of the operations based on a prior failure in an execution of the first one of the operations; and generate an element of the status data that includes the corresponding process identifier and the third temporal data.
(Rickard, “[0079] In one embodiment, validator 120 can be programmed or configured to automatically resolve an issue identified by a validation value [the third temporal data characterizing a re-initiation of the first one of the operations based on a prior failure in an execution of the first one of the operations] if the validation value has changed over time. For example, validator 120 can initially determine a first validation value that indicates that a pipeline subsystem experienced a failure of some sort. Validator 120 can later determine a second validation value that indicates that the pipeline subsystem has experienced a success of some sort, thereby implying that prior failure has been resolved. In one embodiment, validator 120 can send a notification to notifiers 130 and/or dashboard 140 that the first validation value indicating a failure has been resolved [and generate an element of the status data that includes the corresponding process identifier and the third temporal data]. By automatically detecting and resolving validation values that have changed over time, the validator 120 can reduce the need for manual inspection and intervention for prior failures in the pipeline that have since been corrected.”)
In regards to claim 10,
Rickard and Elshawi teach The apparatus of claim 1,
Rickard teaches wherein the at least one processor is further configured to execute the instructions to: obtain a process identifier associated with at least one of the machine learning or artificial intelligence processes (wherein Elshawi teaches the machine learning process as a machine learning production pipeline; see claim 1),
(Rickard, [0065], “In one embodiment, a validation value may also include an identifier that identifies the pipeline subsystem and/or fact data object that it relates to [obtain a process identifier associated with at least one of the machine learning or artificial intelligence processes].”)
Rickard teaches and based on the process identifier, obtain elements of scheduling data and process input data associated with the at least one machine learning or artificial intelligence process; perform operations that execute the one or more data-pipelining operations associated with the at least one machine learning or artificial intelligence process based on the process input data and the scheduling data; and generate, for the at least one machine learning or artificial intelligence process, an element of the process data associated with each of the executed data-pipelining operations, the elements of the process data characterizing a success or a failure of corresponding ones of the executed data-pipelining operations.
(Rickard, “[0079] In one embodiment, validator 120 can be programmed or configured to automatically resolve an issue identified by a validation value [based on the process identifier; wherein the validation value includes the process identifier] if the validation value has changed over time. For example, validator 120 can initially determine a first validation value that indicates that a pipeline subsystem experienced a failure of some sort [obtain elements of scheduling data and process input data associated with the at least one machine learning or artificial intelligence process; wherein the validator obtains information from the collector]. Validator 120 can later determine a second validation value that indicates that the pipeline subsystem has experienced a success of some sort [perform operations that execute the one or more data-pipelining operations associated with the at least one machine learning or artificial intelligence process based on the process input data and the scheduling data; ie a second execution as represented by a second validation value], thereby implying that prior failure has been resolved. In one embodiment, validator 120 can send a notification to notifiers 130 and/or dashboard 140 that the first validation value indicating a failure has been resolved [and generate, for the at least one machine learning or artificial intelligence process, an element of the process data associated with each of the executed data-pipelining operations, the elements of the process data characterizing a success or a failure of corresponding ones of the executed data-pipelining operations; wherein the validation value indicates success/failure]. By automatically detecting and resolving validation values that have changed over time, the validator 120 can reduce the need for manual inspection and intervention for prior failures in the pipeline that have since been corrected.”)
In regards to claim 11,
Rickard and Elshawi teach The apparatus of claim 1,
Rickard teaches wherein the status data further causes the device to: generate and present, and for each of the machine learning or artificial intelligence processes (wherein Elshawi teaches the machine learning process as a machine learning production pipeline; see claim 1), a first graphical representation of a first subset of the one or more metric values within a portion of the digital interface;
(Rickard, [0094], “A summary card may additionally include a graph that that illustrates the historical timeline of validation values for the one or more pipeline subsystems. For example, the graph may be a histogram or line graph of past validation results for the particular pipeline subsystems associated with the summary card. In one embodiment, the graph may be limited to a subset of validation values or statuses. For example, the graph may only display critical status validation values. The summary cards thus provide a quick aggregate overview of the health of pipeline [a first graphical representation of a first subset of the one or more metric values within a portion of the digital interface], by allowing a user computer to quickly view what kinds of issues may be affecting pipeline subsystems.”)
Rickard teaches receive input data indicative of a selection of one or the first graphical representation associated with a corresponding one of the machine learning or artificial intelligence processes;
(Rickard, “[0095] In one embodiment, user interface 200 may include a search field 206 that allows a user to quickly search for summary cards that match a particular criteria. In another embodiment, user interface 200 may include settings (not depicted) that allow a user computer to customize their view of the summary cards [receive input data indicative of a selection of one or the first graphical representation associated with a corresponding one of the machine learning or artificial intelligence processes].”)
Rickard teaches and based on the input data, generate and present a second graphical representation of a second subset of the one or more metric values associated with the corresponding machine learning or artificial intelligence process within an additional portion of the digital interface.
(Rickard, “[0094] User interface 200 may include one or more summary cards 210, 212, 214, 216 [based on the input data, generate and present a second graphical representation of a second subset of the one or more metric values associated with the corresponding machine learning or artificial intelligence process within an additional portion of the digital interface; ie a second filtered summary card]. A summary card is a visual summary of the health of one or more pipeline subsystems.”)
Claims 12 and 20 are rejected on the same grounds under 35 U.S.C. 103 as claim 1 as they are substantially similar, respectively, Mutatis mutandis.
Claim 14 is rejected on the same grounds under 35 U.S.C. 103 as claim 3 as they are substantially similar, respectively, Mutatis mutandis.
Claim 15 is rejected on the same grounds under 35 U.S.C. 103 as claim 5 as they are substantially similar, respectively, Mutatis mutandis.
Claim 16 is rejected on the same grounds under 35 U.S.C. 103 as claim 7 as they are substantially similar, respectively, Mutatis mutandis.
Claim 17 is rejected on the same grounds under 35 U.S.C. 103 as claim 8 as they are substantially similar, respectively, Mutatis mutandis.
Claim 18 is rejected on the same grounds under 35 U.S.C. 103 as claim 10 as they are substantially similar, respectively, Mutatis mutandis.
Claim 19 is rejected on the same grounds under 35 U.S.C. 103 as claim 11 as they are substantially similar, respectively, Mutatis mutandis.
In regards to claim 21,
Rickard and Elshawi teach The apparatus of claim 1,
Rickard teaches wherein the at least one processor is further configured to execute the instructions to: perform operations that initiate an execution of the data-pipelining operations associated with a corresponding one of the machine learning or artificial intelligence processes (wherein Elshawi teaches the machine learning process as a machine learning production pipeline; see claim 1);
Examiner’s note: Examiner interprets the limitation as executing the machine learning pipeline.
(Rickard, “[0129] Functions of the disclosed systems, methods, and modules may be performed by computing device 600 in response to processor(s) 604 executing one or more programs of software instructions contained in main memory 606.”)
Rickard teaches and obtain at least a portion of the process data associated with the corresponding one of the machine learning or artificial intelligence processes during the execution of the data-pipelining operations associated with the corresponding one of the machine learning or artificial intelligence processes.
(Rickard, “[0051] Collectors can collect varied types of fact data objects depending on the type of pipeline subsystem that is providing the fact data object [obtain at least a portion of the process data associated with the corresponding one of the machine learning or artificial intelligence processes during the execution of the data-pipelining operations associated with the corresponding one of the machine learning or artificial intelligence processes]. Examples of fact data objects may include information related to the amount of data being ingested and transmitted by the pipeline subsystem, such as the volume of data (e.g. number of rows in a table, aggregate size of data, etc.) received by the pipeline subsystem during a period of time, the success or failure of a job process”)
In regards to claim 22,
Rickard and Elshawi teach The apparatus of claim 1,
Rickard teaches wherein: the at least one processor is further configured to execute the instructions to, based on the process data, determine, for each of the machine learning or artificial intelligence processes (wherein Elshawi teaches the machine learning process as a machine learning production pipeline; see claim 1), the metric values characterizing the status of each of the data-processing operations that support the execution of the corresponding machine learning or artificial intelligence process;
(Rickard, “[0040] Collector 110 [based on the process data] may be coupled to validator 120. Although depicted as a single validator 120 in pipeline monitoring system 100, in another embodiment, multiple validators may be included. A validator may be defined as a process that is programmed or configured to apply one or more validation criteria to one or more fact data objects in order to determine a validation value that represents whether the pipeline is in a healthy state [metric values ie validation values characterizing a status of the data-pipelining operations].”)
Rickard teaches and the status data further causes the device to present, for each of the machine learning or artificial intelligence processes within the digital interface, a graphical representation of at least one of the determined metric values for each of the data-pipelining operations.
(Rickard, “[0043] One or more of collector 110, validator 120, and/or notifier 130 may be coupled to dashboard 140. A dashboard may be defined as a user interface that allows a user to access and/or view the results of one or more collectors, validators, and/or notifiers in a pipeline monitoring system.”)
In regards to claim 23,
Rickard and Elshawi teach The apparatus of claim 1,
Elshawi teaches wherein the data-pipelining operations comprise, for each of the plurality of machine learning processes, at least one of a data ingestion operation, a data preparation operation, or a data post-processing operation.
(Elshawi, Section 6., “The pre-modeling step includes a number of steps including data understanding, data preparation and data validation [data preparation operation].”)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US Pub No. US20200272947A1 Carullo et al. teaches Orchestrator for machine learning pipeline
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASMINE THAI whose telephone number is (703)756-5904. The examiner can normally be reached M-F 8-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.T.T./Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
Prosecution Insights