FINAL REJECTION, SECOND DETAILED ACTION
Status of Prosecution
The present application, 18/358,302 filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The application was filed in the Office on June 9, 2023 July 25, 2023.
The Office mailed a first detailed action, non-final rejection on May 11, 2026.
Applicant filed amendments with remarks and arguments on August 7, 2026.
Claims 1-20 are pending. Claims 1, 13 and 19 are independent.
Status of Claims
Claims 1-4, 11, 13-16 and 19-20 are rejected under 35 U.S.C. § 103 as being unpatentable over Yuan et al., (“Yuan”) United States Patent 11,599,813 B1, published on March 7, 2023 in view of non-patent literature Miao et al.(“ Miao”), “Towards Unified Data and Lifecycle Management for Deep Learning,” published in 2017 in further view of Negri et al. (“Negri”), United States Patent Application Publication 2022/0270523, published on Aug. 25, 2022.
Claims 5-8 and 17-18 are rejected under 35 U.S.C. § 103 as being unpatentable over Yuan in view of Miao in view of Negri and in further view of non-patent literature Hien Luu, “Beginning Apache Spark 3,” published in 2021 (“Luu”).
Claims 9-10 are rejected under 35 U.S.C. § 103 as being unpatentable over Yuan in view of Miao in view of Negri and in further view of non-patent literature Chen et al. (“Chen”), “Developments in MLflow: A System to Accelerate the Machine Learning Lifecycle,” published in 2020.
Claim 12 is rejected under 35 U.S.C. § 103 as being unpatentable over Yuan in view of Miao in view of Negri and in further view of Davidson et al. (“Davidson”), United States Patent Application Publication 2022/0092043, published on March 24, 2022.
Response to Remarks and Arguments
Examiner thanks Applicant for the submitted remarks and arguments and the amended claims.
Examiner has considered the amended features in the independent claims and has newly rejected the claims with the application of Negri to teach the use of webhooks as applied in different spaces (Remarks: pp. 10-11).
Applicant also contends with regard to the relative progress data that Yuan’s cited portions in the last Office Action that the “binary check of whether a deployment endpoint is available is not relative progress data for the machine learning model generated based on the model activity data,” as required by the claim (Remarks: p. 11). Examiner respectfully disagrees. While Examiner finds Applicant’s characterization of “relative progress” to not be merely a binary check of whether the endpoint is available or not, Examiner asserts that a broadest reasonable interpretation here that “relative progress” as used in claim 1 is commensurate in scope with Yuan’s disclosure, as progress may simply be one of completion or not, relative to the beginning. A review of the Specification further does not indicate special definition of relative progress.
Applicant also further argues that that claim 7’s further limiting of what relative progress is and a reliance on Luu instead indicates the deficiency of Yuan. Examiner respectfully disagrees. Luu is applied and combined to address the narrower scope of “relative progress” accordingly; the application of Luu is not inapposite with the use of Yuan in the parent independent claim, as the combination is appropriate and a further modification of the existing parent claims.
Applicant also contends that Luu itself does not teach the relative progress data (Remarks: p. 12). Applicant characterizes the discrete categorical stage labels as depicted and disclosed by Luu are not indicative of the proportion of the one or more model requirements (Id.). Examiner respectfully disagrees. As noted in the claim mapping from Yuan, the workflow is known with different steps in the training (Yuan: col. 3, lines 33 to 40, the workflow is associated with different steps or stages of the machine learning task including data sourcing, model training, deployment etc.; col. 3, lines 40-49, the series of steps “stitched” together are stage-specific model requirements). Combined with Luu’s display of which stage the workflow is in which is ordinal in nature, Examiner asserts a proportion is captured and displayed for the user. Examiner is not persuaded.
Accordingly, the combination renders the claim obvious and unpatentable.
The claims stand rejected.
Claim Interpretation Notes
Examiner notes the following relevant definitions from the Specification for the reader’s convenience:
[0057]In some embodiments, the term “compute agnostic project workspace” refers to workspace that is at least partially hosted by a first party computing resource and/or at least one third party computing resource. The compute agnostic project workspace may support multiple compute choices for a machine learning project including on-prem, first party, solutions and third party solutions, such as cloud service platforms (e.g., Kubernetes, Spark, AML, Sagemaker, Databricks, etc.). For example, the compute agnostic project workspace may aggregate data and functionality across a plurality of first and/or third party workspaces to allow users (e.g., data scientists, etc.) to take advantage of different compute choices for handling different stages, workloads, and/or the like of a machine learning project from one centralized workspace, while working with consistent contracts for data access, analysis, model building, deployment, and/or the like.
[0066]In some embodiments, the term “canonical representation” refers to a data entity that represents a standardized representation of a machine learning project. The canonical representation may include a plurality of model characteristics that describe one or more aspects of the development and/or performance of a machine learning model orchestrated using a first party computing resource (e.g., a compute agnostic project workspace thereof). For example, the canonical representation may include configuration data, evaluation data, and/or the like, for a machine learning model developed, maintained, and/or at least partially hosted within a compute agnostic project workspace. In some examples, the canonical representation may include interface points (e.g., interactive links, pointer, API endpoints, etc.) for accessing the machine learning model and/or a workspace for a portion of the machine learning model (e.g., hosted by a first and/or third party computing resource, etc.).
Claims Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
A.
Claims 1-4, 11, 13-16 and 19-20 are rejected under 35 U.S.C. § 103 as being unpatentable over Yuan et al., (“Yuan”) United States Patent 11,599,813 B1, published on March 7, 2023 in view of non-patent literature Miao et al.(“ Miao”), “Towards Unified Data and Lifecycle Management for Deep Learning,” published in 2017 in further view of Negri et al. (“Negri”), United States Patent Application Publication 2022/0270523, published on Aug. 25, 2022.
As to Claim 1, Yuan teaches: A computer-implemented method comprising:
generating, by one or more processors of a first party computing resource (Yuan: Fig. 1, the machine learning management system [100] is connected to a client computing device [20] (i.e. a first party computing resource), a representation of a machine learning model associated with a compute agnostic project workspace (Yuan: Fig. 1; col. 3, lines 33-45 workflow templates 111, workflow step library 112, which may include steps that represent machine learning tasks (i.e. a representation of a machine learning model) that is communicatively connected to a plurality of third party workspaces hosted by one or more of a plurality of different third party computing resources (Yuan:, Fig. 1, multi-tenant provider network 190; col. 10, lines 1-9, the provider network may be connected to a distributed set of clients (i.e. different workspaces, different computing resources)).
PNG
media_image1.png
750
584
media_image1.png
Greyscale
in response to user activity within a third party workspace of the plurality of third party workspaces, receiving, by the one or more processors, model activity data from a third party computing resource associated with the third party workspace (Yuan: col. 6, lines 14-17, col. 12, line 62 to col. 13, lines 7, the input and output tracking (i.e. user activity within the workspaces) is used by model monitoring [370] to determine whether to retrain the models);
generating, by the one or more processors, relative progress data for the machine learning model based on the model activity data (Yuan: col. 15, lines32-36, The real-time inference workflow may wait 665 (e.g., for a predetermined period of time) and then check the endpoint
status 670, potentially more than once until the status is determined.” Examiner asserts the status here is “relative progress data”);
modifying, by the one or more processors, the representation of the machine learning model based on the model activity data and the relative progress data (Yuan: col. 12, line 62 to col. 13, lines 7, the input and output tracking (i.e. user activity within the workspaces) is used by model monitoring [370] to determine whether to retrain the model; col. 15, lines 36-45, the status is determined to decide whether the model deployment is successful, which Examiner asserts would be used on the next iteration to determine the retraining);
in response to the representation satisfying a publication threshold, generating, by the one or more processors, a model interface point for accessing the machine learning model (Yuan: col. 15, lines 36-45, the status is determined to decide whether the model deployment is successful (i.e. a publication threshold is satisfied), and thus a status of the particular version of the model is then updated in the model registry [687]); and
providing, by the one or more processors, the model interface point for the machine learning model to one or more users through a model registry (Yuan: col. 5, line 60 to col. 6, line 10, the model registry [160] has versioning with beta and production-ready versions of the model with version identifiers (i.e. model interface point) that is able to be used to utilize the machine learning model).
Yuan may not explicitly teach: generating, by one or more processors of a first party computing resource, a canonical representation of a machine learning model associated with a compute agnostic project workspace that is communicatively connected to a plurality of third party workspaces hosted by one or more of a plurality of different third party computing resources;
modifying, by the one or more processors, the canonical representation of the machine learning model based on the model activity data and the relative progress data;
in response to the canonical representation satisfying a publication threshold, generating, by the one or more processors, a model interface point for accessing the machine learning model
While Yuan does teach the use of workflow templates, and this may reasonably considered to be a “canonical representation,” Examiner notes that Applicant has specially defined “canonical representation” as a “standardized representation of a machine learning project.” If standardized is to be interpreted narrowly to be one that has a formal standards sense, then Yuan may not explicitly teach this.
Miao teaches in general concepts related to a data and lifecycle management system for deep learning(Miao: Abstract). Specifically, Miao teaches a high-level domain specific language to raise the abstraction level and accelerate the modeling process (Miao, Sec. III,A, a data model with a DNN model for the conceptual model and a data model for the version in a repository). A modeler can interact with them via queries (Miao: Sec. III. B), which Examiner asserts requires a standardized representation of the models for interactions.
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the Yuan disclosures and teachings by utilizing a standardized system for querying a model registry as taught and suggested by Miao. Such a person would have been motivated to do so with a reasonable expectation of success to allow for the ease of access and ability to interact with a collection of models readily (Miao: Abstract).
Yuan and Miao may not explicitly teach: the compute agnostic project workspace being communicatively connected to the plurality of third party workspaces via a first party routine set comprising a webhook installed by the first party computing resource within a third party workspace of the plurality of third party workspaces to establish communication between the compute agnostic project workspace and the third party workspace.
Negri teaches in general concepts related to a content management system that manages content types across multiple content spaces (Negri: Abstract). Specifically, a template is able to be used to be applied and installed on different spaces that will include webhooks to allow communication with appropriate servers and other services (Negri: Fig. 4, par. 0093, “ a template 410 is generated from a source content space 400. The content space 400 includes a content type 402 and also includes webhooks 404. These aspects of the source content space 400 can be added/copied to the template 410, so that the template 410 now includes the content type 414 and the webhooks 412 which are defined in accordance with those of the source content space 400.”).
PNG
media_image2.png
594
690
media_image2.png
Greyscale
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the Yuan-Miao disclosures and teachings by implementing webhooks to be installed by the first computing resource to then be used on the third party workspaces as taught and suggested by Negri. Such a person would have been motivated to do so with a reasonable expectation of success to allow for the ease of access communication by using the versatile webhooks.
As to Claim 2, Yuan, Miao and Negri teach the elements of claim 1.
Yuan further teaches: wherein the relative progress data is indicative of a relative development progress of the machine learning model relative to a model stage of one or more model stages (Yuan: col. 3, lines 33 to 40, the workflow is associated with different steps or stages of the machine learning task including data sourcing, model training, deployment etc. Examiner asserts that the se stages are captured in the workflow steps/template of the canonical representation).
As to Claim 3, Yuan, Miao and Negri teach the elements of claim 1.
Yuan, Miao and Negri further teaches: wherein the canonical representation comprises one or more model stage representations that are indicative of one or more model stages for the machine learning model (Yuan: col. 3, lines 33 to 40, the workflow is associated with different steps or stages of the machine learning task including data sourcing, model training, deployment etc.. Examiner asserts that the stages are captured in the workflow steps/template of the canonical representation; Miao, Sec. III.A., “In brief, N;W; M; F are the network definition, weight values, extracted metadata and associated files respectively”), wherein a model stage representation of the one or more model stage representations is indicative of:
(i) a plurality of stage-specific model criteria associated with a model stage of the one or more model stages that corresponds to the model stage representation(Yuan: col. 3, lines 36 to 49, the template may have steps that are particular to certain tasks or aspects that may depend on conditions (i.e. criteria) such as data breach for instance), and
(ii) a stage-specific, third party interface point for accessing one or more third party workspaces associated with the model stage (Yuan: col. 5, line 60 to col. 6, line 10, the model registry [160] has versioning with beta and production-ready versions of the model with version identifiers (i.e. model interface point) that is able to be used to utilize the machine learning model).
As to Claim 4, Yuan, Miao and Negri teach the elements of claim 3.
Yuan further teaches: wherein the one or more model stages comprises a model configuration stage, a model data preparation stage, a model experiment stage, a model review stage, and a model deployment stage (Yuan: col. 3, lines 33 to 40, the workflow is associated with different steps or stages of the machine learning task including data sourcing, model training, deployment etc.).
As to Claim 11, Yuan, Miao and Negri teach the elements of claim 1.
Yuan, Miao and Negri as combined further teach: wherein the canonical representation comprises a model stage representation corresponding to a model deployment stage of the machine learning model, the model deployment stage is associated with one or more stage-specific model attributes that are indicative of a model usage, and the computer-implemented method further comprises:
receiving, using the model interface point, an access request for the machine learning model (Yuan: col. 6, lines 59-67, the application [180] may obtain information from the model registry [160] (and thus from a request));
generating usage data based on the access request (Yuan: col. 12, line 64 to col. 13, line 6, model monitoring [370] observes the input and inferencing metrics); and
modifying the model stage representation corresponding to the model deployment stage based on the usage data (Yuan: col. 12, line 64 to col. 13, line 6, the model monitoring will compare against the actual labels; Miao: Sec. III.B, the models are updated in the repository in the relational backend).
As to Claim 13, it is rejected for similar reasons as claim 1. Yuan further teaches a processor and computer readable media (col. 15, lines 46 to 47).
As to Claim 14, it is rejected for similar reasons as claim 2.
As to Claim 15, it is rejected for similar reasons as claim 3.
As to Claim 16, it is rejected for similar reasons as claim 4.
As to Claim 19, it is rejected for similar reasons as claim 1 and 13.
As to Claim 20, it is rejected for similar reasons as claim 11.
B.
Claims 5-8 and 17-18 are rejected under 35 U.S.C. § 103 as being unpatentable over Yuan et al., (“Yuan”) United States Patent 11,599,813 B1, published on March 7, 2023 in view of non-patent literature Miao et al.(“ Miao”), “Towards Unified Data and Lifecycle Management for Deep Learning,” published in 2017 in further view of Negri et al. (“Negri”), United States Patent Application Publication 2022/0270523, published on Aug. 25, 2022 and in further view of non-patent literature Hien Luu, “Beginning Apache Spark 3,” published in 2021 (“Luu”).
As to Claim 5, Yuan, Miao and Negri teach the elements of claim 3.
Yuan further teaches: wherein the plurality of stage-specific model criteria defines one or more stage-specific model attributes and one or more stage-specific model requirements (Yuan: col. 3, lines 33 to 40, the workflow is associated with different steps or stages of the machine learning task including data sourcing, model training, deployment etc.; col. 3, lines 40-49, the series of steps “stitched” together are stage-specific model requirements).
Yuan, Miao and Negri may not explicitly teach: wherein the model stage representation comprises a stage status indicator that is indicative of a proportion of the one or more model requirements that are satisfied by the machine learning model.
Luu is a book about the distributed processing system Apache Spark, and in particular, the use of machine learning development processes for use with Apache Spark(Luu: p. 399). Of interest is that a user interface is available with MLFlow to manage different models and in particular, their status in which stage and to appropriately transition them (Luu: pp. 424-425, Fig. 9-19).
PNG
media_image3.png
438
648
media_image3.png
Greyscale
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the Yuan-Miao-combination by allowing for a stage indicator associated with the model as it currently is transitioned in as taught by Luu. Such a person would have been motivated to do so with a reasonable expectation of success to allow for the user to visualize and control the model operations and development readily.
As to Claim 6, Yuan, Miao, Negri and Luu teach the elements of claim 5.
Luu further teaches: wherein the one or more stage-specific model attributes are indicative of one or more of a plurality of model attributes for the machine learning model that are associated with the model stage (Luu: Fig. 9-19, the stage column is indicative of the stage).
As to Claim 7, Yuan, Miao, Negri and Luu teach the elements of claim 5.
Yuan and Luu further teaches: wherein:
(i) the model activity data is indicative of one or more modified model attributes that correspond to the model stage(Luu: Fig. 9-19, the stage column is indicative of the stage), and
(ii) the relative progress data for the machine learning model is indicative of an updated proportion of the one or more model requirements that are satisfied by the machine learning model based on the one or more modified model attributes (Yuan: pp. 424-425, Fig. 9-19).
As to Claim 8, Yuan, Miao, Negri and Luu teach the elements of claim 7.
Yuan further teaches: wherein modifying the canonical representation of the machine learning model based on the model activity data and the relative progress data comprises:
augmenting the model stage representation with the one or more modified model attributes; and
modifying the stage status indicator based on the updated proportion (Luu: pp. 424-425, Fig. 9-19, the user is able to transition the model appropriately and the representation is updated accordingly, Examiner asserts per the combination).
As to Claim 17, it is rejected for similar reasons as claim 5.
As to Claim 18, it is rejected for similar reasons as claim 6.
C.
Claims 9-10 are rejected under 35 U.S.C. § 103 as being unpatentable over Yuan et al., (“Yuan”) United States Patent 11,599,813 B1, published on March 7, 2023 in view of non-patent literature Miao et al.(“ Miao”), “Towards Unified Data and Lifecycle Management for Deep Learning,” published in 2017 in further view of Negri et al. (“Negri”), United States Patent Application Publication 2022/0270523, published on Aug. 25, 2022 and in further view of non-patent literature Chen et al. (“Chen”), “Developments in MLflow: A System to Accelerate the Machine Learning Lifecycle,” published in 2020.
As to Claim 9, Yuan, Miao and Negri teach the elements of claim 1.
Yuan, Miao and Negri may not explicitly teach: wherein the model registry comprises a local model registry that is associated with a plurality of visibility levels or a centralized model registry and the one or more users are based on a visibility level of the local model registry or the centralized model registry.
Chen teaches in general concepts related to MLflow, a popular open source platform for managing ML development (Chen: Abstract). Specifically, Chen teaches that MLFlow is able to control access to models in the model registry based on roles (Chen: Fig. 1, Sec. 4.1, Model management, “a per-user or per-role basis, and the Model Registry enables users to request stage transitions from colleagues.”). While Chen’s disclosure here is directed to access, Examiner asserts that the ability to access certain rights would be an obvious variant of visibility.
PNG
media_image4.png
342
730
media_image4.png
Greyscale
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the Yuan-Miao-Negri disclosures and teachings by restricting access and rights to the models as taught by Chen. Such a person would have been motivated to do so with a reasonable expectation of success to allow for the guarding against common pitfalls such as deploying broken or inferior models to production for instance (Chen: Sec. 4.1).
As to Claim 10, Yuan, Miao, Negri and Chen teach the elements of claim 9.
Yuan, Miao, Negri and Chen may not explicitly teach: wherein the publication threshold is based on the visibility level of the local model registry or the centralized model registry.
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have further modified the Yuan-Miao-Chen disclosures and teachings to only display models ready for certain use and thus meeting a publication threshold as suggested by Chen. Such a person would have been motivated to do so with a reasonable expectation of success to allow for the guarding against common pitfalls such as deploying broken or inferior models to production for instance (Chen: Sec. 4.1).
D.
Claim 12 is rejected under 35 U.S.C. § 103 as being unpatentable over Yuan et al., (“Yuan”) United States Patent 11,599,813 B1, published on March 7, 2023 in view of non-patent literature Miao et al.(“ Miao”), “Towards Unified Data and Lifecycle Management for Deep Learning,” published in 2017 in further view of Negri et al. (“Negri”), United States Patent Application Publication 2022/0270523, published on Aug. 25, 2022 and in further view of Davidson et al. (“Davidson”), United States Patent Application Publication 2022/0092043, published on March 24, 2022.
As to Claim 12, Yuan, Miao and Negri teach the elements of claim 11.
Yuan further teaches: wherein the model deployment stage is associated with one or more stage-specific model requirements that are indicative of a model usage threshold and the computer-implemented method further comprises:
generating one or more model usage metrics for the machine learning model based on the usage data Yuan: col. 12, line 64 to col. 13, line 6, model monitoring [370] observes the input and inferencing metrics).
Yuan and Miao not explicitly teach: removing the model interface point for the machine learning model from the model registry based on the one or more model usage metrics.
Davidson teaches in general concepts related to version control in a model registry (Davidson: Abstract). Specifically, Davidson teaches that a review of the endpoints of a model may be analyzed and if a process has failed, the model registry synchronizer will delete them from the model registry (Davidson: par. 0022).
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the Yuan-Miao-Negri disclosures and teachings by deleting the model in the instance of a usage metric (i.e. the failed endpoint detection) as taught and suggested by Davidson. Such a person would have been motivated to do so with a reasonable expectation of success to reduce visual, resource and cognitive clutter of extraneous models.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Additional relevant prior art made of the record:
Matthias Popp, “Comprehensive Support of the Lifecycle of Machine Learning
Models in Model Management Systems,” thesis, 0090073 (May 4, 2019) (describing machine learning model systems in review);
Schlegal et al., “Management of Machine Learning Lifecycle Artifacts: A Survey” (Dec. 2022) (lifecycle stages and MLFlow and other systems).
Vasilache et al. “Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions,” (Feb. 13, 2018);
Ghosh et al., US Patent 11,620,541 (Apr. 4, 2023) (access to model registry).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAMES T TSAI whose telephone number is (571)270-3916. The examiner can normally be reached M-F 8-5 Eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at 571-270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAMES T TSAI/ Primary Examiner, Art Unit 2147
Prosecution Insights