Skip to content

Migrate backend from raw REST API to PythonCall.jl + CondaPkg.jl #79

Description

@pebeto

Historically, MLFlowClient.jl has operated as a pure Julia wrapper around the MLflow REST API. However, keeping pace with MLflow's upstream development has become unsustainable.

While MLflow technically provides an OpenAPI/Swagger specification, their backend relies heavily on Protocol Buffers. In practice, this means the REST API frequently experiences undocumented changes, volatile internal endpoints (e.g., /ajax-api/), and inconsistent payload typing. Maintaining a manual HTTP/JSON mapping against a fast-moving, fundamentally Python-first ecosystem requires constant reverse-engineering and patches, which is no longer feasible for this package's long-term health.

I propose we move the core architecture of MLFlowClient.jl to act as an idiomatic Julia bridge to the official Python mlflow client, utilizing PythonCall.jl paired with CondaPkg.jl.

Workflow:

  • CondaPkg.jl will automatically provision an isolated, invisible Python environment and install the mlflow package when a user installs MLFlowClient.jl. The user does not need to manage Python themselves
  • PythonCall.jl will handle the zero-overhead data translation between Julia types and the underlying Python library
  • End-users will still write pure Julia code (e.g., using do blocks instead of Python with context managers)

Considerations

Pros

  • Parity: When MLflow releases new features (like recent LLM tracking or Model Registry updates), Julia users get access to them immediately without waiting for us to reverse-engineer the REST payloads
  • Artifact logging: Uploading complex artifacts or models becomes significantly easier than raw HTTP multipart/form-data POST requests (a common issue in the history of the package)
  • Reduction in tech debt: Reduced JSON serialization and HTTP boilerplate code

Cons

  • No pure Julia: Python is now a dependency
  • Startup latency: The Python environment will increase the package initialization time

@ablaom @deyandyankov

Activity

  1. self-assigned this
    on Mar 24, 2026
  2. added
    documentationImprovements or additions to documentation
    enhancementNew feature or request
    help wantedExtra attention is needed
    on Mar 24, 2026
  3. added theissue type on Mar 24, 2026
  4. ablaom commented on Mar 24, 2026

    @ablaom
    Member

    Thank you @pebeto for this history and concrete suggestion for moving forward. While maintaining julia wrappers for python code introduces a non-trivial maintenance burden, I agree after our discussions and this post, that the new approach is likely less hassle than what we have been facing recently, and I cannot think of a better alternative.

    Again many thanks for your expertise and all the work on MLJ-mlflow integration to date.

  5. ablaom commented on Mar 26, 2026

    @ablaom
    Member

    @pebeto We have had some excellent help with PythonCall from @tylerjthomas9, who has agreed to be available as a consultant on this migration.

  6. pebeto commented on Mar 30, 2026

    @pebeto
    MemberAuthor

    Here is a POC with working tests calling to the Python mlflow package. For now, the most challenging part of it is to match the types.

  7. Qfl3x commented on Jul 14, 2026

    @Qfl3x

    Is this still in the works?

  8. pebeto commented on Jul 14, 2026

    @pebeto
    MemberAuthor

    Hello @Qfl3x.

    This is something we want to support in the future (and flagged as experimental). Current versions of MLFlow already solved issues that we identified 2 years ago.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

documentationImprovements or additions to documentationenhancementNew feature or requesthelp wantedExtra attention is needed

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions