Skip to content

Versioning Strategies for Data Science & ML Projects

Git records changes to source code well, but large datasets and model files can make a Git repository slow and difficult to clone. Machine learning projects also need to connect a particular code revision to the exact data and parameters used for a run.

I presented my research on Data Version Control (DVC) at an Experts Meetup hosted by Vodafone Germany's Cloud Centre of Excellence in Düsseldorf. This post explains how DVC keeps small metadata files in Git while storing large artifacts separately, and what that design can and cannot reproduce.

Navigate this post

The Data Versioning Problem in ML

In standard software development, source code files are small text documents. Git excels at computing line-by-line diffs for these text files across thousands of commits. Machine learning projects, however, consist of three interdependent assets: source code, training datasets, and trained model artifacts (such as binary model weights).

When data scientists attempt to track raw datasets (like CSV, Parquet, or image archives) directly inside Git, the repository size explodes. Git stores complete copies of modified files in its local history database (.git folder). Committing a modified 2 GB dataset twice inflates the local repository size by 4 GB, making clones and fetches extremely slow for team members.

The Git Large File Trap

Attempting to push files larger than 50 MB to remote hosts like GitHub triggers severe performance warnings, while files exceeding 100 MB are rejected outright. Using Git Large File Storage (LFS) mitigates raw file size errors but incurs steep bandwidth storage charges and lacks built-in pipeline tracking tailored for machine learning.

Without a dedicated data versioning strategy, teams resort to informal file naming conventions (such as data_v1_final_v2.csv or model_weights_oct2024.pkl) stored on shared network drives. This approach leads to unverified model origins, untracked data drift, and an inability to reproduce historical experiment results accurately.

Git vs DVC Architectural Comparison

Data Version Control (DVC) solves data tracking by decoupling lightweight metadata pointers from heavy binary payloads. DVC operates alongside Git: Git tracks source code and DVC metadata files (.dvc pointers), while DVC transfers large data files to external storage targets such as Amazon S3, Google Cloud Storage, Azure Blob Storage, or SSH-accessible file servers.

The diagram below illustrates how Git and DVC collaborate in a unified project structure:

---
title: "Dual-Layer Versioning Architecture (Git + DVC)"
---
flowchart TB
    accTitle: Dual-layer versioning with Git and DVC
    accDescr: Git stores source code and DVC metadata, while DVC stores large datasets and model files in separate remote storage.
    subgraph Local ["Local Workspace"]
        Code["Source Code (.py, .ipynb)"]
        Meta["DVC Metadata (.dvc, dvc.yaml)"]
        Data["Raw Datasets & Model Binaries"]
    end

    subgraph SourceControl ["Source Control"]
        GitRepo["Git Repository (GitHub / GitLab)"]
    end

    subgraph Storage ["Remote Data Storage"]
        CloudStorage["Remote Storage (S3 / GCS / Azure / SSH)"]
    end

    Code -->|"git push"| GitRepo
    Meta -->|"git push"| GitRepo
    Data -->|"dvc push"| CloudStorage

When you add a dataset to DVC using the command dvc add data/train.csv, DVC performs three actions:

  1. Calculates an md5 checksum hash of data/train.csv.
  2. Stores the raw file inside DVC's local content cache (.dvc/cache).
  3. Generates a tiny text pointer file (data/train.csv.dvc) containing the checksum, file path, and byte size.
  4. Adds data/train.csv to .gitignore so Git ignores the raw binary file.

The table below contrasts traditional Git versioning against the combined Git and DVC strategy:

Evaluation Metric Git-Only Approach Combined Git + DVC Approach
Primary Storage Location Git repository object store Remote cloud bucket (S3/GCS) + Git metadata
Git Repository Size Overhead Proportional to total binary data size (Gigabytes) Constant lightweight overhead (Bytes per pointer)
Maximum File Size Limit Hard limit at 100 MB per file Unlimited (constrained only by cloud storage)
Pipeline Reproducibility Manual execution scripts Automated DAG dependency tracking via dvc.yaml
Setup & Integration Effort Native (git init) Lightweight CLI utility (dvc init inside Git repo)

The comparison demonstrates that combining Git and DVC eliminates repository bloat while preserving exact point-in-time snapshots of datasets and model states linked to specific Git commit hashes.

Essential DVC Workflow & Command Reference

Integrating DVC into an existing data science repository requires a few CLI commands. The script below demonstrates initializing DVC, tracking a raw data file, configuring remote storage, and pushing binary assets:

dvc_workflow.sh
# Initialize DVC inside an existing Git repository
dvc init  # (1)

# Track a data file with DVC
dvc add data/training_set.csv  # (2)

# Commit the generated .dvc pointer and updated .gitignore to Git
git add data/training_set.csv.dvc data/.gitignore
git commit -m "Track training dataset version 1 via DVC"  # (3)

# Configure remote cloud storage and push data payload
dvc remote add -d s3remote s3://my-mlops-bucket/dvcstore  # (4)
dvc push
  1. dvc init: Creates the internal .dvc configuration directory and updates .gitignore.
  2. dvc add: Calculates the dataset's md5 hash, moves data to local DVC cache, generates .dvc pointer, and adds binary file path to .gitignore.
  3. git commit: Stores the tiny .dvc text pointer in Git source control alongside your application code.
  4. dvc remote add: Sets up remote storage (such as AWS S3 bucket s3remote) and marks it as default (-d). dvc push uploads cached binary files to cloud storage.
Configuring Remote Storage Destinations

DVC supports diverse remote storage backends depending on team infrastructure:

  • Amazon S3: dvc remote add -d s3remote s3://my-bucket/path
  • Google Cloud Storage: dvc remote add -d gcsremote gs://my-bucket/path
  • Azure Blob Storage: dvc remote add -d azremote azure://container/path
  • Local Shared Network Mount: dvc remote add -d localremote /mnt/shared_drive/dvcstore
  • SSH Server: dvc remote add -d sshremote ssh://user@server.domain:/path

Team members cloning the repository run git pull to fetch code and .dvc pointers, followed by dvc pull to retrieve the exact matching binary datasets from remote storage.

Advanced Features: Pipelines and Experiment Tracking

Beyond file tracking, DVC manages multi-stage machine learning pipelines using Directed Acyclic Graphs (DAGs). Pipeline stages are defined in a dvc.yaml configuration file, specifying stage inputs (dependencies), execution commands, and output data or metrics.

dvc.yaml
stages:
  prepare:
    cmd: python src/prepare_data.py data/raw.csv data/prepared.csv
    deps:
      - data/raw.csv
      - src/prepare_data.py
    outs:
      - data/prepared.csv

  train:
    cmd: python src/train_model.py data/prepared.csv model/model.pkl
    deps:
      - data/prepared.csv
      - src/train_model.py
    outs:
      - model/model.pkl
    metrics:
      - metrics.json:
          cache: false

When running dvc repro, DVC inspects checksums for all dependencies (deps). If raw data or script code has not changed, DVC skips execution for that stage and reuses cached outputs, saving computation time during iterative training.

Tracking Model Metrics and Visualizing Performance Plots

DVC enables comparing model performance metrics across different Git branches or experimental runs:

  • Log Metrics: Save accuracy and loss scores to JSON or YAML files (e.g., metrics.json).
  • Display Metrics: Run dvc metrics show or dvc metrics diff to compare performance between commits.
  • Generate Plots: Run dvc plots diff to create interactive HTML comparison charts for ROC curves and confusion matrices across hyperparameter experiments.

Hands-on Code Repository & Jupyter Tutorial

The research paper presented at the Vodafone Cloud CoE Experts Meetup in Düsseldorf includes interactive demonstrations and code samples.

You can inspect the full implementation and run hands-on exercises directly from the public GitHub repository:

Access the Data Version Control Jupyter Notebook Tutorial on GitHub

The tutorial guides developers through creating local repositories, executing DVC commands, establishing pipeline dependencies, and switching between dataset versions efficiently.

Key Engineering Takeaways

Applying data version control practices provides structural benefits for data science and MLOps teams:

  • Decouple Code from Data
    Track lightweight metadata pointers in Git while storing heavy binary payloads in scalable cloud object storage.

  • Ensure Full Pipeline Reproducibility
    Define deterministic DAG pipelines in dvc.yaml so any team member can recreate exact model outputs with a single command.

  • Track Experiments Systematically
    Compare metrics, hyperparameters, and ROC plots across Git branches without manual spreadsheet record-keeping.

  • Enable Team Synchronization
    Allow distributed team members to retrieve identical dataset states using dvc pull mapped directly to Git commit hashes.

Conclusion

Traditional software version control tools like Git are essential for code tracking but break down when managing large datasets and machine learning model artifacts. Data Version Control (DVC) fills this critical operational gap by introducing lightweight metadata pointers that bridge Git source control with remote binary storage backends.

Using DVC alongside Git can improve traceability and make an experiment easier to reproduce without storing large artifacts in the Git repository. Reproduction still depends on pinned software, available remote data, deterministic steps, and recorded environment details.

References and further reading

Open the complete reference catalog

Primary Sources