Skip to content

🔍️ Extract provenance information (W3C PROV) from GitLab projects.

License

Notifications You must be signed in to change notification settings

DLR-SC/gitlab2prov

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Welcome to gitlab2prov! 👋

License: MIT Badge: Made with Python Badge: PyPi Version Badge: PyPi Downloads Monthly Twitter: DLR Software Badge: Open in VSCode Badge: DOI Badge: W3C PROV Badge: Citation File Format Inside

gitlab2prov is a Python library and command line tool that extracts provenance information from GitLab projects.


The gitlab2prov data model has been designed according to W3C PROV specification. The model documentation can be found here.

️🏗️ ️Installation

Please note that this tool requires Git to be installed on your machine.

Clone the project and install using pip:

pip install .

Or install the latest release from PyPi:

pip install gitlab2prov

To install gitlab2prov with all extra dependencies require the [dev] extras:

pip install .[dev]            # clone repo, install with extras
pip install gitlab2prov[dev]  # PyPi, install with extras

⚡ Getting started

gitlab2prov needs a personal access token to clone git repositories and to authenticate with the GitLab API. Follow this guide to create an access token with the required scopes.

🚀‍ Usage

gitlab2prov can be configured using the command line interface or by providing a configuration file in .yaml format.

Command Line Usage

The command line interface consists of commands that can be chained together like a unix pipeline.

Usage: gitlab2prov [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

  Extract provenance information from GitLab projects.

Options:
  --version        Show the version and exit.
  --verbose        Enable logging to 'gitlab2prov.log'.
  --config FILE    Read config from file.
  --validate FILE  Validate config file and exit.
  --help           Show this message and exit.

Commands:
  combine                  Combine multiple graphs into one.
  extract                  Extract provenance information for one or more...
  load                     Load provenance files.
  merge-duplicated-agents  Merge duplicated agents based on a name to...
  pseudonymize             Pseudonymize a provenance graph.
  save                     Save provenance information to a file.
  stats                    Print statistics such as node counts and...

Configuration Files

gitlab2prov supports configuration files in .yaml format that are functionally equivalent to command line invocations.

To read configuration details from a file instead of specifying on the command line, use the --config option:

# initiate a run using a config file
gitlab2prov --config config/example.yaml

You can validate your config file using the provided JSON-Schema gitlab2prov/config/schema.json that comes packaged with every installation:

# check config file for syntactical errors
gitlab2prov --validate config/example.yaml

Config file example:

- extract:
        url: ["https://gitlab.com/example/foo"]
        token: tokenA
- extract:
        url: ["https://gitlab.com/example/bar"]
        token: tokenB
- load:
        input: [example.rdf]
- pseudonymize:
- combine:
- save:
        output: combined
        format: [json, rdf, xml, dot]
- stats:
        fine: true
        explain: true
        formatter: table

The config file example is functionally equivalent to this command line invocation:

gitlab2prov extract -u https://gitlab.com/example/foo -t tokenFoo \
            extract -u https://gitlab.com/example/bar -t tokenBar \
            load -i example.rdf                                   \
            pseudonymize                                          \
            combine                                               \
            save -o combined -f json -f rdf -f xml -f dot         \
            stats --fine --explain --formatter table

🎨 Provenance Output Formats

gitlab2prov supports output formats that the prov library provides:

🤝 Contributing

Contributions and pull requests are welcome!
For major changes, please open an issue first to discuss what you would like to change.

✨ How to cite

If you use GitLab2PROV in a scientific publication, we would appreciate citations to the following paper:

Bibtex entry:

@InProceedings{SchreiberBoerKurnatowski2021,
  author    = {Andreas Schreiber and Claas de~Boer and Lynn von~Kurnatowski},
  booktitle = {13th International Workshop on Theory and Practice of Provenance (TaPP 2021)},
  title     = {{GitLab2PROV}{\textemdash}Provenance of Software Projects hosted on GitLab},
  year      = {2021},
  month     = jul,
  publisher = {{USENIX} Association},
  url       = {https://www.usenix.org/conference/tapp2021/presentation/schreiber},
}

You can also cite specific releases published on Zenodo: DOI

✏️ References

Influencial Software for gitlab2prov

  • Martin Stoffers: "Gitlab2Graph", v1.0.0, October 13. 2019, GitHub Link, DOI 10.5281/zenodo.3469385

  • Quentin Pradet: "How do you rate limit calls with aiohttp?", GitHub Gist, MIT LICENSE

Influencial Papers for gitlab2prov:

Papers that refer to gitlab2prov:

📜 Dependencies

gitlab2prov depends on several open source packages that are made freely available under their respective licenses.

Package License
GitPython License
click License
python-gitlab License: LGPL v3
prov License: MIT
jsonschema License: MIT
ruamel.yaml License: MIT
pydot License: MIT

📝 License

This project is MIT licensed.
Copyright © 2019 German Aerospace Center (DLR) and individual contributors.