It has been nearly six years since I last wrote about the PROV Python package here, when 2.0.0 dropped Python 2. A lot has happened since, and most of it this summer. 3.0.0 came out in July, 3.1.0 added PROV-JSONLD in August, and 3.1.1 followed in September with a round of fixes and hardening. Three releases in three months deserve at least one post, so here it is.
What changed in 3.x
The 3.x line is where the modernisation work I started in July, and shipped step by step through the 2.x releases of this summer, finally landed. Here is the short version.
- No unconditional dependencies. A bare install brings in the package alone, so it is lighter and quicker to add to a project. RDF, PROV-XML, Graphviz and NetworkX support are extras you add only when you need them.
- Conformance audited. The package was checked against every W3C PROV specification it implements and the gaps were fixed, so what it reads and writes is what the standard says. A conformance matrix records the few limitations that remain in each format.
- Unification follows PROV-CONSTRAINTS. Merging documents now follows the specification's rules and refuses to combine records that conflict, so you learn about a clash instead of getting a silently wrong document.
- PROV-JSONLD. The package reads and writes PROV-JSONLD, so PROV documents can go straight into linked-data tooling with no extra dependency.
- Typed and documented. Full type hints give you editor completion and type checking, and the documentation is reorganised into a tutorial, how-to guides and reference so you can find what you need without hunting.
It runs on Python 3.10 and later, and on PyPy. If you want the full story, the What's new in 3 page has it, and the upgrade guide lists every change and what to do about it. For most code, though, there is nothing to do. It should just work.
Who is using the package?
I once stumbled upon the package's dependency graph on GitHub and was surprised to find over 7,000 public repositories and 75 packages listed as dependents, so I dug further. PyPI adds another 45 packages that require prov in their metadata. The two biggest are nipype, the neuroimaging workflow engine, and cwltool, the Common Workflow Language reference implementation. Each is downloaded over 200,000 times a month, and Galaxy, Renku, StreamFlow, dockstore and the Workflow Run RO-Crate tooling all pick up prov through them. The nipype link is no accident. Satrajit Ghosh, one of the people behind nipype, wrote the RDF serialisation for prov back in 2014, and that one I did know about.
What surprised me more was where it turned up. Seismologists use it, since pyasdf embeds PROV documents in the ASDF data format and SeisMIC and PyGLImER build on that. That one answered a question I had never thought to ask. Lion Krischer contributed the PROV-XML serialisation to prov in the summer of 2014, and I had no idea he was the author of pyasdf. Now I know why he needed it. Climate scientists use it in ESMValCore. Astronomers use it in voprov, for the IVOA provenance model, and in the pipeline of the CTAO LST-1 telescope. It is in fusion research (Indica) and in machine-learning experiment tracking (yProv4ML). And a whole family of provenance tools is built on top of it: gitlab2prov and prov2neo from the German Aerospace Center (DLR), praetor, case-prov, BioProv and the NIDM libraries for neuroimaging.
If your project uses the package and is not on that list, I would love to hear about it. The repository now has Discussions enabled, and a short note there is all it takes.
Please unpin 1.5.1 and 2.1.1
The same digging also showed me where the work is not finished. Most downloads are still pinned to 1.5.1 or 2.1.1, releases from 2017 and 2025. Both predate the modernisation work, and neither is supported any more. The good news is that cwltool is moving to 3.1 now, which will carry a good many of those pinned projects along with it. If your own project pins an old release, please lift the pin to prov>=3 and add the extras you use. The upgrade guide covers the exceptions, and if it does not cover yours, open an issue and I will try to help.
What comes next
Next up is 3.2.0, planned for October, which adds a parser for PROV-N, so the notation becomes readable as well as writable. The roadmap lists what comes after that. The package also has a DOI now, so if it helps your research you can cite it. The citation file in the repository has the details.
If you are new to PROV or to the package, the tutorial is the place to start. As always, please report problems to the issue tracker. And many thanks to everyone who has contributed fixes, reports and ideas over the years.
Go Top
comments powered by Disqus