Dependency Management for C++

Exciting times

After finishing my previous blog I started working on a new article on the subject of performance measurement with micro-benchmarking or perf. Title to be determined 😉

But then I took part in the Standard C++ Foundation Developer survey. And what happend after the survey made me postponing the article and writing the article you are reading now.

You can find a summary of the results on the ISOCPP website. When the summary was published I came across a tweet by Corentin about a post on medium.com about dependency management and a tweet by Bryce Lelbach. So I responded. 

So that’s out. Wait, I just created a liability for myself on a difficult subject where everyone has an opinion about. The missing easy dependency manager is bugging me for years already and that is what made me respond to it. Years of frustration got out in this one tweet. But I’m serious, as a community we should solve this. The article of Corentin gives a lot of good starting points. I agree with most of the article.

This part of the standardization process should be studied by SG15 Tooling, chaired by Titus Winters.

Approach

Before continuing reading it is advised to read the article by Corentin. I’m not going to repeat him here ;-). It is important that we all understand the difference between a package manager and dependency management. Corentin gives some description of dependency management and to me it is clear that what we need is dependency management and tooling around that. So let’s agree that we talk about dependency management from now on only!

We should not try to build a ‘one and only’ dependency manager for C++. Let’s leave the building to tool builders, just as we do with compilers. The ISO C++ Committee doesn’t build compilers, they build specifications and leave the implementation to compiler builders. In this area the same can be done, except for one part – I will get back to that later. We also shouldn’t try to build a ‘one and only’ build system. We should strive to have the build system builders to implement the dependency specification in their tooling and that is it. Existing build tooling can add the dependency management into their tooling and use the standardized dependency specification. This specification can contain a lot of aspects:

  • Libraries these sources depends upon
  • Compiler flags needed to successfully compile these sources
  • Linker flags needed to successfully link these ‘sources’
  • Architectures on which these sources can run without problems
  • Preferred compilers for these sources
  • Versioning and source information
  • Namespace information
  • Etc.

This all is still specification (we as a community should align with tool builders to arrive at a specification that they can implement). The only aspect I think is important to create and support centrally is a repository in which sources are stored. The central repository will be used by to tools to download the sources. It goes without saying the these sources must comply to the specification. I agree with Corentin that the repository must be high available and easily duplicated. I also agree that organizations can have their own in-house repository. But I disagree that this should be a commercial industry initiative. The security and reliability is super important for the success of the initiative and therefore it must be under control of the central standardization body. This repository must be the ‘single’ source of truth.

Let’s do it

I really think that dependency management is important for the future of C++ and I’m willing to contribute to that. So Titus, when do we start.

7 thoughts on “Dependency Management for C++”

  1. I’ve been working with Corentin a bit in the past week to see if a simple-as-could-be package server could be workable. I’ve set up my initial test at accio.link and uploaded an export of Boost. So to look at your list:

    – Libraries these sources depends upon

    Check.

    – Compiler flags and linker flags needed to successfully compile these sources

    Not right now, and given the unportability of them I hope to be able to get away with never supporting them. This relates strongly to the definition of a package / component – do you see a component with a conditional (#ifdef’ed) feature as a configurable component, or as two components? – and as such will be important to decide upon.

    – Architectures on which these sources can run without problems, Preferred compilers for these sources, Versioning and source information, Namespace information, Etc.

    All of these are metadata. They should also include a cryptographic signature of whoever signed off on it – ie, who put it there, because I trust a Boost upload from Jon Kalb, Marshall Clow or Bryce Lelbach a lot more than from Corentin, from myself or from you.

    Are you on the CppLang slack? If so, should we create a channel to discuss this in?

  2. > Compiler flags needed to successfully compile these sources

    You would need to standardized a build system to do this. I dont think a C++ package manager should deal with that, it should focus on how to communicate with the buildsystem.

    > Linker flags needed to successfully link these ‘sources’

    That should be pkgconfig, and that should be generated by the buildsystem not a package manager.

    I also discussed how to standardized such things for a package manager here:

    http://pfultz2.com/blog/2017/10/27/universal-package-manager/

    Ultimately, we need two files with a standard format to allow interoperability between buildsystems and package managers: one for the package meta data, and another for describing the build environment.

    The package meta data should build on top of the debian control file, but tweaked for C++ needs. The toolchain file for the build environment can just be a list of variable assignments or yaml file or something. The more difficult part is deciding the keys.

  3. Are you guys continuing with this initiative? If yes where and how? Is anybody coordinating the activities? It is time to move this forward…

    1. Yes, we are. It is not ‘really’ coordinated at the moment. At the moment I’m gathering opinions and try to get in touch with the chair of SG15. A lot of discussion takes place at cpplang.slack.com in the tools channel.

  4. How should this work? C++ is not a platform specification like Java, Perl etc. There are so many hardware platforms, operating systems and compilers. How do you manage this in one repository? Impossible!

    Another point: there is something you want: https://build2.org/
    I never used it…

    1. I understand that it is difficult to design this, but impossible? No, I don’t believe in impossible. By defining a standard that can be used by build tools, compilers and perhaps package managers. It is possible to create definitions that these tools can use to construct binaries from source for different platforms, provided that only standard language constructs are used.

Leave a Reply

Your email address will not be published. Required fields are marked *