Omnia has relied on the Python-centric Conda distribution platform for getting all our software to users. The Conda platform has allowed us to build, publish, and deliver our software and all dependencies with minimal user effort, allowing users to focus on the important thing: doing awesome science! Someone, however, must put in the effort to compile the software, curate dependencies, ensure cross-platform stability, and distribute updates.
The problems and limits of the old Omnia deployment system
We have maintained all our build infrastructure on the omnia GitHub conda-recipes repository. All of our packages were uploaded to an omnia Conda channel so they could all be fetched from the same place, allowing each Omnia package to seamlessly bring in other Omnia packages as dependencies. However, because we were on our own channel, we also had to compile and publish any and all dependencies that our packages needed. Some of these common packages were available on the default Conda channel, but many were not. We had in total 109 packages on our Omnia channel, which the video above cycles through, of which more than 40 have been published elsewhere on Conda, often by the actual developer of that software. The time and effort spent to curate and update these extra packages takes time away from our developers doing what they do best: making awesome software!
The stability of our distribution framework was also at risk due to changing technology. We use the awesome free Travis-CI and AppVeyor Continuous Integration (CI) services to automate the builds of software on our GitHub page, which allows easy cross-platform deployment. To achieve maximum compatibility in letting omnia packages be used on the widest set of hardware possible, our Linux builds have used a CentOS 5 Docker image based on the Phusion Holy Build Box to ensure a uniform build environment. However, CentOS 5 reached its EOL on March 31, 2017, which presents some unique problems. Conda software itself is also rapidly changing, and so the API hooks and methods change along side them. Coupled with the release of Python 3.6, it had become a full time job just to maintain the distribution platform else the proverbial "gentle breeze" could bring it all crashing down. A better system was needed, both to ensure easy access to the software for users and reduced maintenance of the build infrastructure for Omnia developers.
Conda-forge: The present and future deployment system for Omnia
Conda-forge provides the solution to all the problems we faced as developers and users. Conda-forge is "a community led collection of recipes, build infrastructure and distributions for the conda package manager." The conda-forge community has formed around a common desire to see as much software as possible be centrally accessible on uniform platforms, while also allowing developers to retain full control of updates and maintenance of their software.
From the user perspective, users will notice only a slight change to their install commands as packages come from both omnia and conda-forge channels. Developers get access to all the following features:
- Full control of your software, including all deployment, patches, and other maintenance
- Access to stable and bleeding-edge releases for dependencies built and maintained by other people without the need to curate them yourselves!
- Uniform deployment platforms for Linux with broad glibc compatibility (CentOS 6 or later, Ubuntu, etc.), OSX, and Windows that are maintained by the conda-forge community, and not you!
- Continuous Integration to automate building, and deploying your software for you, once you pass some Lint checks
- Open source development
- Large community of support
Wait! Where are all the packages going!?
Short answer: nowhere.
Longer answer: All packages which we have previously built on the Omnia Conda channel are staying right where they are, for as long as we can possibly keep them there. Even then, we will find a way to archive them and make them accessible to ensure reproducibility.
How does this affect us users right now?
Minimally, if at all
If you want to keep using the current or older versions of packages off the omnia conda channel, nothing will change. Since the previously built packages are not going anywhere, you can continue to use them strait from the Omnia channel, even the old dependencies we built.
For future builds and newer packages, you will need to make a small change to either your Conda configuration, or the command you type to install packages as they will be built against both the Omnia and Conda Forge channels.
Some developers have expressed wanting to keep publishing packages on omnia by copying from conda-forge to omnia, so you may not need to change anything at all! Please check with the individual packages to see their preferred installation method.
What do us developers need to do and worry about right now?
For your own local builds, start by adding the conda-forge channel to your conda commands in addition to the Omnia channel so you can access the myriad of packages available. You can still publish updates to our Omnia GitHub repository for Conda recipes (for now). Just know that new packages will be built against both Omnia and Conda-forge channels.
We encourage our developers to start trying to migrate your software to conda-forge, if possible. We have laid out the steps for the migration on our GitHub page as to what needs to be done, but there is no hard time table for the migration. We should note though that once a package is on conda-forge, it should not depend on any packages exclusively in the Omnia channel (i.e. dependencies come only from default and conda-forge channels).
If you so desire, you can build your packages on conda-forge, then copy them and their dependencies (if needed) over to the omnia channel so your users can continue relying exclusively on the omnia channel. Instructions are maintained on omnia's conda-recipes front page in the "Copying from Conda Forge" section.
In a future blog post, I will go through the steps of migrating a package conda recipe from omnia to conda-forge.
Longer term plans for Conda-forge
Eventually, we want all our packages to be part of the Conda Forge community. There are some technical hurdles we must first overcome before we can rely exclusively on Conda Forge. If you want to help us solve these technical problems, feel free to follow and contribute to the GitHub repository where we have posted the migration plan, and what obstacles we face.
When this is done though, software deployment will be easier for our developers, and the maintenance of the deployment software will be virtually non-existent.