Lightweight Haskell Docker Images with Nix

Nix gives us the opportunity to not only build our Haskell apps more easily, it can as well help to set up a deployment via Docker Images and Containers. This is exactly what I do at Ersocon (my little dev company) and I would like to show in this post how to configure this set-up.

The idea is, to create a CI job on gitlab.com which will build the app on each push to the master branch, create a docker image and promote this image to the destination server by executing a docker-compose pull on the target server. I won’t go into details on how to configure a Gitlab build pipelines, instead we will focus on the Nix part.

First things first. What do we already have?

To create a new image from this project we need to tell Nix how to do this. Let’s create a new file called release.nix and put the following content inside:

let
  config = {
    packageOverrides = pkgs: rec {
      haskellPackages = pkgs.haskellPackages.override {
        overrides = haskellPackagesNew: haskellPackagesOld: rec {
          myApp =
            haskellPackagesNew.callCabal2nix "my-app" (./.) { };
        };
      };
    };
  };

  pkgs = import (builtins.fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/20.09.tar.gz";
  }) { inherit config; };

  
in
  { myAppImage = pkgs.dockerTools.buildImage {
      name = "domainofmyregistry.com/mycompany/my-app-image-name";
      contents = [ 
        pkgs.haskellPackages.myApp
        pkgs.bash
        pkgs.coreutils
        pkgs.htop
        pkgs.cacert
      ];
      tag = "latest";
      config = {
        Cmd = [ 
          ""
        ];
        ExposedPorts = {
          "8000/tcp" = {};
        };
        extraCommands = ''
        '';
        Env = [
          "APP_MODE:production"
        ];
      };
    };
  }

There are three important things here that we need to look at.

First, in line 13 we have chosen a Tarball for the nix packages in the version of 20.09. For absolutely reproducible builds we could as well pin this to a very specific commit.

Second, in line 2 we have added our app to the haskellPackages by overriding the default haskellPackages from Nix packages. Pay attention that we call callCabal2nix and point it to the root directory of out project. This will analyze out cabal file and build the project according to the dependencies we have defined in the cabal file. (I assume that we defined an executable as well).

Third, we use dockerTools from default pkgs to call buildImage. This is more or less straight forward. Here we can add our app pkgs.haskellPackages.myApp and different other useful tools to our image. As you can see, I added htop so I am able to connect to the container and monitor/inspect my server when I need to.

I omitted the Cmd configuration since this depends on what you would like to do. For instance, if you would like to launch a webserver, this would be the point to do so.

With the release.nix file we can now run nix-build release.nix and watch the progress of the build. After it is finished you should see a result folder in your project. (I advice to add this folder to .gitignore to not commit it in the repository). This folder will contain the “cooked” docker image with your app inside. To tell docker to load the image we can simply execute:

docker load < result

This will pull in the local docker image with the name that we have defined in our buildImage call before. We could now push this image to a registry to be able to publish and launch it on our server. But, if you check the image size (it is displayed when we run docker load) it will be around 3 GB, even if your app is small. This is the result of how Nix is linking the packages. And since we used GHC to build our app, it will as well be available in our image, even if we don’t need it for our production app anymore. So, how can we get rid of it?

Unfortunately there is no easy change to get this to work. We need to add one more step to our Nix build. Let’s have a look at line 7 in the release.nix file. We executed callCabal2nix to create our app from cabal and build it. Cabal2nix is a tool that turns a cabal project file into a nix file which describes your project, usually it will be called default.nix This file is then used to call callPackage to create the final result. We need to tweak the default.nix file to get rid of the related dependencies that we only need during the build.

Let’s create our default.nix file by hand by calling cabal2nix in the projects root folder. You should now have a file in your project that looks something like this:

{ mkDerivation, aeson, base, bytestring, http-types, lucid, mtl
, servant, servant-docs, servant-server, stdenv, text, wai
, wai-extra
}:
mkDerivation {
  pname = "my-app";
  version = "1.0.0.0";
  src = ./.;
  isLibrary = true;
  isExecutable = true;
  libraryHaskellDepends = [
    aeson base bytestring http-types lucid mtl servant servant-docs
    servant-server text wai wai-extra
  ];
  executableHaskellDepends = [ base servant-docs ];
  testHaskellDepends = [ base ];
  homepage = "https://myapp.com";
  license = "unknown";
  hydraPlatforms = stdenv.lib.platforms.none;
  doHaddock = false;
}

As you probably can guess, it creates a new derivation. And at this point we can tweak things. Let’s add the following two lines:

enableSharedExecutables = false;
postFixup = "rm -rf $out/lib $out/nix-support $out/share/doc";

With this two lines we switch to static linking and get rid of the unnecessary libraries and documentations after our app builds successfully (postFixup).

Since we have now a modified default.nix file, we cannot use callCabal2nix anymore. We need to use the generated default.nix file and execute callPackage on it, like this:

If you run nix-build release.nix now and docker load the result folder, the docker image should be much smaller (for a small project like this, it should be around 50MB). The size dependes on the size of your app and the dependecies you put into your app/image.

Conclusion

We have been able to bring the size of the final docker image down. It is a big difference if you deploy 3GB or 50MB in production. With a fast network this may be not that important, but still, it can be an issue. Some registry providers charge by the amount of storage space you use. It does not seem a lot, but with time and a fair amount of apps it can be an issue as well.

The advantage of a lightweight docker image came with a tweak. And if you update your app (add new dependencies/change configuration) you will need to run cabal2nix and modify the default.nix file again. Unfortunately we can not pass the two parameters to callCabal2nix (at least I didn’t see the possibility to do so in the source code). Anyway, the advantage of leightweight images are pretty big and worth it. The repetitive task of generating the default.nix file and add two parameters can be scripted into the build process.