Thoughts Heap

TL;DR: Go to the Implement a shell.nix with nixpkgs-mozilla section

Introduction – why use nix if rustup exists?

Rust dependency management is top class, rustup facilitates downloading and managing different versions of Rust without much effort, and also specify what version of Rust you want in your project via the rust-toolchain file. As an author, I can ensure collaborators of a project I’m working on will download the exact version of Rust that I intended to use to develop my project.

That being said, since I’ve become more familiar with the nix package manager, I’ve been striving to use it for all my dependencies; this package manager is like rustup, but for every dependency in your system, not just the Rust toolchain. If you need a C library, it is likely nixpkgs has it.

The cool thing about nix as a package manager, is that you can use it the same way you would use apt or brew, but you can also specify dependencies on an specific sub-directory, like you would do with virtualenv or rbenv files. When you execute nix-shell on a directory that has a shell.nix file, boom! You are on an environment that uses only the specified dependencies, no problems with conflicting libraries installed on your system or anything like that.

Another great feature of nix, is that it allows us to pin down a package repository in a way that will make it’s packages use the same version – always. This is similar to having your project running on a docker image or virtual machine pinned to an specific version of ubuntu, and having the apt install commands always resolving to the same libraries and binaries every time.

How does it do it? Well, this is not a tutorial on nix per-se, just a pitch, and a rationale about why, you, as a Rust developer, would like to try this approach to install the Rust toolchain. There are some great resources out there on how to get started with nix.

What is nixpkgs-mozilla?

It seems some folks at Mozilla also see the value of nix, and use it internally. Given this, they maintain a repository of nix packages that install the latest version of their tools and software. The mechanics about how to integrate it on a project are not trivial if you are not familiar with nix, that’s why I’m writing this blogpost.

Just to clarify, you can install Rust without Mozilla’s repository, but there is a big chance that the version of the Rust compiler you download is not going to be the latest stable version; also I think the Mozilla packages allow you to cutomize the Rust toolchain more easily (target platform, stable vs nightly, etc.) and it understands the rust-toolchain file convention.

Implement a shell.nix with nixpkgs-mozilla

1) Install the niv tool to manage project deps

niv is a tool that sits on top of nix, it allows us to pin a package repository (“attribute set” in nix lingo) and custom packages (“derivations” in nix lingo) using a UX friendly CLI.

Note, niv might become obsolete in the not so distant future, the nix community is working hard on a feature called flakes, which tries to replicate what niv does in a native way.

Is also worth mentioning, niv is not the only way to pin dependencies, you may be able to use nix alone for this, however, I like niv just because it enforces a standard and a way to add packages (derivations) easily.

2) Initialize niv in your directory, and add nixpkgs-mozilla

To add the nixpkgs-mozilla package repository (“overlay” in nix lingo), we are going to use niv add.

$ niv init
$ niv add mozilla/nixpkgs-mozilla

Make sure to also update the nixpkgs repository to nixpkgs-unstable, the nixpkgs-mozilla overlay relies (as of Jul 2020) on code that is not in the stable branch.

$ niv update nixpkgs -b nixpkgs-unstable

3) Add nixpkgs-mozilla overlay

We are going to modify the default nixpkgs “attribute set” with the Mozilla “overlay” overrides. Now, every package that gets installed that has a Rust dependency, will use the project’s Rust version by default. This behavior happens because we are overriding the special rustPlatform derivation, which is used to build Rust programs in nix.

$ cat <<\EOF > nix/pkgs.nix
let
  # import URLs for all dependencies managed by niv
  sources =
    import ./sources.nix;

  # import the package repository (overlay) specified on nixpkgs-mozilla
  mozilla-overlay =
    import sources.nixpkgs-mozilla;

  # define project's package repository
  project-overlay =
    # self and super are convoluted stuff, check out docs for nix overlays
    self: super:
      let
        # use the utilities from the nixpkgs-mozilla to build the "rustup toolchain"
        mozRustToolchain = self.rustChannelOf {
          rustToolchain = ../rust-toolchain;
        };

        # We want to get the rust package with all these extensions
        mozilla-rust = mozRustToolchain.rust.override {
          extensions = [
            "rust-src"
            "rust-std"
            "rustfmt-preview"
            "rls-preview"
            "clippy-preview"
          ];
        };

        # rust-src derivation is a tree of deriviations, we need to get the "src" attribute
        # from one of it's paths
        mozRustSrc = (builtins.elemAt mozRustToolchain.rust-src.paths 0);

        # We need to modify the structure of the rust source package that comes
        # from the nixpkgs-mozilla to work with an structure that works on upstream nixpkgs.
        rustSrc = super.runCommandLocal "${mozRustSrc.name}-compat.tar.gz" {} ''
          # get contents on directory in place
          tar -xf ${mozRustSrc.src} --strip-components 1
          mkdir out

          # modify the directory structure to work with development/compilers/rust/rust-src.nix
          mv rust-src/lib/rustlib/src/rust/* out
          tar -czf rust-src.tar.gz out

          # vaya con dios
          mv rust-src.tar.gz $out
        '';
      in
        {
          rustPlatform = super.makeRustPlatform {
            cargo = mozilla-rust;
            rustc = (mozilla-rust // { src = rustSrc; });
          };
          mozilla-rust = mozilla-rust;
        };

  pinnedPkgs =
    import sources.nixpkgs {
      overlays = [
        mozilla-overlay
        project-overlay
      ];
    };
in
  pinnedPkgs
EOF

4) Now build a shell.nix file that contains all the dependencies you need for development

$ cat <<\EOF > shell.nix
# pkgs contains all the packages from nixpkgs + the mozilla overlay
{ pkgs ? import ./nix/pkgs.nix }:

pkgs.mkShell {
  buildInputs = with pkgs; [
    # the package/derivation we built on nix/pkgs.nix
    mozilla-rust

    # other rust utilities. Because we override the rustPlatform
    # package on nix/pkgs.nix, these dependencies will use the same
    # version as our mozilla-rust dep.
    rust-analyzer
  ];
}
EOF

5) Add a rust-toolchain file

There are various valid formats for this file, to be frank, I didn’t find a place with examples, so there has been a lot of experimenting. Let us use the desired rust version we want for our project

echo '1.45.0' > rust-toolchain

6) Execute nix-shell, and see all your dependencies get downloaded and installed

$ nix-shell
$ which rustc
/nix/store/vqqfzjjnk9zd3ps28pjxcxwzckbwlfvj-rust-1.45.0-2020-07-13-5c1f21c3b/bin/rustc

Note, this process may take longer than usual given you are compiling these tools with the Mozilla’s Rust toolchain, and that is not going to be on the cache server of nixpkgs. If you want to build these packages once and share with the rest of your team, I recommend using cachix or roll out your own binary cache.

7) Bonus: use direnv to call nix-shell.

This will automagically integrate a dynamic dependency environment to your editor for freeeee (if your editor supports direnv, which is likely).

Conclusion

So there you have it, it took me a while to integrate nixpkgs-mozilla’s Rust with the rest of the nixpkgs ecosystem. Hopefully this will empower you to upgrade and keep track of Rust upgrades (as well as your other deps) sanely.

TL;DR: Choosing to use pointer arguments for functions for the sake of better performance is premature optimization, and you should consider the drawbacks of pointers in regards to accidental complexity.

Introduction

When working in Golang, how many times have you considered the choice of receiving a pointer instead of a value as the argument of a function? We usually don’t spend much time thinking about this given it is known that pointer arguments perform faster than values.

However, in other languages (cough, Haskell, cough), doing this, is one of the most pervasive anti-patterns. Why? The repercussions of mutating a variable are many, especially if you are using an awesome language with green threads like Golang. In this blog post, we’ll explore some of the drawbacks of using pointers.

The scope of change

Let’s start with explicitly defining what the pointer contract means in your business logic: whenever we pass a pointer as an argument to a function, we are giving this other function a rubber stamp to make changes on the behavior of their caller.

main()
|
|- fnA(*foo)
|   |- fnB(*foo)
|   `- fnC(*foo)
|     |- fnD(*foo)
|     |- fnE(*foo)
|       `- fnF(*foo)
|
`- fnG(*foo)


# Suddenly the fnG function stopped working as expected because foo is in a state that is not valid/expected, 
# where should we look for the bug?

This approach, by extension, adds a mental overhead to the algorithms you run. As soon as you pass a pointer argument to a function, you cannot guarantee that this function is not going to modify the behavior of an algorithm in unexpected ways (unless the documentation is evident on what the function does, and is also accurate about it).

For sure, this situation is unlikely to happen when you use a general purpose library function, but is not impossible – and I argue, it happens more often than not – to have an unexpected behavior after calling a function from an API that changes for the sake of solving business needs. The priority is not in the UX of the code or the repercussions of its many callers, but the business value that a particular change offers when using a specific set of inputs.

However, now you argue, we have unit tests to solve this kind of issues right? Well yeah, up to a point, the combination of values and mid-states makes it hard to test every single combination that could exacerbate an issue from mutation.

I prefer not to give a function the power to change state like that, and instead, make this issue impossible to happen in the first place by passing parameters by value.

Pointers and concurrency

With the advent of multi-core CPUs, concurrency is becoming more prevalent as a way to squeeze as much performance as we can from our computers.

Golang is one of the few languages that make concurrency easy, via the usage of goroutines; however, it makes it also very easy to shoot yourself in the foot. Values are mutable by default, and if by any chance you use the same reference to multiple goroutines unexpected things tend to happen.

If you add that to the fact that you are passing values as pointers, things become accidentally complex very quickly.

How expensive is it to receive parameters by value?

The points above may sound reasonable, but you still think to yourself, won’t somebody, please think of the performance?, that’s fair, let’s figure out how expensive is it to pass parameters by value to a function.

Let us build a benchmark that allows us to assess what are the performance implications of passing small and big sized values as parameters:

package vals_test

import (
    "testing"
)

type F2 struct {
    f1 int64
    f2 int64
}

type F4 struct {
    f1 int64
    f2 int64
    f3 int64
    f4 int64
}

type F8 struct {
    f1 int64
    f2 int64
    f3 int64
    f4 int64
    f5 int64
    f6 int64
    f7 int64
    f8 int64
}

type F16 struct {
    f1 int64
    f2 int64
    f3 int64
    f4 int64
    f5 int64
    f6 int64
    f7 int64
    f8 int64
    f9 int64
    f10 int64
    f11 int64
    f12 int64
    f13 int64
    f14 int64
    f15 int64
    f16 int64
}

type D2 struct {
    d1 F2
    d2 F2
    d3 F2
    d4 F2
}

type D4 struct {
    d1 F4
    d2 F4
    d3 F4
    d4 F4
}

type D8 struct {
    d1 F8
    d2 F8
    d3 F8
    d4 F8
}

type D16 struct {
    d1 F16
    d2 F16
    d3 F16
    d4 F16
}


func updateValD2(val D2) D2 {
    val.d1.f2 += int64(10)
    return val
}

func updateRefD2(val *D2) {
    val.d1.f2 += int64(10)
}


func updateValD4(val D4) D4 {
    val.d1.f3 += int64(10)
    return val
}

func updateRefD4(val *D4) {
    val.d1.f3 += int64(10)
}

func updateValD8(val D8) D8 {
    val.d1.f3 += int64(10)
    return val
}

func updateRefD8(val *D8) {
    val.d1.f3 += int64(10)
}


func updateValD16(val D16) D16 {
    val.d1.f3 += int64(10)
    return val
}

func updateRefD16(val *D16) {
    val.d1.f3 += int64(10)
}


func BenchmarkPassRefD2(b *testing.B) {
    var d D2
    for i := 0; i < b.N; i++ {
        updateRefD2(&d)
    }
}

func BenchmarkPassValD2(b *testing.B) {
    var d D2
    for i := 0; i < b.N; i++ {
        d = updateValD2(d)
    }
}

func BenchmarkPassRefD4(b *testing.B) {
    var d D4
    for i := 0; i < b.N; i++ {
        updateRefD4(&d)
    }
}

func BenchmarkPassValD4(b *testing.B) {
    var d D4
    for i := 0; i < b.N; i++ {
        d = updateValD4(d)
    }
}

func BenchmarkPassRefD8(b *testing.B) {
    var d D8
    for i := 0; i < b.N; i++ {
        updateRefD8(&d)
    }
}

func BenchmarkPassValD8(b *testing.B) {
    var d D8
    for i := 0; i < b.N; i++ {
        d = updateValD8(d)
    }
}

func BenchmarkPassRefD16(b *testing.B) {
    var d D16
    for i := 0; i < b.N; i++ {
        updateRefD16(&d)
    }
}

func BenchmarkPassValD16(b *testing.B) {
    var d D16
    for i := 0; i < b.N; i++ {s
        d = updateValD16(d)
    }
}

We have two versions of the updateVal function, one receiving a reference, and another receiving a value. We benchmark different sizes to assess how different is the slowness when using small and big values. Executing the above code gives us the following result:

$ go test -bench=.
goos: linux
goarch: amd64
BenchmarkPassRefD2-8            2000000000               1.37 ns/op
BenchmarkPassValD2-8            200000000                6.71 ns/op
BenchmarkPassRefD4-8            2000000000               1.37 ns/op
BenchmarkPassValD4-8            200000000                9.77 ns/op
BenchmarkPassRefD8-8            2000000000               1.35 ns/op
BenchmarkPassValD8-8            100000000               14.7 ns/op
BenchmarkPassRefD16-8           2000000000               1.37 ns/op
BenchmarkPassValD16-8           50000000                24.9 ns/op
PASS
ok      _/home/roman/tmp/playground/vals        19.197s

What do you know? Passing parameters is slow!

The first thing we can notice is that even with small values, passing parameters by value is around 6x slower than passing them by reference. The bigger the value, the slower it becomes, having it 24x slower when having a struct of 16 attributes (not your average use case). Now, is this slowness relevant in the bigger scheme of things? I argue that it may not be.

There is this table about latency numbers every programmer should now that may give us some clarity; for any access to RAM we do in our code, we get a 100ns tax, meaning, if we do have any IO related code around our functions, the latency of those calls is going to eclipse any performance degradation you may get from receiving values as parameters.

Conclusion

Does this mean you should not worry about parameters by value ever? Well, no. If you are executing a pure function that receives big values as parameters in a tight loop, it may make sense to pass parameters by reference to speed up the performance of your algorithm.

Our key takeaway is, before passing pointers as arguments for the sake of performance, make sure that this piece of code is the real bottleneck of your program, this way you reduce accidental complexity from your application.

I wanted to get a sense of how expensive the different mechanisms to build strings are; I’m used to use printf like functions in the languages I work with, and I wanted to check how performant (or not) was in Go. I came up with this benchmark code:

Which gave me the following results when running the code

$ go test -bench=. -benchmem 2>&1 
goos: linux
goarch: amd64
BenchmarkSprintfGreeter-8               20000000               100 ns/op              32 B/op          2 allocs/op
BenchmarkBuilderGreeter-8               30000000                48.9 ns/op            24 B/op          2 allocs/op
BenchmarkConcatGreeter-8                100000000               16.9 ns/op             0 B/op          0 allocs/op
BenchmarkBuilderReplicate2-8            20000000                80.8 ns/op            56 B/op          3 allocs/op
BenchmarkConcatReplicate2-8             20000000               114 ns/op              64 B/op          3 allocs/op
BenchmarkBuilderReplicate4-8            10000000               128 ns/op             120 B/op          4 allocs/op
BenchmarkConcatReplicate4-8              5000000               268 ns/op             176 B/op          7 allocs/op
BenchmarkBuilderReplicate8-8            10000000               191 ns/op             248 B/op          5 allocs/op
BenchmarkConcatReplicate8-8              3000000               607 ns/op             528 B/op         15 allocs/op
BenchmarkBuilderReplicate16-8            5000000               287 ns/op             504 B/op          6 allocs/op
BenchmarkConcatReplicate16-8             1000000              1595 ns/op            1712 B/op         31 allocs/op
BenchmarkBuilderReplicate32-8            3000000               553 ns/op            1016 B/op          7 allocs/op
BenchmarkConcatReplicate32-8              300000              3770 ns/op            6064 B/op         63 allocs/op
PASS
ok      _/home/roman/Projects/playground        23.791s

From what we can tell from the Greeter benchmarks, in the simplest use case, a Sprintf is twice slower than using strings.Builder, with just concatantion using + being almost 10x faster. This is in the case of a single string build (no iterations).

In the scenario where we are accumulating a string in a loop, we can see the benefits of using strings.Builder vs. + at four iterations, where strings.Builder is around 38% faster.

So, the lessons learned are:

• Avoid Sprintf if you can make do with string concatenation (or if performance is not that important compared to convenience);

• Always use strings.Builder when dealing with string accumulation in loops.

Previously this week, the awesome “Matt Of Lambda” published a blog post about the benefits of using ghcid to develop Haskell applications; if you haven’t checked it out, it is an excellent read on how to get started with something like REPL driven development

Personally, I’ve always have used GHCi for development, from trivial tasks like checking my types and experiment with code to running my test suites and the main executable of my projects. This approach was usually way faster than the regular change/compile/run cycle that Haskellers developers typically do, but it came with some challenges:

• Lack of resource de-allocation

Given that most applications are designed to run only once, there are no real incentives to develop our software in a way that can be cleaned up and started again without killing a process.

• Code gets unusually slow after a few runs

Because of the previous point, every time you reload code in the REPL your application leaves resources hanging in memory (e.g., threads), and the REPL starts to get slower (and confusing) if dangling threads are spitting strings to stdout.

• Port already allocated errors

Sometimes, you will get errors like a TCP Socket port is already bound, this happens when you reload your code, and your application leaves socket resources open after a reload.

Solving some of these problems

Given my tenure in Clojure land for a big chunk of the past four years, I had the opportunity to see how “REPL driven development” should be. Their tooling is oriented to have a REPL open at all times, and they run everything there, from unit-tests to executables. Following their example, I started to implement a few libraries that help me find this REPL nirvana.

teardown

Inspired by Reactive Extensions Disposable API, this library is a smart wrapper on top of IO sub-routines whose whole purpose is to perform a cleanup. This library will:

• When a resource cleanup fails with an error, this API makes sure that this error does not affect the cleanup of other unrelated resources

• It keeps track of the resource name and how much time it took to de-allocate; if the de-allocation fails, it also shows the error that made it fail

• It ensures that de-allocation sub-routines are executed exactly once, avoiding errors of the “already de-allocated” nature.

You can take a look at the documentation site of this project, bear in mind though, this API is not a good fit for most applications.

componentm

Inspired by Clojure’s component, this library provides approaches for you to build all the resources of your applications transparently and safely. Some of the features this API offers:

• It keeps track of initialization time for each declared component of your application

• Builds a dependency graph for all your allocated components and then it guarantees they will be cleaned up in the right order (e.g., if component B depends on component A, the library makes sure that B is cleaned up before A)

• If any of the resources fail on initialization (e.g., a Database is not running), this API will rollback all the previously allocated components in the right order and then throw an exception with detailed information

• Makes use of teardown for resource cleanup, getting all the benefits of that API. This dependency is an implementation detail, so users of componentm don’t need to understand teardown.

• It will execute the initialization of your components concurrently if components are built using an Applicative interface.

To create ComponentM value, you can use buildComponent:

buildDatabasePool :: LogFunc -> IORef Etc.Config -> ComponentM (Pool SqlBackend)
buildDatabasePool logFunc configRef = do
  config     <- readIORef configRef
  connString <- getConnString config
  poolSize   <- getPoolSize config

  pool <- buildComponent "databasePool"
    (runRIO logFunc (createPostgresqlPool connString poolSize))
    (destroyAllResources)

  runMigrations logFunc pool

  return pool

Although not all functions are defined in the previous example, we can see that buildComponent, which creates a Pool of PostgreSQL Database Connection and also, a call to destroyAllResources which de-allocates that pool resource.

Following is how the output of your program would look when it cannot connect to the database:

2018-05-22 07:44:02.317225: [error]
# Application Failed

Application failed on initialization, following are the exceptions that made it failed:

  * ComponentAllocationFailed "db migrations" - the following error was reported:
    |
    `- libpq: failed (could not connect to server: Connection refused
                Is the server running on host "localhost" (127.0.0.1) and accepting
                TCP/IP connections on port 5432?
        )

Following, we have the information of application resources cleanup:

✓ servant-playground (0.000036s)
  ✓ http-server (0.000033s)
  ✓ db migrations (empty)
  ✓ databasePool (0.000003s)
  ✓ logReloader (empty)
  ✓ config (empty)

You’ll notice there is an indication of the errors that our initialization script throws, as well as the cleanup of all resources that we have allocated so far. You have total visibility on the lifecycle of your program.

componentm-devel

To tie the knot, I created a library that provides an easy to use API that automatically reloads your application everytime you call it with the ComponentM builder of your application. It provides a single function runComponentDevel that receives the same parameters as runComponentM1

Conclusion

I like to say ComponentM implements a Monad to avoid the usual pyramid of withResourceX functions at the beginning of your application. It composes them all together once, so you don’t need to worry about it. If you like these APIs, make sure to check out a sibling library called etc, it provides an API to manage the configuration values of your app as a map with multiple sources in a declarative way.

You can check out a full-fledged example where we setup an app using servant and persistent-postgresql using the libraries mentioned in this blog post.

Shameless Plug

If you would like to learn more about how to build robust Haskell applications, be sure to join my workshop at LambdaConf 2018

I’ve been working in a few projects at a time in Haskell for the past year, and one point that has been dragging is how much time it takes for a CI job to finish, given this, I started to experiment with other solutions to improve my build feedback loop.

As of a few weeks, my main goto for CI was TravisCI. There is a lot of Work done by others around how to build and test Haskell projects for MacOS, Linux, multiple stackage snapshots, etc. I was however never satisfied with how TravisCI wasted so much time doing the same things over and over again. Some of the builds would take a 20 minute time just to finish, stopping my feedback loop towards quickly closing PRs I was working at the time.

Looking for alternatives

After going through some alternatives, I landed at CircleCI via a tweet from them:

After I started experimenting with CircleCI, there were many things that I liked:

• It allows me to specify my docker images as builders - This is huge, I know most of the time spent in TravisCI is compiling and installing tools and a minimal amount of time was testing my code. By giving me the opportunity to build images with those tools baked in, I could start a PR build from an environment that didn’t spend time compiling the stuff I used to test my libraries.

• It allows you to run several jobs in parallel, and not just that, but also enables to build pipelines. This feature is excellent, I can keep my compilation matrix going, and not just that, I can add other jobs to release my software. I haven’t built an open-source app just yet, but I will keep this in mind for that.

• I got builds to become 95% faster when the build uses a previous cached output (yes, that’s ~2 minutes instead of ~40 minutes). With this flow, I can do quick tweaks to my PR builds and merge them quickly.

What I use to build my software

I rely on two docker images:

• romanandreg/haskell-tools:latest - This image contains hlint, stylish-haskell, brittany and refactor binaries to run all those tools as part of my CI pipeline.

• romanandreg/stack:latest - It contains a stack binary and a cabal-1.24 binary (to run the resolver), and I used this image to build my project in various snapshots.

My CircleCI config file makes use of this docker images that have all the tools I need to build my project.

How I build my project

I heavily rely on Makefiles to build my projects. Why? I do this because I want to be able to run my CI scripts on my local machine. The CI scripts are no different from what I would use on local development.

But seriously, why Makefile and no other tool? Makefiles are the best-supported tool on Unix systems, you have it available everywhere, and it does its job very well; granted the syntax is awful, but I have learned to live with that.

Organization of my Makefiles

After much working with Makefiles, I’ve learned that is a bad idea to keep all your tasks in a single file, this way I cannot re-use well-defined tasks in many projects; re-usability suffers when I need to modify the files per project details. If some common tasks are self-contained in a file, I have to copy/paste that into a new project, and voila, things work as I need them to work. The files I’m currently using:

• make/tools.make - Contains all tasks related to linting and formatting the code of a library

• make/sdist.make - Contains tasks to build a release of the library, and test it with the stack nightly version

• make/solver.make - Contains tasks to execute the resolver so that I can modify my stack.yaml file to support other snapshots dynamically.

I’m planning to add other Makefiles as I go, to script out the cabal-install setup for testing my projects with cabal-install. This functionality is something I need to port from my previous TravisCI build.

Where can you find an example

I believe the best two projects that are implementing this approach are teardown and etc, just go through their .circleci/config.yml file and their Makefiles.

Should you try CircleCI?

I highly recommend spending the time to try CircleCI, if you are typically waiting for a PR build to finish to push a new release up, this tool will provide a fantastic feedback loop for your workflow.