Diagnose memory leaks on PINNED values with GHC 9.2.1 and up

I pride myself on my odd ability for diagnosing memory leaks on Haskell. I have sent some PRs on OSS with some neat diagrams to corroborate the fixes and they are always welcomed. It is a good way to get to know a project and can be considered a low level bug in most cases.

The common way to learn how to do this is reading the GHC user guide section on profiling and experimenting on your own programs. What I usually end up doing is this:

  1. Compile with profiling enabled: Either
    • ghc -prof -fprof-auto
    • stack build --profile
    • cabal v2-build --enable-profiling
  2. Generate some figures with ./program-exe +RTS -xc -L350 -p -hc
  3. Use hp2pretty to generate a diagram. This can involve pruning the uninteresting sections of the program-exe.hp file to get a specific interval in time.
  4. Identify the offending band on the resulting SVG, identify the cost centre stack reported on the .hp file and read the code.
  5. If a more specific cost centre is needed, I add a {-# SCC "slack0" #-} expr annotation at call sites that can be interesting. I repeat the previous points.

This is all fine and dandy! It works for most of the cases that you encounter.

But there was still a thorn on this approach: ByteStrings. On the Haskell heap these are PINNED objects. What this means is that the copying garbage collector gives them special treatment when moving values between the from space to the to space and lets them fixed in place. This is related to foreign pointers and the expectation of C-libraries of not having the values their pointers point moved without notice.

This manifests in diagrams with the following pattern:

Notice the purple band declared as PINNED without any cost-centre information. We would like to have information as in the band 96369, which when read from the .hp file gives us a series of cost-centres names separated by a /.

By the way, this is a real program. I was working on it a $JOB but had to throw my hands on the air as my usual bag of tricks did not work… on GHC 8.10.7.

That is right, my usual bag of tricks got more useful on GHC 9.2.1 as /geekosaur/ informed me #haskell @ libera. Now these PINNED values have normal cost-centres stack associated to them.

Sure, I had to update some dependencies to compile on the new GHC version. But once that was done I was left with a new binary that reported these figures:

Now I got info on the cost-centre stack 85488. I can search up until I get on a cost-centre for my program and force that value.

The leak ended up being a un-demanded selector function over parsed bytestring. The lack of inmediate demand meant that GHC had to leave those values around in case could be needed on the future. Easy to fix once you understand, a bang-pattern is what you need on those cases. This left me with the following diagram:

Win win win.

Conclusion

Now I feel more confident on my odd ability to diagnose memory leaks :-) . This would have taken me at least another afternoon to diagnose under GHC 8.10.7 . On GHC 9.2.3, it took me 15 min, great stuff.

Comentarios

Entradas más populares de este blog

An apologia for lazy evaluation

How to avoid correctness space leaks on a lazy setting in haskell

Presenting data-forced, an alternative to bang patterns for long lived data structures