Haml Benchmark Inaccuracies

Way back when we released Haml 1.7, I said that it was only 1.7 times slower than ERB. In addition, we mentioned that it was 18 times faster than Markaby.

Well, I’m afraid the benchmarks we were running were lacking a little in the accuracy department. See, we had been running them through ActionView, Rails’ rendering manager. Except Markaby, which I couldn’t get set up quite right - that was run as just a templating engine.

This made some sense when we first started out and Haml was pretty much only used with Rails. But now between Merb, StaticMatic, and all the other stuff for using Haml without Rails, this isn’t really the case any more.

So I rewrote the benchmark script in trunk. Now it runs all the template engines on their own, without any ActionView interference. Just give them text and have them go.

The results are very intriguing. A reasonably typical run follows:

---------------------------------------------------
Benchmark: Haml vs. Other Template Engines
---------------------------------------------------
Rehearsal -------------------------------------------
haml:     0.070000   0.010000   0.080000 (  0.083126)
erb:      0.370000   0.010000   0.380000 (  0.386706)
erubis:   0.140000   0.010000   0.150000 (  0.149133)
mab:      0.660000   0.030000   0.690000 (  0.698684)
---------------------------------- total: 1.300000sec

              user     system      total        real
haml:     0.070000   0.010000   0.080000 (  0.075701)
erb:      0.360000   0.010000   0.370000 (  0.365875)
erubis:   0.140000   0.010000   0.150000 (  0.147834)
mab:      0.660000   0.030000   0.690000 (  0.697303)
Haml/ERB:     0.206904         ERB/Haml:     4.83316
Haml/Erubis:  0.512067         Erubis/Haml:  1.95287
Haml/Markaby: 0.108563         Markaby/Haml: 9.21128

Those are some pretty impressive results. Haml appears to be faster than everything. Unfortunately, they don’t entirely reflect reality either.

See, Haml has built-in template-based caching. It skips a considerable portion of the compilation process if it’s fed a template it’s already seen.

Other templating engines don’t have this built in. That’s why they appear to be so slow. However, in most real situations (Rails, Merb, etc.), they’re cached by the client code, so Haml’s caching benefits are mitigated.

As such, I hacked Haml::Engine a bit to get rid of all the caching. That made the results more accurate. Here are, roughly, the current best guesses as to Haml’s performance relative to other engines:

---------------------------------------------------
Benchmark: Haml vs. Other Template Engines
---------------------------------------------------
Rehearsal -------------------------------------------
haml:     0.370000   0.020000   0.390000 (  0.437077)
erb:      0.350000   0.020000   0.370000 (  0.370802)
erubis:   0.150000   0.010000   0.160000 (  0.171015)
mab:      0.620000   0.060000   0.680000 (  0.726201)
---------------------------------- total: 1.600000sec

              user     system      total        real
haml:     0.390000   0.030000   0.420000 (  0.436329)
erb:      0.370000   0.010000   0.380000 (  0.376922)
erubis:   0.150000   0.010000   0.160000 (  0.155158)
mab:      0.670000   0.040000   0.710000 (  0.719252)
Haml/ERB:     1.15761          ERB/Haml:     0.863848
Haml/Erubis:  2.81216          Erubis/Haml:  0.355598
Haml/Markaby: 0.606643         Markaby/Haml: 1.64842

This is more sane. Actually, it’s still better than we had announced for 1.7 in most cases. Haml’s only 1.16 times slower than ERB – much less than the 1.7 times we had thought. It’s not even that bad when compared to Erubis, which is known for its speed. And although it’s not actually better than Markaby by a factor of eighteen, it’s still a good deal faster.

Now, keep in mind that if you go off, check out trunk, and run rake benchmark, you’re likely to get results similar to my first run. Internal caching is still active.

However, after looking at this and the other engines, I’ve realized that it doesn’t really make sense to have it be that way. Really, the client should be the one handling caching, not the engine.

As such, Haml will probably switch over to that model soon. It’ll offer a way to get at the compiled method, and just let clients use that. Then the benchmarks will be much more meaningful.

Update: As of revision 633, Haml trunk doesn’t cache at all, either in Haml::Template or Haml::Engine. This means that it’ll (temporarily) be rather slower than other engines when used in Rails or Merb. I’ll make sure to post a note when other caching options appear.