Introducing RunnerMark: AIR Benchmark Suite

In an effort to benchmark the various Stage3D 2D Frameworks, I’ve created a small test suite called RunnerMark.

It’s hosted on github here, along with a video of it in action:

All contributions are more than welcome. If you feel like something can be optimized, or have an entirely new test suite please submit!

What is it?

Runner Mark benchmarks the Stage3D 2D Frameworks such as ND2D, Starling and Genome2d.

It aims to simulate multiple types of simultaneous load, similar to what you would see in a typical game. Specifically, RunnerMark includes the following elements:

  • 1 Main character with a run animation
  • Enemies with a “Chomp” animation
  • 1 stationary background image (sky)
  • 2 Parallax scrolling backgrounds
  • Scrolling Ground Tiles
  • Scrolling Platforms
  • ~30 small dust sprites as your character runs, to simulate some level of a particles
  • Rudimentary AI and hit detection for all characters

I feel this is a good rendering load to represent all the complex things that happen within a game. If you disagree, or thing something should be added, sound off in the comments and let me know!

Scoring System

RunnerMark awards 580pts for rendering the basic scene at a solid 580 FPS. Then 1 additional point for each animated Enemy added to the scene.

As an example, a score of 650 would indicate the basic scene @ 58fps + 70 animated Enemies. A score of 400, indicates the basic scene was only able to render at 40fps, and no Enemy’s were added at all.



In my initial tests, I compared GPU Mode, Starling, ND2D and Genome2DNativeRenderer.

Here are the results by device:

Galaxy Nexus
GPU – 728
Starling – 674
ND2D – 290
G2DNative – 200

Nexus One
GPU – 410
Starling – 450
G2DNative – 260
ND2D – 350

iPad 2
GPU – 839
Starling – 686
ND2D – 759
G2DNative – 813

iPad 1
GPU – 595
Starling – 490
ND2D – 150
G2DNative – 160

iPhone 4
GPU – 636
Starling – 490
ND2D – 200
G2DNative – 180

* NOTE: The very latest Starling build uses a new optimization technique, which is not present with the other frameworks. This is why the Starling results are so impressive compared to ND2D and G2D. As you can see though, this optimization has no effect at all on iPad 2. It seems like the optimization was relatively easy, so you can (hopefully) expect it to come to the other frameworks within the next couple of weeks.

Initial Conclusions

In short? We’re getting there…

With the latest optimizations, Starling is finally beginning to come into the range of GPU Render Mode, now instead of 50% performance drop, we’re seeing just 10-20%. Although this is still disappointing, (Stage3D should be faster DAMMIT!), this is at least within the general performance  ballpark. That’s good news.

There needs to be further optimizations in the 2D frameworks if they’re going to reach GPU Render Speed, or better yet, Native! Also, Peter from Genome2D is actively looking at this issue right now to see if there’s more gains which can be made on these older devices, *fingers crossed*!

In the upcoming weeks I’ll be adding some more rendering techniques (proper G2D implementation), and then re-testing all the libs once they’ve had a chance to add the new optimization(s).

Written by

5 Comments to “Introducing RunnerMark: AIR Benchmark Suite”

  1. Moko says:

    Thanks for post.

    Starling well-ptimized comparing to previous versions.

  2. Philippe says:

    Hey, I ported your RunnerMark over to haxe NME.

    Here are the results:

  3. Bruno Garcia says:

    Would you share more about the new optimization technique in Starling?

    • shawn says:

      Hey Bruno, I’m not really sure on the details, but it was a low-level optimization to the framework itself, it should improve any Starling based rendering technique.

Leave a Reply