Measuring latency in Colors! – Why game-devices are better for games than smart-phones

Sensationalist title aside, this article will present our measurements of our app Colors! across 10 devices: Nintendo 3DS, PlayStation Vita, Microsoft Surface Pro, Apple iPhone, Samsung Galaxy Note, Samsung Galaxy Note 2, Samsung Galaxy S3, Google Nexus 10, and with special appearance by Nintendo DS and the Nintendo Wii U.

As I’ve posted before, Latency in video-games (and apps) is something I feel strongly about. While the benefit of low latency is easy to understand for video-games where the action is intense, it’s equally true for a painting apps like Colors!. Just try write to write your name on your average note-taking app, and you will see how high latency makes this so much harder than it should be. This video from Microsoft Labs shows that in an excellent way.

Before I continue, I must point out that this is a long and technical article, and if you are looking for info on the Colors! 3D update, you should probably stop reading. Come back in a day or so and I think I should have some good news on that one as well.

With Colors!, we are in a pretty unique position where we have our app up and running on pretty much every major touch-screen device out there. And since I haven’t really seen any comparisons of latency across all the handheld hardware out there before, so I thought I present my findings here. A huge disclaimer though: THIS IS HOW COLORS! PERFORMS ACROSS DEVICES – WHICH MIGHT NOT BE HOW OTHER GAMES/APPS BEHAVE ON THOSE SAME DEVICES. Each device has their own way of doing input and rendering, and we have done more work on latency on some devices than on others. This means that these numbers should be taken with a grain of salt, and I’ll go into details about that later. But first, check out the results:

colors_latency

So let’s kick off and talk what I’m measuring here. By capturing video of Colors! running in 240fps using a GoPro Hero (awesome toy by the way!), I measured how long it takes from when I touch something on the screen, to when something is displayed on the screen based on those screen-coordinates. I do this by counting the number of 240fps frames it takes for the brush-stroke to catch up with the actual stylus (or finger in some cases). For example, if it takes 7 frames (in 240Hz) for the brush-stroke to catch up with the stylus, then that means that the input-to-screen latency is 1000*(7-0.5)/240=27.1ms. In other words, it would take 27.1ms from you touching the screen at a specific point for the device to display something based on that information. (the -0.5 is to account for that the pixels could have been updated anytime between frame 6 and frame 7 in the video-capture, but I’m not sure how sound that math is). Here is an example video of this:

I did such measurements for 10 devices, and multiple times on a couple of devices where I had more than one testing scenario. You can find the full result spread-sheet here.

Results

So what does this data tell us? Well, for one thing we can see that the gaming-devices consistently outperformed the smart-phones. I can only guess why, but it is possible that the Nintendo 3DS and the PSVita can have lower latency since they will never have to render anything on top of your game like a phone sometimes do. Also, the input handlers are usually very effective on game-devices, allowing you to read the latest available input rather than have to wait for it to be sent to you through an event-handler like you have to on iOS and Android.

On the Android side, it is very interesting to see that the various devices perform quite differently, even though they are running the same version of the Colors!. Also that the most popular Android device out there, the Galaxy S3, had the highest latency (except the Galaxy Note 2, but I think there is some bug with that since I could get lower latency with the built-in note app).

Finally, I think it’s nice to see that the handheld gaming devices perform better than a general Console/TV setup. From other articles I’ve read, the lowest latency you can hope for in a game highly optimized for latency like Guitar Hero and Call of Duty, you can at best hope for around 60ms of latency with a good TV.

Platform specific info

Nintendo 3DS – 23 ms

The Nintendo 3DS performs amazingly here, beating all other GPU based platforms. This is especially nice because it’s by a huge margin the most popular platform for Colors!. The reason it’s beating the PSVita by that much is that on the 3DS I spent some time implementing a single-buffer rendering mode. This is normally not viable for games since it causes tearing, but for drawing strokes this is not noticeable and saves one 60Hz frame of latency.

PlayStation Vita – 49 ms

The PSVita is also performing very well. The rendering of the PSVita is built for multi-threading, which made it harder for me to trick my way to lower latency, but with some additional work, it should be possible accomplish the same single-buffering techniques I use on the 3DS which should get it very close to the Nintendo 3DS.

Surface Pro – 100 ms

I wasn’t really expecting the Surface Pro to perform well, since the version of Colors! running on is still in alpha stage. However, we’re using the basic template DirectX implementation used in the samples, so it’s still a little bit surprising that it is not performing better since it’s built around DirectX which was developed with games in mind. Perhaps I’m processing the input incorrectly or something, or maybe the Metro UI adds some latency to it somehow.

iPhone 5 – 81 ms

On the iPhone, I’ve spent a fair amount of time trying to reduce latency. However, most of that time was spent in earlier iterations of iOS, so it is possible that there are better ways of doing thing now. Colors! is set up with the renderer running on a separate thread using DisplayLink, which I think is still the recommended way of doing things.

Galaxy Note – 71 ms

Galaxy Note is the device that has received most attention by Ben, our Android programmer. When using the SPen, MotionEvents are delivered at about 120Hz compared to the 60Hz for finger input. This means that Android is likely to deliver two MotionEvents per screen refresh (60Hz refresh rate). As an experiment Ben created a simple app that moves an OpenGL ES cross in response to input with the MotionEvents used to set the cross’s position. It used a custom version of GLSurfaceView which records the time at which the last elgSwapBuffers returned. The render mode is set to “when dirty”. MotionEvents are continuously processed with requestRender() only being called if at least 12 milliseconds have past since the last elgSwapBuffers returned. When using the SPen this achieved a touch latency of respectable 45 ms. While this suggests that there are improvements to make in Colors!, this sort of latency may not be achievable in since the rendering there is more complex than a simple cross. Ben also had this to say about Android rendering in general.

Galaxy S3 – 104 ms & Nexus 10 – 99 ms

These devices was measured using finger, which makes it harder to accurately measure the number of frames, and also seem to add some latency – probably because there is more processing involved for the OS to process finger input.

Galaxy Note 2 – 132 ms

The Galaxy Note 2 is the reason I started doing these measurements in the first place. We don’t have a Galaxy Note 2 device at the office, but we had gotten reports that latency was bad in Colors! on this device. So I dropped by a T-Mobile store to do make these measurements and luckily, they didn’t mind. Thanks T-Mobile! :) In addition to measuring Colors! I also measured the latency in the built-in S Pen Notes app, which is developed by Samsung specifically for this device and its stylus, which should hopefully be a good reference on how low we should be able to get the latency. Actually, on the original Galaxy Note, we already beat the S Pen Notes App with a small margin, but on the Galaxy Note 2 there is a huge amount of latency (132 ms in Colors! vs 78 ms in the S Pen App) which we are not sure where it’s coming from. Maybe someone reading this article can help us figure that out.

Wii U GamePad – 53 ms

The Wii U GamePad was the one I was most curious to test out the latency on. I know this has been a big point of concern since what is displayed on the GamePad is a video-stream sent over Wi-Fi from the stationary console. We are still in the process of figuring out if we can/should do a Wii U version of Colors! or not, so our test here was performed on the MiiVerse Notes drawing app. To our surprise it performed as well as for the PS Vita, which makes it a very attractive platform for us to bring Colors! to. It’s really impressive work by Nintendo to pull off such a low latency.

Nintendo DS – 9 ms

Talking about Nintendo DS version is this context feels a little bit like cheating. The Nintendo DS version is the earliest version of Colors! and was a public prototype, that was never available in an official way. However, I spent a ton of time testing various latency strategies on that one. The big difference is that this platform has no GPU, so all processing is done on the CPU which allowed me to do things that would not be possible on a GPU based platform. Along with that, I dedicated the whole sub-processor to read out and process input from the hardware at maximum speed (between 500 and 1500 Hz sampling, if I remember correctly). This resulted in less than a 60Hz frame of latency, which surprises a lot of people who say that shouldn’t be possible. However, since I had direct access to VRAM, I could draw directly to the front-buffer which get between 0 – 16.7ms of latency depending on where the LCD refresh scan-line is at the time when I drew a pixel. That ideal result of this would be an average of (0+16.7)/2=8.3 ms of latency, so now that I finally managed to measure this accurately, seeing that I reached 9 ms feels pretty great.

Conclusion

Comparing the dedicated game-devices and the smart-phones it is pretty obvious which one is going to have a better gaming-experience. I also don’t think there is any real reason why the smart-phones should be able to have as good latency as modern handheld gaming consoles. Maybe by highlighting this issue, the hardware and software manufactures for these devices can take another look at what they are doing that is causing this and improve the situation for us all.

I’m personally excited to continue working on this on my end, now that I can easily and accurately measure the latency in Colors!. There is more that I can do to reduce latency, and there are some excellent articles that go into depth about this. This very recent one by Carmack is especially interesting: http://www.altdevblogaday.com/2013/02/22/latency-mitigation-strategies/.

Here are a few more that my friends shared with me while I was writing this article:
http://realtimecollisiondetection.net/blog/?p=30
http://www.gamasutra.com/view/feature/132122/measuring_responsiveness_in_video_.php
http://www.cs.unc.edu/~olano/papers/latency/
http://www.eurogamer.net/articles/digitalfoundry-lag-factor-article

Jens

16 thoughts on “Measuring latency in Colors! – Why game-devices are better for games than smart-phones

  1. I’ve burned through a NDS I bought exclusively for colors! and was considering a Galaxy Note II tomorrow. In light of this post I don’t know if I can spend the money for something that will be 14 times slower than the DS and twice the cost.

    Has anyone at Samsung been of any help?

  2. @Asa, no, we just discovered that this was a real issue with the Note 2. Either way, we definitely intend to get that issue fixed soon, with or without external help, as the Galaxy Note 2 is such great hardware.

  3. It certainly seem impressive, so I hope you can get your fantastic app working with it’s event/update system. It’s fantastic software and is certainly a make-or-break app for any portable hardware I intend to buy. I’d prefer a phablet like the Note due to slimness and not carrying two devices on me, and now that I’ve seen the Note in person I really don’t want to settle for the tiny 3DS screen. But that’s my concern. I’ll certainly be keeping an eye on your RSS feed for solution.

    I really cannot thank you enough for making such a elegant and fun painting app. Thank you.

    And another ‘Thanks’ for the fast reply!

  4. Hmm… interesting… does this mean you are working on a wii u version of colors? If so… It will already be bought by me :D

  5. IMO, the PSvita was dead when it hit the shelves. It suffered from multiple issues, including: Proprietary required memory cards, and bad touch screen feel. There wasn’t really anything it could do that the 3DS couldn’t, and it didn’t have as many amazing titles.

  6. I love the DS version even though I never uploaded anything online. I hope you guys can improve the latency of the galaxy note version. Speaking of which, please allow Galaxy Note 10.1 users to download the note version of Colors! I had to ask from someone for the apk of the note version just to install it on my phone.

  7. Aww, Jens. This update is going to be nice, but it won’t mean anything if the site destroys itself beforehand — Please monitor the site more and keep mature content flagged or removed. Since most users do view the site through 3DS, there have been long-lasting mature or inappropriate drawings uploaded on the site.

  8. @WillowBird,

    We sadly don’t have the resources to check each uploaded painting, so we rely on the community to help us flag inappropriate paintings. Once the update is released, it will be possible to flag paintings directly from Colors! 3D, but until then this is only possible from colorslive.com. And there isn’t much we can do to speed up the release of the update. That is all in Nintendo’s hands.

  9. You wrote:
    original Galaxy Note, we already beat the S Pen Notes App with a small margin, but on the Galaxy Note 2 there is a huge amount of latency (132 ms in Colors! vs 78 ms in the S Pen App) which we are not sure where it’s coming from. Maybe someone reading this article can help us figure that out.

    1. Touch screen sensitivity..video architecture and video driver
    2. OS translation
    3. Image output

    Maybe the galaxy lag is due to a different video architecture since, if I understand correctly, the OS is the same. Or maybe the galaxy has lag due to an added component..an app that bridges between the video architecture input and the OS?

  10. Ah-ha! But the true genius of your studies is that the 3Ds has the best ratings of all devices tested…Comparing the features of other devices tested if nintendo developed a gaming console that had more tablet features, it would knock the competition out of the water. I checked what tablets have creative drawing apps and the Galaxy Note 10.1 was the only one in the common market. In the specialized market of art, the drawing tablets were high on sensitivity and low lag, but miserable in other features and I could not justify buying that and having to also buy a tablet to do other activities. The 3Ds with the internet features is crawling toward a multi use gaming console…like the razer. The two biggest draw backs about the 3Ds is the internet browser unable to display web pages and run script, and the size of the drawing screen and the difference in resolution between the upper and lower screen.

    Colors!3d is the best app for drawing and sharing. I have not seen any in comparison in the handheld device platform..excluding win 8 and adobe..which seems pointless to have on a handheld and desktop..I mean my phone is win 8 and I can share that way…besides Win 8 apps suck compared to android or apple.

    Colors, in my opinion has stuck gold by identifying the need of nintendo users to communicate with one another.

    A app that runs through the wifi and not nintendo connect where users can post requests and leave messages would be extremely popular as is seen by all these kinds of posts in the colors program. Users are using colors to communicate with other users rather than as a art share app.

    Its a dream to have a nintendo tablet..smiles.

    I understand how complicated the whole thing is…another consideration is the expanding market share of nintendo where the majority were teens and now it is expanding to younger and older users so, in my opinion, filtering features need to be considered.

    I use my Galaxy Note 10.1 for its tablet features and my nintendo ds for on the go drawing, gaming and so forth.

    Hmm dual os,,android and nintendo? lol
    I’ll end my ramblings. now…

Leave a Reply to WillowBird Cancel reply

Your email address will not be published. Required fields are marked *