Power Profiling: HTTPS Long Polling vs. MQTT with SSL, on Android

Introduction

A little while ago I performed some power profiling of MQTT on Android to try and put a figure on just how efficient this technology is on mobile devices. I think I got some pretty good results (enough to give an idea anyway), but the next question people always ask is: ‘So how does that compare to the alternatives?’.
Apples Vs Oranges
Well, first of all, it is really Apples vs. Oranges :D

Of my three main choices for Android mobile push notifications: MQTT, HTTP or C2DM, each is designed for a different purpose; with different features, bad points and good points. I’m not going to do a full comparison here, but to summarise:

  • MQTT – designed to provide low latency, assured messaging over fragile networks and efficient distribution to one or many receivers. Protocol focuses on minimising the amount of bytes flowing over the wire and low power usage. Maximum message size of 256MB, but not really designed for sending large amounts of data; better at a high volume of low size messages. Provides a two-way communication channel.
  • HTTP – designed as a request-response protocol for client-server computing. Best known as the foundation of data communication for the World Wide Web (well, duh). It can be ‘abused’ to provide push like capabilities, for example by using Comet style approaches, but it really isn’t designed with this in mind. Provides two-way communication capabilities.
  • C2DM – designed as a one-way ‘shoulder-tap’ notification system. On the plus side, Google takes care of getting the messages to the device and to your application; waking it if necessary. However there are a number of restrictions. The main one is, as the Google Docs themselves say, that ‘C2DM makes no guarantees about delivery or the order of messages’. Therefore, the recommendation is that a C2DM message should never contain the data itself but, rather, be used to provide a notification that there is data available; and applications should then contact their own server to get the data. Other restrictions include:
    • Only available on devices running > Android 2.2.
    • Requires a registered Google account on the device.
    • Message size is limited to 1024 bytes.
    • Google limits the number of messages a sender sends in aggregate.
    • And the number of messages a sender sends to a specific device.

However, fruit rivalry aside, it’s still an interesting question and my points above don’t tend to stop people asking, so I thought I’d try a comparison against the most equivalent & open approach, in my opinion, (and also the easiest one for me to test): HTTP. Also, what with security on mobile being such a big issue, I decided I’d test the secure versions: HTTPS and SSL enabled MQTT.

What was I testing?

On the MQTT side, it was very similar to what I tested previously: a simple application using a custom wrapper around the standard Java MQTT client offered by IBM; but this time using an SSL connection against a SSL enabled instance of Micro Broker and performing mutual certificate based authentication between the client and server.

For the HTTPS side, I decided to use Comet style long polling. If you’re not familiar with this, the client-server interaction looks something like this:

Basically, the client makes a HTTPS request to the server, which is kept open until the server has new data to send to the client. When this data arrives, the server sends it to the client and closes the request. The client then initiates a new long polling request in order to obtain subsequent events.

On the server side, I used a simple SSL enabled Comet style Pub/Sub server I wrote recently in Node.js. And for the client side, I wrote a simple Android app that connects to this as needed using a standard HttpsURLConnection. Again performing mutual certificate based authentication between the two.

How did I test them?

Again, it was very similar to what I did previously; using the Android applications mentioned above, along with simple desktop apps to drive them where required. The tests were kicked off manually, however I’ve improved the overall automation of them. The results from here should be reliably comparable with the previous ones.

Caveats / Specifics

  • The power profiling was performed on my HTC Desire, now running Android 2.2.2 – Build 2.33.161.6 CL345089.
  • ‘% Battery / Hour’ refers to the % of the fully charged capacity of my phone’s battery that is used per hour. My phone has a standard Li-Ion battery that is rated at: 1400mAh & 3.7V.
  • The tool I used to capture the power usage data (PowerTutor) uses real data, combined with a power usage model for some aspects. This model has been tailored against the type of device I used for this testing and is reported to be accurate to within 0.8% on average, with a 2.5% margin of error. If you’re interested, you can find out more here.
  • I’ve made a number of decisions on my implementation of the two approaches. I’ve done this to try and make MQTT and HTTPS more equal and make things fairer; not to bias things one way or the other. I’ve tried to do this sensibly and I’ll explain myself as I go along; however some people may have decided to do things slightly differently.
  • I’ve tried my best to produce correct, consistent and usable results; however I am human and so there’s a chance I have made mistakes. As such, these figures shouldn’t be treated as gospel, but I would expect them to be representative of what you could expect to see and they should provide an accurate comparison between the two approaches.

Results

Establishing & Maintaining a Connection

I decided to start out by looking at the cost of establishing and then maintaining an open connection for each approach. And I pretty much ran straight into one of the main challenges of this testing: how to make it a fair fight? How do I structure things so I’m not being un-necessarily biased to one technology or the other?

Straight out of the box, MQTT is much more feature rich than HTTPS and the main feature involved here is the Keep Alive. Basically this is a way for the client to detect in a timely manner when the server connection has been lost (and vice-versa) without having to wait for the often long TCP/IP timeout. To do this, the MQTT client and server exchange keep-alive messages every so often. This also serves to maintain the raw TCP/IP connection; as in some circumstances (e.g. on some 3G networks) long-running connections that have no data flowing my be purged.

HTTPS doesn’t have this built in, but it seems like useful functionality for an application to have and so I decided to add this into my HTTPS client by using a read timeout of the same duration as the MQTT keep alive interval. This means that if the server doesn’t respond with any data within x seconds then an Exception is thrown and the connection torn down. The client will then establish a new one. If it can’t, then the client knows the server is unreachable.

Hopefully that sounds like a fair thing to do, but don’t worry, it also flows the other way. As, in my implementation, the HTTPS client also ‘subscribes’ as part of it’s connection (the topic of interest being part of the URL), I decided to consider the act of MQTT connecting to include both the connection and a subscription step.

Anyway, enough waffling, onto the results. First of all, the amount of power taken to establish the initial connection to the server:

% Battery Used
3G Wifi
HTTPS MQTT HTTPS MQTT
0.02972 0.04563 0.00228 0.00276

As you can see, HTTPS wins this one hands down; and by quite a significant amount (~30% for 3G). However this isn’t really that much of a surprise:

With the HTTPS approach, all we’re doing is opening a connection to the appropriate URL and exchanging certificates. Whereas, for MQTT, we establish the raw connection, perform the certificate exchange and then flow additional information; including the unique Client Id. We then wait to receive a confirmation from the server and then subsequently send additional messages to subscribe to the test topic (which shouldn’t be required for subsequent reconnections; as the server can remember our subscriptions for us).

Note: just to emphasise, this is an artificial limitation that I’ve placed on MQTT. If you don’t need the subscribe step, then the actual cost of connection is very comparable. However, this is something that I decided to do to make the comparison more fair.

Now onto the cost of ‘maintaining’ that connection (in % Battery / Hour):

% Battery / Hour
3G Wifi
Keep Alive
(Seconds)
HTTPS MQTT HTTPS MQTT
60 1.11553 0.72465 0.15839 0.01055
120 0.48697 0.32041 0.08774 0.00478
240 0.33277 0.16027 0.02897 0.00230
480 0.08263 0.07991 0.00824 0.00112

As you can see, this is where MQTT gains back ground. In all cases it uses less power and in most a fair bit less. So the longer the connection is established, the ‘cheaper’ MQTT is to use.

If we consider the 3G case (the most relevant to the mobile story) and a keep alive / read timeout interval of 240 seconds (my personal favourite) then by my calculations we make up for the difference in the cost of connecting after ~5 ½ minutes of being connected:

Note: You can click & drag on an area of the graph to zoom to it.
Single click anywhere to zoom back out.

3G – 240s Keep Alive – % Battery Used Creating and Maintaining a Connection

mW
1 second period

After that, everything else is gravy, and based on this I reckon you’d save ~4.1% battery per day just by using MQTT over HTTPS to maintain an open stable connection.

The reason for this is simple. While it costs MQTT more to create the initial connection, this is essentially a one off and the cost of the following keep alives is comparatively small. Whereas for HTTPS it needs to perform the ‘expensive’ connection stage every time it has to reconnect (up to once each keep alive interval, in my implementation).

Interestingly, this does seem to suggest that if the phone is constantly cycling / dropping connections and you’ve decided not to let the server remember your connections (so that MQTT has to perform the additional subscribe step on every reconnect – cleanstart=true for those in the know) then HTTPS may be a better choice for that particular scenario; and in that case MQTT would be better in slightly more stable situations. The obvious question then is: how often is the connection likely to drop in typical mobile usage?

I’ve tried searching, but not really found any sensible statistics so far. The mobile network should handle routing TCP/IP connections from tower to tower, so that shouldn’t be an issue; however you might go out of range or hit a tower with no spare capacity. Although I guess that’s not likely to happen very often (?). It’s something that I’d be interested in finding out, but it sounds like it would need a large field trial and that’s a little outside the scope of this blog post :)

Receiving

The next thing I looked at was receiving messages on the phone. To do this, I sent messages to the phone in two different ways:

1. Sporadically

To try and emulate a more realistic style of notification sending, I decided to send 6 messages to the phone, with an average of 1 message per 10 minute interval, but with the message being sent ‘randomly’ during that time. This would enable me to test the long-term performance of each approach, but also to add in some unpredictability. For MQTT, the messages were delivered at QoS 1. The results are shown below:

% Battery Used
3G Wifi
HTTPS MQTT HTTPS MQTT
0.34645 0.27239 0.04817 0.00411

Note: the timing of the messages was calculated beforehand, so both MQTT and HTTPS experienced exactly the same delays, etc.

And here’s some graphs:

3G – Receive 6 x 1 byte messages over 60 minutes – Total mW

mW
1 second period

Wifi – Receive 6 x 1 byte messages over 60 minutes – Total mW

mW
1 second period

As you can see, in both cases MQTT wins; by ~30% for the 3G case and an order of magnitude for Wifi.

Also, I actually could’ve made the MQTT implementation more efficient: Currently I blindly send the keep alive message every x seconds, regardless of when the last message was exchanged between client and server. Whereas I’d actually only need to send the keep alive message x seconds after any message was last sent / received. Interestingly, the HTTPS implementation already exhibits this behaviour and so has a slight advantage.

One final thing of note is that the Wifi graph above nicely shows the difference in the keep alive cost between the two approaches. This is not as obvious in the 3G graph, as the CPU & data transmission costs are overwhelmed by the cost of having the 3G active.

2. As Fast As Possible

To be consistent with the results I produced previously, I also decided to test sending 1024 messages, of 1 byte a piece, to the phone, as quickly as possible. The results are shown below:

3G Wifi
HTTPS MQTT HTTPS MQTT
% Battery / Hour 18.43% 16.13% 3.45% 4.23%
Messages / Hour 1708 160278 3628 263314
% Battery / Message * 0.01709 0.00010 0.00095 0.00002
Messages Received 240 / 1024 1024 / 1024 524 / 1024 1024 / 1024

 
* – % Battery / Message is a bit of a silly metric. There is a fixed cost in having the Wifi or 3G active and so the actual cost of just sending / receiving a single message would be higher. However these figures do serve to indicate the difference between the two approaches and hopefully give you an indication of the battery usage involved.

And here’s some more graphs:

3G – Receive 1024 x 1 byte messages – Total mW

mW
1 second period

Wifi – Receive 1024 x 1 byte messages – Total mW

mW
1 second period

Note: the data in the graphs for each approach stops 5 seconds after all messages have been received.

Again, MQTT wins this one. Despite it being a bit of a silly metric, the % Battery / Message really is the interesting one to look at, and in the 3G case MQTT is over two orders of magnitude better than HTTPS. This highlights both the low power usage of MQTT and also the speed with which the messages were received (averaging 160278 messages per hour for MQTT versus only 1708 for HTTPS).

Another important thing to highlight is the reliability (or perhaps, more accurately, success rate) of delivery. For 3G, every MQTT message got through, whereas HTTPS only managed 240 of the 1024, ~24%. This is because a lot of messages were missed in the interval between when the connection closed with the previous message and was subsequently re-established to receive the next. This is somewhat mitigated on Wifi, as the connection can be re-established more quickly. I could’ve decided to implement a queueing mechanism on the server to help with this, but as MQTT didn’t need it, I thought it was fair not to.

Sending

Finally, I looked at sending messages from the phone. I found it somewhat difficult to get sensible & consistent figures for sending a single message and so I decided to scale up to sending 1024 messages, of 1 byte a piece, as quickly as possible. The results are shown below:

3G Wifi
HTTPS MQTT HTTPS MQTT
% Battery / Hour 18.79% 17.80% 5.44% 3.66%
Messages / Hour 1926 21685 5229 23184
% Battery / Message * 0.00975 0.00082 0.00104 0.00016

And here’s, yes you guessed it, yet more graphs:

3G – Send 1024 x 1 byte messages – Total mW

mW
1 second period

Wifi – Send 1024 x 1 byte messages – Total mW

mW
1 second period

And again, MQTT wins. It uses significantly less power (see % Battery / Message) and is a lot quicker. I don’t think there’s anything else I need to say about this one.

Conclusions

Well, I have to admit I’m somewhat relieved. I’ve been using MQTT on mobile for a while now and I’ve been confident that the power consumption was lower than that of HTTPS, but it’s good to see that the figures support that. I’m definitely an MQTT fan, but I’ve been keen to make this testing fair and unbiased and I would’ve reported the results whichever came out on top (although I know a few people who would be none too happy about that). So definitely relieved :)

As this testing has shown, MQTT uses less power to maintain an open connection, to receive messages and to send them. It also does these last two more quickly and reliably. The only place where it loses out is in establishing the initial connection (with cleanstart=true) and that’s mitigated after ~5 ½ minutes of being connected.

There’s a number of other benefits of MQTT over HTTPS as well, which I’ve not really included in this testing, which include:

  • Assured delivery
  • Retained messages
  • Last will & testament
  • Multiple subscriptions ‘multiplexed’ over one connection

These could theoretically be built into a HTTPS approach, but they’re not there by default.

So I think it’s definitely fair to say that MQTT wins overall and is my technology of choice for providing true push capabilities on Android.

What next?

I’ve addressed a few of the additional use cases I identified in my last bout of MQTT power profiling, but there’s still a couple of things I’d like to look in to:

  • Testing the effect of message size on sending & receiving.
  • Testing power usage when running on fragile networks that keep dropping & being re-established.
  • Performing some longer term live field testing.

So, congratulations on reading this far. That’s a bit of an epic post, my longest to date, but again hopefully it was interesting and informative. And I welcome any comments.


23 Comments

  • 05/31/2012 - 9:13 am | Permalink

    Great work, Steve. I’ve added this to the list of resources about MQTT on Android on the mqtt.org wiki http://mqtt.org/wiki/doku.php/mqtt_on_the_android_platform

    PS when are you going to start submitting bugs and patches for Eclipse Paho? :-)

    • 05/31/2012 - 9:42 am | Permalink

      I’d love to, but I’m not sure how to go about getting on the committers list. Any advice?

      • 05/31/2012 - 4:54 pm | Permalink

        Stephen, great article!

        You don’t have to be a committer to open bugs or provide patches. Just do to https://bugs.eclipse.org/bugs/ and start submitting.

        Ian

      • 05/31/2012 - 5:01 pm | Permalink

        Steve,

        Now that the project is already up and running, you get on Eclipse Paho’s committer list by submitting bugs and patches. Similar to most open source projects, it’s a meritocracy and you earn your way in. But given the level of expertise demonstrated here, I predict it wouldn’t take that long :)

        I for one would love to hear what you would learn from getting the Paho Java client working on Android!

  • Roger
    05/31/2012 - 10:46 am | Permalink

    Very interesting. Can I pester you for the test code at some point in the indefinite future when mosquitto has ssl support?

    • 05/31/2012 - 11:46 am | Permalink

      Sure, I should think so. I might need to check a couple of elements of the Android MQTT side with work, but I’m sure we can sort something out :)

  • 05/31/2012 - 11:15 am | Permalink

    Nice work, it would be interesting to see the comparison between MQTT and WebSockets as well. Did you think about doing this?

    • 05/31/2012 - 11:42 am | Permalink

      Thanks. I did consider WebSockets, but I wasn’t sure if it’s something you would use in a ‘background’ application or whether you’d just use a standard TCP socket. As in this case I’m not considering running in a browser and I don’t know if WebSockets provides any additional benefits to make it preferable (?).

      Also, the current lack of consistent browser support and the fact that HTTPS is probably the most popular current alternative led me to choose it for this round of testing.

      Though if there’s enough demand, and people are happy to help me figure out a sensible and fair scenario / setup, I’d be happy to give it a go :)

  • 06/09/2012 - 9:43 am | Permalink

    Really interesting piece of research. I’d love to get hold of your test code and adapt it for a comparison against web sockets (which can run in background under certain conditions in Android). Clearly HTTP is poorly performing for any scenario where you’re doing a large number of small messages. This is definitely the case for anything doing telemetry and I’m not surprised to see MQTT outperform HTTP in this instance.

    I’d definitely like to see / do a comparison with web sockets as this was the enhancement to HTTP to make it more useful in exactly the type of conditions you’ve discussed for MQTT usage. I’ve done a lot of work with web sockets based systems and prefer the consistency across the stack, however I’d be fascinated to see a comparison based on this type of performance analysis to see how WS compares to both standard HTTP and MQTT. Would be very keen (and able) to help out on this front if you want to extend the test.

  • Ewan Auguste
    06/18/2012 - 3:38 pm | Permalink

    Great work I have added this to our MQdev Community.
    https://www.ibm.com/developerworks/mydeveloperworks/blogs/messaging/?lang=en

  • 06/21/2012 - 3:32 pm | Permalink

    Great article, thorough analysis. Thanks a lot!

  • Eurig Jones
    02/20/2013 - 8:38 pm | Permalink

    Great article. Thanks for sharing!

    I noticed though that you mentioned C2DM (or GCM as it’s called these days) but did not make comparisons with this.

    Have you established any thoughts about GCM vs MQTT specifically in terms of power profiling?

    I would assume that GCM would win only because Google would be silly not to make sure it’s very efficient, but GCM is not available without a Google Account as we know.

    What are your thoughts on dale’s implementation of the client code? Do you think it is production ready?

  • Abhinav
    03/26/2013 - 5:27 pm | Permalink

    Great work!!
    Can you please share how you calculated: (% Battery / Hour) and (% Battery / Message)?
    Is your (% Battery / Message) data dependent upon the size of message as well?

    Cheers!!

  • Jesse Farebrother
    04/06/2013 - 9:08 am | Permalink

    Excellent article by far the best research on the topic. Definitely has made my decision much easier.

  • sathish
    04/26/2013 - 6:28 am | Permalink

    i just wanted to know how mqtt handles if the message sent exceeds the payload length…..

  • Srini
    10/07/2013 - 9:31 am | Permalink

    Excellent analysis! Any data or views on MQTT Vs. SIP power profiles?

  • Pingback: MQTT : the open road to internet of things | ekito peopleekito people

  • Kings
    05/28/2014 - 6:06 pm | Permalink

    Question: does anybody know how to log MQTT topics into a log file (so that the logs could be reviewed later)? Thanks.

    • 06/04/2014 - 2:11 pm | Permalink

      Should be simple enough to e.g. use the Paho Python client to: subscribe to a topic; and log all messages into a file on disk. Is that what you’re asking?

  • King
    05/28/2014 - 6:09 pm | Permalink

    Does anybody know how to log MQTT topics into a log file (so that the log file could be reviewed later, if necessary)?

  • King
    06/04/2014 - 9:11 pm | Permalink

    Hi Andy: Thank you very much for your response. In this case if you are using a Tablet or a Smartphone (which my team will be using when we start testing), where will the data be logged to? I’m trying to find a location (or folder) where the logs will reside). Any help will be appreciated. Thanks again.

  • 06/20/2014 - 7:06 am | Permalink

    Hi Stephen,

    I was looking at your comparison and there are some really important details missing to gain some useful insight, namely concerning security.

    You compared HTTPS with MQTT (with SSL). Of course, using SSL today would be a real problem because it has been replaced by various versions of TLS over the last 15+ years.

    The complexity of the security protocol is much, much higher than HTTP or MQTT and so it would be good to provide details about whether you have used the identical configuration for the two and what that configuration actually was. So, it is quite likely that you have actually compared the performance of two TLS stacks (with different configuration) in the end because the actual data transmission at the HTTP / MQTT layer is almost like noise compared to TLS (particularly when it comes to power consumption).

    Furthermore, whenever you make these types of networking experiments then you need to be prepared to also run into various issues with performance issues that are caused by the network rather than the end devices. This might explain why some of the 3G graphs look a bit strange.

    Finally, it might have been better to compare MQTT against CoAP since both are tailored to the IoT environment and HTTP provides a number of features that MQTT obviously doesn’t have. CoAP of course offers more features than MQTT but maybe that can be dealt with by adding them to MQTT somehow for testing purposes.

    Ciao
    Hannes

  • Tammo
    07/24/2014 - 8:26 pm | Permalink

    Would be nice to know how MQTT power consumption compares to XMPP PubSub. Could imagine that both are almost equal.

    Thanks a lot for this nice article.

  • Leave a Reply

    Your email address will not be published. Required fields are marked *

    You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>