WakeMate: The statistics are in

I’ve been using my WakeMate for 17 days now.  Today they released a CSV export feature and so I’ve gotten a chance to run my data through some basic statistical analysis.  I’m primarily interested in knowing what factors affect my sleep.

My goal is not to publish some new findings in sleep research but rather to get a vague idea of how the things I do affect my sleep.  This is definitely not some super-rigerous study of my sleep habits but mostly some observations that merit further exploration.

Methodology

I wear the WakeMate to bed.  I record five datapoints each night:

• Whether or not I have taken 2.5mg melatonin in the past hour or two–this is often sold as a sleep aid but some criticize its effectiveness
• Whether or not I have had my morning coffee (always, so far)
• Whether or not I have had a weaker cup of coffee in the evening or late afternoon
• Whether or not I am wearing my sleep mask
• Whether or not it is raining

When I wake up, my data is uploaded to WakeMate and they determine a bunch of stuff like how many times I woke up and how many minutes until I fell asleep.  I also fill out a (subjective) slider indicating my morning grogginess.

I should point out that there’s a lot in this methodology that can yield bad data.  For instance, in a truly rigorous experiment each factor to be tested should be randomly determined.  However, I do not roll dice to decide whether to wear a sleep mask, I do it based on my comfort factor with the lighting.  Similarly, I tend not to take melatonin when I am very tired.  This can cause strange trends (for instance, a sleep mask may appear to slightly prolong my “time until sleep” even though it should do the opposite by intuition, however this may be a result of my awakeness which causes me to wear the sleep mask in the first place…)

With that disclaimer out of the way, here are the results so far:

The Bad News

The bad news is there are an awful lot of things that have no statistically-significant effect on anything.  The big elephant in the room is that I can correlate incredibly little to my “User Score”, that is, that subjective slider that I fill out in the morning.  Since this, to me, is the true “Did I sleep well” test, it’s a little disheartening to see that I can’t predict this with any accuracy to speak of.  There are vague indications that with a lot more data I might be able to show an interaction between the number of awakenings in a night and my morning grogginess–but this is a vague supposition (my theory:  morning grogginess is tied to some property of only those specific awakenings that happen near the final awakening, which would explain the “looks like it could eventually be a correlation” siren song without ever actually ever being a correlation..)  What is abundantly clear is that WakeMate’s score and my own have absolutely zero correlation, which is a bit unfortunate.

This means that, sadly, for the purposes of predicting or understanding my morning grogginess I’m about as in the dark as I was before WakeMate.

The Good News

I have managed to demonstrate a few things, however:

• Wearing a sleep mask makes me sleep longer.  This is sort of common-sense, but I do see it clearly in the data (p = .06), which is nice
• Wearing a sleep mask may increase the time taken to fall asleep–I suspect this is a result of my bad methodology as described above (p = 0.08)

Weak Correlations

These weak correlations are worth further study:

• Rain may increase my time asleep by 26 minutes (p=.13)
• Melatonin may take a few points off my WakeMate score (p = .15); many group-level tests report a statistically-significant difference here although the overall difference is not significant
• Rain may increase my awakenings by 1.5 awakenings on an average night (p = .15)

A Strange Result: WakeMate Score as a Caffeine Index?

And then there’s one, very very strange result that I am unable to explain:  although my evening coffee does not correlate well to anything (morning grogginess, awakenings, etc.) it absolutely does correlate with my WakeMate score (p = .003, extremely high correlation).   Drinking coffee in the afternoon or evening adds 12 points to my WakeMate score even though it does not affect any other variable.  The odd thing about this is that I would expect it show up somewhere else too (especially time until sleep) but it’s nowhere to be seen.  Only in the WakeMate score itself does the effect appear and it’s incredibly prominent.

Further Study

Although the last thing I wanted to do when I set out to study my sleep was to try and reverse-engineer the WakeMate score algorithm, my results make me really wonder about it.  It’s a terrible predictor of my morning grogginess, but it’s an excellent indicator of my caffeine metabolism.  This obviously merits further study.

I’m also interested in sifting through some of the timeline data (like total time in Deep vs Average sleep, or perhaps narrowing in on events that occur close to when I wake up) to see if I can find any meaningful correlations to my experimental variables.  Since I suspect the WakeMate score is based in large part on timeline-like data rather than absolute number of minutes asleep and so on, I may be able to dig in and find correlations in timeline-type data to my caffeine intake (and possibly some of these other variables as well, especially my morning grogginess).  Unfortunately it’s early days yet with the data export feature and so this level of detail is not yet available for me to study.

---

Want me to build your app / consult for your company / speak at your event? Good news! I'm an iOS developer for hire.

Like this post? Contribute to the coffee fund so I can write more like it.