Problems worthy of attack prove their worth by hitting back. —Piet Hein

Friday, 27 April 2018

Pastures new-ish

After nine and a half years, today is my last day at Cloudera. It’s difficult to write those words as so much of my life has been bound up with this company. On the day I started, I didn’t meet my co-workers as I was living several thousand miles away in a barn in Wales. (The others were in a borrowed meeting room in San Mateo.) As I leave I am still in a barn in Wales (different barn though), but a lot has happened in the intervening period.

On the personal side, my family and I lived in San Francisco during the early formative years of Cloudera, a time we will always treasure for the lifelong friendships we made.

On the professional side, it is no exaggeration to say that working at Cloudera has been the highlight of my career. I already knew that Hadoop was pretty special when I joined (I may have been biased as I was writing a book on it), but I had no idea how it would transform the industry and how it would be used in every sector you could imagine.

To all of you I have worked with over the last decade—at Apache, Cloudera and elsewhere, on many projects—I consider myself to be incredibly fortunate to have had the opportunity to work with you. Thank you.

So what’s next for me?

Jim Waldo, who worked on distributed systems at Sun, once said that he alternated six month periods between the lab and the outside world: in the lab he and his team built systems software, and in the outside world he saw how people used the system he was building. Doing so gave him valuable feedback on the system design, even though it was time away from being able to build the system.

In some ways this is another way of framing the explore/exploit tradeoff, where you decide between exploring new technological ground—building a new system—and exploiting that system to solve particular problems you are interested in, which is why you built the system in the first place. (Of course, this framing is oversimplified, since there are many people working on both parts simultaneously. It’s a useful way of thinking about things as an individual actor though.)

For the past few years I have been working on a few open source biology and healthcare projects (like GATK, Hail, and OHDSI). I think that the problems in biology are big enough and messy enough that new systems will need to be built. We can’t stop exploring the technological ground since the sheer amount of data will overwhelm even the best of today’s cutting-edge technology. (I like to cite the paper Big Data: Astronomical or Genomical? here for some concrete numbers.)

Having said that, there is still a lot of mileage left in our current crop of tools—which include Spark, TensorFlow, Jupyter, and the cloud. And this is what I am going to do: continue the work to apply tools like these to more bio projects, only now working as a freelancer. I plan to write more about what I’m up to on this blog, so please follow along.

Cloudera Inbox Zero for the first time ever!

Sunday, 15 April 2018

Type 1 Diabetes

On 16 February this year I was diagnosed with Type 1 Diabetes.

I had been feeling under the weather - a bit weak, but also persistently thirsty and hungry. I would gulp down a large glass of water in one go with ease (a bit like I did when I was 10 years old after running around outside for hours on a hot day), and I would do this several times every day. I would eat a sandwich after dinner, even after having seconds. Strangely, I was losing weight despite eating a lot. And I found it hard to complete my usual morning run, and when I did manage it, it was noticeably slower than normal.

In retrospect these are all classic symptoms of type 1 diabetes: weight loss, increased thirst and hunger. My body was not producing enough insulin, which is needed to use the glucose in my blood. The weight loss occurred because my body was using fat reserves for energy. My heightened thirst was my body’s way of trying to flush the excess glucose from my system by getting me to urinate more.

So I went to see the doctor and she arranged a blood test, which I had the next day. That was at lunchtime on Friday, and the nurse who took my blood said it would be a week or two before the results came back. So I was surprised when the phone rang at 6pm, and the doctor’s receptionist asked me to come in. On Friday evening? Weren’t they closed then?

A few minutes later I went in, and she said that my blood glucose level was over 30, and that normally it would be 7. “Tom, you have diabetes.” I didn’t know what to say. I remember asking what the blood glucose level was measured in (millimoles per litre). (I’m always impressing on my daughters the importance of units in science.)

I also asked how they knew it was Type 1. Mainly from the symptoms - I'm quite skinny! Most people with diabetes have Type 2, which is characterised by insulin resistance: the body is still producing insulin, but it can’t use it as effectively. Less than 10% of diabetes sufferers have Type 1, and while most of those are children, adults can get the disease too.

The doctor sent me to the hospital straightaway so they could check if my body was coping with the high sugar levels. Left untreated, high blood glucose levels can lead to a dangerous condition called ketoacidosis.

I rang Eliane to tell her, and we both had a bit of a wobble. She drove over to take me to the hospital. We went to the Emergency Admissions Unit, where they would check my ketone levels (which, thankfully, were normal) and give me a shot of insulin. After that I could go home - there was no need to stay overnight, but they asked me to go back the next day and Sunday to be given insulin again.

The next day, Monday, I saw the Diabetic Specialist Nurse (DSN) who gave me a blood glucose monitor and insulin pens so I could manage the condition myself. She explained my new routine: before every meal and before bed I have to check my blood glucose level and inject insulin. The bedtime insulin is a slow-acting background insulin that lasts for almost a whole day, whereas the mealtime insulin is fast-acting and is meant to compensate for the blood glucose level rise caused by the carbohydrates in the meal. The idea is that you count the number of carbohydrates in the meal you are about to eat, then calculate the number of units of insulin that are needed to cover it.

It doesn’t sound like much, but I hadn’t really paid much attention to the nutritional composition of meals before. And nor had Eliane. We eat healthily, and mainly cook from scratch, but having to analyse each meal was a big change. In the first few weeks it felt like we were spending all our time analysing recipes. It gets easier, but it’s still time consuming.

The goal with diabetes is to keep the blood glucose level between 4 and 7 mmol/l. A person without diabetes has a pancreas that does this for them. Unfortunately, my pancreas has stopped performing this role, hence the need for injected insulin. There are two things to avoid: hyperglycaemia, which is when the blood glucose level is too high, and hypoglycaemia when it is too low.

Broadly speaking, hyperglycaemia has long term effects (such as cardiovascular problems), while hypoglycaemia needs to be treated immediately since its more severe form can require hospitalisation. Normally though, treating a mild “hypo” involves eating something with very fast acting sugar in it (like glucose tablets) and waiting 15 minutes for the level to get back in range. During this time you likely feel weak and shaky.

This chart shows all my blood glucose readings. In the first couple of weeks all my readings were out of range, but then they started stabilising, and now they are mainly in range.

When someone is newly diagnosed with diabetes in the UK, the full support network of the NHS swings into action. In addition to my GP, I have a diabetes consultant, two DSNs, and a dietitian. I saw all of these people in the first week, and I have ongoing support from the DSNs and the dietitian, who I can phone or email if I have a question, or need some help with my insulin dose adjustment. I have a recurring appointment with the consultant every six months. I’ve also been added to the system for annual screening for eye disease. The NHS also provides education programs for carb counting and insulin dose adjustment (one has the fabulous acronym DAFYDD - dose adjustment for your daily diet), as well as an excellent series of videos for newly diagnosed patients. All of this is free. And in Wales, where I live, there are no prescription charges for anyone (for people with diabetes in England, prescriptions are free too), so I don’t have to pay for the medical supplies that I now depend on every day. 

All of the medical staff that Eliane and I have encountered in the last two months have been unfailingly kind and supportive, even when working under pressure (like the first night in the EAU). It’s incredibly reassuring to have access to the resources the NHS provides. I would like to thank everyone there, along with my family, friends, and colleagues from work who have helped me get through the last two months. 

Wednesday, 22 June 2016

Be Part of Something Bigger - Vote #Remain

David Cameron did not have to call this referendum. He did so in an attempt to settle the European issue within his own party, as there is no new EU treaty that we are voting on. The referendum is not binding either, as the legal blogger David Allen Green pointed out.

Far from quelling the debate within the Tory party, the lead up to the referendum has had the opposite effect. The debate over the last couple of months has been increasingly toxic, with both sides making outlandish claims. Parts of the Leave campaign have been xenophobic and racist, in an attempt to scare people to leaving the EU - this is the true Project Fear. And then last week the appalling murder of the Labour MP Jo Cox brought about some reflection on how we’ve moved away from a more respectful, kinder politics. In the words of Stephen Kinnock, "When insecurity, fear and anger are used to light a fuse, an explosion is inevitable.”

But there is a referendum tomorrow, so we have a duty to vote. The vote is about a host of issues, and on all of them I believe we are better staying as a member of the EU. In my mind it boils down to being a part of something bigger than yourself. This is true on a personal level - being part of a company, a team, a club, or an organisation allows you to achieve more than if you go it alone. I’ve seen this in my professional life where loose-knit groups of programmers build open source software that an individual could never dream of. Of course, where there are many personalities pulling in different directions you get conflict, things get messy, compromises are needed, and you don’t always get your own way. But on some decisions you do have influence, and you do get to shape the results.

Being a part of the EU is about the UK being a part of something bigger, and being able to influence policy on issues that affect the UK. The world is a messy chaotic place, and there are many deeply-ingrained, complex problems that require complex policy interventions. Climate change, migration, tax havens, peace - to name a few - all of these need a coordinated approach that cross national borders. Leaving would squander our influence in attacking these problems, while doing nothing to solve them - for us or for the the rest of the world.

One of the more worrying themes of the Leave campaign is not to trust the experts. This allows them to conveniently dismiss the overwhelming opinion amongst economists that Brexit would mean the UK is worse off outside the EU. It’s like climate change denial, and running a country with that kind of gut-feeling policy making is terrifying.

Britain has been at its best when it has been an outward-looking nation, one that works with others and trades with others. That’s why I am going to vote to Remain in the EU.

Sunday, 5 July 2015

The Earth Moon Game

If the Moon were the size of a tennis ball then the Earth would be the size of a basketball. How far apart should the balls be placed so that the distance is to scale?

Before you read on, you might like to have a go yourself. If you don't have a tennis ball and basketball to hand, you can play with this online version I wrote.

My kids and I had a stall at our school fair this Friday where we played this as a game:

The Earth's diameter is 12,742 km and the Moon's is 3,475 km, so the Earth's diameter is about 3.7 larger. (We measured the basketball's diameter to be 23.5 cm, and the tennis ball to be 6.5 cm, so the ratio is about 3.6, which is pretty close!)

The Moon is (on average) 384,400 km from the Earth (the Lunar distance, measured from the centres of the two bodies), which is 111 times the Moon's diameter. Scaling this to the tennis ball, we get a distance of 111 × 6.5 cm = 7.2 metres.

Here's a picture showing the results at the end of the fair:

The basketball representing the Earth is in the bottom of the picture, and the tennis ball just visible at the top is the Moon, 7.2 metres away. To the right of the green tape are white flags that are the players' guesses for where the Moon would be.

It's striking that all the guesses were too low. This seems to be a mixture of two things. Firstly, people really do think that the Moon is closer than it actually is. Secondly, people tend to copy other people, so they would place their flags close to where the others were. (We told everyone that the Moon didn't have to be restricted to the green tape - that just happened to be how long it was.)

We saw a few interesting tactics though. One girl put one flag so it was the closest to the Earth compared to all the other flags, then another so it was the furthest out. She seemed to think that everyone else had either over- or underestimated the distance - which of course they had! (She didn't win though, as someone put their flag even further out later on.) Someone else put five flags over a range of about 25cm where she thought the Moon would be.

The most successful approach seemed to be for the player to stand where the Earth is, and have someone walk away holding the tennis ball until it subtends the same angle as the Moon does in the sky (or your mind's eye). This is easier said than done, however. The player in fourth place (who was about five years old) used this technique.

Here's the data plotted graphically, with each flag shown as a line. The blue line represents Earth, and the orange line the Moon.
Interestingly, the guesses did not benefit from the Wisdom of Crowds effect, where the average tends to be a good predictor of the actual answer:
The opening anecdote [of the book of the same name by James Surowiecki] relates Francis Galton's surprise that the crowd at a county fair accurately guessed the weight of an ox when their individual guesses were averaged
For the Earth Moon Game, however, the median distance was 2.6 metres, and the mean was 2.7 metres, which was 2.3 standard deviations (sd=1.96 metres) from the true distance, 7.2 metres.

Sunday, 19 April 2015

The Hay Dark Skies Festival, Reverend Thomas William Webb, and Jupiter

In 2013, the Brecon Beacons was designated a Dark Sky Reserve, and a year later the first Dark Skies Festival was held in Hay-on-Wye. The second festival took place this weekend, and my family went along to some of the activities.

Young stargazers, Lottie and Millie
In the morning, we found ourselves in a planetarium tent, then we looked at sunspots, and held pieces of meteorite.

The evening event was stargazing at Holy Trinity Church in Hardwicke, just outside Hay. Quite apart from the lack of light pollution, the location was a special one, since the vicar of the parish from 1856 until 1885 was Reverend Thomas William Webb, who in his spare time observed the night sky with telescopes and an observatory he had built himself.
Holy Trinity Church, Hardwicke

In 1859, while at Hardwicke he wrote the classic book, Celestial Objects for the Common Telescope, the object of which was "to furnish the possessors of ordinary telescopes with plain directions for their use, and a list of objects for their advantageous employment".

The book remained in print well into the following century (and was recently republished by Cambridge University Press), and it's probably difficult to overemphasise the importance of this book in encouraging generation after generation of amateur stargazers.

In the words of Janet and Mark Robinson, who used to live in the vicarage and have edited a book about Webb,
Like Patrick Moore, he was an enthusiast who wanted to inspire as many people as possible to look through a telescope. Even at the choir party he "arranged the telescope and acted as showman and all in turn had a look at Saturn".
Webb would no doubt have been pleased to see yesterday's gathering of enthusiastic amateurs (including the Robinsons) with an impressive range of telescopes, on a cold but very clear night. The highlight for us was seeing Jupiter and its four brightest moons (Io, Europa, Ganymede and Callisto) through a large reflecting telescope. We could even see the north and south belts, and the Great Red Spot (or Pink Splodge as Lottie named it).

Sunset. Venus is visible top centre
Thank you to the organisers of the Hay Dark Skies Festival, and the volunteers from the Usk Astronomical Society (the oldest astronomical society in the UK), the Abergavenny Astronomy Society and the Heads of the Valleys Astronomical Society.