Memory Leak in node.js

tl;dr

Never, ever forget that node.js is single threaded and expects every callback to return to the event loop. Otherwise it will never be garbage collected.

Background

This behavior was experienced in the Softwerkskammer platform. In order to show small avatars next to member’s names we load them from gravatar or take an uploaded pic. These are then binhexed and inlined in the page’s content. We started doing this every time the memberlist is requested. This is very slow and additionally causes unnecessary load to gravatar’s site. We then introduced a local cache using node-cache. I wanted to get rid of this caching and persist the gravatar along with other profile data in our DB.

Of course, we do not know, if a gravatar has been updated. Therefor we need to regularly check gravatar’s services.

Code

We had code like that (no caching anymore):

function imageDataFromGravatar (url, callback) {
  request.get(url, function (error, response, body) {
  if (error) { return callback({image: null}); }

  var image = 'data:' + response.headers['content-type'] + ';base64,' + new Buffer(body).toString('base64');

  callback({image: image});
 });
}

module.exports = {
  getImage: function (member, callback) {
    var url = this.avatarUrl(member.email(), 16);
    imageDataFromGravatar(url, function (data) {
      member.setAvatarData(data);
      callback(data);
    });
  }
};

The function “getImage” was called like that (for each member in a loop):

if (!member.getPersistedAvatarData()) {
  avatarProvider.getImage(member, function (imageData) {
    member.persistAvatarData(imageData);
    store.saveMember(member, callback);
  });
} else {
  if (/*actuality check*/) {
    avatarProvider.getImage(member, function (imageData) {
      var oldAvatar = member.getPersistedAvatarData();
      member.persistAvatarData(imageData);
      if (member.getPersistedAvatarData() !== oldAvatar) {
        store.saveMember(member, function () { /* background op */ });
      }
    });
  }
  callback();
}

Do you spot the horror?

        store.saveMember(member, function () { /* background op */ });

I really thought, if I don’t give the “saveMember” function a callback method (which is expected by that method, it eventually calls a mongodb function that expects a callback), it just performs its stuff in the background. – *haha* background, *haha* background, *haha* background – today I am laughing at that. A few weeks ago I have been completely naiv.

So what did I do?

quite simple, changing the last few lines to:

if (/*actuality check*/) {
  avatarProvider.getImage(member, function (imageData) {
    member.setAvatarData(imageData);
    return store.saveMember(member, callback); // never, ever "fork" stuff in node by not having return values *I AM IDIOT*
  });
} else {
  callback();
}

(please note the comment after the important line)

That’s it. I hope it helps you to not do the same mistakes.

P.S.: Finding this took me multiple days, spending lots of hours of effort.

Taking the Bus – a Metaphor for Teamwork

Teamwork and Management

I want to introduce a hopefully helpful metaphor to show the differences between working in a team and trying to control the team from the outside.

Let’s take the bus

Imagine somebody invites you to take a trip with them. They own a bus and want to travel a rather long distance to somewhere in the south. They have a vision and a rough plan. But they do not know whether or where there will be traffic jams, blocked roads or bad weather.

This is how it might feel working in a team.

Sometimes people leave the bus; sometimes new people enter, sometimes people return. From time to time the driver will pass the steering wheel over to another driver. Every now and then you have to repair some parts of the bus.

…feel free to imagine…

I want to know when the bus will arrive

Now imagine local transport: There are strict plans for stops, routes and times. This expands even to long distance buses. These plans are great if you want to take a trip within well-known environments; yet even there delays may happen…

This is the manager’s view. He looks at the bus from the outside and loses contact immediately after departure. The only information of the travel is the plan or second hand information (telephone, gps …)

By this outside view from a controller’s cockpit you get different information. Maybe you learn about a blocked road and you can inform the bus driver to change the route long before she approaches the blockage. You can also see how far it is from the expected destination if you have suitable information.

So why should I care about that?

Often we find ourselves to have rather hostile attitudes towards the other group. No matter if you personally feel more like a member of the team or rather the manager. Neither of these groups is superior to the other. Each has its own strengths and weaknesses.

Reflect – think about it

It’s up to you and your mates to generate a massive impact of this. The impact can be really positively pushing or massively destructive.

If you are aware of your position’s strength and weakness, help the other side grow awareness, too.

 

Bye, I’ll catch the bus now…

Do you take it? – i.T.A.K.E.

I haven’t written an article for a long time. It’s time to come back.

During the last two days the i.T.A.K.E. unconference took place in Bucharest. I had the honor to be one of the presenters there. But I don’t want to speak about what I presented. Instead I want to thank the organizers and the visiting crowd that they created such an awesome event.

Atmosphere

I felt really welcome and taken care of. I want to explicitly mention the organizers of Mozaic Works. They are a family that is open minded and open hearted and they let us be part of their family. The location was pleasant and – apart from minor problems (audio and video) – it went very smooth.

People

You people have been a great mix of experienced and fresh faces. Everyone has been very open and curious. I had many interesting and inspiring conversations about coding and team dynamics. It especially is worth going out with you in the evening.

Contents

In talks, I learned about a frontend developer’s perspective of the code (thanks @gion_13), found an easy way to explain “ports and adapters” (thanks @johan_alps and @sanlaville) and much more. But what really made my week was the keynote of @felienne. I usually do not like to listen to academics talking about software development. She changed my opinion.

Bottom Line

I am sure the event will happen again next year. It is definitely a “Not-To-Miss-Conference” if you are passionate about software craftsmanship and giving and taking.

Currently I am still sitting in my room in the conference hotel. A warm feeling is creeping up my belly when I think back of the last hours. Thank you Bucharest!

 

How I Made My Dream Come True

It’s been a few weeks ago. It has been a highlight in my life: SoCraTes 2012!

It started out as an idea around 2 years ago. I had been visiting quite a lot of events here in Germany: conferences focussing on programming, on agile coaching. – They have been OK. Still, something was missing:

Passion for “Good Programming”

The events were either focussing on technology or on processes. Truly, both are important. Yet, you do not need a fancy process nor do you need the latest technology to create great software. What you definitely need is great developers. People who know their craft. A craft that consists of much more than a programming language or frameworks or mastering the skills. It requires a sharp mind, curiosity, passion as well as communication skills, discipline, creativity.

The dream

Let’s organize an un-conference similar to the Agile Coach Camps focusing on Software Craftsmanship.

It started during XP-days 2010 as an open space session. There we founded the initial organizing group. The result was SoCraTes 2011 with about 50 attendees. During the event some of the attendees decided to continue the experience of learning and sharing by creating user groups all over Germany. That has been the ignition of Softwerkskammer. Currently it consists of more than 350 members and 9 regional groups.

This year we followed up with SoCraTes 2012. The event was sold out in less than two weeks and was really satisfying to me and – at least some folks told me – to others, too. I cannot imagine a life without events like that anymore. It has been energizing to see so many great and passionate developers sharing their knowledge and experience and setting a great stage for themselves.

Every End is a New Beginning

Shortly before SoCraTes 2012 I started to think about withdrawing my engagement for the future. The reason is simple: I always wanted the event to be a community event, not my personal thing.

I am really happy to have found volunteers for next year’s SoCraTes and I have no doubt that it will be great again. Nonetheless I am curios about how it will turn out to be.

Last not least I want to thank everybody for his/her support, especially Nicole who invested lots of her time and always took the load off me. It wouldn’t have been possible without her loving support.