About Andreas Leidig

I am a software developer and respectfully irreverent.

Memory Leak in node.js

tl;dr

Never, ever forget that node.js is single threaded and expects every callback to return to the event loop. Otherwise it will never be garbage collected.

Background

This behavior was experienced in the Softwerkskammer platform. In order to show small avatars next to member’s names we load them from gravatar or take an uploaded pic. These are then binhexed and inlined in the page’s content. We started doing this every time the memberlist is requested. This is very slow and additionally causes unnecessary load to gravatar’s site. We then introduced a local cache using node-cache. I wanted to get rid of this caching and persist the gravatar along with other profile data in our DB.

Of course, we do not know, if a gravatar has been updated. Therefor we need to regularly check gravatar’s services.

Code

We had code like that (no caching anymore):

function imageDataFromGravatar (url, callback) {
  request.get(url, function (error, response, body) {
  if (error) { return callback({image: null}); }

  var image = 'data:' + response.headers['content-type'] + ';base64,' + new Buffer(body).toString('base64');

  callback({image: image});
 });
}

module.exports = {
  getImage: function (member, callback) {
    var url = this.avatarUrl(member.email(), 16);
    imageDataFromGravatar(url, function (data) {
      member.setAvatarData(data);
      callback(data);
    });
  }
};

The function “getImage” was called like that (for each member in a loop):

if (!member.getPersistedAvatarData()) {
  avatarProvider.getImage(member, function (imageData) {
    member.persistAvatarData(imageData);
    store.saveMember(member, callback);
  });
} else {
  if (/*actuality check*/) {
    avatarProvider.getImage(member, function (imageData) {
      var oldAvatar = member.getPersistedAvatarData();
      member.persistAvatarData(imageData);
      if (member.getPersistedAvatarData() !== oldAvatar) {
        store.saveMember(member, function () { /* background op */ });
      }
    });
  }
  callback();
}

Do you spot the horror?

        store.saveMember(member, function () { /* background op */ });

I really thought, if I don’t give the “saveMember” function a callback method (which is expected by that method, it eventually calls a mongodb function that expects a callback), it just performs its stuff in the background. – *haha* background, *haha* background, *haha* background – today I am laughing at that. A few weeks ago I have been completely naiv.

So what did I do?

quite simple, changing the last few lines to:

if (/*actuality check*/) {
  avatarProvider.getImage(member, function (imageData) {
    member.setAvatarData(imageData);
    return store.saveMember(member, callback); // never, ever "fork" stuff in node by not having return values *I AM IDIOT*
  });
} else {
  callback();
}

(please note the comment after the important line)

That’s it. I hope it helps you to not do the same mistakes.

P.S.: Finding this took me multiple days, spending lots of hours of effort.

Taking the Bus – a Metaphor for Teamwork

Teamwork and Management

I want to introduce a hopefully helpful metaphor to show the differences between working in a team and trying to control the team from the outside.

Let’s take the bus

Imagine somebody invites you to take a trip with them. They own a bus and want to travel a rather long distance to somewhere in the south. They have a vision and a rough plan. But they do not know whether or where there will be traffic jams, blocked roads or bad weather.

This is how it might feel working in a team.

Sometimes people leave the bus; sometimes new people enter, sometimes people return. From time to time the driver will pass the steering wheel over to another driver. Every now and then you have to repair some parts of the bus.

…feel free to imagine…

I want to know when the bus will arrive

Now imagine local transport: There are strict plans for stops, routes and times. This expands even to long distance buses. These plans are great if you want to take a trip within well-known environments; yet even there delays may happen…

This is the manager’s view. He looks at the bus from the outside and loses contact immediately after departure. The only information of the travel is the plan or second hand information (telephone, gps …)

By this outside view from a controller’s cockpit you get different information. Maybe you learn about a blocked road and you can inform the bus driver to change the route long before she approaches the blockage. You can also see how far it is from the expected destination if you have suitable information.

So why should I care about that?

Often we find ourselves to have rather hostile attitudes towards the other group. No matter if you personally feel more like a member of the team or rather the manager. Neither of these groups is superior to the other. Each has its own strengths and weaknesses.

Reflect – think about it

It’s up to you and your mates to generate a massive impact of this. The impact can be really positively pushing or massively destructive.

If you are aware of your position’s strength and weakness, help the other side grow awareness, too.

 

Bye, I’ll catch the bus now…

Do you take it? – i.T.A.K.E.

I haven’t written an article for a long time. It’s time to come back.

During the last two days the i.T.A.K.E. unconference took place in Bucharest. I had the honor to be one of the presenters there. But I don’t want to speak about what I presented. Instead I want to thank the organizers and the visiting crowd that they created such an awesome event.

Atmosphere

I felt really welcome and taken care of. I want to explicitly mention the organizers of Mozaic Works. They are a family that is open minded and open hearted and they let us be part of their family. The location was pleasant and – apart from minor problems (audio and video) – it went very smooth.

People

You people have been a great mix of experienced and fresh faces. Everyone has been very open and curious. I had many interesting and inspiring conversations about coding and team dynamics. It especially is worth going out with you in the evening.

Contents

In talks, I learned about a frontend developer’s perspective of the code (thanks @gion_13), found an easy way to explain “ports and adapters” (thanks @johan_alps and @sanlaville) and much more. But what really made my week was the keynote of @felienne. I usually do not like to listen to academics talking about software development. She changed my opinion.

Bottom Line

I am sure the event will happen again next year. It is definitely a “Not-To-Miss-Conference” if you are passionate about software craftsmanship and giving and taking.

Currently I am still sitting in my room in the conference hotel. A warm feeling is creeping up my belly when I think back of the last hours. Thank you Bucharest!

 

How I Made My Dream Come True

It’s been a few weeks ago. It has been a highlight in my life: SoCraTes 2012!

It started out as an idea around 2 years ago. I had been visiting quite a lot of events here in Germany: conferences focussing on programming, on agile coaching. – They have been OK. Still, something was missing:

Passion for “Good Programming”

The events were either focussing on technology or on processes. Truly, both are important. Yet, you do not need a fancy process nor do you need the latest technology to create great software. What you definitely need is great developers. People who know their craft. A craft that consists of much more than a programming language or frameworks or mastering the skills. It requires a sharp mind, curiosity, passion as well as communication skills, discipline, creativity.

The dream

Let’s organize an un-conference similar to the Agile Coach Camps focusing on Software Craftsmanship.

It started during XP-days 2010 as an open space session. There we founded the initial organizing group. The result was SoCraTes 2011 with about 50 attendees. During the event some of the attendees decided to continue the experience of learning and sharing by creating user groups all over Germany. That has been the ignition of Softwerkskammer. Currently it consists of more than 350 members and 9 regional groups.

This year we followed up with SoCraTes 2012. The event was sold out in less than two weeks and was really satisfying to me and – at least some folks told me – to others, too. I cannot imagine a life without events like that anymore. It has been energizing to see so many great and passionate developers sharing their knowledge and experience and setting a great stage for themselves.

Every End is a New Beginning

Shortly before SoCraTes 2012 I started to think about withdrawing my engagement for the future. The reason is simple: I always wanted the event to be a community event, not my personal thing.

I am really happy to have found volunteers for next year’s SoCraTes and I have no doubt that it will be great again. Nonetheless I am curios about how it will turn out to be.

Last not least I want to thank everybody for his/her support, especially Nicole who invested lots of her time and always took the load off me. It wouldn’t have been possible without her loving support.

If it’s Crap, Call it Crap – not Legacy

Tonight we were discussing code quality issues. During this discussion one thing came up to my mind:

Everybody is calling old code “legacy”, and now legacy is used as a synonym for code of poor quality.

Why that?

My first guess is: If it was me who wrote the code, “legacy” sounds much nicer than saying “this code is crap”. “Crap” sounds harsh, brutal, nasty.

But there’s more to it. Many developers have to deal with existing code on a daily basis. Much code. Code that is running at customers’ sites. This code cannot be all crap. People use it, so it must be somehow valuable to them. We even earn money with selling it to new customers. – But it slowly kills us. It kills our ability to change it to modernize it.

Why bother?

Most of us are aware of the fact that today’s produced code will be tomorrow’s legacy. If we allow the use of the word “legacy” as a synonym for “crap” then we have a perfect excuse for producing crappy code. – We should care more of tomorrow’s legacy. We should be proud of yesterday’s work. Today, tomorrow, next year, in five years!

For example: We live in a legacy house, more than a hundred years old. And we are proud of it. We are proud of the people who have built it. Of course there are necessary refurbishments and modernizations, but performing them is usually possible with quite low effort. – I drive a car that is 20 years old. And I love it. – Both the house and the car are legacy, but not crap.

I have a Russian friend that possessed a 7 year old Lada. It was slightly legacy but complete crap. We once bought a brand new replacement part (a rear axle). It was cheap, but crap from the start. Of course the car was running most of the time, but we often had the feeling that we were lucky to reach our destination.

Examples

Let’s imagine a huge class hierarchy. Nowadays we know that in most cases huge hierarchies are not a good idea if you want to write maintainable code. Some years ago this knowledge was not so wide-spread. But the now so big hierarchy started small, believe it or not. So if you start building a hierarchy today you are potentially laying the foundation for tomorrow’s hell. And you should consider yourself a crap-maker, because you should know better than – let’s say – 10 years ago. The knowledge that deep hierarchies are problematic and should be avoided is now a “best practice” and has been around for at least five years. This means that if you are now dealing with a deep hierarchy that is less than five years old, you can frankly call that crap.

If you even think of extending this hierarchy with more depth, that is also crap. No excuses. If a structure you have to deal with already is too complicated, adding more is not a way of producing legacy, it is just a horrible practice and it always has been. As soon as you sense the code you are working on is getting too complex, stop. Refactor or refurbish it. It will never get easier in the future. The more you add, the more difficult it will be.

Remember

Think – don’t just code!

Legacy Software – Why I love working with it!

For years I worked as a consultant and coach. Both on agile methods and coding. I guess I was of some help on a few occasions. At least that is what some customers stated. But in the end I always left them alone. I often felt relief when I left them. But on the other hand it was also like surrender. The software was not finished. Most software is never finished; development goes on and on and on and …

This is probably nothing new to you. Maybe you have similar experiences. But probably you belong to the majority of people in software development: employed developers.

Buried Under Tons of Code

Old code. Untested. Unreadable. Unloved.

Is this what you dreamed of when you started your career? – Probably not. You may have dreamt of creating new shiny software from scratch using great technologies and fancy tools.

But you don’t quit your job. Perhaps you are a masochist? – I don’t think so. There are many reasons to face the challenge working in such an awkward place:

  • Responsibility: You feel responsible for the product. Although it is not the “Big Fun”, it is your job to keep the thing alive and even add more features to it.
  • Challenge: You think it is much harder to work in a situation like yours than to do small projects from scratch. You know that every software will become legacy after a certain age or size.
  • Fear: You are afraid of being the one to produce the nightmare from scratch that you are now working on. For you it is easier to say: “I just clean up other people’s mess!”
  • Hope: You are just staying because you have hope that you will be amongst the chosen few to start the next product from scratch.

In my case, it is a mixture of the first two aspects. In addition I have a strong belief that every legacy product can be changed to the better so that further development will be possible for a long period of time. This work is not funny but deeply satisfying. In our current product everyone in the team is trying his best while working on the huge codebase. And you know what: The software and the code become better each day.

We are all learning. We are all improving ourselves. Maybe some of us will become masters in the end. Who knows…

No IDE: A New Experience

Last week, we were attending a small conference that also included some one-day workshops. I chose to dig a little bit into Node.js. The workshop was held by Golo Roden.

I must admit I’ve had prejudices against JavaScript. And so I chose this topic to gain new insights. Now I know a lot more about JavaScript, have more questions and I really wonder:

Is it possible to conduct large complicated projects without an IDE?

I started developing software more than a decade ago with Smalltalk. For those of you who do not know Smalltalk: There is no separation of the IDE and the product. You don’t even work with files. Have a look yourself: http://www.pharo-project.org/home. – The non-existence of files in the Smalltalk IDE will give you an impression of how I feel with files. About ten years ago I had my first encounter with a different language based on files: Java. – Wow, that was strange at first.

To you this may seem weird. Programming without files? – To me it has been the other way around.

And last Thursday I’ve been attending my first workshop without even an IDE. Only an editor and a command line. – And it worked well for me. At least I was able to type in all the code (or should I say text?) Golo showed on the screen and made only few typos. I was really happy to have a quite sophisticated editor capable of JavaScript: syntax highlighting, able to show brace coherence.

But: I really wonder whether this will scale.

  • How will this work with more than ten files?
  • How will you do refactorings?
  • How can you see relations?

To me one aspect – if not the most important one – of programming is to be able to navigate my code. I definitely do want to see the implementation of a method or function I call. I want to be able to find the potential callers of a function. All this typing is so distracting from the product I’m about to build.

  • I have to know what is already defined to be able to use it.
  • It’s a pain (and slow) to look it up by reading potentially concerned files or documentation.
  • I have to remember the filenames where stuff is defined.
  • I have to track everything inside my head – in detail!

Is this really a professional working style? – What do you think?

 Instead I’d like to be able
  • to see an overview of my code
  • to see the big picture

This is even hard in today’s IDEs. But closer than just an editor.

P.S.: Some googling pointed me to obviously working IDEs for JavaScript with more than syntax highlighting.