« Scaling in the 'cloud', a few case scenarios : Memcached & EC2 | Main | Linux disaster recovery - From upgrade, installation, re-installation to salvaging data »

August 25, 2008

What programming language / platform should you choose for your next project ?

Since I try to post on as many programming languages / platforms as possible on this blog, not to mention some of my clients have also asked me, 'What programming language / platform should I use on 'X' project ?' Here are my opinions on the subject, its a long answer, since its got as much to do with the target business, the people you can bring into a team and the abstraction provided by a programming language / platform.

[Entry continues to the left and below ad ]

To a software stakeholder - the one footing the actual bill - it can become dizzying to hear the amount of available programming languages/ platforms to choose from. In the biggest - or even mid-size - corporations, the question of whether to choose among Java, .NET, PHP, Perl, Python, Ruby, Smalltalk or Vendor X, is often not even up for debate.

In such cases, platforms are chosen and handed down from the 'higher-ups' often with minimal technical analysis. It is as they often say, a corporate policy on using a strategic partner, translation = 'we want no monkey business, 100% accountability and it has to be the biggest bang for the buck'.

In the smallest of companies - or 'one man shows' - the technical merits behind a platform or programming language often weigh the most. There is rarely any preconceived notion on what to choose, and since these are small companies, competitive advantage by means of a technical decision is often the easiest to attain.

In such cases, platforms are generally chosen by the ones that have the most 'buzz' in the news claiming 10x more productivity than any 'incumbent', or often on a contrarian basis, choosing a stalwart platform / programming language that's never been used for a particular problem as the secret weapon to gaining a competitive advantage.

However, whether big corporation or 'one man show' there are many variables that need to be taken into account. Inclusively, there is rarely a clear cut recipe for success in choosing one platform or programming language over another, so what should you ask before deciding on going with a particular platform or programming language ?

What's your abstraction tolerance ?

All programming languages and platforms provide abstractions to solve real world problems. Some are better abstractions than others, the lowest of course is binary 1's and 0's - which is to say no abstraction - and I dare say you won't see anyone cranking out 1's and 0's to create an application now a days. But here comes the first important question, what is you abstraction tolerance ?

Using no abstraction at all -- like 1's and 0's -- is inviable for real world applications, and by the same token, a programming language / platform abstraction can be equally unbearable for certain types of applications or situations.

Take the case of LISP, which is often considered a poster child for programming languages. Many professionals and academics consider certain programming languages to be 'partial' LISP's that implement certain features of this latter language, like object-orientation, garbage collection(memory management), macros, etc. But why on earth would someone want to use a 'partial' LISP if LISP is so great to begin with ? The abstraction level provided by LISP is simply too great.

Similarly, platforms/frameworks like Ruby on Rails and Django (based on Python) have gained tremendous success in the web application space because their abstraction level is just right for the problems confronted in this area. Why would anyone want to keep dealing with Java's JSPs, Servlets and hundreds of web frameworks ? Or for that matter, .NET's lack of an integrated MVC framework and shelling out thousands for the only professional tool of the platform Visual Studio? Someone who doesn't like Rails or Django's level of abstraction level in their projects, that's who.

You see, using abstractions comes at cost. LISP may be the greatest programming language ever, and Rails or Django the most productive web frameworks to date, but each one requires finding competent people to think at that particular level of abstraction.

The head count, are you sure you need that much abstraction ?

Here is a real catch-22. Working at higher-levels of abstraction by its very definition should make anyone more productive, and as a consequence lead to faster delivery times with much less people working on a project. So use LISP, Ruby and Django everywhere! right ? Well not so fast.

Even though there is a certain condescending attitude from those using higher level abstractions, such as this little statement taken from Paul Graham's Beating the Averages : By induction, the only programmers in a position to see all the differences in power between the various languages are those who understand the most powerful one....( the one's using a less powerful one) are satisfied with whatever language they happen to use, because it dictates the way they think about programs.

Sticking to such high-levels of abstractions can equally hurt a project, as much as, using 1's and 0's. Ask yourself, can an application be delivered in the specified time-frame just by using a higher level abstraction programming language/ platform ? And even if it can, is it advisable to develop a project depending on people with experience in such a high level of abstraction?

For the smallest of companies or 'one man shops' there is not much downside to this, in fact, many successful start-ups have depended on using high-levels of abstraction to keep their head count low and deliver products faster and cheaper ( though often with consequences as pointed out later).

For other applications, like those undertaken at medium to large corporations -- or inclusively small non-technical businesses -- high-levels of abstraction can create bottlenecks. Bottlenecks that can either come from trying to solve a technical architecture that is simply not manageable by a small team no matter the level of abstraction used, or depending on harder to find and expensive specialists.

Unlike start-ups, which are always hitting for the fences or home-run ball, and use these higher levels of abstractions as a weapon. In most businesses, where there are more than technical careers at stake, using higher levels of abstraction can be like tightrope walking, with the higher levels equaling a potentially deadly or harder fall.

Choosing lower levels of abstraction for a programming language or platform provides safety. Even if such choices often take more time(money), having the capability to easily expand or replace a project's team is always seen as beneficial. Its a reality people move on and change jobs, and having a project held by niche or 1-2 inside experts is generally frowned upon by the 'higher-ups' ( Unless it has to be, which is mentioned next)

This is one reason why application development in programming languages like Java, .NET or PHP will not be superseded - at least in the coming years. There is simply a vaster - less expensive - pool of people that know these platforms. So by all accounts everything should be developed in these last programming languages / platforms ? Well no, I didn't say that either, there is one reason why large and small companies need to make use of higher level abstraction or particular languages.

May be you really need a domain language / platform

A domain language refers to an abstraction choice that fits exactly the problem you are trying to solve. There are programming languages / platforms that are uniquely designed to deal with real world problems, and inclusively, some of these problems are almost impossible to solve were it not for such levels of abstraction. I'll cite a few examples to give you an idea.

Erlang is one programming language that's been getting a fair share of attention lately, given the multi-core processor architectures that are becoming more common in the industry. The attention is warranted, because using multi-core processor architectures entails dealing with many of the issues related to concurrency.

The abstractions for solving concurrency problems in other programming languages / platforms are not as optimal, in fact, Erlang provides a stronger language for concurrency programming because it was designed with these issues in mind.

Another case scenario would be a problem requiring macros, which are abstractions of programs that create programs, or in other words, the capability to generate a logical sequence(program) out of a given input to a program. The problems tackled by airlines and banks often require dealing at such an abstraction level (e.g. Quote for flight from Los Angeles to Johannesburg, the outcome for such results is so complex given the number of airlines, times and stops, that having a pre-generated program to solve this particular query is simply too vast. Generating a logical sequence(program) at each given possibility(input) via macros is often seen as the optimal solution).

This is one reason why big corporations simply can't forgo the use of higher-level abstractions like macros, which are among the primary benefits of using languages like LISP and Smalltalk , the problems they face simply require what is often qualified as a domain language.

Don't choose 'just for sport', choose 'out of need' -- and even then it may not be right

Then there is of course the 'just for sport' or 'out of need' use of a programming language / platform, which is also very common. Here its the case of a language / platform being over-kill for the problem its trying to solve or undershooting the programming language / platform that is actually needed. This problem also afflicts big and small companies.

For small companies this is a double edged sword, especially those with technical founders. Most tend to go down the 'just for sport' path, using the hottest available programming language / 'platform' for everything, creating applications in Erlang or Scala that 90%+ of the time could have been built faster using Java or .NET.

Of course, others choose the hottest available programming language / platform 'out of need' -- like Django or Rails -- since they have little money, want to get the biggest productivity gains offered by the newest platforms, and crank out prototypes into user's hands fast!. However, this can also have consequences once a company grows.

Its very well documented that most start-ups once they become successful or are bought by a bigger company, are forced into using or replacing everything with lower abstraction languages / platforms. Here its a classic 'back to basics' approach where an application is no longer manageable by a small team no matter the level of abstraction. In order to accommodate growth, safer and lower level abstraction programming languages / platforms are brought in ( This is a borderline case though, since most start-ups err on getting things done faster no matter what platform, having technology supplanted once profitable or acquired is a moot point for many in this segment)

Then there is the undershooting problem in bigger corporations, occurring in places where its procedure to get two-three signatures in order to install any software on a PC. Such places are fertile ground for using a one size fits all technical platform based on corporate policy.

Its here that you will tend to see overrun, over-budgeted and overstaffed projects, attempting to create applications in .NET or Java that are simply not adequate for a programming languages / platform's abstractions, and hear technical outsiders gasp 'Shheesh...that was how much $$$ ?!? You could have used X platform...', but so is the fear of using a newer -- often unproven -- abstraction for tackling certain problems in bigger corporations.

As you'll realize, there are no easy answers in choosing your next programming language / platform for your next project. Analyze the problem you are trying to solve thoroughly, then attempt to identify/map certain programming languages / platforms that can solve such a problem more easily. Adjust your selection based on the number of people you can bring into team, and hope your abstraction analysis and selection was correct and stays manageable throughout the entire project.

Posted by Daniel at August 25, 2008 3:13 PM