Integration innovation - a history

Integration between disparate, heterogenous systems is pretty complicated. Like anything else in IT, maturity in this area has progressed through phases. First we had disconnected integration, where one system would leave a message, and another would pick it up. This was non-transactional and batchy, but most importantly data format became a concern, although we didn’t have to worry about things like protocols of transmission, because messages were not transmitted from system to system, and required an intermediate location such as an FTP site, or a filesystem. Ultimately, it was the batchiness of this integration technique that caused the impetus to innovate. The notion that we could transmit data from system to system in ‘real-time’ (or close to, and certainly better than batch) was hugely attractive – but how to do this when all systems are different, use different platforms, programming languages, data constructs and so on?

Web Services rapidly emerged to fill this gap, and I remember writing an academic paper on its advent in the mid-90s, not really that long after the web started to gain public popularity. Web services allowed us to communicate using an emerging open standard, using a platform independent data model (although necessarily basic) over HTTP. Web service the concept is often confused with the benefit that could be afforded it – most people were talking web services when what they really meant was something akin to a forerunner to Service Oriented Architecture (SOA), but I will get to that later. So all our systems communicate with each other using synchronous web service calls – great…. actually not that great! The more systems we have, the more un-reusable work we have to conduct to get them to talk; but the more success we create, the more value we realise and the more systems we need to integrate. This cycle meant that we rapidly ended up with a vast spaghetti architecture communication model, and a greater homegrown integration codebase (with all the operational and maintenance work that goes with that) than anything else. So we go back to basics, with an intermediate location or system to simplify the number of integrations we need to write.

Enterprise Application Integration (EAI) and later the Enterprise Service Bus (ESB) patterns (they are very similar) do just that – allowing us to reduce the number of connections we need to write because each system only need communicate with the ESB, rather than every other system it might need to talk to. The ESB can intelligently route requests, mutate data as it travels, enforce compliance, quality, security and other NFRs, and the ESB at the centre of the overall enterprise architecture, is ideally positioned to keep an eye on everything, for monitoring, metrics and so on. We aren’t even mandated to use a given protocol or data format as the modern ESB is ‘aware’ of lots of them. Sounds great so far? Well, again we are the victim of our own success! With the promise of integration allowing seamless data flow with a given architecture, what about when things change? One system gets replaced by another and we then need to undertake a lot of migration work both on and off the ESB. Also what about partnering? How can other businesses communicate with us, and us with them more effectively and meaningfully?

SOA again, had been around for a little longer, but again the growing business imperative of finding a solution to the above issues fuelled its growth in uptake. The key notion of SOA is that you create a brittle architecture by getting vendor A system talking to vendor B system whether using an ESB or not; and so we integrate business services afforded by IT applications and packages, not the applications and packages themselves. Thus, we integrate CRM and ERP, with vanilla easy to understand interfaces, rather than SuperWhizz CRM with HugeComplicated ERP using complicated and proprietary APIs. This way we can break free of any one-to-one mapping between application and business service – some business services may be realised in a combination of several systems and databases. SOA acts as a proxy to IT estate, making it easy to understand and therefore easier to integrate; and aligning business and IT more effectively.

So what does the future hold? Watch this space...

This is a guest blog post. Views expressed in this post are original thoughts posted by Stuart MacPherson, Pre-Sales Consultant at Imtech ICT Ltd. These views are his own and in no way do they represent the views of the company he works for.