Service Oriented Architectures: The Real Deal

Service oriented architectures, or SOAs, are just another in a long line of software systems engineering means of dealing with complexity. In software, the trail of abstraction goes from basic bits through functions, objects, modules, components, until today where we are trying to abstract entire systems. The history of this increasing abstraction is fascinating and perhaps worth talking about, but I won't here, for now. I want to talk about where we are today and what I believe has really happened with our notion of abstraction and how that has affected entire industries (okay, so that sounds grand and sweeping, but I believe it's not far off the mark).

Service orientation is the view that things provided by software and hardware can be made available as a "service" to be consumed by those who need whatever capability the service offers. Services are not necessarily applications but can be as simple as some function that is made available for other uses beyond the original intended use. We have identified categories of services that include basic or simple services (think of a function that simply looks up and returns an address) and composite services. A composite service can be thought of as occurring in 2 flavors (a bit of an oversimplification); orchestrated and otherwise. Orchestrated services are services that require the use of multiple other services in some sequence that must be adhered to. The term orchestration indicates that there is some entity outside of the service that directs the sequencing and ensures that the service completes successfully or not. Unorchestrated composite services are services requiring use of other services but no special sequencing is required. An example of an orchestrated service might be the end-to-end process of acquiring a loan. An example of an unorchestrated service might be the delivery of a complete customer profile (used in making a loan determination). This is just a brief highlight of some service basics. I mention this because it's important to understand how I think about services for the rest of this to make any sense.

The notion of services seems to have come about with the increasing complexity of IT systems which we knew about but weren't really overly concerned with until...businesses started having to deal with systems other than their own. This naturally occurs when there is a significant amount of merger and acquisition activity and when a need is identified for something that already exists but is not readily available. Merger and acquisitions create businesses with multiple IT systems that are not in general compatible. Leveraging investment in these systems has taken the form of identifying ways to use the system capabilities without a significant expenditure on rework. If I can take existing capabilities and make them available without completely rewriting the original software that implements that capability, then I have used resources to make that capability available, but it's less and different resources than I would have used if I had to start from scratch.

This is nothing new, it's just that we have started thinking about how to really deal with this complexity in a way that doesn't involve enormous outlays of time and other resources. And yet, it's still hard and many organizations that undertake to "implement" a SOA, seem to fail or give up. Why? Much has been written about the why but I want to state my observations about the "why." So what follows are my thoughts on SOA success or failure.

1. SOA is not a product! There are companies that sell SOA technologies but these are not SOA; they are enablers of SOA, perhaps, but the market hype around them leads you to believe they do everything. NOT SO!
2. SOA isn't for everyone. If you have several incompatible legacy systems, applications, and organizations offering different types of capabilities and you need to make them appear as a somewhat unified whole, then you should probably consider taking a SOA approach. Otherwise, it remains to be seen that the investment will be worth the ensuing chaos.
3. SOA is not just a technical solution. This is the biggie from my experience. Technically, there are some hard things to do when implementing a service oriented approach. However, they are not things that are entirely new. In fact, many of them have been done many times over in different ways. The real challenge in SOA is that it will nearly always incur a significant investment in social change within the participating organizations. This social change cannot be mandated from the senior executive level nor can it be entirely effected by a grassroots effort from the lower levels. The change must occur as a natural change to the way the entire organization thinks about itself and in the way the various parts interact with each other. This is significant! Traditional IT organizations may well see this as a threat to their very existence. Middle management may see it as a loss of control or power and senior level management may believe that it will be possible by just saying it will be so. The end result is that the effort will fail because expectations and understanding follow all of the things about human nature that make anything involving more than 1 human, difficult. This failure is important to understand because at all levels of involvement, SOA will be perceived as not useful since we tried and it didn't work. This is something that I want to explore in other posts, even though it's not overly new or innovative. It's still interesting.
4. SOA isn't the end. We are still abstracting and thinking about how to reduce overall costs in using software and the hardware that makes software possible. The most current trend is to go beyond software services and consider everything as a service. Thus, we have Software As A Service (SaaS), Platform As A Service (PaaS), and Infrastructure As A Service (IaaS). These are loosely grouped together and termed the "cloud." In each of these, the level of abstraction looks at software, hardware, and support as something that is offerable (not a great word) as a service. This means that I no longer have to consider spending small budgets on equipment and infrastructure that I can now outsource. Beware, though. This approach is not to be taken lightly and serious organizational consideration should be put into understanding the tradeoffs and risk with taking this approach.

None of this is groundbreaking and as I mentioned, there are copious quantities of publications on the subject. What I believe, though, is that SOA is the first in a line of abstractions where we as engineers cannot be oblivious to the "people" issues that surround the technical work. I also believe there are integrity and ethical issues that surround this work that we as engineers must step up to and not just peddle a technical solution because it's sexy technology. Pitching SOA as the answer to all problems is something at which marketing has excelled. As engineers, we haven't stepped up to answer this blatant nonsense with anything other than just sword rattling. We need to do a better job of answering incorrect marketing claims and letting those without the knowledge drive the industry. This means we have to educate those making the claims, we have to speak out loudly as a cohesive group when these claims are made, we need to better educate future engineers to taking a contrary stand when necessary, and we need to make the average engineer more articulate.

Feel free to disagree. These are my opinions and observations from having worked in this area for the last 10 years and having seen the progression since I started writing software in 1976.

Is it Engineering or not?

I find it remarkable that we still can call the development of software, "software engineering" given how it's usually done in most organizations. What I've seen is that there's very little planning and what planning is made is focused on details that are miss the larger picture. To use an overused analogy: if we were building a bridge like we built software, we'd have very detailed plans for how to construct suspension cables but we're really building a cantilevered bridge. To top it off, instead of actually buying the cables, we're building them from scratch including mining the ore that supplies the metal!

Agile methods are even worse (in my opinion). Agile methods are touted as a great tool in the face of rapidly changing requirements. Probably, but they're rarely used that way. Instead they seem to supply a good excuse to management about why we don't need to do pesky documentations, requirements, etc. And worse, management actually seems to believe it. Agile methods are more like a couple of kids in the backyard who decide that they need a treehouse, so they start collecting scraps of wood and start nailing them to the tree in hopes that something vaguely treehouse-like will result.

If requirements are changing rapidly, perhaps the best engineering approach would be to NOT do anything until some of the larger fluctuations have died down. Requirements churn is usually a sign that the concept is either ill-conceived or less than half-baked. I understand the necessity of getting to market fast (although I have yet to understand why everybody's in such a rush) but getting there with crap is not the way to do it.

Some of the problem I think lies in how we educate people for careers in software development. We teach them all kinds of great theory (I love theory) and all kinds of great languages (Java, Ruby, what's next) and give them only 1 or 2 courses in how to actually conduct software development efforts. Certainly the ethics of how to responsibly develop software are rarely discussed.

Responsibility for change in this area lie with each individual developer, each development organization, each company hosting a development organization, and every consumer of software. Individual developers need to stop whining about how documentation is useless, testing doesn't make sense, and we just need to get it out the door. Organizations need to stop supporting sloppy development practice, shoddy products, and developers with the personalities of slugs. Consumers need to reconsider whether continuing to buy and thus support shoddy products is worth the money or whether it's better to just not use it and send a clear message to the developers. We did that in this country with domestic cars in the 70's and 80's when imports were clearly of better quality. When are we going to do it with software (not that imports are better in this case)???

An Odd Thing About Mozart

I've listened to classical music most of my life and a considerable amount of that has been Mozart.  For years I've been able to listen to Mozart and discern when he's being dark and when he's being more playful or lighthearted.  Yesterday, however, is the first time I've listened to a choral work of his and heard...sarcasm??? Mockery???

This was very surprising to me and made me wonder if my interpretation of the music I hear is more in tune with my mood or if the music tunes my mood.  Sarcasm is not something I've ever really heard in choral works.  Specifically, the Kyrie and Laudamus Te portions were almost mocking in their tone and movement.  I don't know if it was intended and if so who was being mocked: the church, god, his sponsor?  It was almost as if Mozart was saying, "I'm writing this because you want it but I'm so much better than this."  Even that isn't quite what I heard.

Anyone ever hear this in other classical works?  It's a curious thing and now I'm going to be on the lookout for more of it.

Do You Care???

So I wanted to start blogging and it occurs to me that it may not matter what I think or say except to me. That, however, will not stop me from putting down what falls/dribbles/oozes out of my mind since I need a place to collect "brain droppings" (thank you, George Carlin).

I work in software as a "software architect" and so am very close to technology and how it evolves and advances. Yet I find that I have become something of a "neo-Luddite." The longer I've worked creating software systems and fixing other people's problems, the more I've convinced myself that technology hasn't really added anything positive to our society. If anything, I believe it has removed things that were once precious and replaced them with things that appear precious on the surface but are not capable of standing up to close scrutiny.

Please disagree with me - I would like to be persuaded otherwise! What I hope to explore in this blog is what I think about technology and how it has made us less as a society. Along the way, I hope that my examination will also show me how it has made us more than we were; but today I'm not confident. So please, post your critiques and disagreements. And please, don't get personal: opinions are personally held but they are just opinions and not great truths or even profound ones. Perhaps we'll even come to some understanding of how to take where we are now and make something really good out of it! That would be (for me, anyway) the best outcome.

Being a blog, it will also be a place to explore bizarre and other things that I may think are interesting, weird, unexplainable, and completely off-topic. So bear with me.