What should we do to handle the constantly changing domain and technical requirements? Patrick Kua talks about some interesting aspects on the architectural level.
It is not uncommon that an existing piece of non-trivial software gets re-written from scratch. Personally, I find this preference for a greenfield approach interesting, because it is not something I share. In fact, I believe that it is a fundamentally flawed approach in many cases, so why are people still doing this today?
Probably because they just like it more and also it is perceived, at least when things get started, as more glorious. But when you dig deeper the picture changes dramatically. Disclaimer: Of course there are situations when re-starting is the only option. But it typically involves aspects like required special hardware not being available any more.
When I started writing this post with making some notes, it all started with technical arguments. They are of course relevant, but the business side is much more critical. Just re-phrase the initial statement about re-writing to something like
Instead of gradually improving the existing software and learn along the way, we spend an enormous amount of money on something that delivers no additional value compared to what we have now. For this period, which we currently estimate to be 2 years (it is very likely to be longer), apart from very minor additions, the business will not get anything new, even if the market requires it; so long-term we risk the existence of the organization. And yes, we may actually loose some key members of staff along the way. Also, it is not certain that the new software ever works as expected. But should it really do, the market has changed so much, that the software is not very useful for doing business anyway and we can start all over again.
Is there anyone who still thinks that re-writing is generally a good idea?
Let us now change perspective and look at it from a software vendor’s point of view. Because the scenario above was written with an in-house application in mind. What is different, when we look at a company that develops and sells enterprise software? For this text the main properties of enterprise software are that it is used in large organizations to support critical business processes. How keen will customers be to bet their own future on something new, i.e. not tested? But even if they waited long enough for things to stabilize, there would be the migration effort. And if that effort comes towards them anyway, they may just as well look at the market for alternatives. So you would actively encourage your customer base to turn to the competition. Brilliant idea, right?
What becomes clear looking at things like that, is what the core value proposition of enterprise software is: investment protection. That is why we will, for the foreseeable future, continue to have mainframes with decades-old software running on them. Yes, these machines are expensive. But the alternatives are more expensive and in addition pose enormous risk.
In the context of commercial software vendors one argument for a re-write is that of additional revenue. It is often seen as easier to get a given amount of money for a new product than an improved version of something already out there. But that is the one-off view. What you typically want as a software vendor is a happy customer that pays maintenance every year and, whenever they need something new, first turns to you rather than the competition for options. Also, such a happy customer is by far the best marketing you can get. It may not look as sexy as getting new customers all the time, but it certainly drives the financial bottom line.
Switching over to the technical side, there are a few arguments that are typically made in favor of a restart. My favorite is the better architecture of the new solution, which will be the basis for future flexibility, growth, etc. I believe that most people are sincere here and think they can come up with something better. But the truth is that unless someone has done something similar successfully in the past, there is a big chance that the effort is hugely underestimated. Yes, technology has advanced and we have better languages and frameworks. But on the other hand the requirements have also grown dramatically. Think about high availability, scalability, performance and all the others. Even more important, though, is the business side. With something brand new people will have greatly increased expectations. So giving them something like-for-like will probably not be seen as success.
The not-invented-here syndrome is also relevant in this context and particularly with more junior teams. I have seen a case when an established framework used in more than 9,000 business-critical (i.e. direct impact on revenue) installations was dismissed in favor of something the team wanted to develop themselves. And I can tell you that the latter was a really bad implementation. Whether it was a misguided sense of pride or a fundamental lack of knowledge I cannot say. But while certainly being the most extreme incarnation I have seen so far, it was certainly not the only one.
So far my thoughts on the subject. Please let me know what you think about this topic.
Two years after reading Eliyahu M. Goldratt‘s famous book “The Goal” for the first time, I had a second go at it. It proved, again, to be an entertaining and at the same time enlightening read. I will not come up with yet another summary, you will find plenty of those already. Instead I want to point out a few interesting links to other areas.
Let me start with an interesting connection to my article about personal objectives. One of the statements I had made there, was that a global optimum cannot be reached by local optima everywhere; some local deficiencies would need to be accepted to reach the overall goal. (Apart from the common-sense aspect, this is also formally proven in systems theory.) Well, exactly the same point is mentioned in “The Goal” when Dr. Goldratt writes that at the end of the day, only the amount of shipped goods, i.e. goods sold, counts. Any intermediate over-achievements (“We beat the robot”), is really completely irrelevant.
Another interesting link is one with the book “Turn the Ship Around” by David Marquet (see this for a quick summary). Mr Marquet makes the point that in his experience a deciding factor for an organization’s (in his case the crew on a nuclear attack submarine) performance is, whether its members work to avoid individual mistakes or to achieve a common goal. Similar, in “The Goal” there is a paragraph about the success of various changes in a manufacturing context. It is basically about the different intent (and Mr Marquet calls his style intent-based leadership, by the way). Instead of trying to reduce costs, the manufacturing team looked at things from a revenue generation point of view.
In the addendum “Standing on the Shoulders of Giants” a short introduction is given into the Toyota Production System (TPS). One interesting comment in this section is, that other car manufacturers which have also implemented a comparable system, have not achieved the same level of improvement. This is attributed to the guiding principle: Toyota focused on throughput (and by that ultimately on revenue), whereas all the others looked at cost reduction. The analogy to that is a study (source unknown as of this writing) about customer satisfaction vs. profit optimization. Allegedly, those companies that focus on customers not only win on this front. But in the mid- to long-term they also constantly outperform those that focus on financials.
A very entertaining, yet serious talk about what kinds of “projects” there are and why you need different people for them.
At least here in Germany there has been a recent shift away from personal objectives in compensation. Some very big companies like Daimler, Bosch and Infineon have announced that they will abandon personal objectives in this context and only look at the overall result of the organization. (I am pretty certain that sales is exempt from that, though.) They will, of course, keep personal objectives as a tool for individual and career development, but there will be no more financial impact.
Personally, I think this is a move into the right direction for various reasons and would like to reflect on a few of them (in descending order of importance).
- Local optimization: This is the part of individual, financially relevant objectives that has by far the biggest impact. You have all seen it at work when different parts of the organization seem to fight each other. With the odd exception, they do not do that out of actual dislike but because it is needed for them to achieve their goals. One might think now, that in this case the goals just need to be adjusted. Well, not really. It is a major research topic in the field of systems theory and the boiled-down answer is that for a global optimum it is required that at least some sub-systems run below their local optimum. (For details see e.g. Systems theory and complexity: Part 3 by Kurt A. Richardson.) Besides, who in business would have the time to develop such an elaborate system?
- Employee demotivation: I have yet to meet someone who thinks that personal goals, which impact their salary, are a good and motivating thing. (There is also the fundamental point that motivation is by definition intrinsic, but this is another discussion.) The best you can hope for is a neutral judgement. However, the normal approach is that objectives trickle down through the hierarchy and the manager more or less gives the direct the same goals he or she got. I see two main problems here. First, if I get pretty much the same goal as my boss, how much influence do I still have to really control the result and by that my payment? Second, all of a sudden I have to deal with abstraction levels that are in a different “hemisphere” than my actual work. How does that fit?
- Time spent: If you as a manager are at least somewhat serious about giving your directs proper objectives and balancing your own financial outcome with their needs, you spend an enormous amount of time on this. You have to to be careful how to sell them their goal (which you first had to devise), discuss the details and make adjustments. And the real fun comes with “judgment day”. Be prepared for discussions that likely scar your relationship with the direct. And we also have all the administrative details around this, although I am least concerned about them.
So in total you probably know by now my position on the subject. I consider myself a highly motivated employee and love most of my work. So how can anyone assume that I do not throw all my energy behind what I am doing? And even if I did not do this, would an objective linked to money change anything? No, I would just play the system.
Let’s see if the companies mentioned at the beginning of this post are just the tip of the iceberg.
I have recently watched a very interesting presentation by Gunter Dueck (former CTO of IBM in Germany), where he talks about the future of work. When looking at how many consulting organizations structure their business, you can see a trend that seemingly high-level parts, usually the interaction with the customer and where business domain knowledge is required, stay in your own country. And those aspects that can, allegedly, be performed by less qualified people are moved to countries with lower salaries.
This may seem attractive from a cost-management point of view for a while. And it certainly works right now – to the extent possible. Because the organizations still have people locally who had been “allowed” to start with relatively simple coding tasks and grow into the more complex areas of the business. But what if those folks retire or leave? How do you get people to the level of qualification that allows them to understand the customer’s business on the one hand, and at the same time be able to have an intelligent conversation with the offshore folks who do the coding? Because for the latter you must have done coding yourself extensively.
Quite recently I came across a number of texts (e.g. “Is Giving the Secret to Getting Ahead?“), which suggest that trying to be helpful to others is not only altruistic, but also helps the giving person. This is certainly true for me, so let me tell you my story.
My employer is running a number of internal mailing lists that cover all our products and are used by folks to discuss all sorts of (mainly technical) stuff. I have been subscribed to many of those lists for close to ten years now and they helped me to learn an awful lot about our products and their real-world application.
Most of this information cannot be found in official documentation or training, simply because it is relevant only in a very specific context. What I realized, though, is that it is exactly this context-specificity, which helped me understand the products better. Because if you discuss, along with your own use-case, the border conditions, you develop a “feeling” how the software works internally. And this, in turn, allows you to analyze totally new use-cases.
So on average I spend about 30-60 minutes per day on those mailing lists (you typically get 150-200 mails per day in total). Some of them I just scan through rather quickly. But others I follow much more closely. And of course after some time I had identified various folks that stood out in terms of amount and quality of their contributions. Many of them I have never met personally in all those years because they live on another continent. But still we have a relationship.
After some time, when I had learned enough from following the lists as well as my actual work with some of the tools, I started answering questions myself. Soon enough people knew my areas of expertise and also added my personal email address next to the mailing lists when sending their questions, to make sure I would take a look.
So apart from my ego, how did helping others on the mailing lists help me? The obvious thing is that I am known to be an expert on various topics (e.g. ALM, performance, HA) for our products. But more importantly, when I have a question, people are willing to spend time and return the favor. Because, on the other hand, we all know the lone wolf character who only asks, but never helps others. And guess how many replies they usually get …
The same approach also works on public mailing lists, by the way. In the mid-1990s I had basically the same experience in the Novell NetWare group of the German FidoNet (basically a proprietary version of UseNet). I was a university student then and also had my own small company. Following the group was critical to building up my NetWare expertise, which I then used to charge a nice (at the time) hourly rate.
In conclusion I cannot overestimate the role that helping others played on my professional success. Plus, it feels really good :-).
There is an “interesting” perception of software development in that quite a few people think it is an activity suitable for graduates but not highly qualified professionals. I am in violent disagreement with this. But since it is a widespread belief, just dismissing it as dumb does not help very much. So how does it come that so many people, who have a somewhat limited knowledge about the subject, make such a judgement and act according to it?
From a psychological point of view there seems to be a mechanism in place that either makes those people believe that they actually understand the subject enough to make an informed decision. Or they do not see it as important enough to invest more time in the whole process and just go for the easy way. Or something in the middle? I will look at some thoughts I have had on the topic over the last couple of years and hope they make sense to you.
If you look at how most companies handle careers, you will soon realize that many talk about career paths for managers and professionals. And they are usually eager to say that both are valued equally in their organization. Well, I have yet to see a place where this statements, typically coming from HR, survives the reality check. If the entire organization is run such that managers from day one experience that the “techies” (sometimes lovingly referred to as code monkeys) are just there to serve management without questioning its wisdom, guess what happens.
Most professionals will simply do as they are told and not start a discussion why something is possibly not as easy or brilliant as it looks from the business side. Managers, and that is absolutely the techies’ fault, will then start to believe that they know enough about technology to decide on details. And the next time, when they do so and the techie just nods and goes away to deliver, they expect a good result. If the result is not so good though, because many details were not taken into account, it is indeed the techie’s fault to not have brought them up. And, voila, there you have the vicious circle.
I must say that I have been fortunate enough so far and not run into that situation. But that not only requires experience and the “standing” in the organization, it also depends on your financial independence. I live in Germany and we have laws against my employer just firing me, because I am not obedient enough but quite a nuisance instead. So in that respect I am just lucky. But your standing with management is something you can and should control. If you are too quiet and never give them a chance to learn how thoughtful and interested in the company’s progress you are, how should they know?
What has worked nicely for me is not trying to be someone else but find opportunities where I was on my home turf and still talk about something they were interested in. And in a nutshell the message I got across was: I understand and want to support what you need to achieve; and for the technical details you can rely on my experience and let me decide on the nitty-gritty stuff, which you are not interested in anyway.
And with this you are not talking about coding any more but supporting the business. Yes, actually performing this support means coding. But once people see that they are much better off with you doing this, rather than someone less experienced, they will happily accept it.
So how can you proof that your experience is relevant to the business? By keeping your promises and deliver on time, quality and budget. Because you have gone through the learning curve and know all the potential pitfalls that can happen for a given task. Having them taken care of upfront and not let surprise anyone, you keep things under control. Just a few weeks ago one of my projects went live and my counterpart from the business side was thrilled that there was only one bug discovered, and fixed immediately, so far.
Fixing things quickly and effectively is, by the way, one of the most powerful tools in your arsenal. Nobody expects a bug-free software, especially if it is custom development. But what people want, and rightly so, is that you can fix things fast. So invest time and have a system in place that allows you to ship fixes in no time and without interrupting normal operations. If you need to tell the business that a two-hour downtime is required for you to bring a fix into production, you are really not worth your salary.
That’s it for now. I know that I barley scratched the surface of a very diverse topic. But everything needs to start somewhere …
What is a good size for a development team? Of course there is no truly universal answer to this question. Software projects are just too diverse. But still my gut feeling as well as several articles I read on the subject, tell me that in many cases two to five people is probably a good initial estimate.
Obviously this excludes things above a certain size, like operating or ERP systems in their entirety. But change the focus to a module of the aforementioned and all of a sudden the team with five people is not so ridiculously small any more. So, like in many other cases, it all comes down to scope.
My general line of thinking is this: When something is too big for five people, it comes with an inherent complexity that will increase the likelihood of failure considerably. Of course, you can throw more resources at the problem. But you will get all the by-products of that. Think about communication overhead between developers, additional management layers, and the increased risk of low-performers among all the people that are poring in.
Interestingly, you can also approach things from the opposite direction: I was once told a story about a CEO who reduced the size of a development team to get things done better. At the time I thought this was a ridiculous approach. But I have long changed my mind since then and wholeheartedly agree. It is a bit like the inverse of what is described in “The Mythical Man-Month“, where Frederick P. Brooks, Jr. describes the naive idea that adding developers will shorten the time needed to accomplish a certain goal.
So in my view the goal should absolutely be to to have development teams no larger than five people. If the overall project is too big for this, you need to slice it into parts that can be delivered by small-enough teams. This will also help to enforce clean interfaces/contracts between the various parts.
All of the above assumes that we are talking about people with considerable experience (5+ years) in the relevant area. Given that the average experience of software developers is about five years, it will often be challenging to staff such a team. (And five years average is for their entire career, not the relevant subject area.) But dealing with this, is the topic for a different post.
There is a widespread notion, that developers at some point in their career evolve into something “better”, called architect. This term has the connotation of mastery on the subject, which I think is ok. What is not ok for me, is that in many cases there is the expectation that an architect’s time is too valuable for something as mundane as coding. Instead architects should develop the overall architecture of the solution.
Ok, so what is the architecture? Most people believe that some documentation (often a mixture of prose and high-level pictures) is the architecture. I would argue that this not the architecture but a more or less accurate abstraction from it. When enough work has gone into things, it may may even become the to-be architecture (but certainly not the as-is one). However, outside of regulated industries I have never seen a document that was thorough enough. A friend of mine used to work on embedded systems for car breaks where lives are at stake; and he told me some interesting stories about the testing and documentation efforts these guys take.
In my view the architecture is, by definition, in the code itself. Everything else is, I repeat myself, something that has some relationship with it. So how can an architect not be coding? You could argue that instead of doing the work him- or herself, mentoring and guiding less experienced developers is a good use of the architect’s time. For me that works fine up to a certain level of complexity. But if we talk about an STP (straight-through processing) solution, is that really something you want to give to a mid-level developer? (It would probably be an ideal piece of work for pair-programming, though.)
I do certainly not want to demean people who call themselves architects. It is a not-so-common capability to look at an IT landscape and see the big picture. Instead many people will be lost in the details. So we definitely need this this perspective! But it is a different kind of architecture, the so-called Enterprise Architecture (EA). I know folks who started as (really good) developers and are now very successful at Enterprise Architecture.
So, in closing, my core point is that the architecture of a solution and its code are basically two sides of the same coin. Everybody involved on the technical level should understand both aspects. And if the level of detail varies, depending on the official role, that is probably ok.