All posts by chris

Theory of Constraints and “The Goal”

Two  years after reading Eliyahu M. Goldratt‘s famous book “The Goal” for the first time, I had a second go at it. It proved, again, to be an entertaining and at the same time enlightening read. I will not come up with yet another summary, you will find plenty of those already. Instead I want to point out a few interesting links to other areas.

Let me start with an interesting connection to my article about personal objectives. One of the statements I had made there, was that a global optimum cannot be reached by local optima everywhere; some local deficiencies would need to be accepted to reach the overall goal. (Apart from the common-sense aspect, this is also formally proven in systems theory.) Well, exactly the same point is mentioned in “The Goal” when Dr. Goldratt writes that at the end of the day, only the amount of shipped goods, i.e. goods sold, counts. Any intermediate over-achievements (“We beat the robot”), is really completely irrelevant.

Another interesting link is one with the book “Turn the Ship Around” by David Marquet (see this for a quick summary). Mr Marquet makes the point that in his experience a deciding factor for an organization’s (in his case the crew on a nuclear attack submarine) performance is, whether its members work to avoid individual mistakes or to achieve a common goal. Similar, in “The Goal” there is a paragraph about the success of various changes in a manufacturing context. It is basically about the different intent (and Mr Marquet calls his style intent-based leadership, by the way). Instead of trying to reduce costs, the manufacturing team looked at things from a revenue generation point of view.

In the addendum “Standing on the Shoulders of Giants” a short introduction is given into the Toyota Production System (TPS). One interesting comment in this section is, that other car manufacturers which have also implemented a comparable system, have not achieved the same level of improvement. This is attributed to the guiding principle: Toyota focused on throughput (and by that ultimately on revenue), whereas all the others looked at cost reduction. The analogy to that is a study (source unknown as of this writing) about customer satisfaction vs. profit optimization. Allegedly, those companies that focus on customers not only win on this front. But in the mid- to long-term they also constantly outperform those that focus on financials.

Revisiting Software Architecture

Quite recently I heard a statement similar to

“The application works, so there is no need to consider changing the architecture.”

I was a bit surprised and must admit that in this situation had no proper response for someone who obviously had a view so different from everything I believe in. But when you think about it, there is obviously a number of reasons why this statement was a bit premature. Let’s have a look at this in more detail.

There are several assumptions and implicit connotations, which in our case did not hold true. The very first is that the application actually works, and at the time that was not entirely clear. We had just gone through a rather bumpy go-live and there had not yet been a single work item processed by the system from start to finish, let alone all the edge cases covered. (We had done a sanity test with a limited set of data, but that had been executed by folks long on the project and not real end users.) So with all the issues that had surfaced during the project, nobody really knew how well the application would work in the real world.

The second assumption is that the chosen architecture is a good fit for the requirements. From a communication theory point of view this actually means “a good fit for what I understand the requirements to be”. So you could turn the statement in question around and say “You have not learned anything new about the requirements since you started the implementation?”. Because that is what it really means: I never look back and challenge my own thoughts or decisions. Rather dumb, isn’t it?

Interestingly, the statement was made in the context of a discussion about additional requirements. So there is a new situation and of course I should re-evaluate my options. It might indeed be tempting to just continue “the old way” until you really hit a wall. But if that happens you have consciously increased sunk costs. And even if you can “avoid the wall”, there is still a chance that a fresh look at things could have fostered a better result. So apart from the saved effort (and that is only the analysis, not a code change yet) you can only loose.

The next reason are difficulties with the original approach and of that there had been plenty in our case. Of course people are happy that things finally sort-of work. But the more difficulties there have been along the way, the bigger the risk that the current implementation is either fragile or still has some hidden issues.

And last but not least there are new tools that have become available in the meantime. Whether they have an architectural impact obviously depends on the specific circumstances. And it is a fine line, because there is always temptation to go for the new, cool thing. But does it provide enough added value to accept the risks that come with such a switch? Moving from a relational database to one that is graph-based, is one example that lends itself quite well to this discussion. When your use-case is about “objects” and their relationships with one another (social networks are the standard example here), the change away from a relational database is probably a serious option. If you deal with financial transactions, things look a bit different.

So in a nutshell here are the situations when you should explicitly re-evaluate your application’s architecture:

  • Improved understanding of the original requirements (e.g. after the first release has gone live)
  • New requirements
  • Difficulties faced with the initial approach
  • New alternatives available

So even if you are not such a big fan of re-factoring in the context of architecture, I could hopefully show you some reasons why it is usually the way to go.

My WiFi Setup with Ubiquiti Networks UAP-AC-PRO

After almost nine months it is time for a verdict on my “new” WiFi access, the Ubiquiti Networks UAP-AC-PRO. I can honestly say that it works extremely well here and all the WiFi issues I have had for years, have simply gone. The device is comparatively expensive (I paid about 140 Euros at Amazon) and unless my existing solution had not caused so many issues, I probably would have not spent the money. But for me it was definitely worth it.

I was initially made aware of the Ubiquiti Networks UAP-AC-PRO by an article on a German website that covers Apple-related topics. The guys there were quite enthusiastic about it and especially its graceful and uninterrupted handover of connections from one access point to the other. The latter had been a particularly nasty issue for me, with a Fritz!Box Fon WLAN 7390 covering the ground floor and an  FRITZ!WLAN Repeater 450E, configured as a pure access point, covering the first floor. There simply was no handover, so I had to effectively configure two completely separate WiFi networks. In addition the FRITZ!WLAN Repeater 450E needed a regular power-cycle because for no apparent reason it would stop working every couple of days. Its predecessor, a FRITZ! 300E WLAN Repeater, was much better in that respect, but it had died after a bit more than two years.

So all in all the situation was not too great on the WiFi front. This changed dramatically when I replaced both the Fritz!Box and the FRITZ!WLAN Repeater with just a single UAP-AC-PRO. Or in other words: Juts one UAP-AC-PRO gave me better WiFi than the both Fritz components combined. Impressive! In consequence, the seamless handover of connections from one access point to the other was not relevant any more at all. So instead of buying a second UAO-AC-PRO, I just have one and all is well. The flip-side is that I cannot play around with this feature ;-).

To sum things up, I am extremely satisfied with the UAP-AP-PRO. For administration I run the Unifi program on a Raspberry Pi 3 (model 2 worked just as well for me) and I will write another post on some of the setup aspects of that later. If you search on the Internet or look at Youtube, you will also find a lot of additional information.

Configuration Management – Part 8: The Maintenance

How do I maintain my configuration data? It is one thing to have them stored somewhere and being able to maintain stuff, if you are a developer or a technical person in general. In which case you will be fine with a plain text editor, sometimes even something like vi (I am an Emacs guy 😉 ). But what if you want business users be able to do this by themselves? In fact, quite often these folks will also tell you that they do not want to be dependent on you for the job.

When trying to identify the requirements for a maintenance tools suitable for business users, I came up with the following points (and in that order of priority):

  • Familiarity: People do not want to learn an entirely new tool for just maintaining a few configuration values. So exposing them to the complete functionality of a configuration management system will not work. And even if the latter has something like role-based configuration or user profiles, it will typically just hide those parts, that “normal” users are not supposed to see. But it will still, tacitly, require a lot of understanding about the overall system.
  • Safety: In many cases business people will tell you that while they want to do changes by themselves, they still expect you (or the system you develop) to check what they entered. In a number of cases the argument went so far that people, from my perspective, were trying to avoid responsibility for their actions altogether. Whether this was just an attempt to play the blame-game or rather an expression of uncertainty, I cannot say. But for sure you need to add some automated checks.
  • Auditing: In many cases you must be able to maintain records of changes for legal reasons (e.g. Sarbanes–Oxley Act). But even if that is not the case, you as the technically responsible person absolutely want to have records, what configuration were in the system when.
  • Extensibility: Especially today, when most of us try to work in an agile manner, this should not need mentioning. But unfortunately the reality I mostly witness is quite different. Anyway, what we as developers need to do here, is scrutinize what the business folks have brought up as their requirement. It is not their job to tell us about it, but the other way around.

So what is the solution? I see basically two options here: custom-developed web UI and Excel. The Excel approach may surprise you, but it really has a number of benefits, also from a technical perspective. Remember that it is file-based, so you can simply leverage your normal VCS for auditing. How the business users “upload” the updates, may vary. But you can start with something simple like a shell script/batch file that acts as a VCS wrapper, and many people will already be happy. More sophisticated stuff like a web upload and built-in verification is nicer of course.

A web UI has the benefit that you can provide feedback on rule violation easier, compared to the Excel approach. But what is often overlooked is the requirement for mass-updates. It is a big pain to click through 50 records whereas in Excel that would simply be a copy-paste operation. And the latter is less susceptible to errors, by the way.

Sometimes Business Rules Management System (BRMS) and the web UI they usually come with can be an alternative. But if you are not already using it on the project, the overhead of bringing in an entire additional system, will typically outweigh the benefits. If you still want to explore that route, pay particular attention to how the web UI changes can be synced back into the artifacts in VCS the developers work with.

How Useful Are Personal Objectives?

At least here in Germany there has been a recent shift away from personal objectives in compensation. Some very big companies like Daimler, Bosch and Infineon have announced that they will abandon personal objectives in this context and only look at the overall result of the organization. (I am pretty certain that sales is exempt from that, though.) They will, of course, keep personal objectives as a tool for individual and career development, but there will be no more financial impact.

Personally, I think this is a move into the right direction for various reasons and would like to reflect on a few of them (in descending order of importance).

  • Local optimization: This is the part of individual, financially relevant objectives that has by far the biggest impact. You have all seen it at work when different parts of the organization seem to fight each other. With the odd exception, they do not do that out of actual dislike but because it is needed for them to achieve their goals. One might think now, that in this case the goals just need to be adjusted. Well, not really. It is a major research topic in the field of systems theory and the boiled-down answer is that for a global optimum it is required that at least some sub-systems run below their local optimum. (For details see e.g. Systems theory and complexity: Part 3 by  Kurt A. Richardson.) Besides, who in business would have the time to develop such an elaborate system?
  • Employee demotivation: I have yet to meet someone who thinks that personal goals, which impact their salary, are a good and motivating thing. (There is also the fundamental point that motivation is by definition intrinsic, but this is another discussion.) The best you can hope for is a neutral judgement. However, the normal approach is that objectives trickle down through the hierarchy and the manager more or less gives the direct the same goals he or she got. I see two main problems here. First, if I get pretty much the same goal as my boss, how much influence do I still have to really control the result and by that my payment? Second, all of a sudden I have to deal with abstraction levels that are in a different “hemisphere” than my actual work. How does that fit?
  • Time spent: If you as a manager are at least somewhat serious about giving your directs proper objectives and balancing your own financial outcome with their needs, you spend an enormous amount of time on this. You have to to be careful how to sell them their goal (which you first had to devise), discuss the details and make adjustments. And the real fun comes with “judgment day”. Be prepared for discussions that likely scar your relationship with the direct. And we also have all the administrative details around this, although I am least concerned about them.

So in total you probably know by now my position on the subject. I consider myself a highly motivated employee and love most of my work. So how can anyone assume that I do not throw all my energy behind what I am doing? And even if I did not do this, would an objective linked to money change anything? No, I would just play the system.

Let’s see if the companies mentioned at the beginning of this post are just the tip of the iceberg.

Configuration Management – Part 7: The Time

An often overlooked aspect of configuration management is time. There are many scenarios where the correct value is dependent on time (or date for that matter). Typical examples are organizational structures (think “re-org”) or anything that is somehow related to legislation (every change in law or regulation becomes active at a certain time).

There are several approaches to deal with this situation:

  • Deployment: This is the simplest way and its ease comes with a number of drawbacks. What you basically do is make sure that a deployment happens exactly at the time when the changes should take effect. Obviously this collides with a typical CD (Continuous Deployment) setup. And while it is certainly possible to work around this, the remaining limitations usually make it not worthwhile. As to the latter, what if you need to process something with historical data? A good example is tax calculation, when you need to re-process an invoice that was issued before the new VAT rate got active. Next, how do you perform testing? Again, nothing impossible but  a case of having to find ways around it. And so on…
  • Feature toggle: Here the configuration management solution has some kind of scheduler to ensure that at the right point in time the cut-over to the new values is being made. This is conceptually quite similar to the deployment approach, but has the advantage of not colliding with CD. But all the other limitations mentioned above still apply. WxConfig supports this approach.
  • Configuration dimension: The key difference here is that time is explicitly modeled into the configuration itself. That means every query not only has the normal input, usually a key or XPath expression, but also a point in time (if left empty just take the current time as default). This gives full flexibility and eliminates the aforementioned limitations. Unfortunately, however, very few people take this into consideration when they start their project. Adding it later is always possible, but of course it comes with an over-proportional effort compared to doing it right from the start. WxConfig will add support for this soon (interestingly no one asked for it so far).

That was just a quick overview, but I hope it provided some insights. Comments are very welcome (as always).

 

webMethods Integration Server: Continuous Deployment

For more than nine years I have been working on a package for webMethods Integration Server. With the experience gained there, I want to discuss a number of aspects about Continuous Deployment.

Versioning

I recommend the use of semantic versioning, which at its core is about the following (for a lot of additional details, just follow the link):

  • The version number consists of three parts: Major, minor, and patch (example v1.4.2).
  • An increase in the major version indicates a non-backwards-compatible change.
  • An increase in the minor version indicates a backwards-compatible change.
  • A increase in the patch version indicates bug fixes only, no functional changes at all.

It is a well-known approach and makes it very easy for everyone to derive the relevant aspects from just looking at the version number. If the release in question contains bug fixes for something you have in use, it is probably a good idea to have a close look and check if a bug relevant to you was fixed. If it is a minor update and thus contains improvements while being backwards compatible, you may want to start thinking about a good time to make the switch. And if it is major update that (potentially) breaks things, a deeper look is needed.

Each Integration Server package has two attributes for holding information about its “version”. One is indeed the version itself and the other is the build. The latter is by default populated with in auto-increase number, which I find not very helpful. Yes, it gives me a unique identifier, but one that does not hold any context. What I put into this field instead is a combination of a date-time-stamp and the change set identifier from the Version Control System (VCS). This allows me to see at a glance when this package was built and what it contains.

Build

Conceptually the work on a single package, as opposed to a set of multiple ones that comprise one application, is a bit different in that you simply deal with only one artifact that gets released. In many projects I have seen, people take a slightly different approach and see the entirety of the project as their to-be-released artifact. This approach is supported by how the related tools (Asset Build Environment and Deployer) work: You simply throw in the source code for several packages, create an archive with metadata (esp. dependencies), and deploy it. Of course you could do this on a per-package basis. But it is easier to have just one big project for all of them.

Like almost always in live, nothing comes for free. What it practically means is that for every change in only one of the potentially many packages of the application, all of them need a re-deployment. Suppose you have an update in a maintenance module that is somewhat unrelated to normal daily operation. If you deploy everything in one big archive, this will effectively cause an outage for your application. So you just introduced a massive hindrance for Continuous Deployment. Of course this can be mitigated with blue-green deployments and you are well advised to have that in place anyway. But in reality few customers are there. What I recommend instead is an approach where you “cut” your packages in such a way that they each of them performs a clearly defined job. And then you have discrete CI job for each of them, of course with the dependencies taken into account.

Artifact Storage

Once your build has been created, it must reside somewhere. In a plain webMethods environment this is normally the file system, where the build was performed by Asset Build Environment (ABE). From there it would be picked up by Deployer and moved to the defined target environment(s). While this has the advantage of being a quite simple setup, it also has the downside that you loose the history of your builds. What you should do instead, is follow the same approach that has been hugely successful for Maven: use a binary repository like Artifactory, Nexus, or one of the others (a good comparison can be found here). I create a ZIP archive of the ABE result and have Jenkins upload it to Artifactory using the respective plugin.

To have the full history and at the same time a fixed download location, I perform this upload twice. The first contains a date-time-stamp and the change set identifier from the Version Control System (VCS), exactly like for the package’s build information. This is used for audit purposes only and gives me the full history of everything that has ever been built. But it is never used for actually performing the deployment. For that purpose I upload the ZIP archive a second time, but in this case without any changing parts in the URL. So it effectively makes it behave a bit like a permalink and I have a nice source for download. And since the packages themselves also contain the date-time stamp and change set identifier, I still know where they came from.

Deployment

Depending on your overall IT landscape there are two possible approaches for handling deployments. The recommended way is to use a general-purpose configuration management tool like Chef, Puppet, Ansible, Salt, etc. This should then also be the master of your webMethods deployments. Just point your script to your “permalink” in the binary reposiory and take it from there. I use Chef and its remote file mechanism. The latter nicely detects if the archive has changed on Artifactory and only then executes the download and deployment.

You can also develop your own scripts to do the download etc., and it may appear to be easier at a first glance. But there is a reason that configuration management tools like Chef et al. have had such success over the last couple years, compared to home-grown scripts. In my opinion it simply does not make sense to spend the time to re-invent the wheel here. So you should invest some time (there are many good videos on YouTube about this topic) and figure out which system is best for your needs. And if you still think that you will be faster with your own script, chances are that you overlooked some requirements. The usual suspects are logging, error handling, security, user management, documentation, etc.

With either approach this makes deployment a completely local operation and that has a number of benefits. In particular you can easily perform any preparatory work like e.g. adjusting content of files, create needed directories, etc.

Summary

All in all, this approach has worked extremely well for me. While it was first developed for an “isolated” utility package, it has proven to be even more useful for entire applications, comprised of multiple packages; in other words, it scales well.

Another big advantages is separation of concerns. It is always clear which activity is done by what component. The CI server performs the checkout from VCS and orchestrates the build and upload to the binary repository. The binary repository holds the deployable artifact and also maintains an audit trail of everything that has ever been built. The general-purpose configuration management tool orchestrates the download from the binary repository and the actual deployment.

With this split of the overall process into discrete steps, it is easier to extend and especially to debug. You can “inject” additional logic (think user exits) and especially implement things like blue-green deployments for a zero-downtime architecture. The latter will require some upfront thinking about shared state, but this is a conceptual problem and not specific to Integration Server.

One more word about scalability. If you have a big-enough farm of Integration Servers running (and some customers have hundreds of them), the local execution of deployments also is much faster than doing it from a central place.

I hope you find this information useful and would love to get your thoughts on it.