Category Archives: Technical Stuff

Configuration Management – Part 3: The Journey Begins

Today I want to tell you about a configuration management solution I have developed. It is built for Software AG‘s ESB (the webMethods Integration Server). The latter did, for a very long time, not have built-in capabilities for dealing with configuration values. So what many projects did, was to hack together a quick-and-dirty “solution” that was basically a wrapper around Java properties.

At the time (early 2008) I was working in a “SWAT” team in presales, where we dealt with large prospects that typically looked for a strategic solution to their integration requirements. Of course configuration management, although not as prevalent as today, came up more and more often. So instead of trying to dodge the bullet, I started working on a solution. And since webMethods Integration Server is built on an open architecture, you can easily add things.

So let’s dive into some of the details of my configuration management solution, called
WxConfig. Its overall design goal was to have packages (roughly the platform’s equivalent to a web application on servlet containers) that could be moved around without any additional manual setup or configuration. The latter is a requirement stemming from the fact that we are in an ESB/integration context. By definition most packages therefore have some “relationship” to the outside world, and are not an application running more or less in isolation. Typical example would be a connection to a database, or an ERP system.

Traditionally there were two approaches for dealing with these external environment dependencies. On development (DEV) environments, there was some kind of document, telling the developer what setup/configuration they would have to apply to their DEV environment. For all “higher” stages like test (TEST) or production (PROD) this was hopefully handled by some kind automation; although back in 2008 that was rarely case and more often than not also there a manual checklist was used.

From a presales perspective (that is where it all started) the DEV part was the relevant one. My idea was to come up with something that could transform the document with the setup instructions into some kind of codification. So I basically devised an XML format (or rather a convention) that did just that. It started out quite small, with only schedulers being supported. (In that specific case we needed a periodic execution of logic for a demo.)

Since then a lot of other things were added. The approach was so successful that  many customers have moved their entire environment setup (well, at least those parts that are supported) to the WxConfig solution. And there are, interestingly enough, quite a few customers who use WxConfig solely for environment configuration but not for configuring their actual applications.

You may now ask: Hang on, what about the other environments, like TEST and PROD? Very good question! This is one of the core strengths of the WxConfig solution and therefore warrants a separate post. Stay tuned …

Configuration Management – Part 2: Splendid Isolation

After the first post on configuration management, today I want to write about what I consider one of the biggest flaws in all systems I am aware about so far. They all live pretty much in “splendid isolation” and ignore whatever surrounds them.

My professional background is integration software and I attribute it to this fact, that for me it is a rather obvious real-world requirement to not ignore what is around you, but connect with it. In the past this was typically applied to applications on the business level. So having the ERP system talk with the CRM and SCM applications is well-established. But of course only your own imagination stops you from doing the same for other software. For an example, please check my post on integration of ALM software.

So what specifically makes integration an interesting aspect of configuration management? There are many different “layers” of configuration management out there. It starts with the infrastructure where you typically find a CMDB (Configuration Management Database), that contains information about computers and other components (esp. network). Then there are tools like Chef, Puppet, Ansible etc. that manage the operating system. (I know that is quite a simplification, because you can obviously handle other aspects that way as well; and I have done that extensively.)

A whole “new” level comes into play with containers like Docker. They come, again, with their own tools to manage configuration and operations. Then there is infrastructure-level software like databases, middleware, application servers, etc.

And last but not least, you finally hit the actual application. This is, by the way, the only, if any, level that the business will be interested in. They do not care about the vast complexity underneath, the interdependencies, etc. But of course they expect us to deliver value fast, without interruptions or failures, and for as little money as possible.

If there is no well-managed link between all those layers (and of course the number of layers needs to be multiplied by the number of environment types), how could you be able to handle things, let alone handle them well?

This is why I firmly believe, you need integration, governance, workflow capabilities etc. around configuration management.

Configuration Management – Part 1: The Horrors

Why is Configuration Management (CM) so important? The simple answer is: Because the correct configuration is one of the three critical parts for running a non-trivial piece of software successfully. (The other two are the actual code and the data.)

I still remember my first project when I was fresh out of university. While the team was great, everything else was a bloody nightmare. In particular the software, for which we developed, did not support configuration management in any way. Staging was basically done by exporting from the development system (DEV), manually finding and changing values in the export (at least it was plain ASCII), and importing it on the test system (TEST), and so on.

As if this wasn’t bad enough, there actually was no TEST environment. The customer had decided to save the money, so we had a combined DEV+TEST system. So more than once we ended up with the production system (PROD) accessing the DEV+TEST database. You may ask why we did not simply write a script to change the various settings automatically. Well, we were not allowed to, because it was not budgeted for.

And it gets even worse, because I actually lied to you when describing the staging process before. The problem was that while the export from one environment worked fine, the import into another did not after some time. It was basically a limitation of the maximum size the import file could could have. Our solution (after consultation with the vendor!) was to shut down both environments (DEV+TEST and PROD) and copy the binary storage files that held the development from one to the other. Then all the relevant values had to be adjusted manually on PROD, and then a test to be performed.

Of course that test failed a couple of times until all the changes had been done and recognized by the system (caching can be a dangerous thing). And because we were interfacing with at least five other systems (all PROD!) such a test run was not so easy. To make things more complicated, every set of test data (to be manually prepared on all surrounding systems) could only be used once. So we made a lot of “friends” with the folks operating the other systems.

We pulled a few all-nighters and somehow got things up and running (there were many other nice things with the software, of course). But it was frustrating and very inefficient. Since then configuration has been something that I deeply care about. And you probably understand why now.

FreeNAS Upgrade from 9.2.1.7 to 9.10.2-U2

I have been an extremely satisfied user of FreeNAS for many years, but had not updated my system for quite a while. Having a free Sunday was the perfect opportunity to do just that.

There were two steps involved: First go to 9.3.1 and then to 9.10.2-U2. As my box sits in the basement and is headless, I opted for update via GUI and it went extremely well.

The only issue was that on 9.10.2 the jail template list was empty. A quick online search revealed that this was, unfortunately, expected. But on the other hand, in such a case there is a good chance, that someone else has already found a solution.

But it all proved a bit more challenging than expected. And  nothing I found worked completely. So I ended up with a combination of things from other people (in particular this link) and my own discoveries:

  • I deleted the ZFS volume .warden-template-standard via the web UI
  • Then from the command line the ZFS was re-created with
    zfs create zpool/jails/.warden-template-standard
  • Deleted the template
    warden template delete standard
  • Re-created the standard template
    warden template create -nick standard -tar http://download.freenas.org/jails/10/x64/freenas-standard-10.3-RELEASE.tgz

At this point everything should have been good – but it was not. Still no template showed up when I tried to create a new jail. So I added a jail template in the web UI, too.

The URL is exactly the same as for the command line above.

With this I was able to create a new FreeBSD jail.

Choosing a Technology

I recently started a new hobby project (it is still in stealth mode, so no details yet) and went through the exercise to really carefully think about what technology to use for it. On a very high level the requirements are fairly standard: Web UI, persistence layer, API focus, cross-platform, cloud-ready, continuous delivery, test automation, logging, user and role management, and all the other things.

Initially I was wondering about the programming language, but quickly settled for Java. I have reasonable experience with other languages, but Java is definitely where most of my knowledge lies these days. So much for the easy part, because the next question proved to be “slightly” more difficult to answer.

Looking at my requirements it was obvious that developing everything from the ground up would be nonsense. The world does not need yet another persistence framework and I would not see any tangible result for years to come, thus loosing interest to soon. So I started looking around and first went to Spring. There is a plethora of tutorials out there and they show impressive results really quickly. Java EE was not really on my screen then, probably because I still hear some former colleagues complain about J2EE 1.4 in the back of my mind. More importantly, though, my concern was more with agility (Spring) over standards (JEE). My perception with too many Java standards is that they never outgrow infancy, simply because they lack adoption in the real world. On the other hand Spring was created to solve real-world problems in the first place.

But then, when answering a colleague’s question about something totally different, I made the following statement:

I tend to avoid convenience layers, unless I am 100% certain that they can cope with all future requirements.

All to often I have seen that some first quick results were paid for later, when the framework proved not to be flexible enough (I call this the 4GL trap). So this cautioned myself and I more or less went back to the drawing board: What are the driving questions for technology selection?

  • Requirements: At the beginning of any non-trivial software project the requirements are never understood in detail. So unless your project falls into a specific category, for which there is proven standard set of technology, you must keep your options open.
  • Future proof: This is a bit like crystal ball gazing, but you can limit the risks. The chances are bigger that a tier-3 Apache project dies than an established (!) Java standard to disappear. And of course this means that any somewhat new and fancy piece must undergo extreme scrutiny before selecting it; and you better have a migration strategy, just in case.
  • Body of knowledge: Sooner or later you will need help, because the documentation (you had checked what is available, right?) does not cover it. Having a wealth of information available, typically by means of your favorite search engine, will make all the difference. Of course proper commercial support from a vendor is also critical for non-hobby projects.
  • Environment: Related to the last aspect is how the “landscape” surrounding your project looks like. This entails technology but even more importantly the organization which has evolved around that technology. The synergies from staying with what is established will often outweigh the benefits that something new might have when looked at in isolation.

On a strategic level these are the critical questions in my opinion. Yes, there are quite a few others, but they are more concerned with specifics.

ESP8266 Management Platform

There is an awful lot of people doing hobby projects with the ESP8266. I did not want to come up with the 47th incarnation of something, so the question came up, what an interesting project could look like. In the end I decided to develop a solution for managing a multitude of ESP8266-based devices. From a high-level perspective this will include the following components:

  • CMDB: holds information about all devices
  • Bootstrapper: prepares the raw module for all further work
  • Security Manager
  • Lifecycle Manager: Reference processes “from cradle to grave”

In my professional life I have been doing a lot of work on SDLC (Software Development Lifecycle) and configuration management, so the above project choice seemed a natural fit. It also happens that I am deeply interested in both topics and strongly think that poor application of them is often responsible for IT project failures.

Arduino: Ethernet with ENC28J60

The two main approaches for connecting an Arduino to an Ethernet network are the “normal” Ethernet shield or a module based on the ENC28J60 chip. While the Ethernet shield is easier to start with, it is comparatively expensive and also of little use for the smaller Arduino models (e.g. Arduino Nano or Pro Mini).

p1020887a

In contrast the ENC28J60 modules are cheap, small, and universally connectable. This comes, however, with the small downside that they are not supported by the Arduino IDE out-of-the-box. Of course, there is a myriad of web pages out there that deal with it in one aspect or another. There are two reasons why I add yet another:

  • My ENC28J60 module (bought from AliExpress) is slightly different from the one usually seen. It does not need 3.3 volts but 5 volts, so the product description is wrong in that respect. Also, unlike all the other modules I have seen, it does not come with 10 but 12 connections.

    p1020877

  • The usual recommendation for the library is EtherCard, but this did not work for me.

What finally worked for me was using the UIPEthernet library, which can also replace the standard Arduino Ethernet library. I connected the ENC28J60 module with a recently purchased Arduino Nano v3 clone using the following connections

Arduino Nano ENC28J60
SS (Pin 10) CS
MOSI (Pin 11) SI
MISO (Pin 12) SO
SCK (Pin 13) SCK
5V 5V
GND GND

The Arduino Nano pin layout can be found here. The ENC28J60 module module offers a few more connections, as can be seen on the photo below. However, I used only the “usual suspects”.
p1020883a

The next step is to put this thing to use. I am currently planning some kind of data logger for 433 MHz signals, that are typically used for entry-level remote control purposes.

LaTeX: Hyphenation of \texttt

I have been using \LaTeX since I was 17 and still love it. But one thing has been bugging me for a long time. For a while I had been manually tweaking documents to get proper linebreaks when using the \texttt{} macro. Now I had finally come to the point where this did not work any longer. A search gave me this snippet:

\renewcommand{\texttt}[1]{%
 \begingroup
 \ttfamily
 \begingroup\lccode`~=`/\lowercase{\endgroup\def~}{/\discretionary{}{}{}}%
 \begingroup\lccode`~=`[\lowercase{\endgroup\def~}{[\discretionary{}{}{}}%
 \begingroup\lccode`~=`.\lowercase{\endgroup\def~}{.\discretionary{}{}{}}%
 \catcode`/=\active\catcode`[=\active\catcode`.=\active
 \scantokens{#1\noexpand}%
 \endgroup
 }

It works great!