Structuring a VCS Repository

My main programming hobby project will soon celebrate its tenth birthday, so I thought a few notes on how I structure my VCS repository might be of interest. The VCS I have been using since the beginning is Subversion. (When I started, Git had already been released, but was really not that popular yet.) So while some details of this article will be specific to Subversion, the general concepts should be applicable elsewhere as well.

When it comes to the structure of a repository, Subversion does not impose anything from a technical point of view. All it sees is a kind-of file system, with all the pros and cons that come with the simplicity of this approach. It makes it easy for people to start using it, which is really good. But it also does not offer help for more advanced use-cases, so that people need to find a way how to map certain requirements onto that file system concept.

As a result, a convention has emerged and been there for many years now. It says that at the top-level of the project there should be only the following folders:

  • trunk: Home of the latest version (sometimes called the HEAD revision)
  • branches: Development sidelines where work happens in isolation from trunk
  • tags: Snapshots that give meaningful names to a certain revision

You will find plenty of additional information on the subject when searching the Internet. I can also recommend the book “Pragmatic Version Control: Using Subversion“, although it seems to be out of print now.

With these general points out of the way, let me start with how I work on my project. There are only a few core rules and despite their simplicity I can handle all situations.

  • The most important aspect for the structure of my SVN repository is that all active development on the coming version happens at trunk. See this article for all the important details, why you really want to follow that approach in almost all cases.
  • Once a new version is about to be released, I need a place where bug-fixes can be developed. So I create a release branch (e.g. /branches/releases/v1.3) with major and minor version number but not the patch version (I use semantic versioning). From this release branch I then cut the release (v1.3.0 in this case) by pointing the release job of my CI server to the release branch.
  • Once the release is done, I create a tag that also includes the patch version. In this example the tag will be from /branches/releases/v1.3 to /tags/releases/v1.3.0 .
  • Now I return to working on the next release (v1.4) by switching back to trunk.
  • Bug fixing happens primarily on trunk with fixes being back-ported to released versions. There are cases when this is not practical, of course. The two main reasons are that significant structural changes were done on trunk (you do refactor, don’t you?) or another change has implicitly removed the bug there already. But that is the exception.
  • When a bug-fix is needed on a released version, I temporarily switch my working copy to the release branch and do the respective work there. Unless the bug is critical I do not release a new version immediately after that, though. So this may repeat a few times, before the maintenance release.
  • The maintenance release is then cut, again, from /branches/releases/v1.3 . And after that a new tag is created to /tags/releases/v1.3.1 .

Those rules have proven to be working perfectly and I hope they will continue to do so for the next ten years. I have been quite lucky in that, although for me this is still a hobby project, the result is used by many global companies and organizations in a business-critical context. There are at least 11.000 installations in production that I am aware about, so I cannot be casual about reliability of the delivery process.

 

Changing the Web Hosting Company

After many years I have decided to move my web sites to a different web hosting company. While I am happy with the service provided in the past, the commercial side of things was not so bright. And even when I canceled the automatic renewal of a domain registration, the special offer I immediately got was not exciting.

Which is why I finally started the move this weekend. I have a plan to make the cut-over as smooth as possible, but there is a small chance of temporary interruption.

Raspberry Pi Cluster – Part 1: The Hardware

For  a long time I had wanted my own Raspberry Pi cluster and I finally managed to get started. There will be five nodes, which will give me 20 cores to play with. The overall plan is to manage the nodes with Ansible and run a Kubernetes cluster on them.

Let’s start with putting these boards together. I quickly settled for using spacer bolts. There are special ones for Raspberry Pi available, but I just got some cheap ones from AliExpress, as shown below. They have two disadvantages, though. Firstly, their diameter is a little bit too big for the wholes in the PCB (printed circuit board). So I used a wood drill size 3 to extend the wholes by just a fraction of a millimeter. You can somewhat see the white left-overs from the drilling at the lower left whole in the picture below.

The second problem is that my spacer bolts are pretty short. So in order to avoid short circuiting things, I put insulation tape on top of the USB ports.

With these two small modifications I went ahead and assembled the Raspberry Pis into a nice stack.

As power supply I chose a 5-port 50 watts USB model from Aukey, which I got a while ago from Amazon when they were on special offer. Aukey does such promotions quite frequently, so if you can wait a bit, you will be able to safe a few bucks.

And this is what things look like connected (excluding network of course).

That’s it for today. Stay tuned for the next part in this series.

On Micro Services

Just a couple of thoughts on micro services, triggered by a German blog post on heise Developer. So let’s go …

I have read quite a few articles on micro services, and literally all of them were overly optimistic and just scratching the, of course, positive surface. The examples are usually just slightly more complex than “Hello World” and also completely focused on technical aspects. For a non-deep dive technical exercise these things are ok. But nobody should fool him- or herself: the real world is much more complex.

There is, by the way, a real déjà vu here for me. Many things sound remarkable similar to what was said ten years ago in the context of SOA (Service-Oriented Architecture). Then, also, things like modularity and increased agility were cited all the time. (Re-use was the other standard one, but that did not work in most cases.) So what is really different now? I honestly fail to see anything new.

Instead there are arguments, again remarkably similar to SOA, in favor of faster possibility to adopt, easier deployment of change, and smaller and less complex code. Well, they are wrong. The key here is complexity and what its consequences are. The core complexity comes from the business requirements and not the chosen technology or architecture. And no matter how you slide, dice, and re-arrange things – the business complexity simply stays. Your only chance for survival is to find a clever way to layer things in a manner that makes sense for your business.

You will find that there are things that lend themselves well to (micro) services and of course that is how they should be approached. But you do not really have to have a separate deployment unit for every Mickey Mouse service. Just apply common sense and also keep in mind that things are in constant flux anyway.

So instead of spending much time on micro services, I would rather recommend to think about ways, which ensure that your system/architecture can easily adjust. And from a business perspective that is, of course.

Configuration Management – Part 9: The Audit Trail

Keeping track of  changes is a critical functionality in every configuration management system because there are legal requirements like  SOX (Sarbanes-Oxley Act) that require it. It can be accomplished in several ways. Basically you can either use an existing tool like a VCS (version control system) or have something custom-built.

When possible, I tend to prefer a VCS because it is (hopefully) already part of your process and governance approach. A typical workflow is that the underlying assets (i.e. configuration files) will be changed and then the VCS client be used to commit the change. The commit message allows to record the intent here, which is the critical information.

But there are cases when you need to be able to track things outside the VCS. In all cases I have seen so far the reason was that some information should not be maintained within the VCS for security or operational reasons. While organizations are often relaxed about data like host names in non-PROD environments, this changes abruptly when PROD comes into play. While I always think “security by obscurity” when I have that discussion, it is also a fight not worth having.

The other reason is operational procedures. The operations team often has a well-established approach that maintains configuration files for many applications in a unified way. The latter typically involves a dedicated location on network storage where configuration data sit. Ideally, there should also be a generic mechanism to track changes here. A dedicated VCS is of course a good option, but operations staff without a development background often (rightly) shy away from that route.

So it comes down to what the configuration management system itself offers. What I have implemented in WxConfig is a system where every operation that changes configuration data results in an audit event that gets persisted to disk. It includes metadata (e.g. what user initiated the change from which IP address), the actual change (e.g. file save from UI or change of value via API), and the old and new version of the affected configuration file.

The downside compared to a well-chosen commit message for VCS is that the system cannot record the intent. But on the other hand no change is lost, because no manual activity is needed. In practice this far outweighs the missing intent, at least for me. Also it has proven to be helpful during development when I had accidentally removed data. It was far easier to restore the latter from an audit record compared to looking them up in their original source.

All audit data get persisted to files and the metadata is recorded as XML. That allows automated processing, if required by e.g. a GRC system (Governance, Risk Management, and Compliance) or legal frameworks like the aforementioned Sarbanes-Oxley Act.

Ubiquity Networks Unify Controller on Raspberry Pi: Startup exits with RC=1

As mentioned in My WiFi Setup with Ubiquiti Networks UAP-AC-PRO I run the Unify Controller software on a Raspberry Pi 3. There is a ready-made package available for Debian and Ubuntu Linux, that can easily be used for this and I have been doing so for more than a year.

Just a yesterday, though, I broke things by overdoing it a bit with the removal of unneeded software from the Raspberry Pi. Through some “chain” the Unifi Controller had been removed and after re-installation it did not work anymore. Instead I saw a constant CPU utilization of an entire core by Java and also errors in /var/log/unifi/server.log :

[2017-12-26 13:07:04,783] <launcher> INFO system - *** Running for the first time, creating identity ***
[2017-12-26 13:07:04,791] <launcher> INFO system - UUID: yyyyyyy-yyyy-yyyy-yyyyyy-yyyyyyy
[2017-12-26 13:07:04,817] <launcher> INFO system - ======================================================================
[2017-12-26 13:07:04,819] <launcher> INFO system - UniFi 5.6.26 (build atag_5.6.26_10236 - release) is started
[2017-12-26 13:07:04,819] <launcher> INFO system - ======================================================================
[2017-12-26 13:07:04,867] <launcher> INFO system - BASE dir:/usr/lib/unifi
[2017-12-26 13:07:05,057] <launcher> INFO system - Current System IP: xxx.xxx.xxx.xxx
[2017-12-26 13:07:05,059] <launcher> INFO system - Hostname: zzzz
[2017-12-26 13:07:05,071] <launcher> INFO system - Valid keystore is missing. Generating one ...
[2017-12-26 13:07:05,072] <launcher> INFO system - Generating Certificate[UniFi]... please wait...
[2017-12-26 13:08:33,574] <launcher> INFO system - Certificate[UniFi] generated!
[2017-12-26 13:08:53,004] <UniFi> ERROR system - [exec] error, rc=1

The last couple of lines were showing up repeatedly, so obviously the system tried to restart over and over again. When you search the Internet for this problem, you will find out that you are not alone. Most solutions address available memory and not all people succeed with the various approaches to increase it (typically by removing memory from graphics and increasing swap space).

What I realized was that most discussions were for older versions and a recurring theme was that things changed between minor versions. So something that had worked for v5.6.19 did not necessarily work for v5.6.22 and vice versa. Also, changes to how Java was dealt with were mentioned quite often. Running Java-based applications on Linux can be somewhat delicate, so I do not blame the folks at Ubiquity Networks for that.

This was when I realized that the JVM on my system had changed. Before the accidental cleanup I had used the Oracle 8 JVM that gets installed via the Debian package oracle-java8-jdk. So I re-installed the latter and configured it as the default JVM via

sudo apt-get install oracle-java8-jdk
sudo update-alternatives --config java

This solved my problems instantly and things are up and running again.