Category Archives: Tools

Configuration Management – Part 6: The Secrets

Every non-trivial application needs to deal with configuration data that require special protection. In most cases they are password or something similar. Putting those items into configuration files in clear is a pretty bad idea. Especially so, because these configuration files are almost always stored in a VCS (version control system), that many people have access to.

Other systems replace clear-text passwords found in files automatically with an encrypted version. (You may have seen cryptic values starting with something like {AES} in the past.) But apart from the conceptual issue that parts of the files are changed outside the developer’s control, this also not exactly an easy thing to implement. How do you tell the system, which values to encrypt? What about those time periods that passwords exist in clear text on disk, especially on production systems?

My approach was to leverage the built-in password manager facility of webMethods Integration Server instead. This is an encrypted data store that can be secured on multiple levels, up to HSMs (hardware security module). You can look at it as an associative array (in Java usually referred to as map) where a handle is used to retrieve the actual secret value. With a special syntax (e.g. secretValue=[[encrypted:handleToSecretValue]]) you declare the encrypted value. Once you have done that, this “pointer”will of course return no value, because you still need to actually define it in the password manager. This can be done via web UI, a service, or by importing a flat file. The flat file import, by the way, works really well with general purpose configuration management systems like Chef, Puppet, Ansible etc.

A nice side-effect of storing the actual value outside the regular configuration file is that within your configuration files you do not need to bother with the various environments (add that aspect to the complexity when looking at in-file encryption from the second paragraph). Because the part that is environment-specific is the actual value; the handle can, and in fact should, be the same across all environments. And since you define the specific value directly within in each system, you are already done.

Configuration Management – Part 5: The External World

A standard requirement in configuration management is using values that already exist somewhere else. The most obvious places are environment variables (defined by the operating system) and Java system properties. Other are existing (!) databases, e.g. from an ERP system, or files.

The reason for referencing values directly from their original sources instead of duplicating them in a copy-paste fashion is the Don’t-Repeat-Yourself (DRY) principle. Although the latter is typically discussed in the context of code, it applies to configuration at least as well. We all know those “great” applications, which require manual updates of the same value in different places. And if you miss only one, all hell breaks loose.

For re-use of values within a file, the standard approach is variable interpolation. A well-known syntax for that comes from the build tool Apache Ant, where ${variableName} can be placed anywhere into a value assignment. Apache Commons Configuration, and therefore also WxConfig, support this syntax. In WxConfig you can even reference values from other files that belong to the same Integration Server package.

But what to do for other sources? Well, for the aforementioned environment variables and Java system properties, the respective interpolators from Apache Commons Configuration can also be used in WxConfig. But the latter also defines several interpolators of its own.

  • Cross package: Assuming you have an Integration Server package that holds some general values (often referred to as “global”), those can be referenced. (There are multiple ways to deal with truly global values and the specifics really determine which one is used best.)
  • Current date/time: Gets the current date and/or time, which is admittedly a bit of an edge-case. One could argue that this is not really a configuration value, and that is correct. However, there are scenarios when the ability to quite easily have such a value comes in very handy. Think about files that get created during processing of data. Instead of manually concatenating the path, the base filename, the date/time stamp and the file suffix in the code, you could just have something like this in your configuration file: workerFile=${tmpDir}/appFile_${date:yyyyMMdd-HHmmssSSS}.dat Doesn’t that make things a lot cleaner to read?
  • File content: Similar to date/time this was added primarily for convenience and clarity reasons. The typical use-case are templates, where a file (possibly maintained by another application) is just read directly into a configuration value.
  • Code invocation: When all else fails, you can have an arbitrary piece of logic be executed and the result be placed into the configuration value.

It is worth noting that all interpolators resolve at invocation. So you always get current results. In the case of the file content interpolator, this has the downside of file I/O; but if that really becomes a bottleneck, you are still not worse off than when having the read somewhere else in the code. And the file system caching is also still there …

With this set of options, pretty much all requirements for a redundancy-free configuration management can be met.

Configuration Management – Part 3: The Journey Begins

Today I want to tell you about a configuration management solution I have developed. It is built for Software AG‘s ESB (the webMethods Integration Server). The latter did, for a very long time, not have built-in capabilities for dealing with configuration values. So what many projects did, was to hack together a quick-and-dirty “solution” that was basically a wrapper around Java properties.

At the time (early 2008) I was working in a “SWAT” team in presales, where we dealt with large prospects that typically looked for a strategic solution to their integration requirements. Of course configuration management, although not as prevalent as today, came up more and more often. So instead of trying to dodge the bullet, I started working on a solution. And since webMethods Integration Server is built on an open architecture, you can easily add things.

So let’s dive into some of the details of my configuration management solution, called
WxConfig. Its overall design goal was to have packages (roughly the platform’s equivalent to a web application on servlet containers) that could be moved around without any additional manual setup or configuration. The latter is a requirement stemming from the fact that we are in an ESB/integration context. By definition most packages therefore have some “relationship” to the outside world, and are not an application running more or less in isolation. Typical example would be a connection to a database, or an ERP system.

Traditionally there were two approaches for dealing with these external environment dependencies. On development (DEV) environments, there was some kind of document, telling the developer what setup/configuration they would have to apply to their DEV environment. For all “higher” stages like test (TEST) or production (PROD) this was hopefully handled by some kind automation; although back in 2008 that was rarely case and more often than not also there a manual checklist was used.

From a presales perspective (that is where it all started) the DEV part was the relevant one. My idea was to come up with something that could transform the document with the setup instructions into some kind of codification. So I basically devised an XML format (or rather a convention) that did just that. It started out quite small, with only schedulers being supported. (In that specific case we needed a periodic execution of logic for a demo.)

Since then a lot of other things were added. The approach was so successful that  many customers have moved their entire environment setup (well, at least those parts that are supported) to the WxConfig solution. And there are, interestingly enough, quite a few customers who use WxConfig solely for environment configuration but not for configuring their actual applications.

You may now ask: Hang on, what about the other environments, like TEST and PROD? Very good question! This is one of the core strengths of the WxConfig solution and therefore warrants a separate post. Stay tuned …

LaTeX: Hyphenation of \texttt

I have been using \LaTeX since I was 17 and still love it. But one thing has been bugging me for a long time. For a while I had been manually tweaking documents to get proper linebreaks when using the \texttt{} macro. Now I had finally come to the point where this did not work any longer. A search gave me this snippet:

\renewcommand{\texttt}[1]{%
 \begingroup
 \ttfamily
 \begingroup\lccode`~=`/\lowercase{\endgroup\def~}{/\discretionary{}{}{}}%
 \begingroup\lccode`~=`[\lowercase{\endgroup\def~}{[\discretionary{}{}{}}%
 \begingroup\lccode`~=`.\lowercase{\endgroup\def~}{.\discretionary{}{}{}}%
 \catcode`/=\active\catcode`[=\active\catcode`.=\active
 \scantokens{#1\noexpand}%
 \endgroup
 }

It works great!

VMware Fusion 4 on OS X Mavericks

Having upgraded to OS X Mavericks just recently, one of the main questions for me was, whether I would need to upgrade from VMware Fusion 4 to the current version (which is Fusion 7). Initially the thought had not really occurred to me, but after the second crash of my MacBook I started to look around. The majority of content I found was about the support status and some people had gotten really agitated during the discussion. But little was said about the technical side of things, i.e. whether the stuff was working or not.

What I can say today is that for me VMware Fusion 4.1.4 works nicely and without issues so far. I use it almost exclusively to run a company-provided Java development environment on Windows 7 64 bit. So no 3D graphics, USB drives, or other “fancy” things.

The crashes were most like caused by an old version of the keyboard remapping tool Karabiner (formerly KeyRemap4MacBook). Once I had upgraded to the appropriate version, the crashes were gone.

Keyboard Tools

Just a quick post to share two awesome utilities that I have been using for a while now. I like working with the keyboard over the mouse in many cases, simply because I am faster with that, and was recently pointed to a a tool that will make things even better:

Auto Hotkey: This allows you to “play” an arbitrary set of key strokes and commands either by replacing text you just typed or by re-assigning a key (combination). Very handy for emails where you can now simply type “brc” and get “Best regards,<NEW_LINE>Christoph”

Another tool worth mentioning is Key Tweak, which provides a GUI over the registry that allows to remap the keyboard. My use-case was a very old IBM Model M keyboard that does not have a Windows key (I was made aware of the tool by this post)

JMeter 2.5 Release

I am quite happy that my first contribution to a pretty well known open source program has made it into the latest release of JMeter, the famous load-testing tool from the Apache folks. At the beginning of this year I needed to do some load-testing on JMS topics with durable subscriptions. At the time those weren’t available in JMeter so I dug into the code and added the functionality, which was less difficult than anticipated.

Custom target directory for unattended Java installation on Windows

There is a bunch of information available that explain how to perform an unattended (aka silent) installation of Java on Windows  (examples for JRE and JDK).

What is usually not mentioned is the fact that when you specify a non-standard target directory, it must not contain any spaces. The easiest way to achieve this is the use of the 8.3 format. So instead of “C:\Program Files” you can use “C:\Progra~1”.

WebDAV Client

If you are looking for a small cross-platform WebDAV client, you should probably check on DAV Explorer. It looks a bit old-fashioned but is small (less than 600 KB), does not require installation, and simply worked for me. The only requirement is a JDK and its “bin” sub-directory in the path, but you have that anyway, don’t you?

It is probably not the best choice if you are a non-technical person, want to work with it every day, and look for a nice integration into your system’s GUI. But for quick tests or the occasional development-centric work it’s absolutely sufficient.

Automate hMailServer Setup

For a set of demo systems (running on Windows) I needed a local mail server. I have been using hMailServer for quite a while in such scenarios. It integrates nicely with Windows (runs as a service), has an administration GUI and overall a good reputation. What was different this time? Well, I simply wanted to do the setup in a fully automated (also referred to as unattended) fashion. This comprises the installation and all following configuration steps that I so far used to perform using the GUI.

Some research on the web revealed that automating the  installation itself is quite easy but several requests for doing the same with the configuration were unanswered. This is funny because hMailServer comes with a COM API that allows you to do pretty much everything you would normally do with the GUI. I did not want to emerge as a .Net developer from this and quickly found that VBS (Visual Basic Script) is probably the quickest way. Back in 1998 I had done some serious development with VBA (Visual Basic for Applications) and Access 97 which made things relatively easy. So here we go …

The installation is automated using the /verysilent command line switch, which will install

  • server and administration tools
  • in the default location (%PROGRAMFILES%\hMailServer)
  • using the embedded MS SQL database
  • with an empty password for administration access

If you want a custom location for the installation, you can add /DIR="c:\pathto\hmailserver".

Then you  have to run the VBS script (download here and rename to hmail-create-accounts.vbs) which will create the account. The syntax to call from a command line window (cmd.exe) is start hmail-create-accounts.vbs USERNAME PASSWORD DOMAINIf you need other things automated the script and the API documentation should make this sufficiently easy.

If you want to go one step further, simply put the hMailServer installable and the script into your own installer. I usually recommend NSIS (Nullsoft Scriptable Install System) and its Eclipse plugin for this. The latter provides a nice wizard, so you can have your individual setup.exe in a just a few minutes.