Category Archives: Technical Stuff

Ubiquity Networks Unify Controller on Raspberry Pi: Startup exits with RC=1

As mentioned in My WiFi Setup with Ubiquiti Networks UAP-AC-PRO I run the Unify Controller software on a Raspberry Pi 3. There is a ready-made package available for Debian and Ubuntu Linux, that can easily be used for this and I have been doing so for more than a year.

Just a yesterday, though, I broke things by overdoing it a bit with the removal of unneeded software from the Raspberry Pi. Through some “chain” the Unifi Controller had been removed and after re-installation it did not work anymore. Instead I saw a constant CPU utilization of an entire core by Java and also errors in /var/log/unifi/server.log :

[2017-12-26 13:07:04,783] <launcher> INFO system - *** Running for the first time, creating identity ***
[2017-12-26 13:07:04,791] <launcher> INFO system - UUID: yyyyyyy-yyyy-yyyy-yyyyyy-yyyyyyy
[2017-12-26 13:07:04,817] <launcher> INFO system - ======================================================================
[2017-12-26 13:07:04,819] <launcher> INFO system - UniFi 5.6.26 (build atag_5.6.26_10236 - release) is started
[2017-12-26 13:07:04,819] <launcher> INFO system - ======================================================================
[2017-12-26 13:07:04,867] <launcher> INFO system - BASE dir:/usr/lib/unifi
[2017-12-26 13:07:05,057] <launcher> INFO system - Current System IP: xxx.xxx.xxx.xxx
[2017-12-26 13:07:05,059] <launcher> INFO system - Hostname: zzzz
[2017-12-26 13:07:05,071] <launcher> INFO system - Valid keystore is missing. Generating one ...
[2017-12-26 13:07:05,072] <launcher> INFO system - Generating Certificate[UniFi]... please wait...
[2017-12-26 13:08:33,574] <launcher> INFO system - Certificate[UniFi] generated!
[2017-12-26 13:08:53,004] <UniFi> ERROR system - [exec] error, rc=1

The last couple of lines were showing up repeatedly, so obviously the system tried to restart over and over again. When you search the Internet for this problem, you will find out that you are not alone. Most solutions address available memory and not all people succeed with the various approaches to increase it (typically by removing memory from graphics and increasing swap space).

What I realized was that most discussions were for older versions and a recurring theme was that things changed between minor versions. So something that had worked for v5.6.19 did not necessarily work for v5.6.22 and vice versa. Also, changes to how Java was dealt with were mentioned quite often. Running Java-based applications on Linux can be somewhat delicate, so I do not blame the folks at Ubiquity Networks for that.

This was when I realized that the JVM on my system had changed. Before the accidental cleanup I had used the Oracle 8 JVM that gets installed via the Debian package oracle-java8-jdk. So I re-installed the latter and configured it as the default JVM via

sudo apt-get install oracle-java8-jdk
sudo update-alternatives --config java

This solved my problems instantly and things are up and running again.

ESXi 6.5: Intel NICs Not Found

While playing around with an ESXi 6.5 test system, I accidentally killed all network connectivity by setting the NICs to pass-through. This post gives a bit of background and the solution that worked for me.

The system is home-built with a Fujitsu D3410-B2 motherboard and an Intel dual-port Gigabit NIC (HP OEM). The motherboard has a Realtek RTL8111G chip for its NIC, which does allegedly work with community drivers, but not out-of-the-box. One of the things I want to run on this box is a pfSense router. So, when I discovered, that the Realtek NIC was available for pass-through, I enabled this. I also enabled one(!) of the two ports of my Intel dual-port NIC. At least, that is what I had intended to do.

Because what really happened was that all three NICs were set to pass-through, which of course meant that ESXi itself had no NIC available to itself any more. This issue showed after the next reboot, when the console told me that no supported NICs had been found in the system. Perhaps not wrong in strict terms, but certainly a bit misleading, when you are not very experienced with ESXi.

Searching the net did not provide a real answer. But after a couple of minutes I realized that perhaps my change about pass-through might be the culprit. The relevant file where these settings are stored is /etc/vmware/esx.conf. I searched for lines looking like this

/device/000:01.0/owner = "passthru"

and replaced them with

/device/000:01.0/owner = "vmkernel"

After that I just had to reboot and things were fine again.

 

My WiFi Setup with Ubiquiti Networks UAP-AC-PRO

After almost nine months it is time for a verdict on my “new” WiFi access, the Ubiquiti Networks UAP-AC-PRO. I can honestly say that it works extremely well here and all the WiFi issues I have had for years, have simply gone. The device is comparatively expensive (I paid about 140 Euros at Amazon) and unless my existing solution had not caused so many issues, I probably would have not spent the money. But for me it was definitely worth it.

I was initially made aware of the Ubiquiti Networks UAP-AC-PRO by an article on a German website that covers Apple-related topics. The guys there were quite enthusiastic about it and especially its graceful and uninterrupted handover of connections from one access point to the other. The latter had been a particularly nasty issue for me, with a Fritz!Box Fon WLAN 7390 covering the ground floor and an  FRITZ!WLAN Repeater 450E, configured as a pure access point, covering the first floor. There simply was no handover, so I had to effectively configure two completely separate WiFi networks. In addition the FRITZ!WLAN Repeater 450E needed a regular power-cycle because for no apparent reason it would stop working every couple of days. Its predecessor, a FRITZ! 300E WLAN Repeater, was much better in that respect, but it had died after a bit more than two years.

So all in all the situation was not too great on the WiFi front. This changed dramatically when I replaced both the Fritz!Box and the FRITZ!WLAN Repeater with just a single UAP-AC-PRO. Or in other words: Juts one UAP-AC-PRO gave me better WiFi than the both Fritz components combined. Impressive! In consequence, the seamless handover of connections from one access point to the other was not relevant any more at all. So instead of buying a second UAO-AC-PRO, I just have one and all is well. The flip-side is that I cannot play around with this feature ;-).

To sum things up, I am extremely satisfied with the UAP-AP-PRO. For administration I run the Unifi program on a Raspberry Pi 3 (model 2 worked just as well for me) and I will write another post on some of the setup aspects of that later. If you search on the Internet or look at Youtube, you will also find a lot of additional information.

Configuration Management – Part 7: The Time

An often overlooked aspect of configuration management is time. There are many scenarios where the correct value is dependent on time (or date for that matter). Typical examples are organizational structures (think “re-org”) or anything that is somehow related to legislation (every change in law or regulation becomes active at a certain time).

There are several approaches to deal with this situation:

  • Deployment: This is the simplest way and its ease comes with a number of drawbacks. What you basically do is make sure that a deployment happens exactly at the time when the changes should take effect. Obviously this collides with a typical CD (Continuous Deployment) setup. And while it is certainly possible to work around this, the remaining limitations usually make it not worthwhile. As to the latter, what if you need to process something with historical data? A good example is tax calculation, when you need to re-process an invoice that was issued before the new VAT rate got active. Next, how do you perform testing? Again, nothing impossible but  a case of having to find ways around it. And so on…
  • Feature toggle: Here the configuration management solution has some kind of scheduler to ensure that at the right point in time the cut-over to the new values is being made. This is conceptually quite similar to the deployment approach, but has the advantage of not colliding with CD. But all the other limitations mentioned above still apply. WxConfig supports this approach.
  • Configuration dimension: The key difference here is that time is explicitly modeled into the configuration itself. That means every query not only has the normal input, usually a key or XPath expression, but also a point in time (if left empty just take the current time as default). This gives full flexibility and eliminates the aforementioned limitations. Unfortunately, however, very few people take this into consideration when they start their project. Adding it later is always possible, but of course it comes with an over-proportional effort compared to doing it right from the start. WxConfig will add support for this soon (interestingly no one asked for it so far).

That was just a quick overview, but I hope it provided some insights. Comments are very welcome (as always).

 

Configuration Management – Part 6: The Secrets

Every non-trivial application needs to deal with configuration data that require special protection. In most cases they are password or something similar. Putting those items into configuration files in clear is a pretty bad idea. Especially so, because these configuration files are almost always stored in a VCS (version control system), that many people have access to.

Other systems replace clear-text passwords found in files automatically with an encrypted version. (You may have seen cryptic values starting with something like {AES} in the past.) But apart from the conceptual issue that parts of the files are changed outside the developer’s control, this also not exactly an easy thing to implement. How do you tell the system, which values to encrypt? What about those time periods that passwords exist in clear text on disk, especially on production systems?

My approach was to leverage the built-in password manager facility of webMethods Integration Server instead. This is an encrypted data store that can be secured on multiple levels, up to HSMs (hardware security module). You can look at it as an associative array (in Java usually referred to as map) where a handle is used to retrieve the actual secret value. With a special syntax (e.g. secretValue=[[encrypted:handleToSecretValue]]) you declare the encrypted value. Once you have done that, this “pointer”will of course return no value, because you still need to actually define it in the password manager. This can be done via web UI, a service, or by importing a flat file. The flat file import, by the way, works really well with general purpose configuration management systems like Chef, Puppet, Ansible etc.

A nice side-effect of storing the actual value outside the regular configuration file is that within your configuration files you do not need to bother with the various environments (add that aspect to the complexity when looking at in-file encryption from the second paragraph). Because the part that is environment-specific is the actual value; the handle can, and in fact should, be the same across all environments. And since you define the specific value directly within in each system, you are already done.

Configuration Management – Part 5: The External World

A standard requirement in configuration management is using values that already exist somewhere else. The most obvious places are environment variables (defined by the operating system) and Java system properties. Other are existing (!) databases, e.g. from an ERP system, or files.

The reason for referencing values directly from their original sources instead of duplicating them in a copy-paste fashion is the Don’t-Repeat-Yourself (DRY) principle. Although the latter is typically discussed in the context of code, it applies to configuration at least as well. We all know those “great” applications, which require manual updates of the same value in different places. And if you miss only one, all hell breaks loose.

For re-use of values within a file, the standard approach is variable interpolation. A well-known syntax for that comes from the build tool Apache Ant, where ${variableName} can be placed anywhere into a value assignment. Apache Commons Configuration, and therefore also WxConfig, support this syntax. In WxConfig you can even reference values from other files that belong to the same Integration Server package.

But what to do for other sources? Well, for the aforementioned environment variables and Java system properties, the respective interpolators from Apache Commons Configuration can also be used in WxConfig. But the latter also defines several interpolators of its own.

  • Cross package: Assuming you have an Integration Server package that holds some general values (often referred to as “global”), those can be referenced. (There are multiple ways to deal with truly global values and the specifics really determine which one is used best.)
  • Current date/time: Gets the current date and/or time, which is admittedly a bit of an edge-case. One could argue that this is not really a configuration value, and that is correct. However, there are scenarios when the ability to quite easily have such a value comes in very handy. Think about files that get created during processing of data. Instead of manually concatenating the path, the base filename, the date/time stamp and the file suffix in the code, you could just have something like this in your configuration file: workerFile=${tmpDir}/appFile_${date:yyyyMMdd-HHmmssSSS}.dat Doesn’t that make things a lot cleaner to read?
  • File content: Similar to date/time this was added primarily for convenience and clarity reasons. The typical use-case are templates, where a file (possibly maintained by another application) is just read directly into a configuration value.
  • Code invocation: When all else fails, you can have an arbitrary piece of logic be executed and the result be placed into the configuration value.

It is worth noting that all interpolators resolve at invocation. So you always get current results. In the case of the file content interpolator, this has the downside of file I/O; but if that really becomes a bottleneck, you are still not worse off than when having the read somewhere else in the code. And the file system caching is also still there …

With this set of options, pretty much all requirements for a redundancy-free configuration management can be met.

Mac OS X Mavericks: Mouse Right-Click Stopped Working

Not sure how many people still use Mac OS X Mavericks, but anyway. I had just had the issue that the right-click had stopped working, on the mouse as well as the trackpad. There are many posts on the subject, so it seems a not too uncommon problem. The one that helped me suggested to simply turn the mouse off and on. Strangely enough, this fixed it for both, mouse and trackpad. Thanks!