Site Menu

 

Links

Recent Comments

Social Links

Site search

Recent Posts

Categories

Tags

Agile Architecture Book BPM BPM and SOA C-64 CIFS Commodore Communication Directory EA ESB Fedora Gartner Git gitosis Java LDAP Lifecycle Management Linux Management Meeting NAS OpenSolaris Palm Pilot Password Pattern Performance Perl Pictures Podcast Prague Requirements Restaurant Review Samba Security SMB SOA Software TOGAF Travel VCS Web Service Zachman

Oracle DB does not start

Sometimes people wonder why Oracle DB (both 10g and 11g) will not start after installation on Linux/Unix. In many cases the simple reason is the content of /etc/oratab. For each database it contains a line in the format $ORACLE_SID:$ORACLE_HOME:START_DB_FLAG:The last column is set to “N” by default. Just change it to “Y” and run $ORACLE_HOME/bin/dbstart again. Your database should come up now.

Ant Filter to Include Files with Configurable Names

I have been using Ant for a lot of things in the past. Just recently, I decided to also leverage it for an important aspect of my approach to shell script programming. This had been accomplished by a combination of Perl and Bash in the past. The “old” approach is ok, except that it doesn’t integrate with Maven as well as Ant does. Also, I have the requirement to run the build process on various operating systems, incl. Windows and Unix flavours that don’t necessarily have Bash installed.

So what needs to happen during the build? A lot of things in shell script programming come up time and again. So I want to encapsulate them as functions and build a library. Sourcing libary files in at run-time is really hard to achieve, if you want to be portable (your favourite search engine has loads of information on the topic). So instead of throwing a lot of time on these sometimes very subtle problems, I decided to have include statements in the source code and pre-process the scripts during the build.

There is include as well as replace functionality in Ant in Maven. But nothing I found gave me the capability to easily retrieve the file name from the currently processed source file. Yes, it would have been possible to come up with something based on normal Ant tasks (perhaps with some support from ant-contrib), but the result would have been quite ugly. So, after some investigation, I decided to write my own Ant filter reader for that. To speed things up, I borrowed from two of the filters that come with Ant, ExpandProperties and ReplaceTokens. (Needless to say my stuff comes with the same licence terms as the rest of Ant.)

Here is how it works:

  • Check the source content (in most cases coming from a file) for a regular expression. This has to match the format in which you have the include statements in your source code. So while I have set the default regex to fit my needs, you can configure it to whatever you need; just remember that Java regex handling is a bit different from e.g. Perl. This also means, that the thing can not only be used for shell scripting but any purpose that requires parametrised includes.
  • Retrieve the file name from the regex match. This is done by using a regex group. By default the first group is taken, but you can also override this if needed.
  • Read the include file line by line and insert them into the original content.
  • Optionally, you can add prefixes and suffixes to every line. By default both are set to an empty string, which effectively switches things off. My personal use case was that I want to be able to include instructions from a file and have them marked as a comment (prefix = “# “). Probably this could have been achieved differently as well. But it was easy to implement on top of the rest and makes things more comfortable for me.

Now, let’s look at an example Ant build file:

<?xml version="1.0"?>
<project name="TestAntTools" basedir="." default="" >

<property name="base.dir" value="../../.."/>
<property name="lib.dir" value="${base.dir}/target"/>

<property name="test.file"
   value="${base.dir}/src/test/resources/main_file.txt"/>

<typedef resource="antlib.xml">
  <classpath>
    <pathelement location="${lib.dir}/ant-0.0.1.jar" />
  </classpath>
</typedef>

<!-- =================================
target: TestProcessIncludes
================================= -->
<target name="TestProcessIncludes" depends=""
    description="Test the filter processIncludes">
  <loadfile srcfile="${test.file}" property="test.file.processed">
    <filterchain>
      <processIncludes>
        <setting key="prefix" value=""/>
        <setting key="suffix" value=""/>
        <searchdir value="."/>
        <searchdir value="../resources"/>
      </processIncludes>
    </filterchain>
  </loadfile>
  <echo>${test.file.processed}</echo>
</target>

</project>

The main file has this content:

# !/bin.bash

# This is before include

##include "incl_file.txt"

# This is after include

And the included file looks like this:
# ===========================
#   This was included
# ===========================

The result from the Ant script above:
Buildfile: /home/XXXXX/workspace/AntTools/src/test/ant/build.xml
TestProcessIncludes:
[echo] # !/bin.bash
[echo]
[echo] # This is before include
[echo]
[echo] # ===========================
[echo] #   This was included
[echo] # ===========================
[echo]
[echo] # This is after include
BUILD SUCCESSFUL
Total time: 142 milliseconds

As you can see, nothing special here. To download the jar containing the filter, please click here. You will need to rename the file upon saving to end up with a proper extension (Wordpress does not allow to save attachments with .jar as extension). And for those intersted, here is the Eclipse project with source code etc.

Linux Network Mounts for a Laptop

When you are running Linux (or any other Unix-like OS) on a mobile computer, you may have a bit of a problem with mounting shares from a server. On a normal PC with a fixed connection to the server, you would simply add the appropriate entries to /etc/fstab and have them mounted automatically during startup. But since I am not “always on”, this was not an option. So I added the “noauto” option to the entries in /etc/fstab, which effectively means that they get ignored during system startup. Of course I could later call mount /my/mountpoint manually for each entry. But this is cumbersome. So I went and wrote a small script (download here). For security reasons the extension of the script is set to .sh.txt instead of .sh, so you will need to rename it before execution.

There are a few conditions for the script to work:

  • Effectively the script will call “mount /my/mountpoint” for each relevant entry in /etc/fstab. All configuration details for the actual mount operation need to be provided in /etc/fstab. So before running the script, make sure those manual calls go through without problems.
  • For CIFS mounts the credentials should be provided using the “credentials” option from mount.cifs (see respective man page for details). Those can then be protected properly. And should there be several people using the system, they can maintain their passwords by themselves.
  • Currently there is connectivity check to the server(s). So you will get delays/errors when calling the script without a proper connection.

The script will detect whether a particular mount point is in use and skip the respective entry for mounting. The same goes for umount when the mount point is not used.

For CIFS mounts you will most likely need to call the script with sudo. At least on Fedora 12 I could not get mount.cifs to work for normal users, event with the SETUID bit set.

Git Links

Here is a number of links to resources I found useful.

OpenSolaris Upgrade

I just upgraded my OpenSolaris box from 2009.06 (snv_111b) to the latest development build (snv_131 from January 14th, 2010). The main reason was the broken CIFS server that comes with 2009.06. And although the next release is pretty much around the corner (2010.02), I needed the beast in a working state now.

The good thing about such upgrades is that thanks to ZFS and its snapshot functionality, a roll-back is trivial. The package manager will automatically create the snapshot and also create a new entry in the Grub menu for the upgraded OS. So if things go badly wrong, you can just select the old version and continue from there.

The upgrade itself is a three-step process

  • Change repository URL from release to development builds
  • Upgrade package manager
  • Upgrade other packages

The URL change for the repository is done with the following command
$ pfexec pkg set-publisher -O http://pkg.opensolaris.org/dev opensolaris.org
The update of the package manager is done like that
$ pfexec pkg install SUNWipkg
To start the upgrade from a shell, run this command
$ pfexec pkg image-update
If you want the GUI version, please execute
$ gksu packagemanagerand press the “Update All” button.

Part of the upgrade (bug or feature?) is that the permissions of /dev/ptmx will be changed. The result is that shell access via an SSH login is not possible any more. Since my box is headless, SSH is the only way to work on it. So it’s a real problem. Fortunately, it is only the shell access that is blocked (and this not only via SSH but universally) and giving the error message:
PTY allocation request failed on channel 0

The execution of commands via SSH is still working. So you just have to issue the following command:
$ ssh -l user host "pfexec chmod 666 /dev/ptmx"

which will fix the problem. It seems that for people working directly on the machine, the solution is to blindly enter the command from above.

“How Open Source Projects Survive Poisonous People”

Interesting video from Google Tech Talks. It not only covers what you might anticipate from the title. Instead a good portion is actually about how to run an open source project in the first place.

Using Git without Shell Access

If you want to host a Git repository there should normally be no shell access for all the people that need access to the repositories. So far many people have used gitosis to achieve this. Now there is a “new kid on the block”, called gitolite. I have not really used it in practice so far, but the added functionality looks promising and I also like the fact that it’s written in Perl. There is also a chapter about it in the Pro Git book.

Mount Windows Share with Fedora 12

The file system type for the mount command has been changed from smbfs to cifs. So using “-t smbfs” will not work any more. Instead you need to enter something like this:

mount -t cifs -o username=user[,uid=500,gid=500] //machine/share[/path] /mountpoint

There seems to be a documentation bug in the man pages, because they still mention smbfs.

Building a NAS with OpenSolaris and ZFS

A few weeks ago I have finally made the big step and built my own NAS with OpenSolaris using the ZFS file system. The main reason for this was the hassle of having multiple USB hard disks for redundancy purposes and also the relatively poor performance of USB 2.0. In the course of the decision process I had looked into the NAS systems that are available off-the-shelve. But they were either too slow or too expensive. If you want proper speed and resilience against hardware failure you pretty soon end up with systems that are 500+ Euros without any hard disk. And those are still somewhat inflexible. Of course they have the advantage of relatively low power consumption and special-purpose cases with mostly hot-swappable drives. In my case neither was relevant because I am the only person using it and the system is running only when my main machine is up.

When looking into the possible solutions I quickly realized that I want ZFS. It’s reliable, easy to set up, has fantastic features and does not require a hardware RAID controller (which alone saves you about half a grand). There is a ZFS module available for Linux but at the time people seemed to agree that it was in its infancy (relatively speaking). So it was down to FreeNAS and OpenSolaris. There were performance issues with FreeNAS (a known problem at the time), so I decided to go for the reference implementation, which of course meant Solaris. Interestingly, FreeNAS has recently announced to switch to Debian instead of FreeBSD and will therefore cease to support ZFS in the future.

There is bunch of great stuff on the net for exactly my use-case and I recommend you use your favourite search engine to find out more about the details. The one place that really stuck out was and still is Simon Breden’s blog. Make sure to check it out!

For those interested here is my setup:

  • 5* 1 TB WD Green configured as RAID-Z2 (equiv. RAID-6) with 2.8 TB net capacity
  • Asus M3A78-CM with AMD Athlon II X2 250
  • 8 GB ECC RAM
  • Intel® PRO/1000 GT Desktop Adapter (OpenSolaris did not work reliably with the built-in NIC)
  • Case Tacens Prior M

I have not really done any performance tuning so far and get around 55 MB/sec using CIFS server (not Samba). Neither have I used FTP or NFS so far (solely for lack of time). For those that have only worked with Linux before, be advised that Solaris is sufficiently different to greatly increase the time needed to set up things. Nevertheless I think it was the right decision and gives me a pretty flexible system.

The Non-SOA View on SOA: Part 1

There is a bunch of folks out there that don’t like SOA (Service-Oriented Architecture) for various reasons. So I try to look at things without all the buzz and distill out a few aspects that in my view provide value. The goal is to provide an alternative perspective that is hopefully hype-free.

I want to split this in two parts: First (in this post) comes the conceptual side of things, which looks at what SOA in my view is about at its core. Second are the more practical benefits we get from the way SOA is approached in real life. This will be a separate post.

Concepts

As a “preface” let me point out that I look at things mostly from a technical perspective. So everything along the lines of “SOA is about business and not technology” gets pretty much ignored. There are two reasons for that: Firstly I am a technical person and more interested in the technical aspects. Secondly, the business argument is probably the one most resented by skeptics. So let’s get started …

The core thing about SOA is that all functionality is made available as a service (surprise, surprise). This is really trivial and hardly worth mentioning. However, it has far-reaching consequences once you dig a bit deeper. And it’s those secondary aspects that provide the advancements.

  • The right level of slicing up things: Of course there were many other approaches and technologies before (e.g. OO and  CORBA). However, none of those has kept its promise. Admittedly it remains to be seen to what extent SOA can fulfill expectations. On the other hand, those expectations are still in flux as we all continue to improve our understanding. So there is a chance that expectations and reality will actually meet some time (where?). Also, the criticism I am aware of is not about this aspect. In fact it seems pretty much undisputed, at least I have never heard anything from customers or prospects.
  • The service is the application: A direct consequence of the previous point is that the service gets elevated to the place that was formerly held by an entire application; at least from a user’s perspective. Whether the implementation reflects this or not is irrelevant to the consumer.
    For new development it is usually desirable that the internals match the services exposed to the outside. For “legacy” stuff that just gets a service interface, the wrapper logic takes this place. In either case, however, the exposed logic is much smaller than an entire application.
  • State management: There has been a lot of talk about loose coupling. This principle can be applied at many levels, transport protocol and data format being the obvious ones. A slightly more subtle place is the handling of state, which pretty much depends on the aforementioned transport protocol and data format.
    The interface, at least of the initial operation, must be exposed in a way that it can just be called.  In other words it is kind of stateless. Of course from that point on everything could be state-full. There is no difference between SOA and traditional applications here. In reality, however, people pay much more attention to keeping things stateless when working in a SOA context.
  • Life cycle management: This one is strongly related to the point “the service is the application”. The notion of a life cycle has formerly mostly been on the level of entire applications. The obvious disadvantage is that this huge scope almost demands big bang-approaches for new versions. With the effort and costs associated for bringing a new version into production, there are only very few such releases and much stuff gets crammed into it. The risks increase, business agility is close to zero and nobody is really happy. With a service as the “unit to be delivered”, the scope is drastically reduced and so are things like risk, complexity etc. Thus I think the concept will be brought to much more practical relevance than before.
  • Standards: Although a real SOA can be built using an entirely proprietary technology, in reality a set of standards has established itself. Yes, there are probably to many WS-* things around that only work together in particular versions. So I would not recommend to jump onto too many things. But with the basic things you are certainly on the right track.

There is probably a whole lot more to say, but let’s keep it here for now. I am looking forward to any comments on the subject!