I always wanted to build a cluster and now that I have a few projects that could make use of a cluster, I'm started the process.

Half the fun is being clever with hostnames, right?

The domain for my network is vay8, but the clusters will get their own subdomain, zodiac.vay8.

Each node will be named for an animal from the Chinese zodiac.

While I like the idea of running stable, enterprise software, I don't like the idea of running 10+ year old code. I have not ruled out Scientific Linux as a possibility, but right now I'm leaning towards archlinux.

Either way, I'll probably also do docker containers to allow each project to have its own dedicated environment without contaminating the servers themselves, plus it makes it easier to distributed an environment to each node. I can probably make a read-only docker image and stash it on the NAS for each node to use as needed.

• Will utilize MOSIX for automated process load balancng: http://www.mosix.cs.huji.ac.il/ Can't get MOSIX to work after a dozen too many hoops
• SSH keys will let each cluster run arbitrary commands to other bits of the cluster
• rsync (or NFS + bind mounts) will be used to keep system configs (/etc/passwd,shadow,hosts et al) in sync
• NFS, via autofs, will provide shared, common storage (may be problematic as the NAS is pretty low-performance)
• Infiniband will provide PtP Ethernet connections between nodes

Custom (tcl) programs will be (re)written using knottcl which itself uses MPI to implement a multiprocessing-like package/library for tcl.

mpi can be used as a job dispatcher. Knottcl can act as an orchestration layer so things like aircrack-ng and john the ripper can be ran in a distributed fashion (via split dictionaries).

OpenCL will be leveraged so additional processing power can be harnessed from GPUs. PCI and PCIe graphics cards can fill the extra slots for low-cost additions.

There's even a compatibility layer for OpenCL that lets OpenCL programs run directly on an x86_64 cpu–so code only has to be written fro one arch!