As far as routing goes, it is a very bad idea to place multiple network interfaces with IPs of a single network (subnet). The routing table, which decides which interface to route the data through, reads the table line by line, thus – all traffic goes through a single interface of the batch (-> of the interfaces “living” on the same network).
Assume a server has two network interfaces as follows:
eth0 : 192.168.0.1/24
eth1 : 192.168.10.1/24
Let’s assume these interfaces reside on the different VLANs. Lets assume they were connected incorrectly, in such a way that eth0 is connected to VLAN 10, which servers 192.168.10.0/24 and eth1 is connected to VLAN 2, which serves 192.168.0.0/24.
You would expect that queries by other hosts on VLAN 2 (which is connected to eth1, but serves 192.168.0.0/24!) would not get responses from the server. You are wrong.
kSar is one of the best tools to display an easy chart based on your system’s SAR metrics collection. kSar has not been developed since sometime in 2013, and it lacks the ability to parse modern Linux sysstat charts, leaving us with some missing info. However – some good soul has taken kSar and forked in into development. This tool can be found here, and it’s working nicely with RHEL6 metrics, as far as I could see. Grab it if you haven’t already!
I have created an RPM package, and SRPM package, which I will share here, for smartmontools version 6.5 on RHEL 6. Note that the official version is 5.43 which is clueless with many modern SSD disks. I have yet to test it correctly, and in general – use at your own risk.
Below I will supply a small shell script to run on a XenServer node which will create a file /tmp/storage_map.csv you can import into Excel or LibreOffice without problems. The delimiter is comma (,).
echo "VM Name,VM UUID,VM Power Status,Disk Name,Disk UUID,Disk size in GB,SR Name" > /tmp/storage_map.csv
for i in `xe vm-list --minimal is-control-domain=false | tr , ' '`
VM_NAME="`xe vm-param-get uuid=$VM_UUID param-name=name-label`"
Recently, with the release of a new 8.x version of Dell SrvAdmin tools, the Centos/RHEL5 (and XenServer, by the way) repositories disappeared. It appears that Dell will not support the RHEL5.x brands anymore.
The proper solution is to install the last 7 SrvAdmin tools, as can be shown in this site.
This solves the problem for XenServer as well.
When installing Oracle RAC (or in its more modern name – GI) version 22.214.171.124 and above, you can use Oracle ASM DiskGroup as your CRS+Voting file location.
It is fairly simple changing the disk membership in Oracle ASM DiskGroup, however, when you face some unknown bugs which prevent you from doing just that, or when you are required to modify the ASM DiskGroup on which the CRS+Voting files are placed, the article below is the one for you. You would have to remember, in addition, the ASM spfile.
Let’s say you have old shelves of either EMC or NetApp with SAS or SATA disks in them. And let’s say you want to connect them via FC to a Linux machine and have some nice ZFS machine/cluster, or whatever else. There are few things to know, and to attend in order for it to work.
While searching the web for how to perform PCI-Passthrough on XenServers, we mostly get info about previous versions. Since I have just completed setting up PCI-Passthrough on XenServer version 6. 5 (with recent update 8, just to give you some notion of the exact time frame), I am sharing it here.
I wanted to write down a guide for RHCS on RHEL/Centos6 and XenServer.
If you want to do that, you need to go through two major challenges which you will encounter. I want to save on the search and sum it all up together here.
When using Ricci as the engine for ‘cman_tool version -r’ command, you will experience timeouts (and practically – you will be unable to use ricci to update the cluster configuration across the nodes) when the ricci user password contains XML-sensitive characters, like <>&, etc.
As they say – FYI
I was wandering for a long while about the connection between NetApp’s LUN Serial and the identifier the host sees, aka “Word 83″. There was an obvious connection, but I figured it out only today.
The LUN Serial is an ASCII representation of the hexadecimal Word 83, or, to be exact, the last 22 hex characters of it.
lun serial /vol/volume/qtree/lun
When querying the multipath device represented there, we get:
Following some unknown problems, I had recently several XenServer machines (different clusters, different sites and customers, and even different versions) with a VDI-END-of-File issues. It means that while you can start the VM correctly, perform XenMotion to another server you are unable to do any storage-migration task – neither Storage XenMotion, nor VDI copy or VM-move commands. In some cases, snapshots taken from the “ill” disks were misbehaving just the same. This is rather frustrating, because the way to solve it is by cloning the disk into a new one, and your hands are bound.
This is a very nice project I have been working on. The hardware at hand - two servers, with a shared SAS bus containing several SAS disks. Since it's a shared bus, no RAID solution would cut it, and as I don't want to waste disks with ASM ("normal" redundancy meaning half the size...), I went to ZFS storage.
ZFS is a wonderful technology, with many advantages, but with some dangerous pitfalls. As I prefer Linux, I did not bother with any Sloaris solutions, and went directly to Centos 6. I will describe my cluster setup below.
Well, tricks is not the right word to describe advanced shell scripting usage, however, it does make some sense. These two topics are relevant to Bash version 4.0 and above, which is common for all modern-enough Linux distributions. Yours probably.
These ‘tricks’ are for advanced Bash scripting, and will assume you know how to handle the other advanced Bash topics. I will not instruct the basics here.
Trick #1 – redirected variable
What it means is the following.
This post will describe the process of placing SSH keys using the internal ‘systemshell’ command of NetApp. As always – when doing something which the vendor did not intend you to do, do it very carefully. This data was obtained from NetApp forums, and while I do not have the original post to link (I usually link to the original, as a courtesy to the original author), this is the content, as is.
First, set to advanced mode:
filer> priv set advanced
Then, unlock and set a password to diag account:
filer*> useradmin diaguser unlock
I love XenServer. I love the product, I believe it to be a very good answer for SMBs, and enterprises. It lacks on external support, true, but the price tag for many of the ‘external capabilities’ on VMware, for instance, are very high, so many SMBs, especially, learn to live without them. XenServer gives a nice pack of features, at a very reasonable price.
The following procedure was tested by me, and was found to be working. The version of the XenServer I am using in this particular case is 6.1, however, I belive that this method is generic enough so that it could work for every version of XS, assuming you're using iSCSI and LVM (aka - not NetApp, CSLG, NFS and the likes). It might act as a general guideline for fiber channel communication, but this was not tested by me, and thus - I have no idea how it will work.
In particular – Oracle UEK, which “claims” to be 2.6.39-xxx, but is actually 3.0.x with a lower version number. Several misbehaviors (or differences) of version 3 can be found. One of them is related to BackupExec. The service would not start on OEL6 with UEK kernels. The cause of it is an incorrect use of a function – getIfAddrs. Everything can be seen in this amazing post.
When using ‘disk show -v’ on a NetApp filer version 7.3.x, following replacement or addition of disk(s), you might see the above mentioned message. It is caused by incorrect disk label – of OnTap version 8, on an OnTap version 7.3.x system. The system cannot handle the incorrect label, and thus – ignores the disk.
SABnzbd is a nice tool. I just replaced my previous nzbget with it, due to its better handling of the obfuscated names in usenet groups. However, on an Atom CPU, the max download speeds did not go over ~5MB/s on a 100Mb/s link. This is rather sad, because nzbget did get the whole ~11MB/s speeds.
It is an amazing news to me. I really love XenServer. I think that Citrix were able to make a good use of Linux mechanisms for the purposes of virtualization, without abusing the OS layer (like some of the other virtualization solutions did). The file locations are decent (for example – most parts are located in /opt, which is the right place for it to be at), and in general, it always felt to me as if Citrix developers (and the original XenSource developers) had respect for the OS. I liked it, and still do.
Due to a major disk crash, I have had to use my ‘other’ computer for VPN connections. It meant that I have had to prepare it for the operation. I attempted to login to aJuniper-based SSL-VPN connection, however, I did get a message saying that my 64bit Java was inadequate. I had a link, as part of the error message to Juniper KB, to which I must link (remembering how I have had to search for possible solutions in the past).
Today there will be a few different posts. This is a day full of events, so…
My first – to allow tsclient to work under Ubuntu 12.04, you should follow this guide: http://superuser.com/a/547102
To sum it up:
I have been using the excellent template and scripts from this forum post, however, when the NetApp device is loaded with LUNs and volumes, the script will cause the Cacti to timeout, and during that time, consume CPU. The original cause of this problem was a workaround to some NetApp Perl API bug the original author found, which forced him to query the entire data set for each sub-query.
When Connecting Linux to a large array of SAS disks (JBOD), udev creates default persistent names in /dev/disk/by-* . These names are based on LUN ID (all disks take lun0 by default), and by path, which includes, for a pure SAS bus – the PWWN of the disks. It means that an example to such naming would be like this (slightly trimmed for ease of view):
I have been implementing Nas4Free recently, and found this system to be a very nice one. I might try to port its web interface to Linux, as it completes a set of requirements (regarding graphic interface) I do not find in Linux, and wish I could…
However, I have had to add a driver for ConnectX 10GbE interface, which, unfortunately, was not included.
This might show as a simple task, however, for a person unfamiliar with FreeBSD, it was a challenge.
ndmpcopy is a wonderful command. It allows a fine-grained copy of files or directories between NetApp devices, across network, even if they do not use (or unlicensed) SnapMirror, SnapVault and the rest of the Snap* products NetApp offer.
In this example I will show how to copy a LUN from one filer to the other.
First, set the LUN to offline on the source filer. Make sure that it is not mounted, disconnected, etc – whatever prevents any major data loss. As you can deduce – setting a LUN to offline state will prevent write access to it. Also – take its parameters. For example:
One of the worst things you can have in XenServer, is some wize-guy performing a ‘forget storage’ on a storage device still holding virtual disks related to VMs. As XenServer database is internal (for the whole pool) and not per-VM, all references to this virtual disks disappear, and you remain with bunch of VMs without disks, and later on, when the recovered from the shock and restored the SR, with a bunch of virtual disks you have no clue as to where they belong. Why?
I have had a single node of a pool using a different license server. Temporary, unfortunately. It has expired, and as the purchase process was somewhat prolonged, I have had to extend it. I did not want to disconnect my other-four-hosts-pool from the permanent license server, which works so fine for the last year or so, so I have had to change the license only for a single host.
XenServer 6.1 XenCenter does not allow changing the license server for a single host in a licensed pool. I have had to search for a solution. The solution looks like this:
I have had recently the pleasure and challenge of setting up VPN server for mobile devices on top of Linux. the common method to do so would be by using IPSec + L2TP, as these are to more common methods mobile devices allow, and it should work quite fine with other types of clients (although I did not test it) like Linux, Windows and Mac.
I have decided to use PSK (Pre Shared Key) due to its relative simplicity when handling multiple clients (compared to managing certificate per-device), and its relative simplicity of setup.
Using the GUI, it could be somewhat complex identifying a VM based on its MAC address. There are several solutions on the network using PowerShell, but I will demonstrate it using a simple bash script, below. Save, make executable, and run.
I find it that identifying a missing something in the fridge, and adding it to a list does not work well for me. It’s either that I take a mental note of the missing groceries, and then, almost immediately, forget them until the unpacking of the just-purchased groceries, back home, several days later, or that I actually move myself into writing it down on a note, placed on the fridge, and then, of course, forget to take the note with me to the supermarket. Not working.
The concept of running a virtual machine, KVM-based, in this case, under RHCS is acceptable and reasonable. The interesting part is that the directive replaces the directive and acts as a high-level directive for VMs. This allows for things which cannot be performed with regular 'service', such as live migration. There are probably more, but this is not the current issue.
In my post here, I have explained (actually – created a shell script) to map USB disks to VMs directly. While this is easy and simple, it becomes more challenging when you want to map internal SATA disks. They are not attached to the “Removable Storage” SR, and thus, behave differently.
There is a very nice script for Windows dealing with attaching XenServer USB disk to a guest. It can be found here.
This script has several problems, as I see it. The first – this is a Windows batch script, which is a very limited language, and it can handle only a single VDI disk in the SR group called “Removable Storage”.
In the past I hacked around a problem with the order of starting (and with several bugs) a network stack combined of network bonding (teaming) + VLAN tagging, and then with network bridging (aka – Xen bridges). This kind of setup is very useful for introducing VLAN networks to guest VMs. This works well on Xen (community, Server), however, on RHEL/Centos 5 versions, the startup scripts (ifup and ifup-eth) are buggy, and do not handle this operation correctly.
This is a tricky one. I have had to waste a lot of time. The trick is to use 32bit browser (extract firefox into $HOME/firefox and run it from there. Details below), and a full JDK package from Sun. JRE is not good enough!
I have a $HOME/bin/firefox32 script which looks like this:
Unlike previous versions, VMs do not have a visible property in the GUI allowing autostart. This has been claimed to collide with the HA function of the licensed version. While I believe there is a more elegant way of doing that (like – ignoring this property if HA is enabled), the following method can allow your free XenServer allow autostart of VMs:
xe pool-param-set uuid=UUID other-config:auto_poweron=true
xe vm-param-set uuid=UUID other-config:auto_poweron=true
DRBD is a low-cost shared-SAN-like solution, which has several great benefits, among which are no single point of failure, and very low cost (local storage and network cable). Its main disadvantages are in the need to constantly monitor it, and make sure it does what’s expected. Also – in some cases – performance might be affected greatly.
As my work takes me to various places (where technology is concerned), I collect lots of browser tab of things I want to keep for later reference.
I have to admit, sadly, that I lack the time to sort them out, to make a real good and nice post about them. I do not want to lose them, however, so I am posting now those which I find or found in the past as more useful to me. I might expand either of them one day into a full post, or elaborate further on them. Either or none. For now – let’s clean up some tab space:
This post is for the users of the great dm-multipath system in Linux, who encounter a major availability problem when attempting a resize of mpath devices (and their partitions), and find themselves scheduling a reboot.
This documented is based on a document created by IBM called "Hot Resize Multipath Storage Volume on Linux with SVC", and its contents are good for any other storage. However - it does not cover the procedure required in case of a partition on the mpath device (for example - mpath1p1 device).
I have encountered several methods of doing stateless RedHat Linux systems. Some of them are over-sophisticated, and it doesn’t work. Some of them are too old, and you either have to fix half the scripts, or give up (which I did, BTW), and after long period of attempts, I have found my simple-yet-working-well-enough solution. It goes like that (drums please!)