Linux/Windows Troubleshooting part 2

Network troubleshooting part 2:

The next article is about some basic DNS troubleshooting.
First we will do it on Linux with dig command, then we will check out nslookup on Windows too.

dig any @

This command will check domain at google’s DNS server(@ and will ask for all available records (any) on this domain.
I have highlighted every important parts of this command. All in all 7 records been found as you can see on this picture above:



You can change the server easily with the @ part. You can put your own DNS server if you want to check your updated local DNS server.
The fully DNS zone propagation(update) theoretically takes 2 days, but usually enough few hours to get updated nearly everywhere.
If you completely lose the @server-IP-address then dig will use the current DNS server address from /etc/resolv.conf.
For example:

dig any

To check only the MX records for the domain change the any to mx like this:

dig mx


The next one is how to check the reverse record for the domain.

dig -x

As you can see in the answer section the command found the reverse record for the domain which is


So let’s take a look at this with Windows nslookup:



set type=any



You can see that in the answer parts all the nameserver addresses and A records are there, also both MX records have been presented.
To check only MX records then you could easily change the type to mx, like this:

set type=mx

You will get only the MX records result from the server:



Windows Update troubles:

I was just updating few servers at my workplace remotely at the datacenter and 1 of them didn’t reboot properly.

So the issue was this:

– Server updated with new service packs.
– Reboot has been processed and started via RDP(remote desktop).
– The RDP can’t be reachable anymore, because that service has been shut down already and connections has been shut down.
– Server still pingable.
– No any other way to reach the server anymore. (IPMI/KVM/DRAC)


– Go to datacenter and restart the server manually. On Saturday is not a good fun, let’s be honest
– Phone up the datacenter to ask for remote hand… Takes ages to explain everything, server number, rack location etc…
– Download PsTools from here: and kill the winlogon process which stuck on the server.

Extract PsTools and first try this command:

psexec \\REMOTE_SERVER_NAME shutdown /r /t 0

This will try to execute shutdown command on the remote box and restart the server. The /r means reboot the /t switch is the time which is zero.
If this wont help for some reason then you could try to use the pskill.exe command.

pskill [-t] [\\computer [-u username [-p password]]] <process ID | name>

pskill \\ -u mydomain\Administrator -p mylovelypassword Winlogon

This should work and you wont need to go to datacenter neither to phone them up and asking for the reboot.
You can monitor the server with ping command and you will see when the server really reboots, because you will lose ping from it.

This one saved me so many times on my weekends, when I usually make Windows updates. ( Just like right now:) )
Weekdays you can’t really do Windows updates on corporate servers, because they are heavily used by users, so reboot is not a good idea that time.

Next issue will be posted shortly…

Linux/Windows Troubleshooting part 1

Network troubleshooting part 1:

Checking open ports on box:

netstat -natlp

It will tell you the locally open ports with the running daemon name also.
As you can see on the picture the first red cubic is the named daemon DNS name server is listening on the local port. The second red cubic is the ( ssh daemon which is the remote session terminal.You can see the local and the remote address on these picture.

If you want to check all the listening udp and tcp ports then you need to add and extra u at the switches which will provide the udp connections also.

netstat -natlpu



On this picture you can clearly see the Asterisk and Named servers are listening on 5060 and 53 ports.

So if you want to check any running daemon or application on your box just issue any of those commands and you will see if they are listening on the local ports or not. This is the easiest way to figure this out.
If you stop for example smtp daemon or imap daemon, then the port(s) will disappear straight away.

Netstat is available by default in any major Linux distro also available in every old and new Windows.
Under Windows you need much less switches:

netstat -an -p tcp


netstat -an -p udp


The first command will show you all listening, connected and waiting tcp connections.
If you have so many wait/close connections then probably someone is attacking your box remotely.
I have seen it on Exchange server 2010, it had more then 100 wait/close connection against port 110, because someone tried  to brake into that server with brute force attack. The sysadmin didn’t lover down the maximum available connection per IP, it was on default which is 2000.


The next command will show you all the listening UDP connections. If you have a DNS server or an Asterisk SIP server they will listen on UDP and you will see those ports up and open.


My other favorite command on Linux and Windows is the nmap. This application can scan local and remote IP addresses or domain names to show you the open/listening ports. Also you can trace your whole local or remote network which IP addresses are used or which port(s) listening/open. By default nmap is not installed on any major Linux distro, so you need to install it.


yum install nmap

On Debian/Ubuntu:

apt-get install nmap

On Windows:

Few examples for port scans:

nmap localhost

This will show you all locally listening applications/daemons on your box:


 nmap IP_address or domain name

This command will show you all the listening ports on the host that you check. (IP or name)


Nmap local network scan to discover up and running hosts on your local network:

nmap -sP


Also Nmap can do remote IP address scan. Be careful with this, just only do it for testing /troubleshooting purpose not for fun!
Port scans usually logged by firewalls/servers and your IP address will be logged, so you can be traced if you did something.

nmap -sP



Next troubleshooting tools is traceroute/tracert:

Linux: traceroute, Windows: tracert.

Lets do a traceroute first on Linux box:




I used capital I and n to force traceroute to use icmp protocol for tracing and n to avoid name resolution.
As you can see on the picture the jump via routers showing the whole route how the package reach my server.
The last address is my server address at Hetzner. This is the best way to figure out for example how the packets reaching your server.
If you have multiple path like two ISP broadband, then how you will know that what way the packets getting into your box?
Also when you are testing firewalls this and something is not correct, then you can check with this if the packets getting on the appropriate way.
For example you are connected into a VPN and in the meantime you are testing a firewall and the remote VPN default gateway address has been used.
So every packet will travel via VPN instead of your local network gateway (if the VPN’s configured that way) and you got lost with the firewall troubleshooting.

On Windows you can use this tool also:

tracert -d

The -d switch is compulsory I would say otherwise every IP address will be resolved into names and takes ages to get back the results.
I guess you don’t really want to wait, so use all the time -d switch  otherwise takes too long the whole tracing process.



Back to Linux again.

How to check the running processes under Linux box?
You have the ps tool which can list every background processes in your running system.
So try to run and check the output, you will see many processes in the screen.

ps  ufxa

This command will show you the background process name including the directory which is it running from.
Also it will show you the user name who runs the actual process and also the process’s memory and cpu usage will be shown by this command.


For example check the mysql daemon. The third red cube shows the 2.9 which is the percentage of the current memory usage by mysql daemon.


Next command is the mtr that we will take a look more closely.
Mtr is stands for my traceroute which is a real time traceroute application.
This is not installed by default on any distro you need to do it from repository.


yum install mtr


apt-get install mtr

So let’s do some mtr tracerouting:


This will traceroute the whole route till
As you can see there are some packet losses at but not much.
The other route path look clean and fine, no packets loss whatsoever.


The next mtr trace is to google’s public DNS server address:



To be continued…

ESXi backup free solution from Thinware

Hi everybody,

It’s been a long time with out any new post. I’ve been quiet busy with my family and with my new job in the last few months. 🙂

Anyway let’s get going and hit the wall.

I just managed to create a new totally free and automated backup solution for VMware ESXi server which is also free by the way.
So I’m posting my scripts and researches regarding to this solution and share with you guys.

Let’s assume you have your VMware ESXi server up and running with installed free license on it. (unlimited time, but limited features)
If you have this setup then you can’t use for example veeambackup as a free backup/restore solution anymore because the license limit these kind of features. So the solution for this is the Thinware vbackup application. You can download this application from here: You need to login in to do this although.  (right side / login)
So download and install this application and set it up for remote SSH VMware backup. Also you need to download and install VMware virtual disk development kit version: VMware-vix-disklib-5.1.0-774844.i386.exe and VMware Vcenter converter stand alone version: VMware-converter-all-5.0.1-875114.exe. You must use these versions because the latest one wont work properly and you either way wont be able to backup or neither restore the virtual machines properly!  These were discussed on Thinware forum more briefly look it up if you want.

So after you installed Thinware and Vcenter converter and virtual disk development kit is time to set up the settings in Thinware.
Go to tools/settings in Thinware and set the disk development kit and Vcenter converter’s directory.



If these settings not been setup then the backup/restore wont work. It will complain straight away regarding to these missing settings.
So it’s time to browse up the ESXi server and set up the hosts also.



Only need to fill up the name which can be an IP address if your server has not been added to local DNS server.
And also put your ESXi server username(root) and password.

In the next window you will see the hosts from the ESXi server, so you can add all of them or whichever you want to make backup of it.
Next one create the actual backup job. Add a name of like backup and choose from the drop down type menu Backup-Image-SSH.



Then choose the backup root drive where the host(s) will be backed up.



Then finish the wizard and right click on the backup job and execute it. Choose the debug brake point after option when it asks for it.
You will have a new terminal window where you can see the actual logs of the full backup process like: Creating snapshot, converting the disks etc…



You can schedule this backup with Windows scheduler, the current Thinware does not support this yet.

This is the script for the Windows scheduler to run the backup every week or day or whatever time schedule you would need:


cd C:
\Program Files (x86)\Thinware\vBackup

vBackup.exe -v Nagios -j backup
date /t >> C:\Users\Administrator\Desktop\backup-check.txt
time /t >> C:\Users\Administrator\Desktop\backup-check.txt
eventcreate /T Success /L APPLICATION /ID 100 /D “VMware backup was successfull”


The next post will be about how to restore host with this backup system.
Enjoy it! 🙂


DNS tools for sysadmins. Coming soon: SPAM database check.

– Nmap firewall check

– Nslookup domain check. (All records shown)

– Traceroute, tracing packets route

– Reverse DNS check

– Ping check

– Whois tool domain checker

–  Password generator

– Send test mail via mail server  user: admin password: admin

Scanning logged… 

This tools for checking, please do not use it for abuse! 🙂

Sendmail server setup

Sendmail mail server setup and configuration step by step.
Let’s start and setup our first sendmail server.

First thing to do is install the packages that we need.

yum install sendmail sendmail-cf

Then edit the file and make those changes above.

Find the line:

dnl define(`confAUTH_OPTIONS’, `A p’)dnl

and change it to:

define(`confAUTH_OPTIONS’, `A p’)dnl

Then change those lines above. This is need to get outlook express and Mozilla to work with plain text authentication.
Use it only in a secure local network, otherwise use a different authentication mechanism for example ssl.

From this:


To this:


Next step is change the queue parts for fine tuning from these:

dnl define(`confTO_QUEUEWARN’, `4h’)dnl
dnl define(`confTO_QUEUERETURN’, `5d’)dnl
dnl define(`confQUEUE_LA’, `12′)dnl
dnl define(`confREFUSE_LA’, `18′)dnl

To these:

define(`confTO_QUEUEWARN’, `4h’)dnl
define(`confTO_QUEUERETURN’, `5d’)dnl
define(`confQUEUE_LA’, `12′)dnl
define(`confREFUSE_LA’, `18′)dnl

Next thing is the maximum children numbers:

dnl define(`confMAX_DAEMON_CHILDREN’, `20′)dnl


define(`confMAX_DAEMON_CHILDREN’, `20′)dnl

Then the maximum connection number per IP address from this:

dnl define(`confCONNECTION_RATE_THROTTLE’, `3′)dnl

To this:

define(`confCONNECTION_RATE_THROTTLE’, `3′)dnl

Next is the local daemon copy the line and insert your server’s IP address:

DAEMON_OPTIONS(`Port=smtp,Addr=, Name=MTA’)dnl
DAEMON_OPTIONS(`Port=smtp,Addr=, Name=MTA’)dnl

If you are using public IP address then put that instead of the private one.
If you want to get IPv6 support then uncomment this line:

dnl DAEMON_OPTIONS(`port=smtp,Addr=::1, Name=MTA-v6, Family=inet6′)dnl

To this:

DAEMON_OPTIONS(`port=smtp,Addr=::1, Name=MTA-v6, Family=inet6′)dnl

Obvious the Address part should be filled properly.

Next find the line:

dnl FEATURE(`relay_based_on_MX’)dnl

and change this to:


If your server runs on a DSL or cable connection then you need to change the smart host part too:

define(`SMART_HOST’, `’)dnl

Obvious change the part to your provider. Then your server will be able to communicate through smtp protocol.
Save the file, this part is done.

Next thing to do is edit the access file and insert your server’s IP address like this:

Connect:            RELAY
Connect:192.168.0                   RELAY

Change the addresses to your server’s IP address and for your local network address too.
The second line needs for the clients computers on your network to be able to relay with the server.
If you are using public IP address then put that IP instead of the private one.
This file enables the sendmail server to reject and accept domain(s) and IP addresses.

When this done you need to hash the access file with this command:

makemap  hash /etc/mail/access.db &lt; /etc/mail/access

Next thing to do is enter the domain name(s) that you have into the local-host-name file like this:

vi /etc/mail/local-host-names

Then save the file and close it.
Create a test user for this email account:

useradd –s /sbin/nologin test1

Change the password for the test1 user:

passwd test1

Then edit the aliases file under the /etc directory and put these line into it:

test1:         test1

Save and close it then issue the newaliases command:


/etc/aliases: 78 aliases, longest 10 bytes, 781 bytes total

After this you got a email address.
If you want more addresses for this account just edit the /etc/aliases file and save it.
Then issue the newaliases command to be accept the new lines from the aliases file.

We are finished with the sendmail part. Let’s compile the sendmail and start it.

make clean
make all
make restart

You can start any service in CentOS Linux with these commands:

service sendmail start


/etc/init.d/sendmail start

Check the sendmail service because it should be switched on otherwise the next reboot the servcie wont start with the chkconfig command:

chkconfig sendmail on

Be carefull with postfix, exim and any another mail servers. Only one mail server can run on the 25 smtp port. So check the service to be switched off with the command:

chkconfig postfix off
chkconfig exim off

And check the firewall smtp port that should be enabled. You can check this with iptbales command:

iptables -L

If it says “ACCEPT     tcp  —  anywhere             anywhere            state NEW tcp dpt:smtp” that is fine.

Last thing to do is change the server name for the appropriate one.
Go to /etc/sysconfig and edit the network file.

vi /etc/sysconfig/network

Change this:



That’s it we are done. After this you should restart the server otherwise the server name wont change.
Before that save every open files!!!
You can restart the server with this command:

shutdown -r now (-r means restart the box. If you put -h the server wont restart it will stay in halt state)

After you rebooted the server test the sendmail server with this command:

telnet localhost 25

Connected to localhost.localdomain (
Escape character is ‘^]’.
220 ESMTP Sendmail 8.13.8/8.13.8; Sun, 16 Jan 2011 19:14:03 GMT
helo me
250 Hello localhost.localdomain [], pleased to meet you
250 2.1.0… Sender ok
rcpt to:test1
250 2.1.5 test1… Recipient ok
354 Enter mail, end with “.” on a line by itself
250 2.0.0 p0GJE39C006422 Message accepted for delivery

To check the open ports on your box use the nmap command.

nmap localhost
nmap myexternalipaddress

Hints for DSL/Broadband connections:

If your server is behind a firewall don’t forget to forward the 25 (smtp) port into your box.
On few DSL line you might have a problem with the MTU size. If the mails stuck into the Linux box then change the MTU size to 1420 in your router.
The DSL line is not a real Ethernet network. It is a pppoe (&lt;a href=””>;/a>) line so the MTU size is smaller then a real Ethernet and it can cause trouble for sendmail.
So if you are having this kind of problem(mail can’t go out from the box)  try to change this in the router and in the Linux box too.
In the Linux box the MTU size at the /etc/sysconfig/networking/devices/ifcfg-eth0

You can download all the configuration files from here:




Next blog will be about the Dovecot POP3/IMAP server.

Linux Bonding Interfaces for High availability.

Well this article is for bonding Ethernet interfaces into one for high availability and performance improvement on your Linux box.
Bonding is important if you want to have a high available server. If one interface goes down you still have backup interfaces left.
The interfaces can be programmed as: Balance Round-Robin, Back-up slave, Balance-tlb, Balance-alb, Balance-xor, etc…

So there are the steps to get it work on RedHat, Fedora and CentOS based systems.

Create the ifcfg-bond0 file at /etc/sysconfig/network-scripts/

touch /etc/sysconfig/network-scripts/ifcfg-bond0

Edit the file and change the IP address to match for your needs.


Next step is to modify the interface cards configuration files.
cat /etc/sysconfig/network-scripts/ifcfg-eth0

eth0 should look like this:


Then change the eth1 as well.
cat /etc/sysconfig/network-scripts/ifcfg-eth1

eth1 should look like this:


If you want more device to bond just change the ifcfg file(s).

After you setup the interfaces you need to setup the kernel module parameters.

Add the following lines to /etc/modprobe.conf file.

alias bond0 bonding
options bonding mode=balance-alb miimon=100

Next thing is load the kernel module.

modprobe bonding

Then restart the network service.

service network restart

You are set!

To test the bonding devices, list them with this command:

cat /proc/net/bonding/bond0

You can change the mode at the modules.conf file to fit for your system.
Take a look at this link for more reference: and search for the “Bonding Driver Options” part to change the bonding mode.

VNC secure connect with putty. Part 2 “The client box”

Well if you already setup your Linux box you can start to setup the client side.
Let’s assume you are using Windows on the client side.

First of all download the latest putty.exe from here: PUTTY.EXE

After you started the application put your Linux box’s IP address into the “Host Name (or IP address)” box.

And put a name into the “Saved Sessions” box that you want to save for the later connections and save it.
It must be saved otherwise the next time you start the putty it will be lost and you need to refill every part of the configuration.


Next step is to go to the Connection/SSH/Tunnels tab and fill the source and destination ports.
We will forward the local 5900 port to the Linux box 5901 port.
The VNC communication will be encrypted through the SSH protocol.

Next step click on the X11 tab and tick the “Enable X11 forwarding

Next step click on the “Session” tab then save the session as you named it before!
I already mentioned it at the top of this guide.

Well we can check the connection so click the open tab and log into the Linux box through SSH.

After you logged in with the user that you added on your Linux box start the vncviewer on the client box.
If you don’t have VNC yet you can download it here: RealVNC

Then in the “Server tab” type localhost:5900 and click on the “ok” tab.
The VNC client will ask for the password that you typed into the Linux box so type it and click “ok
You should get an Xdesktop:

If your connection is refused by the Linux box check the firewall. The port 22 has to be opened.
As well as on the client box don’t install the vnc server because if you do the local vnc server will be forwarded to itself and it wont work.
So check that service to be sure it is not running on the client box.

VNC secure connect with putty. Part 1 “The Linux box”

This guide will show you how to setup a VNC server on your Linux box and how to connect to it with a ssh putty client.

The server to connect is a Linux CentOS box. And the client is a Windows box(version really does not matter in this case)

After you logged in into your Linux box with root account edit the /etc/sysconfig/vncserver file.
open it with your prefered editor and change these parameters:

VNCSERVERARGS[1]=”-geometry 1152×864 -depth 16 -nolisten tcp -nohttpd”

In the first line the 1 means the default port number + 1 ==>> it will run on port 5901.
The “ok” is the user name who will connect into the box. Change it to your user name that you already added into your Linux box.

Edit the file under the /root/.vnc/xstartup and make sure that your file looks like this one:


# Uncomment the following two lines for normal desktop:
exec /etc/X11/xinit/xinitrc

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
xterm -geometry 80×24+10+10 -ls -title “$VNCDESKTOP Desktop” &
twm &

If you use kde desktop change the last line “startx” to “startkde“.

Then copy the xstartup file into the home directory of the user

that you made to connect with into the Linux box.

In my case this is the ok user.
So copy the xstartup file from the /root/.vnc/xstartup to /home/ok/.vnc/ directory.

cp /root/.vnc/xstartup /home/ok/.vnc/

Don’t forget to change the user’s directory name into the user that you used!
If the user that you used called myvncuser for example it vill be look like

cp /root/.vnc/xstartup /home/myvncuser/.vnc/

Now we can start the vnc server.
service vncserver start
chkconfig vncserver on

Now check the service that is running. In our case the VNC server runs on port 5901.
So we check just that port with the netstat command.
netstat -a | grep 5901

If it says something like that:
tcp 0 0 *:5901 *:* LISTEN
That means the server is ready to server the clients.

By the way you can use this command to check the open ports on your Linux box:
nmap localhost
of course the is should be your Linux box’s IP address.

The next step is to setup the password for your VNC service.
Issue this command in your terminal:
Then type the password for the VNC server.

Well the next step to do is setup the firewall on your box.
Start the system-config-securitylevel or system-config-securitylevel-tui (this is the terminal version)
Make sure the SSH service is ticked as a trusted service. As well as add the VNC port 5901 if you want to use the VNC in your local network.
In this guide is not necessary to open the 5901 port because we will connect through secure SSH service. (port 22)

After you setup the firewall save it and restart it.
service iptables restart
Then check the ports that is accepted by the firewall with this command:
iptables -L
this will list the whole firewall rules.
An important thing with the firewall don’t use the iptables and the ip6tables together!

This is a RedHat recommendation!

Solution for this problem is that:
service ip6tables off
service iptables on

Then check them with this command:
chkconfig –list | grep tables
this will list the iptables and the ip6tables services too.
Make sure the iptables is on and the ip6tables is off.

The Linux box part is done by this.
To be continued with the next box. That will be the Windows box with putty ssh client.


SquidGuard + Squid proxy installer script with LDAP integration

SquidGuard + Squid proxy integration into Windows Active directory.

#Preinstall the requriements to work with LDAP

yum install -y flex bison openldap* gcc make

#ORACLE Berkeley DB. The 3.2.9 is the stable and tested for squidguard
#Don’t use higher or lower versions cause squidguard wont be stable or wont start at all


tar xzvf db-3.2.9.tar.gz

cd db-2.7.7

cd build_unix

#Building the Oracle BerkeleyDB for Linux
../dist/configure && make && make install


tar xzvf squidGuard-1.4.tar.gz

cd squidGuard-1.4

./configure –with-db=/usr/local/BerkeleyDB.3.2/ –with-ldap=yes && make && make install

# Blacklists

tar xzvf blacklists.tgz

mv blacklists /usr/localsquidGuard/db/

chown -R squid:squid /usr/local/squidGuard/*
chmod -R 740 /usr/local/squidGuard/db/
chmod -R 755 /usr/local/squidGuard/log/



# Change the ldapbinddn
# Ldapbindpass and the ldapusersearch part to fit for your configuration
# If you make a group in the AD named InternetAccessGroup and you put your users into it that users wont be filtered at all
# If you make a group named InternetAccessGroup2 and the “regular” user linked into it that users will be filtered as the rules say in the SquidGuard

To get it work make an organization unit in AD called myorg.

Make a group in myorg called InternetAccessGroup. (This group users wont be filtered at all)

Make a group called InternetAccessGroup2. (This group users will be filtered)

The users in the first group will reach the Internet with out any restrictions.
The second group can reach the Internet but will be filtered.



Iptables based Linux Firewall.

This firewall was posted on site.
It has based on Iptables and it has all the features that has to have a firewall.

Just few things to mention:

– Packet mangling
– Access controlling
– Port forwarding

You can download it from here:

The installation manual:

The configuration manual:

The advance features like access control, port forwarding:

Also one of my favourite iptable based firewall is:  CSF firewall .

Check out the status report file here:


Show Buttons
Hide Buttons