Archive for November, 2013

Linux/Windows Troubleshooting part 1

Network troubleshooting part 1:

Checking open ports on box:

netstat -natlp

It will tell you the locally open ports with the running daemon name also.
As you can see on the picture the first red cubic is the named daemon DNS name server is listening on the 127.0.0.1:53 local port. The second red cubic is the (78.46.184.202:22) ssh daemon which is the remote session terminal.You can see the local and the remote address on these picture.

netstat-1
If you want to check all the listening udp and tcp ports then you need to add and extra u at the switches which will provide the udp connections also.

netstat -natlpu

netstat-2

 

On this picture you can clearly see the Asterisk and Named servers are listening on 5060 and 53 ports.

So if you want to check any running daemon or application on your box just issue any of those commands and you will see if they are listening on the local ports or not. This is the easiest way to figure this out.
If you stop for example smtp daemon or imap daemon, then the port(s) will disappear straight away.

Netstat is available by default in any major Linux distro also available in every old and new Windows.
Under Windows you need much less switches:

netstat -an -p tcp

netstat-3

netstat -an -p udp

netstat-4

The first command will show you all listening, connected and waiting tcp connections.
If you have so many wait/close connections then probably someone is attacking your box remotely.
I have seen it on Exchange server 2010, it had more then 100 wait/close connection against port 110, because someone tried  to brake into that server with brute force attack. The sysadmin didn’t lover down the maximum available connection per IP, it was on default which is 2000.

#######################################################

The next command will show you all the listening UDP connections. If you have a DNS server or an Asterisk SIP server they will listen on UDP and you will see those ports up and open.

Nmap:

My other favorite command on Linux and Windows is the nmap. This application can scan local and remote IP addresses or domain names to show you the open/listening ports. Also you can trace your whole local or remote network which IP addresses are used or which port(s) listening/open. By default nmap is not installed on any major Linux distro, so you need to install it.

CentOS/RedHat/Fedora:

yum install nmap

On Debian/Ubuntu:

apt-get install nmap

On Windows:

http://nmap.org/download.html#windows

Few examples for port scans:

nmap localhost

This will show you all locally listening applications/daemons on your box:

nmap-1

 nmap IP_address or domain name

This command will show you all the listening ports on the host that you check. (IP or name)

nmap-2

Nmap local network scan to discover up and running hosts on your local network:

nmap -sP 192.168.0.1-254

nmap-local-scan

Also Nmap can do remote IP address scan. Be careful with this, just only do it for testing /troubleshooting purpose not for fun!
Port scans usually logged by firewalls/servers and your IP address will be logged, so you can be traced if you did something.

nmap -sP 86.1.80.1-20

nmap-remote-scan

#######################################################

Next troubleshooting tools is traceroute/tracert:

Linux: traceroute, Windows: tracert.

Lets do a traceroute first on Linux box:

traceroute 7layer.org

traceroute-1

 

I used capital I and n to force traceroute to use icmp protocol for tracing and n to avoid name resolution.
As you can see on the picture the jump via routers showing the whole route how the package reach my server.
The last address is my server address at Hetzner. This is the best way to figure out for example how the packets reaching your server.
If you have multiple path like two ISP broadband, then how you will know that what way the packets getting into your box?
Also when you are testing firewalls this and something is not correct, then you can check with this if the packets getting on the appropriate way.
For example you are connected into a VPN and in the meantime you are testing a firewall and the remote VPN default gateway address has been used.
So every packet will travel via VPN instead of your local network gateway (if the VPN’s configured that way) and you got lost with the firewall troubleshooting.

On Windows you can use this tool also:

tracert -d 7layer.org

The -d switch is compulsory I would say otherwise every IP address will be resolved into names and takes ages to get back the results.
I guess you don’t really want to wait, so use all the time -d switch  otherwise takes too long the whole tracing process.

w-traceroute

#######################################################

Back to Linux again.

How to check the running processes under Linux box?
You have the ps tool which can list every background processes in your running system.
So try to run and check the output, you will see many processes in the screen.

ps  ufxa

This command will show you the background process name including the directory which is it running from.
Also it will show you the user name who runs the actual process and also the process’s memory and cpu usage will be shown by this command.

ps-process

For example check the mysql daemon. The third red cube shows the 2.9 which is the percentage of the current memory usage by mysql daemon.

#######################################################

Next command is the mtr that we will take a look more closely.
Mtr is stands for my traceroute which is a real time traceroute application.
This is not installed by default on any distro you need to do it from repository.

CentOS/Fedora/RedHat:

yum install mtr

Debian/Ubuntu:

apt-get install mtr

So let’s do some mtr tracerouting:

mtr yahoo.com

This will traceroute the whole route till yahoo.com.
As you can see there are some packet losses at core1.hetzner.de but not much.
The other route path look clean and fine, no packets loss whatsoever.

mtr-1

The next mtr trace is to google’s public DNS server address:

mtr-2

 

To be continued…

ESXi backup free solution from Thinware

Hi everybody,

It’s been a long time with out any new post. I’ve been quiet busy with my family and with my new job in the last few months. 🙂

Anyway let’s get going and hit the wall.

I just managed to create a new totally free and automated backup solution for VMware ESXi server which is also free by the way.
So I’m posting my scripts and researches regarding to this solution and share with you guys.

Let’s assume you have your VMware ESXi server up and running with installed free license on it. (unlimited time, but limited features)
If you have this setup then you can’t use for example veeambackup as a free backup/restore solution anymore because the license limit these kind of features. So the solution for this is the Thinware vbackup application. You can download this application from here: http://www.thinware.net/Products/vBackup/tabid/202/Default.aspx You need to login in to do this although.  (right side / login)
So download and install this application and set it up for remote SSH VMware backup. Also you need to download and install VMware virtual disk development kit version: VMware-vix-disklib-5.1.0-774844.i386.exe and VMware Vcenter converter stand alone version: VMware-converter-all-5.0.1-875114.exe. You must use these versions because the latest one wont work properly and you either way wont be able to backup or neither restore the virtual machines properly!  These were discussed on Thinware forum more briefly look it up if you want.

So after you installed Thinware and Vcenter converter and virtual disk development kit is time to set up the settings in Thinware.
Go to tools/settings in Thinware and set the disk development kit and Vcenter converter’s directory.

thinware_pref

 

If these settings not been setup then the backup/restore wont work. It will complain straight away regarding to these missing settings.
So it’s time to browse up the ESXi server and set up the hosts also.

host_srv

 

Only need to fill up the name which can be an IP address if your server has not been added to local DNS server.
And also put your ESXi server username(root) and password.

In the next window you will see the hosts from the ESXi server, so you can add all of them or whichever you want to make backup of it.
Next one create the actual backup job. Add a name of like backup and choose from the drop down type menu Backup-Image-SSH.

backup

 

Then choose the backup root drive where the host(s) will be backed up.

backup_root

 

Then finish the wizard and right click on the backup job and execute it. Choose the debug brake point after option when it asks for it.
You will have a new terminal window where you can see the actual logs of the full backup process like: Creating snapshot, converting the disks etc…

backup_terminal

 

You can schedule this backup with Windows scheduler, the current Thinware does not support this yet.

This is the script for the Windows scheduler to run the backup every week or day or whatever time schedule you would need:

backup.cmd

####
cd C:
\Program Files (x86)\Thinware\vBackup

vBackup.exe -v Nagios -j backup
date /t >> C:\Users\Administrator\Desktop\backup-check.txt
time /t >> C:\Users\Administrator\Desktop\backup-check.txt
eventcreate /T Success /L APPLICATION /ID 100 /D “VMware backup was successfull”
####

 

The next post will be about how to restore host with this backup system.
Enjoy it! 🙂

 
Show Buttons
Hide Buttons