Linux/Windows Troubleshooting part 1

Network troubleshooting part 1:

Checking open ports on box:

netstat -natlp

It will tell you the locally open ports with the running daemon name also.
As you can see on the picture the first red cubic is the named daemon DNS name server is listening on the 127.0.0.1:53 local port. The second red cubic is the (78.46.184.202:22) ssh daemon which is the remote session terminal.You can see the local and the remote address on these picture.

netstat-1
If you want to check all the listening udp and tcp ports then you need to add and extra u at the switches which will provide the udp connections also.

netstat -natlpu

netstat-2

 

On this picture you can clearly see the Asterisk and Named servers are listening on 5060 and 53 ports.

So if you want to check any running daemon or application on your box just issue any of those commands and you will see if they are listening on the local ports or not. This is the easiest way to figure this out.
If you stop for example smtp daemon or imap daemon, then the port(s) will disappear straight away.

Netstat is available by default in any major Linux distro also available in every old and new Windows.
Under Windows you need much less switches:

netstat -an -p tcp

netstat-3

netstat -an -p udp

netstat-4

The first command will show you all listening, connected and waiting tcp connections.
If you have so many wait/close connections then probably someone is attacking your box remotely.
I have seen it on Exchange server 2010, it had more then 100 wait/close connection against port 110, because someone tried  to brake into that server with brute force attack. The sysadmin didn’t lover down the maximum available connection per IP, it was on default which is 2000.

#######################################################

The next command will show you all the listening UDP connections. If you have a DNS server or an Asterisk SIP server they will listen on UDP and you will see those ports up and open.

Nmap:

My other favorite command on Linux and Windows is the nmap. This application can scan local and remote IP addresses or domain names to show you the open/listening ports. Also you can trace your whole local or remote network which IP addresses are used or which port(s) listening/open. By default nmap is not installed on any major Linux distro, so you need to install it.

CentOS/RedHat/Fedora:

yum install nmap

On Debian/Ubuntu:

apt-get install nmap

On Windows:

http://nmap.org/download.html#windows

Few examples for port scans:

nmap localhost

This will show you all locally listening applications/daemons on your box:

nmap-1

 nmap IP_address or domain name

This command will show you all the listening ports on the host that you check. (IP or name)

nmap-2

Nmap local network scan to discover up and running hosts on your local network:

nmap -sP 192.168.0.1-254

nmap-local-scan

Also Nmap can do remote IP address scan. Be careful with this, just only do it for testing /troubleshooting purpose not for fun!
Port scans usually logged by firewalls/servers and your IP address will be logged, so you can be traced if you did something.

nmap -sP 86.1.80.1-20

nmap-remote-scan

#######################################################

Next troubleshooting tools is traceroute/tracert:

Linux: traceroute, Windows: tracert.

Lets do a traceroute first on Linux box:

traceroute 7layer.org

traceroute-1

 

I used capital I and n to force traceroute to use icmp protocol for tracing and n to avoid name resolution.
As you can see on the picture the jump via routers showing the whole route how the package reach my server.
The last address is my server address at Hetzner. This is the best way to figure out for example how the packets reaching your server.
If you have multiple path like two ISP broadband, then how you will know that what way the packets getting into your box?
Also when you are testing firewalls this and something is not correct, then you can check with this if the packets getting on the appropriate way.
For example you are connected into a VPN and in the meantime you are testing a firewall and the remote VPN default gateway address has been used.
So every packet will travel via VPN instead of your local network gateway (if the VPN’s configured that way) and you got lost with the firewall troubleshooting.

On Windows you can use this tool also:

tracert -d 7layer.org

The -d switch is compulsory I would say otherwise every IP address will be resolved into names and takes ages to get back the results.
I guess you don’t really want to wait, so use all the time -d switch  otherwise takes too long the whole tracing process.

w-traceroute

#######################################################

Back to Linux again.

How to check the running processes under Linux box?
You have the ps tool which can list every background processes in your running system.
So try to run and check the output, you will see many processes in the screen.

ps  ufxa

This command will show you the background process name including the directory which is it running from.
Also it will show you the user name who runs the actual process and also the process’s memory and cpu usage will be shown by this command.

ps-process

For example check the mysql daemon. The third red cube shows the 2.9 which is the percentage of the current memory usage by mysql daemon.

#######################################################

Next command is the mtr that we will take a look more closely.
Mtr is stands for my traceroute which is a real time traceroute application.
This is not installed by default on any distro you need to do it from repository.

CentOS/Fedora/RedHat:

yum install mtr

Debian/Ubuntu:

apt-get install mtr

So let’s do some mtr tracerouting:

mtr yahoo.com

This will traceroute the whole route till yahoo.com.
As you can see there are some packet losses at core1.hetzner.de but not much.
The other route path look clean and fine, no packets loss whatsoever.

mtr-1

The next mtr trace is to google’s public DNS server address:

mtr-2

 

To be continued…

 
Show Buttons
Hide Buttons