VMware ESXi & VCenter used disk percentage monitoring

GitHub: https://github.com/7layerorg/Monitoring/tree/master/Nagios/ESXi

The next article is about SNMP monitoring with Nagios to check VMware ESXi and VCenter servers used disk size.

I’ve had some trouble recently with our VCenter server because the logs just filled up one of the volume and I was unable to log in to the server at all.
Even the main console didn’t work and it was complaining about the storage filling up with logs.
VCenter server do not log rotate the old logs unfortunately by default, so sooner or later the volume will be filled up.

Well there are some solutions to monitor used disk size but the default Nagios won’t give you straight and appropriate answer from the VCenter server or form the ESXi boxes either.
This is why I made a script with SNMPwalk to be able to monitor any kind of ESXi or VCenter servers.
There are some “tricks” in the script because for VCenter need a different OID to check than on ESXi boxes, but this is built in to the script already.
Also there is a way to monitor disk size with SNMP under Cacti monitoring server but to receive an email regarding to the triggers you will need to modify way too many thresholds and would take ages if you have several servers.

With this script you will need to add only the host names and the disk number and you are all set, Nagios will take care all of the rest.
The check_vmware_disk script needs to be uploaded into your Nagios libexec folder, so Nagios will be able to run the script automatically when it’s been scheduled.

There are some pictures regarding to this service:

Also the second script which is available via my GitHub account is to check the ESXi host’s network up-link.
root@nagios: ./check_esxi_vnic 10.0.4.71 1
up
0 status:0; Triggers: down; OK – up

root@nagios: ./check_esxi_vnic 10.0.4.71 2
down
1 status:1; Triggers: down; CRITICAL – down

 

 

VMware Replication & Recovery

The following three videos show how to create virtual machine offsite replication in your vCenter server on any available storage drive in your server.
This solution is available freely from VMware and can be integrated into all type of vCenters, even into VMware Small Business Essentials Plus.
To download VMware replication follow this link: VMware Replication

The backup storage can be a network share mount or a local drive.
If you use for example an NFS share or iSCSI storage, then this gives you the benefit of an offsite backup for your virtual machines.
The replication is automatic and scheduled to run in the background on the vCenter server.

The offsite backup, can be restored with out the vCenter server, if for any reason vCenter is unavailable.
In this case you need to add manually the backed up machine onto your vCenter or stand alone ESXi server’s inventory.

 

 

Networking Tools

Packet capture analyzing tools:

https://www.wireshark.org/  Wireshark is a Network protocol analyzer to capture and analyze network packets in deep details, available under Linux/Windows/macOS.

http://www.tcpdump.org/  Tcpdump is a command line packet analyzer, similar to Wireshark but witout GUI interface.

 

Security Tools

Online Security scanners:

https://www.ssllabs.com/ssltest/
Deep detailed SSL check for sites. It will give you a very detailed SSL check and also shows the issue on the vulnerability itself.


https://sitecheck.sucuri.net/
Website Malware and Security Scanner to analyze sites.


https://app.upguard.com/webscan
Website Security Scanner. It will analyze and score the site with detailed results E.G.: SSL enabled or Suspected Malware, Phishing on the site or an unwanted software.


http://www.unmaskparasites.com/
Exploit security, vulnerabilities, SQL Injection, Cross Site Scripting scanner. It’s totally free and gives you a quick scan results to check up your site against those type of vulnerabilities and attacks.

This scanner checks against Phishing, Malware, Blacklist, Worms, Trojans, Backdoors, Suspicious frames and Suspicious connections.

https://nmap.org/
Nmap port scanner to check open/closed ports locally and remotely on any computer.
Available under Linux/Windows/macOS.

Website Vulnerability Scanner, TCP/UDP Port Scan with Nmap, OpenSSL Heartbleed/POODLE/DROWN scanning.

DNS Tools

DNS/Mail server:

https://mxtoolbox.com/ DNS, MX record, reverse zone, openrelay check. This is a very detailed domain checker tool site. Almost everything available for testing a new domain regarding to any records.

https://intodns.com/ DNS zone checker. Basic domain check to see if a domain setup correct to the current RFC standards. They update their DNS zones pretty quickly (maybe Gooogle’s DNS used 8.8.8.8/8.8.4.4), so zone updates  can be seen almost instantly on this site.

https://www.dmarcanalyzer.com/  Create and analise spf and dmarc records for DNS server. Also you can register Dmarc zone for domains free.

https://www.fraudmarc.com/spf-record-check/ Spf record check.

Network auditing and vulnerability management with 7layer

Automate network audit scanning and check vulnerability daily with 7layer

We offer you Network audit and network vulnerability scheduled scanning to get alerts in real time about hosts/OS vulnerability and out of date updates, expiring certificates, new open ports etc.

  • Scheduled Network vulnerability Monitoring (SNVM)

  • Daily email reports of vulnerabilities

  • OS update check

  • Malware detection service

  • Website Security Check

 

 

 

Contact us for more info:

lszabo (at) 7layer (dot) org

 

ESP8266 Temperature logger for Nagios

Previously I had a post about ESP8266 microcontroller and I had some brief review of this lovely hardware here.
Now I’m going to post a new schematics and a Nagios module to use it as a real time temperature logger for Nagios server in Data-centres.

Briefly what it does and how it works:

– ESP8266 module reads the temperature sensor in every 10 seconds and sends data via UDP
– Nagios server processes the received UDP data  from the ESP module and compares with the settings in Nagios

If Nagios server picks value that triggers the alarm, than it will send warning or alarm to the Nagios admin.
So here we have the electronics schematics and the related program codes for the ESP and the Nagios server.

Connect the ESP and the Dallas sensor as on this picture below. This is the easiest way to wire up them. (1-Wire )
I’m not going to go into details of the ESP module programing, there are dozens articles on the net regarding to this.
Myself I use the LuaLoader, which I think is the easiest one to use. If you just need the related code files, then jump to the end of this article, there you can download all lua files.
You must correct the Nagios server’s address, which is this: ” cu:connect(7,”10.0.4.252″) ” and also your SSID and Password to connect to your access point.

esp8266-ds18b20-2_bb

For power supply I used an old USB cable to power up the ESP module from a server in the Data-centre. After all this is to check the racks and server’s temperature in the DC. 🙂
The USB has 5V as we know, so you would need to lower this up to 3.3V. You can use an AMS1117 5V to 3V stabilizer, please check the link below about this.
Temperature Sensor = Dallas
ESP module = ESP
AMS1117-3.3 = AMS

And from here the codes for the ESP8266 module and for the Nagios server as well.
Two files need to be uploaded to ESP8266:

Github links for source code: https://github.com/7layerorg/ESP8266/tree/master/Temperature_Logger

first file init.lua:

#####

function startup()
if abort == true then
print(‘startup aborted’)
return
end
print(‘Starting xmitTemp’)
dofile(‘xmitTemp.lua’)
end
abort = false
print(‘Startup in 5 seconds’)
tmr.alarm(0,15000,0,startup)
#####

Second file xmitTemp.lua:

#####

function getTemp()
local addr = nil
local count = 0
local data = nil
local pin = 3 — pin connected to DS18B20
local s = ”
— setup gpio pin for oneWire access
ow.setup(pin)
— do search until addr is returned
repeat
count = count + 1
addr = ow.reset_search(pin)
addr = ow.search(pin)
tmr.wdclr()
until((addr ~= nil) or (count > 100))
— if addr was never returned, abort
if (addr == nil) then
print(‘DS18B20 not found’)
return -999999
end
s=string.format(“Addr:%02X-%02X-%02X-%02X-%02X-%02X-%02X-%02X”,
addr:byte(1),addr:byte(2),addr:byte(3),addr:byte(4),
addr:byte(5),addr:byte(6),addr:byte(7),addr:byte(8))
–print(s)
— validate addr checksum
crc = ow.crc8(string.sub(addr,1,7))
if (crc ~= addr:byte(8)) then
print(‘DS18B20 Addr CRC failed’);
return -999999
end
if not((addr:byte(1) == 0x10) or (addr:byte(1) == 0x28)) then
print(‘DS18B20 not found’)
return -999999
end
ow.reset(pin) — reset onewire interface
ow.select(pin, addr) — select DS18B20
ow.write(pin, 0x44, 1) — store temp in scratchpad
tmr.delay(1000000) — wait 1 sec
present = ow.reset(pin) — returns 1 if dev present
if present ~= 1 then
print(‘DS18B20 not present’)
return -999999
end
ow.select(pin, addr) — select DS18B20 again
ow.write(pin,0xBE,1) — read scratchpad
— rx data from DS18B20
data = nil
data = string.char(ow.read(pin))
for i = 1, 8 do
data = data .. string.char(ow.read(pin))
end
s=string.format(“Data:%02X-%02X-%02X-%02X-%02X-%02X-%02X-%02X”,
data:byte(1),data:byte(2),data:byte(3), data:byte(4),
data:byte(5),data:byte(6), data:byte(7),data:byte(8))
–print(s)
— validate data checksum
crc = ow.crc8(string.sub(data,1,8))
if (crc ~= data:byte(9)) then
print(‘DS18B20 data CRC failed’)
return -9999
end
— compute and return temp as 99V9999 (V is implied decimal-a little COBOL there)
return (data:byte(1) + data:byte(2) * 256) * 625
end — getTemp
function xmitTemp()
local temp = 0
temp = getTemp()
if temp == -999999 then
return
end
cu:send(tostring(temp))
end — xmitTemp
function initUDP()
— setup UDP port
cu=net.createConnection(net.UDP)
cu:connect(7,”10.0.4.252″)
— cu:connect(7,”10.0.4.252″)
end — initUDP
function initWIFI()
print(“Setting up WIFI…”)
wifi.setmode(wifi.STATION)
wifi.sta.config(“Your SSID”,”SSID Password”)
wifi.sta.connect()
tmr.alarm(1, 1000, 1,
function()
if wifi.sta.getip()== nil then
print(“IP unavailable, Waiting…”)
else
tmr.stop(1)
print(“Config done, IP is “..wifi.sta.getip())
end
end — function
)
end — initWIFI
initWIFI()
initUDP()
tmr.alarm(0, 10000, 1, xmitTem

Here follows the Nagios server modules:

Add to your localhost.cfg the following configuration.
This is usually at /usr/local/nagios/etc/objects/

define service{
use                             local-service         ; Name of service template to use
host_name                       Telehouse
service_description             Telehouse_Temperature
check_command                   check_temp
#check_interval                 0.5
#retry_interval                 1
#max_check_attempts             5
notification_interval           1
check_interval          1
retry_check_interval    1
max_check_attempts      5
}

#####

Create a file called check_temp in the libexec directory and make it executable.  (check_temp at /usr/local/nagios/libexec)

#!/bin/bash
DIRS=”/var/log /tmp”

temp1=`/usr/bin/cut -c 1-2 /home/nagios/current_temp.txt`
temp2=`/usr/bin/cut -c 3-4 /home/nagios/current_temp.txt`

op1=2200
op2=2500

count=$(/usr/bin/tail -n 1 /home/temp/current_temp.txt)

count2=$count

if [[ “$count2” < “$op1” ]] ; then

status=0
statustxt=OK

elif [[ “$count2” < “$op2” ]] ; then

status=1
statustxt=WARNING
else

status=2
statustxt=CRITICAL
fi

echo “$status Temperature:$temp1.$temp2; Triggers: 22.00;25.00;0; $statustxt – $count2”
exit $status

######

Add a new crontab to run tshark which will check the UDP echo messages from the ESP module.
If you don’t have tshark/wireshark installed, then make it available for your box.
CentOS: yum install wireshark
Debian: apt-get install wireshark

nano /etc/crontab

01 * * * * root cd /home/temp && /usr/bin/tshark -a duration:3600 -i eth0 src 10.0.4.30 -T fields -e data -w temp2.pcap & > /dev/null
* * * * * root /home/temp/temp.sh

######

Create a new directory in /home as temp

mkdir /home/temp

Create a file called temp.sh

#!/bin/bash

cat /home/temp/temp2.pcap | tr -dc ‘[:alnum:]\n\r’ | cut -c 2-5 | awk ‘length($0) > 2’ | tail -n 1 -c 5 > /home/temp/current_temp.txt

To check ESP8266 sending the correct UDP packet run this command:

tcpdump -i eth0 udp

You need to see similar UDP packets from the ESP module every 10 seconds:

18:10:40.248116 IP 10.0.4.30.45908 > nagiosnew.echo: UDP, length 6

And also in the Nagios you will hopefully see this:

telehouse

References:

https://bigdanzblog.wordpress.com/2015/04/29/snmp-environmental-monitoring-using-esp8266-based-sensors/
http://www.instructables.com/id/Low-cost-WIFI-temperature-data-logger-based-on-ESP/?ALLSTEPS
http://benlo.com/esp8266/
https://github.com/nodemcu/nodemcu-firmware/tree/master/


https://www.7layer.org/downloads/init.lua

https://www.7layer.org/downloads/xmitTemp.lua
https://www.7layer.org/downloads/esp8266_flasher.exe
https://www.7layer.org/downloads/v0.9.2.2 AT Firmware.bin

 

VMware networking setup for vMotion/iSCSI & VM traffic

VMware ESX/ESXi network setup.

In the following post I will show you some networking setup regarding to VMware servers.
This will involve Cisco switches(2960/3750 series) and HP or Dell servers setup.
I got these configurations in production running for quite some times now(2+years) without any issues.

As we know the networking setup for VMware servers, got much more complicated, than any other regular server setup earlier we had with “classic” Linux or Windows physical boxes.
Classic only one uplink connection with regular vlan is not enough for vmware anymore.
You must separate virtual machine traffic from the management traffic and also you must separate the storage and vmotion traffic.
Although VMware says you can have separated vswitches for all physical connections with different vlans, but the failover to other physical connections is more complicated, than if you have one or two vswitches. VMware server needs minimum 2 network uplinks for VM traffic and management traffic, but VMware recommends 4 uplinks for the physical servers.

The following picture shows briefly the current setup.

7layer

So let’s take a look the 4 uplink configuration in the VMware ESXi host:

two

We got all 4 uplinks connected to the same vswitch. With this configuration is very easy to create the failover for the management traffic and to separate the storage and vmotion traffic as well. Let’s take a look the vswitch properties:

vswitch0-1

Also take a look the NIC teaming for the vswitch.
As you can see all adapters are active in this vswitch:

nic-teaming

 

Now take a look the management uplink settings.
The management network has one active adapters and two standby adapters.
If the active vnic0 adapter physical connection fails(switch issue or cable connection issue), then VMware kernel will activate one of the other standby adapters.
With this setup the management network will always be available and you cannot lose the connection to the VMware box.
       mgmt-ipstorage

Now we check the vMotion settings.
Here we have an added VMkernel port with vMotion and IP storage which contains extra IP address for the vMotion.
As you can see here we have one active adapters and three unused adapters. To properly separate this kind of traffic by the kernel you must tick the failover order and move down the adapters, that you don’t want to use in the kernel. This settings is the same with iSCSI storage.
vmotionvmotion-ip

Now take a look the Storage IP kernel settings.
Here we have also an extra added VMkernel port with extra IP address.
In this setup also the extra active vswitch adapters have been disconnected and unused as you can see on the picture.
Without this you won’t be able to add properly the iSCSI software storage. The VMkernel IP settings creates a point to point 1 to 1 connection to the storage and therefore only one active adapters should be enabled in any VMkernel port groups. With this setup you can have more than one path to the iSCSI storage, but for this you need to enable this feature in iSCSI setup.

storageiscsi

So now take a brief look to the Virtual Machine Port Group settings regarding to the Vlan settings.
You can add new vlans here to the kernel and create load balance and failover for the virtual machines.
I used two adapters from the physical adapters for the virtual machines and they are activated as vnic5/vnic1 and vnic1/vnic5 opposite to each other.
But if you have 4 or 6 uplink adapters, then you could active 3-4 adapters for the virtual machines, it’s up to you.
Also this depends on how heavily loaded your virtual boxes, obviously if the boxes are pretty loaded then, it’s better if you separate the loads and leave out vmotion and management traffic from the physical uplink connections.
1-x-network-vlan

4-x-network-vlan

 

I know it’s getting a bit confusing, so here we are again some binding regarding to the VLAN, management traffic and vMotion traffic:
mgmt-vlan

vmotion-vlan

The storage traffic is not added to any of those traffics, it is just connection via the VMkernel port group IP address as a one to one connection:

storage-vlan

In this setup I use the same vlan for the vMotion, because this is only used for maintenance, but if you use heavily the vMotion then it is better to be separeted into a different vlan.You might as well create a new physical uplink for traffic, which could help you to separate this traffic not just on a vlan level, but on the physical level also.

And finally the physical uplink ports to the Cisco switch:

interface GigabitEthernet1/0/22
description 10.0.4.92 vnic0
switchport trunk allowed vlan 100,200,300,400
switchport trunk native vlan 999
switchport mode trunk
switchport nonegotiate
speed 1000
duplex full

interface GigabitEthernet1/0/23
description 10.0.4.92 vnic2
switchport trunk allowed vlan 100,200,300,400
switchport trunk native vlan 999

switchport mode trunk
switchport nonegotiate
speed 1000
duplex full

The native vlan 999 command is used to change the default untagged vlan traffic which is vlan1.
With this command you can avoid unnecessary layer 2 traffic to the VMware server, like flooding and broadcast.
Also if you have a system already configured with vCenter, then sometimes you cannot change the management vlan, because vCenter won’t be able to reach the box anymore and the changes goes into error or the box could get dropped from vSphere. In that case you would need to disconnect the connected server from vCenter and create a second VMkernel interface with a different IP subnet with different physical interface, than the currently running one and connect to the box via that KMkernel. With this you can do any major changes to the main interface. (native vlan, vlan tagging etc) I have seen few times, when I wanted to do changes, then I lost the connection to the server and I needed to either reset the VMkernel management or rollback the switch configuration or change the native vlan on the switch. So you need to be careful with this changes, if you cannot reach your physical box for any reason (server is in a data-center or a different office)

So now let’s take a look the Cisco switch side, after the native vlan configuration and the trunking configuration:

Port        Mode             Encapsulation  Status        Native vlan
Gi1/0/22    on               802.1q         trunking      999

Port        Vlans allowed on trunk
Gi1/0/22    100,200,400

Port        Vlans allowed and active in management domain
Gi1/0/22    100,200,400

 

References:

https://www.vmware.com/files/pdf/support/landing_pages/Virtual-Support-Day-Best-Practices-Virtual-Networking-June-2012.pdf
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2038869
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2045040

IoT Temperature logger with ESP8266 and DS18B20 sensor

Room temperature:

I will post the circuit schematics and coding shortly, in the meantime this is the module that I used.
Also I’m posting the firmware flasher and the firmware that I used for this project.
There are many available on the net and you could get confused easily, so there you go follow this links and check what I bought and used.

ESP8266 used for this project: ESP-01: http://www.esp8266.com/wiki/doku.php?id=esp8266-module-family#esp-01

esp8266-pinout

ESP8266 on ebay: http://www.ebay.co.uk/sch/items/?_nkw=esp8266&_sacat=&_ex_kw=&_mPrRngCbx=1&_udlo=&_udhi=&_sop=12&_fpos=&_fspt=1&_sadis=&LH_CAds=&rmvSB=true

DS18B20 sensor on ebay: http://www.ebay.co.uk/sch/i.html?_fspt=1&_mPrRngCbx=1&_from=R40&_sacat=0&_nkw=DS18B20+sensor&_sop=15

Nodemcu Firmware: https://github.com/nodemcu/nodemcu-firmware/tree/master/pre_build/latest

esp8266_flasher

Programming and testing with Lualoader: http://benlo.com/esp8266/
download: http://benlo.com/esp8266/LuaLoader.zip

Programming and testing with ESPlorer: http://esp8266.ru/esplorer/
download: http://esp8266.ru/esplorer-latest/?f=ESPlorer.zip  

 

 

 

Contact

Please email me at:

lszabo (at) 7layer (dot) org

Thanks.

 
Show Buttons
Hide Buttons