Configuring the VMA as a syslog server for ESXi

I wanted to use our VMware vSphere Management Assistant (vMA) as a syslog server for ESXi hosts. Central logging is a good thing, and since there is already a VMA in our environment, it may as well receive logs while it sits there.

Why use syslog / what are the alternatives?

  • Although you can define a datastore where ESXi will write it’s logs, I am not sure that datastore should be on the SAN. I still want logs to persist after ESXi reboots, whether or not I happen to have SAN connectivity.
  • IF there is a problem with an HBA or local disk controller being deadlocked, I would still like to see logs about it – syslog will give me those logs.
  • Logging to an existing central syslog server, or something like Splunk, is also an option, but using VMA is a simple and quick path to saving your logs, since you probably use the VMA already.
  • The VMA can be configured to collect logs from ESXi hosts, but I have never liked this – what if a host experiences problems, logs something interesting, but those logs weren’t “collected” before the host crashed or rebooted? Having ESXi send logs in real time, using syslog, will get me the most log data.

Here are the changes I made to VMware vSphere Management Assistant 5.0.0:

Add a disk to hold logs

The VMA disk is not that large, so ideally a second disk should be added to store the ESXi logs. Skipping this step is ok if you just want to experiment, but your VMA disk will fill up eventually, with only a Gb or so free on a typical VMA. Use the df -h /var/log command to see how much space is free on your VMA.

  • Shutdown the VMA, edit the VM’s settings, and add a second hard disk. I used a 10 Gb second disk.
  • Optionally snapshot the VMA, just in case. Remember to delete the snapshot when you are done.
  • Power on and log back into the VMA,. FOrmat and mount the disk under /var/log/esxi:
    • Run fdisk to partition the new disk: sudo fdisk /dev/sdb
    • Use the p command to verify that there is currently no partition on the disk (there is nothing under the columns at the bottom of the output).
    • Use the ncommand to create a new primary partition, number 1, using the default start and end cylinder values?
      Command (m for help): n
      Command action
         e   extended
         p   primary partition (1-4)
      p
      Partition number (1-4): 1
      First cylinder (1-1305, default 1):
      Using default value 1
      Last cylinder, +cylinders or +size{K,M,G} (1-1305, default 1305):
      Using default value 1305
      
    • Use the p command to view the partition, then use the wcommand to write the changes to the disk:
      Command (m for help): p
      
      Disk /dev/sdb: 10.7 GB, 10737418240 bytes
      255 heads, 63 sectors/track, 1305 cylinders
      Units = cylinders of 16065 * 512 = 8225280 bytes
      Disk identifier: 0xc68e9f4d
      
         Device Boot      Start         End      Blocks   Id  System
      /dev/sdb1               1        1305    10482381   83  Linux
      
      Command (m for help): w
      
  • Format the new partition:
    mke2fs -j /dev/sdb1
    
  • Add a line to /etc/fstab, for mounting the partition:
    /dev/sdb1            /var/log/esxi                    ext3       defaults        1 2
    
  • Create the mount-point, and mount the filesystem:
    mkdir /var/log/esxi
    mount /var/log/esxi
    df -h /var/log/esxi
    

Configure syslog-ng

Add the following to /etc/syslog-ng/syslog-ng.conf, and then restart the syslog service:

# Collect syslog messages from ESXi hosts and store them in /var/log/esxi/
# Define a filter to exclude debug and info log levels.
# Otherwise, ESXi logs become verbose and are more difficult to use.
filter f_esxi { not level(debug,info); };
# Define a source for remote UDP port 514 syslogs.
# This line exists (commented out) earlier in this config file,
# it is shown here for completeness.
# Any remote logs are assumed to come from ESXi, since this is VMA after all.
source src_remote { udp(port(514)); };
# Put logs in an esxi directory, and name logs after the host name.
# Remember to mkdir /var/log/esxi
destination esxi_host_logs {
 file("/var/log/esxi/$HOST.log"
# THe use of a template allows reformatting of the log output.
# $TAG and $PRIORITY includes log facility and level for tuning / testing.
#   remove these from the template for production.
# Sample output from this template is:
# b4 warning Jul 21 13:55:15 esx1 vmkernel: 1:00:03:30.566 cpu0:255265)WARNING: UserLinux: 1638: UNIMPLEMENTED!  write-back of mmap regions unsupported
 template("$TAG $PRIORITY $S_DATE $HOST $MSG\n"));
# For logs to look like those stored on the ESXi host,
#  exclude the facility, level, and host name tags from the template:
# template("$S_DATE $MSG\n"));
 };
# PUt it all together:
log { filter(f_esxi); source(src_remote); destination(esxi_host_logs); };

Remember to restart syslog-ng after editing the config file: sudo service syslog restart

The f_esxi filter leaves out debugging and info syslog messages, but ESXi logs are still pretty verbose. YOu may use additional syslog-ng configuration to remove particularly irritating log messages, but as soon as you remove those messages you may wish you had them for troubleshooting. Here is an example of some verbose messages that still show up in logs:

Jul 21 09:14:41 esx1 14:18 [issue_cmd           ]   07/21/12 15:14:18 [issue_cmd           ] --- 192.168.8.1 ping statistics ---  07/21/12 15:14:18 [issue_cmd           ] 1 packets transmitted, 1 packets received, 0% packet loss  07/21/12 15:14:18 [issue_cmd           ] round-trip min/avg/max = 0.567/0.567/0.567 ms  07/21/12 15:14:18 [issue_cmd           ]   07/21/12 15:14:18 [myexit              ] VMwareresult=success  07/21/12 15:14:18 [elapsed_time        ] Total time for script to complete:  0 minute(s) and 0 second(s)

Yes, that is how the single log message looks – messy, but potentially useful if you think that network latency may have been causing ESXi to lose heartbeats, or trigger an isolation response./p>

Configuring logrotate

Create a configuration file in /etc/logrotate.d so that the new log files in /var/log/esxi will be rotated and compressed. I have placed this in /etc/logrotate.d/esxi – you may want to adjust the retention and other options. See the logrotate man page for more information on the keywords you can use in logrotate config files:

# Rotate ESXi logs, which we have configured syslog-ng to put in /var/log/esxi
# Rotation is weekly, compressing all but the latest rotated log.
# Five logs are kept.
/var/log/esxi/*.log {
        weekly
        missingok
        rotate 5
        compress
                delaycompress
        notifempty
        nocreate
        sharedscripts
        postrotate
                /etc/init.d/syslog reload
        endscript
}

This logrotate configuration will give you file names like this in /var/log/esxi:

  • esx1.log-20120708.bz2
  • esx1.log-20120715.bz2
  • esx1.log-20120722 (this is not compressed because it is the latest, rotated log. This is logrotate’s DelayCompress option.)
  • esx1.log (This is the current log.)

COnfiguring ESXi to log to the VMA

COnfigure ESXi hosts to send syslog messages to VMA, and verify that new files are created in /var/log/esxi for each host.

I configure ESXi to use a syslog server as part of a post-installation script, using this command. YOu can also run this from a tech support shell (local console or SSH):

vim-cmd hostsvc/advopt/update Syslog.Remote.Hostname string 192.168.8.17

You can also set this in the VMware client by clicking a host and then: Configuration -> Advanced Settings, expand Syslog in the tree, and enter the VMA’s IP address in the “Remote” field.

Advertisements
This entry was posted in VMware vSphere and tagged . Bookmark the permalink.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s