User Guide - Resources Overusage Alerts

19/01/2010 11:57 AM
- Server Configuration

User Guide – Resource Overusage Warnings

Overview

This User Guide will help explain the meaning of the e-mails you have gotten regarding the resource overages in your VPS.

These e-mails are sent to alert you of a potential issue in your VPS that is consuming more than your allotted system resources for CPU, memory, disk space, daily traffic and network traffic (or sometimes a combination of these resources).

It is expected that you will take the time to read these warnings and act upon them. Your hosting plan guarantees you a specific allocation of available CPU, memory and disk space, and allows you to burst beyond the CPU and memory limits for short periods of time.

However, if you are consistently using more than your allotted resources, you are taking resources that another customer should be using, and potentially impacting their service.

If you continue to use more than your allotted resources, even after multiple warnings, you may be moved to a Quarantine server so that you are no longer impacting the service for other users.

A Quarantine server has other users who are also exceeding their allotted resources, so you may find that your service is impacted by being on a Quarantine server.

Also included in this User Guide is a script that you can install on your VPS that will run from a cron job and send you e-mails with the results of the Linux commands top, free -m and df -h so that you can better monitor your VPS and see what is using the resources. See the section titled Simple Resources Monitoring Script for more information.

CPU Overusage Warning Notice
Memory Allocation Warning Notice
Disk Allocation Warning Notice
Pro-rata Daily Traffic Allowance Notice
Network Traffic Rate Allocation Notice

Simple Resources Monitoring script

Creating the load-alert.sh script
Creating the cron job
Other Configurations
Testing the script
Disabling or removing the load-alert.sh script

The load-alert.sh script
Example report from the load-alert.sh script

Links to other information

CPU Overusage Warning Notice

The CPU Overusage Warning notice alerts you that you have exceeded your allotted CPU allowance. The threshold for the warning is a load factor of 10, which means that you have exceeded your allotted CPU allowance by 1000% (one thousand percent).

This overage may have come during a spike in your web traffic, or due to a process on your VPS that needed to burst beyond your allotted CPU to run.

However, it is possible that your site or applications require more resources than your current plan offers. If this is the case, then you will either need to upgrade your plan or optimize your application to lower your CPU usage.

The current version of the Resource Overage script does not identify what was causing the CPU overage, just the time frame when it occurred and the Peak Usage. It is your responsibility as the site owner or administrator to determine what is causing the CPU overage, and to correct the issue by either optimizing your site and code, or by upgrading to a plan with adequate resources.

eApps support can assist with this, but cannot take on the full responsibility of finding the root cause for your CPU overages. It is recommended that you use the Simple Resources Monitoring Script at the end of this User Guide to assist you in your investigations.

This is the text of the CPU Warning notices:

Notes on warnings:
CPU Overusage - Spike: Each VPS container has an allocation of CPU, measured by Load Factor, which you may exceed (burst) for short periods of time. A load factor of 1, represented as 100%, is the normal allocation. This warning means that your CPU usage spiked more than 1000% sometime during the one hour time period indicated. Occasional large spikes do not necessarily represent a problem as they can be caused by search engine spiders, peaks of web traffic, and short duration processes running inside of your VPS container. However, if your CPU bursts (spikes) more than 1000% (which represents a load factor of 10) frequently within a short period of time, you should investigate the reason for the spike, or contact eApps Support for assistance. Customers that continuously spike very high are subject to being moved to a quarantine server. If you frequently get CPU spike warnings you should evaluate your programming and use `of CPU intensive processes and make changes to use less CPU, upgrade to a larger plan, or consider a load balanced cluster of VPS instances.

CPU Overusage - Average: Each VPS container has an allocation of CPU, measured by Load Factor, which you may exceed (burst) for short periods of time. A load factor of 1, represented as 100%, is the normal allocation. Our guidelines are that you should should take action if your CPU consistently averages more than 200% (which represents a load factor of 2). This warning means that your CPU usage exceeded 200% as an average for the one hour time period indicated. Customers that consistently exceed the CPU guidelines are subject to being moved to a quarantine server. Also, if you exceed the CPU allocation, CPU cycles will automatically be rationed to your VPS container, which may impact performance. You should evaluate your programming and use of CPU intensive processes and make changes to use less CPU, upgrade to a larger plan, or consider a load balanced cluster of VPS instances.

Memory Allocation Warning Notice

The Memory Allocation Warning notice alerts you that you are using 90% or more of your available memory.

If you use all of the memory in your VPS, the Linux kernel will run a process called the OutOfMemory Killer (OOM) that will kill processes using the most memory in your VPS, in order to keep the entire VPS itself from crashing.

However, the process using the most memory is usually the process that is most important to you, such as the Apache web server or Tomcat. When the OOM Killer kills that process, this shuts your site or application down until it is restarted. That is why it is very important make sure your plan has enough memory to handle your peak usage.

To get more memory, you can upgrade your plan to one that has more memory available. But before doing that, look at all the applications you have installed, especially if you are on one of the Standard plans. Installed applications consume memory, even if they are not being used. If you are not using an application that you have installed, consider removing it. If you are not sure if an application that is installed is safe to remove, please contact eApps support for more information.

To see what applications you have installed, go to the Control Panel, and click on the System Tab. If necessary, click on the Select Another System (Subscription) link on the left and choose the correct Virtuozzo system. Then, click on All Applications. This will show you what is currently installed. To remove an application, click on the application itself, and click Uninstall. You may get an alert message saying that it is not completely safe to remove this application. Click OK and continue.

Applications that say “Embedded in System” cannot be removed. If you are not 100% sure what an application does, please contact eApps support before trying to remove the application. Removing the wrong application can cause problems with your sites.

You can also take advantage of the Burst Memory feature, which will allow you to add a few megabytes of extra memory that your VPS can take advantage of if needed, but only for short periods of time (in the 10 to 15 seconds range). This is available to you at no charge.

To activate the Burst Memory feature, go to the Control Panel, and click on the System Tab. If necessary, click on the Select Another System (Subscription) link on the left and choose the correct Virtuozzo system.

Then click on the Upgrade Center tab, and then on Buy Resources. In this screen, look for the value of SLM memory limit, which is usually the last item in the list. The Current Limit shows your current memory allocation. Look at the Range, and that will show you the amount of Burst Memory you can allocate. In the New Limit box, put the maximum value shown in Range, and click Next.

Click the box for I accept all terms of the User Agreement, and then click Place Order. Allow a few hours for the order to be processed, and then the burst memory will be available to you.

If you have removed the applications you are not using, and added the burst memory feature, and you are still running low on available memory, it is time to consider upgrading to a higher plan.

This is the text of Memory Allocation Warning notice:

Notes on warnings:
Memory Allocation: Each VPS container has a hard (finite) allocation of memory that can not be exceeded. You are near this allocation. If you hit the Memory Allocation limit, your VPS container is likely to crash or encounter processing errors. To avoid this undesirable situation, you should remove unnecessary processes, optimize your program code to use less memory, or upgrade to a larger plan.

Spike Memory Allocation: Each VPS container has a hard (finite) allocation of memory that can not be exceeded. You are near this allocation. If you hit the Memory Allocation limit, your VPS container is likely to crash or encounter processing errors. To avoid this undesirable situation, you should remove unnecessary processes, optimize your program code to use less memory, or upgrade to a larger plan.

Disk Allocation Warning Notice

There are two types of hard disk limits – a soft limit and a hard limit. The soft limit is the amount of hard disk space that is guaranteed to you per your plan. This is the amount included for free with your plan. Once you exceed the soft limit, you are charged an overage fee of $.01 (one cent) per megabyte, or $10.24 per gigabyte. The hard limit is generally twice the soft limit, but this varies per plan.

The Soft Disk Allocation Warning notice first alerts you that you are using 90% (ninety percent) of the hard disk space allotted to you for your plan (the soft limit). Once you get near the Hard Disk Allocation limit allotted to you per your plan, you will receive the Hard Disk Allocation warning.

Once you reach the hard limit of your disk allocation, your VPS will essentially shut down because it will no longer be able to process files.

eApps support can assist you with finding where the disk space is being used, but it will be your responsibility to determine what files should be removed. We can give you guidance on this, but do not want to be responsible for removing any of your mission critical files.

This is the text of the Disk Allocation Warning notices:

Notes on warnings:

Soft Disk Allocation: Each VPS container has a soft allocation of disk space that is free. You are near, or have exceeded this allocation. Disk space used in excess of the soft allocation is billable, up to the physical hard allocation which can not be exceeded. If you do not want to pay the disk space overage fee, you should delete unnecessary files or upgrade to a larger plan.

Hard Disk Allocation: Each VPS container has a hard (finite) allocation of disk space that can not be exceeded. You are near this allocation. If you hit the Hard Disk Allocation allocation limit, your VPS container is likely to crash or encounter processing errors. To avoid this undesirable situation, you should delete unnecessary files or upgrade to a larger plan.

Pro-rata Daily Traffic Allowance Notice

The Pro-rata Daily Traffic Allowance notice alerts you that you may exceed your monthly bandwidth and data transfer allowance.

Each VPS has a certain amount of bandwidth allotted to them per their specific plan that is free to use. Once you exceed that limit, you are charged an overage fee.

The Pro-rata Daily Traffic Allowance notice is a script that monitors the bandwidth used by your VPS, and calculates based on that usage whether or not it appears that you will exceed your allotted bandwidth.

The script assumes that the data transfer rates it sees will continue throughout your 30 day billing cycle. You are then alerted to this fact, and if you are regularly getting these notices, can take measures to avoid any overage fees, such as reducing the amount of downloads from your VPS or upgrading to a higher plan with more resources.

The Pro-rata Daily Traffic Allowance is based on the billing cycle for your VPS, and will reset every 30 days. Note that 30 days does not mean that it resets at the end of the calendar month. For example, if you purchased your VPS on the 12th of the month, then your 30 day billing cycle runs from the 12th of the month to the 12th of the next month.

This is the text of the Pro-rata Daily Traffic Allowance notice:

Notes on warnings:

Pro-rata Daily Traffic Allowance: Each VPS container has a monthly allocation of data transfer that is free. Your pro rata daily usage of data transfer indicates that you may exceed your monthly allocation (assuming your daily usage profile continues throughout the monthly billing period). Data transfer used in excess of the monthly allocation is billable. There is no hard limit for Data Transfer, you may use as much as you need subject to billings in excess of your Monthly Data Transfer Allowance. If your daily usage indicates that you may have to pay data transfer overage fee for the total month, you should modify the usage profile of your VPS container, or upgrade to a larger plan.

Network Traffic Rate Allocation Notice

The Network Traffic Rate Allocation notice alerts you that you have exceeded or come close to exceeding the rate limit or traffic rate for your VPS. The rate limit is measured in Kbps, or Kilobits per second, and the allowed rate limit varies by plan. This is the amount of data/traffic that is allowed outbound from your VPS per second.

This is the text of the Network Traffic Rate Allocation notice:

Notes on warnings:
Hard Traffic rate limit allowance have been exceeded

Simple Resources Monitoring script

The following script and the associated cron job can be used to monitor your VPS and its resource usage at a predetermined interval, and the results e-mailed to you (or to multiple people).

The script uses the Linux commands top , free -m and df -h to check the current load on the VPS, the amount of memory being used and the amount of disk space available at the time the script was run.

Note that the amount of disk space shown in the Size column for Filesystem /dev/vzfs is the hard limit for your VPS. If you reach that limit, your VPS will shut down. The soft limit, which is the amount of disk space allotted to you per your plan, is usually 50% (fifty percent) of the hard limit. Please check your actual plan limits for the exact totals.

This script requires that you install the Development Tools application on your VPS. The Development Tools can be installed from the Control Panel, System Tab, Add Application (if you don't see it there, check All Applications - it may already be installed)

Using this script and interpreting the output requires some basic Linux knowledge. If you need assistance interpreting the output of the top command, there are many online resources that can assist.

The installation and configuration of this script requires that you are able to log in to your VPS using SSH, and can work as the root user. A knowledge of basic Linux commands and the ability to use the vi or vim editor (or any other plain text editor you may have on your VPS) is also required.

The script needs to be run from the /root directory, and the cron job has to be installed for the root user. The permissions on the script will need to be set correctly so that it can run.

eApps Support has tested this script, and verified that it does work. However, this script is not officially supported by eApps.

If you would like to use this script but do not have the necessary Linux skills to install and configure it, please contact eApps Support. The script can be installed and the cron job set up for a fee of $15.

Note that this script is offered AS IS, with NO WARRANTY and no implied support. Use this script at your own risk. If you do request assistance with this script, that could be considered a billable support request, at our standard rate of $15 per 10 minute increment.

Creating the load-alert.sh script

To create the script, connect to the VPS via SSH, and as the root user, do the following:

[root@example ~]# cd /root
[root@example ~]# vim load-alert.sh

Copy and paste the script at the end of this document (the load-alert.sh script) into the load-alert.sh file. Change only the following three items:

NOTIFY="0.00" - set this to the load average you wish to be alerted at. Note that a load average of 1.00 represents 100% of your CPU being used.

NOTIFY2="000000" - set this to the amount of available free memory that you wish to be alerted at, in Kilobytes (KB). For example, if you wished to be alerted when your VPS went below 200 MB of free memory, you would set this value to 200000.

EMAIL="user@example.com" - set this to the e-mail address you want the alerts to go to.

Once those changes have been made, save and exit the file. Then change the permissions on the file so that the script can execute.

[root@example ~]# chmod 755 load-alert.sh

Creating the cron job

To set up the cron job that will run the script and send the e-mails, as the root user do the following:

[root@example ~]# crontab -e

Add a line similar to this to the crontab file:

*/2 * * * * /root/load-alert.sh > /dev/null 2>&1
This will run the cron job every two minutes. If you wanted to run this cron job every 5 minutes, substitute a 5 for the 2.

If you wanted to run the cron job every hour, the cron file would look like this:
0 * * * * /root/load-alert.sh > /dev/null 2>&1

If you wanted to run the cron job every two hours, the cron file would look like this:
0 */2 * * * /root/load-alert.sh > /dev/null 2>&1

The syntax for cron can be fairly complex. There are many examples online that will explain how to set up your cron job with the interval you wish to use. Search for "cron syntax examples".

Once the cron job is created, save and exit the file.

This starts the cron job. Now the script will run at the specified time, and if the load average matches the threshold in the NOTIFY value, an e-mail will be generated to the person in the EMAIL value.

Other Configurations

If you want the script to send an e-mail every time the cron job is run, no matter what the load average is, set NOTIFY to "0.00".

If you want the cron job to run every minute (cron has no concept of seconds) so that it is constantly scanning for the specified load average, set the crontab file to:
* * * * * /root/load-alert.sh > /dev/null 2>&1

Be aware that depending on what you have the NOTIFY value set to, this could possibly send you an e-mail every minute of every hour of every day – 1440 e-mails a day, 10,080 e-mails a week, and approximately 525,600 e-mails a year.

If you want to send the e-mail alerts to more than one user, two things in the script have to change.
1. Add another EMAIL value to the CONFIGURATION section of the script. In this example the value of EMAIL1 was added:

# Set e-mail address to send alerts to
EMAIL="user@example.com"
EMAIL1="other_user@example.com"

2. Add that new EMAIL1 value to this line in the E-MAIL MESSAGE section of the script:

mail -s "$SUBJECT" "$EMAIL" < $TEMPFILE
so that it looks like this:
mail -s "$SUBJECT" "$EMAIL" "$EMAIL1" < $TEMPFILE

and save and exit the file. You can add as many new EMAIL users as you want.

Testing the script

If the script doesn't appear to be working, set the NOTIFY value back to "0.00", and then from the command line in the /root directory, run the following command:
[root@example ~]# ./load-alert.sh

That will run the script, and should send out the required e-mail. If no e-mails are being received, check the /var/log/maillog for any errors. If you still need assistance, then contact eApps Support.

Disabling or removing the load-alert.sh script

To disable the load-alert.sh script from running, remove or comment out the cron job that runs the script. As the root user, do the following:

[root@example ~]# crontab -e

Comment out the line for the load-alert.sh script, or remove it entirely. Save and exit the file, which will restart the cron daemon. The script should no longer run.

To remove the load-alert.sh script completely, as the root user do the following:

[root@example ~]# rm -rf load-alert.sh

This removes the script from the VPS. Then disable or remove the cron job so that it does not try to run a script that no longer exists.

The load-alert.sh script

The load-alert script. Save this file as load-alert.sh in /root on your VPS.

#-------------- COPY BELOW HERE ---------------------------------------#
#!/bin/bash
#
# Script to notify admin user if their eApps VPS load has crossed a
# certain limit. It will send an email notification to the designated
# admin user.
#
# This script is based on one found at http://bash.cyberciti.biz/ and
# modified to fit the needs of eApps customers.
#
# This is a free script under GNU GPL version 2.0 or above.
# http://www.gnu.org/licenses/gpl-3.0.txt
# This software is offered AS IS, with NO WARRANTIES
#
#----------------------------------------------------------------------#
#
# CONFIGURATION
#
# The values for NOTIFY, NOTIFY2 and EMAIL are the only values that should be
# changed. Change NOTIFY to the load average you want to alert on. Change
# NOTIFY2 to the memory usage you want to alert on. Change EMAIL to the
# admin e-mail address that the alerts should go to.
#
# Set load average threshold. A value of 1.00 equals 100% of your CPU

NOTIFY="0.00"

# Memory usage threshold. Set value in KB (Kilobytes)

NOTIFY2="000000"

#
# Set e-mail address to send alerts to

EMAIL="user@example.com"

#
# Subject line for the e-mail alerts

SUBJECT="Alert $(hostname) load average $(date)"

#
#----------------------------------------------------------------------#
#
OS="$(uname)"
TRUE="1"
        TEMPFILE="$(mktemp)"
        FTEXT='load average:'

# get first 5 min load
F5M="$(uptime | awk -F "$FTEXT" '{ print $2 }' | cut -d, -f1)"

# 10 min
F10M="$(uptime | awk -F "$FTEXT" '{ print $2 }' | cut -d, -f2)"

# 15 min
F15M="$(uptime | awk -F "$FTEXT" '{ print $2 }' | cut -d, -f3)"

# Free memory
FMEM="$(grep "^MemFree" /proc/meminfo|awk '{print $2}')"
#
#----------------------------------------------------------------------#
#
# E-MAIL MESSAGE
#

echo "Load average crossed specified limit: $NOTIFY" >> $TEMPFILE
echo "Hostname: $(hostname)" >> $TEMPFILE
echo "Local Date & Time : $(date)" >> $TEMPFILE
echo "Memory usage crossed specified limit: $NOTIFY2 kb" >> $TEMPFILE

# Look if it crossed limit
# compare it with last 15 min load average and last memory average
RESULT=$(echo "$F15M >= $NOTIFY" | bc)
RESULT2=$(echo "$FMEM <= $NOTIFY2" | bc)

# if so send an email
if [ "$RESULT" == "$TRUE" ]; then
    echo "-------------------------------------------" >> $TEMPFILE
    echo "System snapshot from top:" >> $TEMPFILE
    echo "-------------------------------------------" >> $TEMPFILE
    top -b -n1 >> $TEMPFILE
    echo "-------------------------------------------" >> $TEMPFILE
    echo "System snapshot for memory:" >> $TEMPFILE
    echo "-------------------------------------------" >> $TEMPFILE
    free -m >> $TEMPFILE
    echo "Process list by memory usage:" >> $TEMPFILE
    echo "-------------------------------------------" >> $TEMPFILE
    ps -e -orss=,args= | sort -b -k1,1n >> $TEMPFILE
    echo "-------------------------------------------" >> $TEMPFILE
    echo "Disk space usage:" >> $TEMPFILE
    echo "-------------------------------------------" >> $TEMPFILE
    df -h >> $TEMPFILE

        mail -s "$SUBJECT" "$EMAIL" < $TEMPFILE

fi

#
#----------------------------------------------------------------------#
#
# CLEAN UP
#

rm -f $TEMPFILE

#
#--------------- COPY ABOVE HERE --------------------------------------#

Example report from the load-alert.sh script

The following is an example of what the output of the load-alert.sh script looks like. This is just an example - your reports will have different information, but the basic format and structure will be the similar.

Subject: Alert example.com load average Wed Jan 13 07:28:50 EST 2010
Load average crossed specified limit: 0.00
Hostname: example.com
Local Date & Time : Wed Jan 13 07:28:50 EST 2010

-------------------------------------------
System snapshot from top:
-------------------------------------------
top - 07:28:51 up 74 days, 8:51, 1 user, load average: 0.97, 0.22, 0.07
Tasks: 38 total,   1 running, 37 sleeping,   0 stopped,   0 zombie
Cpu(s): 1.2%us, 0.7%sy, 0.0%ni, 98.0%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem:    442368k total,   165688k used,   276680k free,        0k buffers
Swap:        0k total,        0k used,        0k free,        0k cached
PID USER   PR NI   VIRT    RES    SHR    S   %CPU %MEM   TIME+     COMMAND
1    root   15   0   2056    592    568    S    0    0.1    0:14.91   init
2016 root   18   0   10020   2828   2268   S    0    0.6    0:00.04   sshd
3101 root   15   0   3740    1584   1284   S    0    0.4    0:00.03   bash
3448 root   18   0   3604    1312   1132   S    0    0.3    0:00.01   mysqld_safe
3620 rpc    19   0   1800    432    428    S    0    0.1    0:00.00   portmap
3681 mysql 15   0   124m    18m    6388   S    0    4.4    2:11.02   mysqld
9269 apache 18   0   61408   21m    4744   S    0    4.9    0:00.57   httpd
9272 apache 18   0   61408   21m    4744   S    0    4.9    0:00.55   httpd
9273 apache 18   0   61408   21m    4744   S    0    4.9    0:00.56   httpd
9361 root   18   0   3740    948    644    S    0    0.2    0:00.00   bash
9429 root   18   0   2176    924    732    R    0    0.2    0:00.00   top

-------------------------------------------
System snapshot for memory:
-------------------------------------------
          total       used       free     shared    buffers     cached
Mem:       432        161        270          0          0          0
-/+ buffers/cache:    161        270
Swap:        0          0          0

-------------------------------------------
Disk space usage:
-------------------------------------------
Filesystem            Size Used Avail Use% Mounted on
/dev/vzfs              17G 2.2G   14G 14%    /
none                  7.9G 4.0K 7.9G   1%    /dev

Links to other information

Linux top command man page - http://linux.die.net/man/1/top
Linux Crontab examples - http://www.thegeekstuff.com/2009/06/15-practical-crontab-examples/
Please login to comment