Go back

Create a Splunk app to monitor Nginx

Using the web interface of Splunk you can easily add a new app. This will create the following structure:

nginxwatcher/
|-- bin
|   `-- README
|-- default
|   |-- app.conf
|   |-- data
|   |   `-- ui
|   |       |-- nav
|   |       |   `-- default.xml
|   |       `-- views
|   |           `-- README
|   |-- props.conf
|-- local
|   `-- app.conf
`-- metadata
    |-- default.meta
    `-- local.meta

To the nginxwatcher/default folder the files indexes.conf and inputs.conf should be added.

The content of indexes.conf should be something like the following:

[nginxwatcher]
coldPath = $SPLUNK_DB\nginxwatcher\colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB\nginxwatcher\db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB\nginxwatcher\thaweddb

The content of inputs.conf should indicate we are monitoring the Nginx logging folder:

[monitor:///var/log/nginx/*.log]
disabled = false
host = nginxwatcher
index = nginxwatcher
sourcetype = nginxwatcher_logs

The contens of props.conf looks like this:

[nginxwatcher_logs]
NO_BINARY_CHECK = true
TZ = UTC
category = Structured
pulldown_type = 1
KV_MODE = none
disabled = false

After restarting Splunk we can start using the new index of the Nginx app and query with:

index="nginxwatcher"

To retrieve the values of the log files, we can use regular expressions and follow the description on the Nginx website. 

index="nginxwatcher" | rex field=_raw "^(?<remote_addr>\d+.\d+.\d+.\d+)\s-\s(?P<remote_user>.*?)\s\[(?<localtime>\d+\/\w+\/\d{4}:\d{2}:\d{2}:\d{2}\s\+\d+)\]\s+\"(?<request>.*?)\"\s(?<status>\d+)\s(?<bytes_sent>\d+.*?)\s\"(?<http_refererer>.*?)\"\s\"(?<http_user_agent>.*)\""

Obviously we do not want to do this every time and we extract the values using the props.conf. The file is changed to:

[nginxwatcher_logs]
NO_BINARY_CHECK = true
TZ = UTC
category = Structured
pulldown_type = 1
KV_MODE = none
disabled = false
EXTRACT-e1 = ^(?<remote_addr>\d+.\d+.\d+.\d+)\s-\s(?P<remote_user>.*?)\s\[(?<localtime>\d+\/\w+\/\d{4}:\d{2}:\d{2}:\d{2}\s\+\d+)\]\s+\"(?<request>.*?)\"\s(?<status>\d+)\s(?<bytes_sent>\d+.*?)\s\"(?<http_refererer>.*?)\"\s\"(?<http_user_agent>.*)\"

The following query will show a timechart for the status over time, grouped by 30 minutes.

index="nginxwatcher" 
| timechart count(status) span=30m by status

Top visitors:

index="nginxwatcher"
| top limit=20 remote_addr

Pages which produce the most errors:

index="nginxwatcher"
| search status >= 500 
| stats count(status) as cnt by request, status
| sort cnt desc

All these graphs can of course be added to a dashboard to keep a close watch on the webserver.

Splunk tutorial dashboard Nginx
Go back