Server Logs on Ubuntu & Introduction to Central Logging

GNU/Linux sysadmins need to look at the log files for various purposes. Logs are usually located at /var/log. You can run ls -l /var/log on your server to realize the big size of list on a production server. It is not practical to run UNIX command to check the logs when the number of servers are not exactly few. It is impossible to detect a WordPress XML RPC attack without proper central logging system. Logging & Monitoring is a Part of Security and Optimization. Here is a Guide on Server Logs on Ubuntu & Introduction to Central Logging.

Server Logs on Ubuntu & Introduction to Central Logging

There are basic tools like syslog or rsyslog. We talked about Urban Airship before, that can be combined with rsyslog :

Second way is to use some typical Free softwares and paid softwares intended to be installed on own server which will act as server for centralized, distributed logging for the other servers. Scribe, Flume, logstash, Chukwa, fluentd, Kafka, Graylog2, Splunk are typical examples of such distributed logging.

The above softwares are used with a typical combination of softwares to visualize, analyze, search and have a web GUI. With $7/month 6GB VPS, you really can not hugely complain about the pricing. We recommend to understand the need for your situation, plan for the infrastructure and use these self hosted typical stacks for centralized, distributed logging. Setup may be time consuming, painful but it is cost effective, security is more. These are suitable for a typical website like that of this one – we will run the website virtually forever. We are not going in to dissection of each type of system in this article.

Third category for centralized logging is fully different – you need not to install much softwares. They are typical Logging as a Service – a kind of SaaS.

Server Logs on Ubuntu & Introduction to Central Logging : Ready to Use SaaS

Typical example of SaaS loggers are Loggly, Splunk Storm, Paper Trail, Bugsnag and the list is endless. Usually they have a free usage tier. Thereafter they have a paid tier.

These services, which we marked as Ready to Use SaaS, can be setup within few minutes. This may be good for monitoring an application running on Heroku, but they are not really cost effective when you already have many servers, using CDN etc.

Unlike the transactional email services, where we give importance more to Mandrill for WordPress over self hosted, using Ready to Use SaaS for Central Logging is not exactly a great idea for long term usage. It may be good if you have one domain and one server or need to monitor just an application running on PaaS. Apart from cost, there are security and bug related matters.