To manage and process logs of multiple webservers, Rsync provides the best method for transferring the logs from servers to centralized log processing server.
There are two major steps involved
a) Configuring the Log Processing Server (IP: 192.168.1.1)
b) Configuring the client (say, www.myserver.com) to transfer the logs to the central server.
Configuring the Log Processing Server
Let this server have IP address 192.168.1.1. We create a directory , /usr/local/logs, where the log files would be downloaded. We create a subdirectory for www.myserver.com, under /usr/local/logs. (mkdir /usr/local/logs/www.myserver.com)
a. create a group logman and add user logman to it. This will be the uid/gid for the log files
b. edit/create /etc/rsyncd.conf, with the following details:
uid = logman
gid = logman
use chroot = yes
max connections = 4
log file = /var/log/rsyncd.log
pid file = /etc/rsyncd.pid
comment = here are the apache access logs from www.myserver.com downloaded
path = /usr/local/logs/www.myserver.com/
hosts allow = www.myserver.com
read only = no
c. Now, run rsync:
# rsync –daemon
We have now successfully configured our server to received log files.
Configuring the clients to transfer the logs
on the client system (www.myserver.com), run this command periodically, to transfer the logs:
rsync -azvu /usr/local/apache/logs/access_logs 192168.1.1::www.myserver.com_logs
This way, the logs would be transferred to 192.168.1.1, and would be updated (not deleted and recreated, differential transfer) everytime.