Menu
Article

Simplifying Nginx Access Log Management with AWS CloudWatch

Martin Karrer

Martin Karrer

• 2 min read

Sometimes, as website administrators, we want to review our Nginx access logs without the need to SSH into our instance. What if I told you that you could seamlessly send these logs to AWS CloudWatch? Let's dive into how you can achieve this!

Step 1: Generate JSON-formatted Logs on your EC2 Instance Nginx Server

To begin, we'll need to configure your EC2 instance's Nginx server to generate logs in JSON format. If you're using Session Manager, you're in luck, but you can also access your instance via SSH for the configuration. Here's what you need to do:

Use the following command to open the Nginx configuration:

sudo vim /etc/nginx/nginx.conf

Within the Logging Section, add the following configuration to define the JSON log format:

log_format json_log escape=json '{"time_iso8601":"$time_iso8601",'
                                 '"remote_addr":"$remote_addr",'
                                 '"request_time":$request_time,'
                                 '"request":"$request",'
                                 '"status":$status,'
                                 '"body_bytes_sent":$body_bytes_sent,'
                                 '"http_user_agent":"$http_user_agent"}';
access_log /var/log/nginx/access.json.log json_log;

Restart Nginx to apply the changes:bashCopy codesudo systemctl restart nginx

Open a webpage hosted by Nginx to verify that the logging is functioning as expected: cat /var/log/nginx/access.json.log

Step 2: Grant EC2 Instance IAM Role Access to CloudWatch

For your instance to send logs to CloudWatch Logs, you'll need to modify the IAM Role associated with the EC2 instance. Ensure that the role includes the CloudWatchAgentServerPolicy policy, which authorizes the instance to send logs to CloudWatch.

Step 3: Install and Configure CloudWatch Log Agent

Now, let's bridge the gap between your instance's disk and CloudWatch. To do this, we'll need to install the CloudWatch Log Agent:

Amazon Linux:

sudo yum install amazon-cloudwatch-agent

Ubuntu:

wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
sudo dpkg -i -E ./amazon-cloudwatch-agent.deb amazon-cloudwatch-agent-ctl -a status

Step 4: Write and Load the Configuration

We're almost there! To finalize the setup, you'll need to configure the CloudWatch Log Agent to collect and send the Nginx logs to CloudWatch. Here's how:

  1. Write the configuration to a file (for example, cwa.conf): 
{
    "logs": {
        "logs_collected": {
            "files": {
                "collect_list": [
                    {
                        "file_path": "/var/log/nginx/access.log",
                        "log_group_name": "nginx",
                        "log_stream_name": "access_log",
                        "timezone": "UTC"
                    }, {
                        "file_path": "/var/log/nginx/access.json.log",
                        "log_group_name": "nginx",
                        "log_stream_name": "access_json_log",
                        "timezone": "UTC"
                    }
                ]
            }
        },
        "log_stream_name": "default" 
    }
}
  1. Load the configuration into the CloudWatch Agent:
sudo amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file://home/ssm-user/cwa.conf

Step 5: Check CloudWatch Logs

With the setup complete, head to your CloudWatch Logs console. You should now see a new Log Group containing both your standard Nginx logs and the newly formatted JSON logs.

By following these steps, you've seamlessly integrated your Nginx logs with AWS CloudWatch, making log analysis and troubleshooting a breeze!

Happy log tracking!

Share