Skip to main content
Skip to main content
Edit this page

Monitoring EC2 Host Logs with ClickStack

TL;DR

Monitor EC2 system logs with ClickStack by installing OpenTelemetry Collector on your instances. The collector automatically enriches logs with EC2 metadata (instance ID, region, availability zone, instance type). You'll learn how to:

  • Install and configure OpenTelemetry Collector on EC2 instances
  • Automatically enrich logs with EC2 metadata
  • Send logs to ClickStack via OTLP
  • Use a pre-built dashboard to visualize EC2 host logs with cloud context

A demo dataset with sample logs and simulated EC2 metadata is available for testing.

Time Required: 10-15 minutes

Integration with existing EC2 instance​

This section covers installing OpenTelemetry Collector on your EC2 instances to collect system logs and send them to ClickStack with automatic EC2 metadata enrichment. This distributed architecture is production-ready and scales to multiple instances.

Running ClickStack on the same EC2 instance?

If ClickStack is running on the same EC2 instance whose logs you want to monitor, you can use the all-in-one approach similar to the Generic Host Logs guide. Mount /var/log into the ClickStack container and add the resourcedetection processor to your custom config to automatically capture EC2 metadata. This guide focuses on the more common distributed architecture for production deployments.

If you would like to test the EC2 host logs integration before configuring your production instance, you can test with our preconfigured setup and sample data in the "Demo dataset" section.

Prerequisites​
  • ClickStack instance running (can be on-premises, cloud, or local)
  • EC2 instance running (Ubuntu, Amazon Linux, or other Linux distribution)
  • Network connectivity from EC2 instance to ClickStack's OTLP endpoint (port 4318 for HTTP or 4317 for gRPC)
  • EC2 instance metadata service accessible (enabled by default)

Verify EC2 metadata is accessible​

From your EC2 instance, verify the metadata service is accessible:

# Get metadata token (IMDSv2)
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

# Verify instance metadata
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/placement/region
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-type

You should see your instance ID, region, and instance type. If these commands fail, verify:

  • The instance metadata service is enabled
  • IMDSv2 is not blocked by security groups or network ACLs
  • You're running these commands from the EC2 instance itself
Note

EC2 metadata is available at http://169.254.169.254 from within the instance. The OpenTelemetry resourcedetection processor uses this endpoint to automatically enrich logs with cloud context.

Verify syslog files exist​

Verify that your EC2 instance is writing syslog files:

# Ubuntu instances
ls -la /var/log/syslog

# Amazon Linux / RHEL instances
ls -la /var/log/messages

# View recent entries
tail -20 /var/log/syslog
# or
tail -20 /var/log/messages

Install OpenTelemetry Collector​

Install the OpenTelemetry Collector Contrib distribution on your EC2 instance:

# Download the latest release
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.114.0/otelcol-contrib_0.114.0_linux_amd64.tar.gz

# Extract and install
tar -xvf otelcol-contrib_0.114.0_linux_amd64.tar.gz
sudo mv otelcol-contrib /usr/local/bin/

# Verify installation
otelcol-contrib --version

Create collector configuration​

Create a configuration file for the OpenTelemetry Collector at /etc/otelcol-contrib/config.yaml:

sudo mkdir -p /etc/otelcol-contrib

Choose the configuration based on your Linux distribution:

sudo tee /etc/otelcol-contrib/config.yaml > /dev/null << 'EOF'
receivers:
  filelog/syslog:
    include:
      - /var/log/syslog
      - /var/log/**/*.log
    start_at: end
    operators:
      - type: regex_parser
        regex: '^(?P<timestamp>\S+) (?P<hostname>\S+) (?P<unit>\S+?)(?:\[(?P<pid>\d+)\])?: (?P<message>.*)$'
        parse_from: body
        parse_to: attributes
      
      - type: time_parser
        parse_from: attributes.timestamp
        layout_type: gotime
        layout: '2006-01-02T15:04:05.999999-07:00'
      
      - type: add
        field: attributes.source
        value: "ec2-host-logs"

processors:
  resourcedetection:
    detectors: [ec2, system]
    timeout: 5s
    override: false
    ec2:
      tags:
        - ^Name
        - ^Environment
        - ^Team
  
  batch:
    timeout: 10s
    send_batch_size: 1024

exporters:
  otlphttp:
    endpoint: "http://YOUR_CLICKSTACK_HOST:4318"
    headers:
      authorization: "${env:CLICKSTACK_API_KEY}"

service:
  pipelines:
    logs:
      receivers: [filelog/syslog]
      processors: [resourcedetection, batch]
      exporters: [otlphttp]
EOF

Replace the following in the configuration:

  • YOUR_CLICKSTACK_HOST: The hostname or IP address where ClickStack is running
  • For local testing, you can use an SSH tunnel (see Troubleshooting section)

This configuration:

  • Reads system log files from standard locations (/var/log/syslog for Ubuntu, /var/log/messages for Amazon Linux/RHEL)
  • Parses syslog format to extract structured fields (timestamp, hostname, unit/service, PID, message)
  • Automatically detects and adds EC2 metadata using the resourcedetection processor
  • Optionally includes EC2 tags (Name, Environment, Team) if present
  • Sends logs to ClickStack via OTLP HTTP
EC2 Metadata Enrichment

The resourcedetection processor automatically adds these attributes to every log:

  • cloud.provider: "aws"
  • cloud.platform: "aws_ec2"
  • cloud.region: AWS region (e.g., "us-east-1")
  • cloud.availability_zone: AZ (e.g., "us-east-1a")
  • cloud.account.id: AWS account ID
  • host.id: EC2 instance ID (e.g., "i-1234567890abcdef0")
  • host.type: Instance type (e.g., "t3.medium")
  • host.name: Instance hostname

Set ClickStack API key​

Export your ClickStack API key as an environment variable:

export CLICKSTACK_API_KEY="your-api-key-here"

To make this persistent across reboots, add it to your shell profile:

echo 'export CLICKSTACK_API_KEY="your-api-key-here"' >> ~/.bashrc
source ~/.bashrc

Run the collector​

Start the OpenTelemetry Collector:

CLICKSTACK_API_KEY="your-api-key-here" /usr/local/bin/otelcol-contrib --config /etc/otelcol-contrib/config.yaml
For production use

Configure the collector to run as a systemd service so it starts automatically on boot and restarts on failure. See the OpenTelemetry Collector documentation for details.

Verifying Logs in HyperDX​

Once the collector is running, log into HyperDX and verify logs are flowing with EC2 metadata:

  1. Navigate to the search view
  2. Set source to Logs
  3. Filter by source:ec2-host-logs
  4. Click on a log entry to expand it
  5. Verify you see EC2 metadata in the resource attributes:
    • cloud.provider
    • cloud.region
    • host.id (instance ID)
    • host.type (instance type)
    • cloud.availability_zone

Demo dataset​

For users who want to test the EC2 host logs integration before configuring their production instances, we provide a sample dataset with simulated EC2 metadata.

Download the sample dataset​

Download the sample log file:

curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/host-logs/journal.log

The dataset includes:

  • System boot sequence
  • SSH login activity (successful and failed attempts)
  • Security incident (brute force attack with fail2ban response)
  • Scheduled maintenance (cron jobs, anacron)
  • Service restarts (rsyslog)
  • Kernel messages and firewall activity
  • Mix of normal operations and notable events

Create test collector configuration​

Create a file named ec2-host-logs-demo.yaml with the following configuration:

cat > ec2-host-logs-demo.yaml << 'EOF'
receivers:
  filelog/journal:
    include:
      - /tmp/host-demo/journal.log
    start_at: beginning
    operators:
      - type: regex_parser
        regex: '^(?P<timestamp>\S+) (?P<hostname>\S+) (?P<unit>\S+?)(?:\[(?P<pid>\d+)\])?: (?P<message>.*)$'
        parse_from: body
        parse_to: attributes
      
      - type: time_parser
        parse_from: attributes.timestamp
        layout: '%Y-%m-%dT%H:%M:%S%z'
      
      - type: add
        field: attributes.source
        value: "ec2-demo"

processors:
  # Simulate EC2 metadata for demo (no real EC2 instance required)
  resource:
    attributes:
      - key: service.name
        value: "ec2-demo"
        action: insert
      - key: cloud.provider
        value: "aws"
        action: insert
      - key: cloud.platform
        value: "aws_ec2"
        action: insert
      - key: cloud.region
        value: "us-east-1"
        action: insert
      - key: cloud.availability_zone
        value: "us-east-1a"
        action: insert
      - key: host.id
        value: "i-0abc123def456789"
        action: insert
      - key: host.type
        value: "t3.medium"
        action: insert
      - key: host.name
        value: "prod-web-01"
        action: insert

service:
  pipelines:
    logs/ec2-demo:
      receivers: [filelog/journal]
      processors:
        - resource
        - memory_limiter
        - transform
        - batch
      exporters:
        - clickhouse
EOF
Note

For demo purposes, we manually add EC2 metadata using the resource processor. In production with real EC2 instances, use the resourcedetection processor which automatically queries the EC2 metadata API.

Run ClickStack with demo configuration​

Run ClickStack with the demo logs and configuration:

docker run --name clickstack-demo \
  -p 8080:8080 -p 4317:4317 -p 4318:4318 \
  -e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
  -v "$(pwd)/ec2-host-logs-demo.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
  -v "$(pwd)/journal.log:/tmp/host-demo/journal.log:ro" \
  docker.hyperdx.io/hyperdx/hyperdx-all-in-one:latest

Verify logs in HyperDX​

Once the collector is running:

  1. Open HyperDX and log in to your account (you may need to create an account first)
  2. Navigate to the Search view and set the source to Logs
  3. Set the time range to 2025-11-10 00:00:00 - 2025-11-13 00:00:00
  4. Filter by source:ec2-demo
  5. Expand a log entry to view EC2 metadata in the resource attributes
Timezone Display

HyperDX displays timestamps in your browser's local timezone. The demo data spans 2025-11-11 00:00:00 - 2025-11-12 00:00:00 (UTC). The wide time range ensures you'll see the demo logs regardless of your location. Once you see the logs, you can narrow the range to a 24-hour period for clearer visualizations.

You should see logs with simulated EC2 context including:

  • Instance ID: i-0abc123def456789
  • Region: us-east-1
  • Availability Zone: us-east-1a
  • Instance Type: t3.medium

Dashboards and visualization​

To help you get started monitoring EC2 host logs with ClickStack, we provide essential visualizations with cloud context.

Download the dashboard configuration​

Import the pre-built dashboard​

  1. Open HyperDX and navigate to the Dashboards section
  2. Click Import Dashboard in the upper right corner under the ellipses
  1. Upload the host-logs-dashboard.json file and click Finish Import

View the dashboard​

The dashboard will be created with all visualizations pre-configured:

You can filter dashboard visualizations by EC2 context:

  • cloud.region:us-east-1 - Show logs from specific region
  • host.type:t3.medium - Filter by instance type
  • host.id:i-0abc123def456 - Logs from specific instance
Note

For the demo dataset, set the time range to 2025-11-11 00:00:00 - 2025-11-12 00:00:00 (UTC) (adjust based on your local timezone). The imported dashboard will not have a time range specified by default.

Troubleshooting​

EC2 metadata not appearing in logs​

Verify the EC2 metadata service is accessible:

# Get metadata token
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

# Test metadata endpoint
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id

If this fails, verify:

  • The instance metadata service is enabled
  • IMDSv2 is not blocked by security groups
  • You're running the collector on the EC2 instance itself

Check collector logs for metadata errors:

# If running as systemd service
sudo journalctl -u otelcol-contrib -f | grep -i "ec2\|metadata\|resourcedetection"

# If running in foreground, check stdout

No logs appearing in HyperDX​

Verify syslog files exist and are being written:

ls -la /var/log/syslog /var/log/messages
tail -f /var/log/syslog

Check collector can read the log files:

cat /var/log/syslog | head -20

Verify network connectivity to ClickStack:

# Test OTLP endpoint
curl -v http://YOUR_CLICKSTACK_HOST:4318/v1/logs

# Should get a response (even if error, means endpoint is reachable)

Check collector logs for errors:

# If running in foreground
# Look for error messages in stdout

# If running as systemd service
sudo journalctl -u otelcol-contrib -f | grep -i "error\|failed"

Logs parsing incorrectly​

Verify your syslog format:

For Ubuntu 24.04+:

# Should show ISO8601 format: 2025-11-17T20:55:44.826796+00:00
tail -5 /var/log/syslog

For Amazon Linux 2 / Ubuntu 20.04:

# Should show traditional format: Nov 17 14:16:16
tail -5 /var/log/messages

If your format doesn't match, use the appropriate configuration tab in the Create collector configuration section based on your distribution.

Collector not starting as systemd service​

Check service status:

sudo systemctl status otelcol-contrib

View detailed logs:

sudo journalctl -u otelcol-contrib -n 50

Common issues:

  • API key not set correctly in environment
  • Config file syntax errors
  • Permission issues reading log files

Next steps​

After setting up EC2 host logs monitoring:

  • Set up alerts for critical system events (service failures, authentication failures, disk warnings)
  • Filter by EC2 metadata attributes (region, instance type, instance ID) to monitor specific resources
  • Correlate EC2 host logs with application logs for comprehensive troubleshooting
  • Create custom dashboards for security monitoring (SSH attempts, sudo usage, firewall blocks)