DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
View Events Video Library
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Integrating PostgreSQL Databases with ANF: Join this workshop to learn how to create a PostgreSQL server using Instaclustr’s managed service

Mobile Database Essentials: Assess data needs, storage requirements, and more when leveraging databases for cloud and edge applications.

Monitoring and Observability for LLMs: Datadog and Google Cloud discuss how to achieve optimal AI model performance.

Automated Testing: The latest on architecture, TDD, and the benefits of AI and low-code tools.

Related

  • Auto Remediation of GuardDuty Findings for a Compromised ECS Cluster in AWSVPC Network Mode
  • Building an ETL Pipeline With Airflow and ECS
  • Microservices on AWS Fargate
  • Containerize Gradle Apps and Deploy to Kubernetes With JKube Kubernetes Gradle Plugin

Trending

  • Five Tools for Data Scientists to 10X their Productivity
  • Writing Reusable SQL Queries for Your Application With DbVisualizer Scripts
  • Memory Management in Java: An Introduction
  • Selecting the Right Automated Tests
  1. DZone
  2. Software Design and Architecture
  3. Cloud Architecture
  4. Amazon ECS Log Analysis (Part 2): Now With More Docker

Amazon ECS Log Analysis (Part 2): Now With More Docker

Getting the logs out of a Docker container is only the first step, but it's an important first step. See how you can use Docker to help with your logging.

Daniel Berman user avatar by
Daniel Berman
·
Oct. 22, 16 · Tutorial
Like (2)
Save
Tweet
Share
8.14K Views

Join the DZone community and get the full member experience.

Join For Free

in the previous post in this series, i explained how to log ecs container api actions with cloudtrail and elk . in this second part of the series, i will describe how to log the docker containers themselves.

there are various ways of logging docker containers, each with its own advantages and disadvantages. of course, the challenge in a dockerized environment, whether on ecs or in a local docker environment, is the fact that containers are transient, dynamic, and distributed in nature. not only that, containers produce various types of outputs, so a logging solution that tracks docker container output will not necessarily track docker daemon events, for example.

in the steps below, i will describe two different methods for logging ecs docker containers — one uses a docker log collector that pulls container logs, stats, and daemon events into the logz.io elk stack . the second option, uses the aws-logs logging driver to log container logs only to cloudwatch.

prerequisites

this article assumes that you have an ecs cluster setup in aws. if you do not have that setup and are just getting started with the service, i highly recommend this getting started with ecs console walk-through — it will take you through the steps for setting up a cluster that will service a simple web application. i also recommend reading up on the ecs service on the aws documentation site .

logging to the logz.io elk stack

to pass container logs to the logz.io elk stack using the log collector , we need to create a new task definition that includes a volume definition and passes the logz.io token as an environment variable.

in the ecs console, select the task definitions tab on the left and then click the create new task definition button.

name the new task definition (e.g., ‘logzio-docker’), and if you have a specific iam role you wish to attach to the task, enter it in the task role field.

before we configure a new container, we are going to define a new data volume for our task to make sure that the data will persist even after all of the containers that referenced it have stopped.

under the volumes section, click add volume , and enter the following volume:

add volume

next, enter the following json configuration:

{
   "networkmode": "bridge",
   "taskrolearn": null,
   "containerdefinitions": [
      {
         "volumesfrom": [],
         "memory": null,
         "extrahosts": null,
         "dnsservers": null,
         "disablenetworking": null,
         "dnssearchdomains": null,
         "portmappings": [],
         "hostname": null,
         "essential": true,
         "entrypoint": null,
         "mountpoints": [
            {
               "containerpath": "/var/run/docker.sock",
               "sourcevolume": "docker-sock",
               "readonly": true
            }
         ],
         "name": "logzio-docker",
         "ulimits": null,
         "dockersecurityoptions": null,
         "environment": [
            {
               "name": "logzio_token",
               "value": "logz-token"
            }
         ],
         "links": null,
         "workingdirectory": null,
         "readonlyrootfilesystem": false,
         "image": "logzio/logzio-docker",
         "command": null,
         "user": null,
         "dockerlabels": null,
         "logconfiguration": null,
         "cpu": 0,
         "privileged": true,
         "memoryreservation": 128
      }
   ],
   "volumes": [
      {
         "host": {
            "sourcepath": "/var/run/docker.sock"
         },
         "name": "docker-sock"
      }
   ],
   "family": "logzio-docker"
}


make sure that you replace the ‘logz-token’ placeholder with your logz.io account token before saving the configuration.

also, please note that some of the container options used in this configurations are optional — change them as you see fit in terms of however it will suit your requirements. if this is your first time, you might want to use the new container wizard (just click add container on the new task definition page).

the next step is to create (or update) the task and run it in your ecs cluster. if all goes well, you should see a green running status alongside the task in your cluster view.

all that’s left to do now is to open your logz.io user interface and begin to analyze the logs:

analyze the logs

remember, the log collector collects a mixture of logs — docker daemon events, docker logs, and docker stats. be sure to differentiate the different output types using the type field, as shown in the image above.

analyzing and visualizing the logs

great! you’ve shipped the ecs container logs into elk. what next?

the next step is to begin to analyze the data, a task for which elk is renowned due to its rich querying options and visualization features. use kibana to search container logs for events taking place in your environment, and use the visualizations to create your own monitoring dashboard.

to help you hit the ground running, you can use one of the elk apps — a library of free customized kibana searches, alerts, visualizations, and dashboards — to install a pre-built dashboard for logs collected by the log collector .

here’s what it looks like:

log collector dashboard

logging to aws cloudwatch

another way you can log ecs containers is by using cloudwatch and docker’s built-in log drivers. as described in this post, there are a number of drivers for logging docker containers, each of which allows one to export the stderr and stdout output for containers to a specific logging tool.

using the aws-logs driver, you can send these container logs to cloudwatch for storage and analysis in third-party monitoring and logging tools. note, this option does not allow you to aggregate docker stats and docker daemon events in the same pipeline.

to log to cloudwatch, you will need to update the logging section of your ecs task definition (set to “null,” usually).

when you edit the task’s json configuration, the logging section will look as follows:

"logconfiguration": {
         "logdriver": "awslogs",
         "options": {
            "awslogs-group": "log-group",
            "awslogs-region": "aws-region",
            "awslogs-stream-prefix": "log-stream-prefix"
         }
      },


make sure to replace the placeholders above with your cloudwatch configurations:

  • log-group : the name of your cloudwatch log group.
  • aws-region : the the aws region for cloudwatch.
  • log-stream-prefix : the prefix and name of your cloudwatch log stream.

create and run the task in your ecs cluster. open the log stream in cloudwatch, and you should start to see your container logs:

cloudwatch log stream

once in cloudwatch, you can tap into any other monitoring and logging system to analyze the logs. you could, for example, ship the logs to an s3 bucket for storage and extraction into the elk stack.

for related information, i recommend reading our blog post on shipping s3 logs into the logz.io elk stack as well .

Docker (software) Entity component system Log analysis Task (computing)

Published at DZone with permission of Daniel Berman, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Auto Remediation of GuardDuty Findings for a Compromised ECS Cluster in AWSVPC Network Mode
  • Building an ETL Pipeline With Airflow and ECS
  • Microservices on AWS Fargate
  • Containerize Gradle Apps and Deploy to Kubernetes With JKube Kubernetes Gradle Plugin

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: