Otis Gospodnetic

There is an updated version of this at https://sematext.com/blog/top-docker-metrics-to-watch/ and https://sematext.com/blog/open-source-docker-monitoring-logging/ lists open source Docker monitoring tools.

Daniel, see https://sematext.com/docs/logagent/config-file/#section-output - you can point to your own Elasticsearch from the output plugin.

> One of the biggest challenges of building an ELK deployment is making it scalable.

Yup! More precisely, it's that E in ELK that's hard to scale. L and K are simple, relatively easy to set up run, inexpensive to operate. Elasticsearch is the one that requires expertise to scale, requires more babysiting, bigger infra, and is more expensive to run an operate. There are a number of log management solutions out there that, like https://sematext.com/cloud that give you Elasticsearch without management.

Nice list, Philip! For some organizations logging can indeed be expensive. We interact with a lot of companies who get stuck with building, running, and managing their own Elasticsearch clusters, for example. Nowadays, there are managed services (e.g. https://sematext.com/cloud ) that do that for you for much less when you add up all the infra and people time that gets sunk towards running something like Elasticsearch on your own.

Here is something fresh with a lot of Rsyslog + Elasticsearch how-tos: Rsyslog eBook: Centralized Logging with Rsyslog - https://sematext.com/blog/rsyslog-ebook-centralized-logging-with-rsyslog/

Nice overview! Sematext Docker Agent (http://sematext.com/docker ) does a good job of collecting logs (+ events and metrics) and here is nice Docker Enterprise Reference Architecture for monitoring & logging https://sematext.com/docker-enterprise-monitoring-and-logging/

Sounds useful and very similar to https://sematext.com/blog/2016/03/17/on-demand-java-profiling/

As does Sematext - see http://sematext.com/

Kimi - that's becase neither Apache Spark or Python are log shippers (and this is what Logstash is most often used for). Apache Spark is a framework for processing streaming data, and Python is a programming language. Sure, one can ship data from point A to point B with both of them.... but one can also do that with C, Bash, Perl, Ruby, and so on...

Oh, and Logsene has S3 archiving now.

A few corrections.

1) Sematext's Logagent provides built-in log parsing. See https://github.com/sematext/logagent-js

2) Logz.io limits users. I don't think other services limit that.

3) Logz.io base paid plain is 1 GB/month. Retention is not specified, I assume 1 week. Cost: $107/month when paid monthly. Sematext Logsene Pro plan is $60/mo. Standard plan is $50/mo. I didn't check the other services.

Hope these corrections help others.

I've got to point out Sematext, which Docker recently added as its Ecosystem Technology Partner for monitoring. Here is the Sematext's open-source Docker monitoring: https://github.com/sematext/sematext-agent-docker . The nice thing about it is that it also collect Docker and app logs and Docker events.

To take one of the above examples, say a developer decides to take these field boosts:

qf=title^5 content^2 comments^0.5

And change them to:

qf=title^10 content^5 comments^2

How is this developer going to tell whether the new boosts ultimately produce better results overall (i.e. not just in a limited number of tests one tries and eyeballs)?

Here is something related:
Relevance Tuning and Competitive Advantage via Search Analytics.

What else are people using to measure the effectiveness of search config changes?

Thanks.

To take one of the above examples, say a developer decides to take these field boosts:

qf=title^5 content^2 comments^0.5

And change them to:

qf=title^10 content^5 comments^2

How is this developer going to tell whether the new boosts ultimately produce better results overall (i.e. not just in a limited number of tests one tries and eyeballs)?

Here is something related:
Relevance Tuning and Competitive Advantage via Search Analytics.

What else are people using to measure the effectiveness of search config changes?

Thanks.

You can also flip through the slides here.

And you may want to see other presentations on Search Analytics (some with more technical info) here.

Oh, and if you want to actually get Search Analytics for your search app or website, the Search Analytics SaaS described above is still free - http://sematext.com/search-analytics/index.html

Enjoy!