On Mon, 22 Aug 2016, 3:40 p.m. Thomas Güttler, <guettliml@xxxxxxxxxxxxxxxxxx> wrote:
Am 19.08.2016 um 19:59 schrieb Andy Colson:
> On 8/19/2016 2:32 AM, Thomas Güttler wrote:
>> I want to store logs in a simple table.
>>
>> Here my columns:
>>
>> Primary-key (auto generated)
>> timestamp
>> host
>> service-on-host
>> loglevel
>> msg
>> json (optional)
>>
>> I am unsure which DB to choose: Postgres, ElasticSearch or ...?
>>
>> We don't have high traffic. About 200k rows per day.
>>
>> My heart beats for postgres. We use it since several years.
>>
>> On the other hand, the sentence "Don't store logs in a DB" is
>> somewhere in my head.....
>>
>> What do you think?
>>
>>
>>
>
> I played with ElasticSearch a little, mostly because I wanted to use Kibana which looks really pretty. I dumped a ton
> of logs into it, and made a pretty dashboard ... but in the end it didn't really help me, and wasn't that useful. My
> problem is, I don't want to have to go look at it. If something goes bad, then I want an email alert, at which point
> I'm going to go run top, and tail the logs.
>
> Another problem I had with kibana/ES is the syntax to search stuff is different than I'm used to. It made it hard to
> find stuff in kibana.
>
> Right now, I have a perl script that reads apache logs and fires off updates into PG to keep stats. But its an hourly
> summary, which the website turns around and queries the stats to show pretty usage graphs.
You use Perl to read apache logs. Does this work?
Forwarding logs reliably is not easy. Logs are streams, files in unix are not streams. Sooner or later
the files get rotated. RELP exists, but AFAIK it's usage is not wide spread:
https://en.wikipedia.org/wiki/Reliable_Event_Logging_Protocol
Let's see how to get the logs into postgres ....
> In the end, PG or ES, all depends on what you want.
Most of my logs start from a http request. I want a unique id per request
in every log line which gets created. This way I can trace the request,
even if its impact spans to several hosts and systems which do not receive http requests.
You may decide not to use Elasticsearch but take a look at other components of Elastic Stack like logstash and beats. They can be helpful even when you use Postgres as the end point. Otherwise (IMHO), you would spend a lot of time writing scripts and jobs to capture and stream logs. If I were you, I would not want to do that.
Regards,
Thomas Güttler
--
Thomas Guettler http://www.thomas-guettler.de/
--
Sent via pgsql-general mailing list (pgsql-general@xxxxxxxxxxxxxx)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
--
--
Best Regards
Sameer Kumar | DB Solution Architect
ASHNIK PTE. LTD.
101 Cecil Street, #11-11 Tong Eng Building, Singapore 069 533
T: +65 6438 3504 | M: +65 8110 0350
Skype: sameer.ashnik | www.ashnik.com
