Re: Optimizing grep, sort, uniq for speed

[m.roth@xxxxxxxxx]

Sean Carolan wrote:
> This snippet of code pulls an array of hostnames from some log files.
> It has to parse around 3GB of log files, so I'm keen on making it as
> efficient as possible.  Can you think of any way to optimize this to
> run faster?
>
> HOSTS=()
> for host in $(grep -h -o "[-\.0-9a-z][-\.0-9a-z]*.com" ${TMPDIR}/* |
> sort | uniq); do
>     HOSTS+=("$host")
> done

For one, do the sort in one step: sort -u. For another, are the hostnames
always the same field? For example, if they're all /var/log/messages, I'd
do awk '{print $4;}' | sort -u

       mark


_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos