All,
The following script can be used to check for dead or
otherwise problematic HTTP URLs in the scanModem utility.
--
Nick Zhuravlev
www.auriga.com
============================================
#!/bin/bash
if [ $# -ne 1 ]; then
echo "Usage: $0 path_to_scanModem"
exit 1
fi
HTTP_URLS=$(awk '{
if (match($0,/(http[^[:space:],]+)/, url)) {
printf "%s\n", url[0];
}
}' $1 | sort -u)
for i in $HTTP_URLS; do
echo === $i ===;
wget -T 7 --tries=1 --spider --force-html $i 2>&1 \
| egrep '(ERROR 404|failed|timed out)';
done | tee /tmp/full.log
