Well.. I tried.
I need to ban 8613 URLs. Because a law.
If I put one per line in a file and set the filename for an url_regex acl it works. But when the traffic goes up the cpu load goes 100% (even using workers) and the proxy turns unusable.
I tested and saw my squid can't parse regexes with more than 8192 characters.
I managed to combine the 8000 uris in 34 regexes using a ruby gem, and the cpu load stays almost at the same level it is without any acl (same traffic).
The problem is that of the 34 regexes 33 works fine and 1 matches everything.
"http://something" matches.
the regex is:
^http:\/\/(m(i(n(i(\.free\-teen\-pussy\.com\/scj\/thumbs\/(0\/(253_Mini_Mini\.jpg|306_Mini_Mini\.jpg|468_Pree_Pree\.jpg|519_Models_Models\.jpg|688_Models_nn\.jpg|7(17_Models_Models\.jpg|22_Models_Mini\.jpg)|880_nn_Models\.jpg)|1\/(170_little_models\.jpg|271_Young_Young\.jpg|330_models_little\.jpg|412_little_models\.jpg|6(28_little_models\.jpg|43_little_little\.jpg)|742_Young_Young\.jpg|877_littlemodels_littlemodels\.jpg|903_littlemodels_littlemodels\.jpg)|2\/(0(47_models_little\.jpg|92_models_models\.jpg)|1(44_little_little\.jpg|75_Young_Young\.jpg)|2(04_littlemodels_littlemodels\.jpg|29_russian_russian\.jpg|76_russian_little\.jpg)))|girls\.biz)|t(ladies\.com\/|teens\.com\/))|ragetube\.com\/content\/(35\/907_un_pequena\.jpg|75\/939_sex_Simpsons\.jpg|95\/473_Babe_039\.jpg))|mmgay\.com\/|o(dels(\.world\-collections\.com\/index\.html\?(4[69]|7(34|7)|88)|\-(hot\.net\/|me\.com\/index\.html\?(23|838)))|e(img(1\.moesearch\.net\/imgs1\/01058000\/1058747\.jpg|2\.moesearch\.net\/imgs2\/(0(1058000\/1058604\.jpg|2095000\/2095432\.jpg|4027000\/4027262\.jpg)|12509000\/12509478\.jpg))|search\.net\/img\.php\?mode=view&id=1058747)|ist\.lolajunior\.com\/images\/cache\/240x180\/(1(00\.125\.1231425553\.jpg|11\.85\.1921663855\.jpg|40\.76\.1735890287\.jpg|50\.115\.1167057586\.jpg|6(1\.133\.2028798891\.jpg|6\.130\.165837020\.jpg|7\.111\.1548016707\.jpg)|72\.107\.88564122\.jpg)|2(42\.142\.777367129\.jpg|76\.170\.1873449959\.jpg)|95\.80\.959860653\.jpg)|n(dayporn\.com\/f\/teen\/|stercams\.in\/)|ppetdollz\.com\/|r(asmovie\.com\/|e\-porn\.net\/)|ther\.taboo\.cc\/images\/cache\/300x250\/191\.170\.32154737\.jpg|v(\.(18\-21\-teens\.com\/|ftvcash\.com\/Free\-Movies\/galleries\/FTV\/1\/165\/)|ie(1820\.com\/trifuns\/sabine\.61sec\.shtml|galls(1(\.t(eensexreality\.com\/665\/\?nats=MjgwOjI6NA|inseks\.com\/video2\/070\/\?nats=MjI6Mjo2,0,0,0,5506)|2\.teenburg\.com\/262\-265\/265\/070\/\?nats=Mjc6Mjox)|3\.teen(burg\.com\/videos15\/093\/\?nats=MTc6Mjox,0,0,0,4396|sexreality\.com\/videos3\/042\/\?nats=MjU6Mjo0))|pornshop\.com\/?|sroom\.net\/index2\.shtml)))|pxgirls\.com\/stream\/thumbs\/(196\/196614\.jpg|96\/96855\.jpg)|rpornsite\.com\/tube\/teen\/|snwhores\.com\/|u(lti(\.xnxx\.com\/gallery\/132750\/ad59\/|grab\.olimptraffic\.com\/thumbs\/(003a\/8e8675a8f2fb40dfedbde11abedb4685\.jpg|new(\/00(02\/1fe44498d20b8ca368b23b517f03af47\.jpg|14\/07df98c7a77ab7069dc236273948fe7d\.jpg|37\/a209f10ecc66fa95c4e5d0c56c6eb6b5\.jpg|52\/0a61ea933a3b7a5febcfc94f3e225c98\.jpg)|2\/0(0(07\/0bae6ad1630ca38eafe01383eeb0d860\.jpg|89\/(86365a52f500fd038170116e4026fe94\.jpg|a9bb7df30ede34bb568978495968ed7b\.jpg))|1(36\/e00a6e199a1655fb757448e857b63607\.jpg|59\/3434c873b5544bfef18c58518b89eb81\.jpg)))))|n\-da\.com\/\?ref=nuvilon\.com|s(etoons\.com\/|icnet\.sil\.at\/phpbb\/viewtopic\.php\?p=35246\#35246)|ychicas\.com\.ar\/imgoct2013\/17\.jpeg)|y(18teens\.com\/(movies\/(amazing\-teenage\-whore\-kacey\-gives\-head\/\?nats=NTM6Mjox,0,0,0,10461|brenda\-blowjob\/\?nats=NjY1OjI6MQ|gorgeous\-teen\-girl\-katya\-pornmovies\/\?nats=NTM6Mjox,0,0,0,10507|teenage\-model\-gina\-porn\-clips\/\?nats=NTM6Mjox,0,0,0,10492)|pictures\/(naked\-teen\-hoe\-sabrina\/\?nats=NTM6Mjox,0,0,0,10440|sweet\-teen\-babe\-sabrina\/\?nats=NTM6Mjox,0,0,0,10431|teen\-fuck\-movie\/\?nats=MTI6Mjox,0,0,0,1189))|3(dsex\.com\/|xxx\.com\/(dtr\/galls\/(03104b\/|6(0298a\/|64bda\/)))?)|boyssite\.com\/\?id=10gay\.com|crazyvids\.com\/mov\/best\/petite\-1\.html|e(mogirl\.com\/|xbaby\.com\/)|firstteens\.com\/|\-fruits\.info\/topics\/(HOT%20NEW%203D%20SITE%20!!%20Photos%20of%20young%20nonude%20models%20in%203D%20!%20%28Anaglyph%20and%20MPO%29\/|More%20young%20talents%20Vinka%20Model\/|NEW%20star%20Tiana%20Model\/|Preteen%20art%20from%20Talent%20Young%20%28NEW%29\/|S(ummer%20days%20\-%20best%20nonude%20preteen%20photos%20and%20videos%20from%20summer%20beaches\/|veta%20with%20piggy%20tails\/)|Talent%20Young%20very%20HOT%20preteens%20collection%20%28NEW%29\/)|idealgirl\.com\/archives\/1|juniorsister\.com\/videos\/(0(29\.jpg|61\.jpg)|1(3(7\.jpg|8\.jpg)|54\.jpg|88\.jpg)|2(4(4\.jpg|6\.jpg)|89\.jpg|98\.jpg)|375\.jpg)|s(ilverteens\.com\/index\.php|luts\.net\/(\?x=7106\.8911\.3297)?)|teen(ie\.com\/|pics\.co\.tv\/tgp\/|tv\.com\/|video\.com\/pictures\/(amateur\-teenage\-girl\-nesti\-sex\/\?nats=bogk84q9;827:notrials50:myteenvideo,0,0,0,19752|sweet\-teen\-babe\-aletta\/\?nats=NTM6Mjoz,0,0,0,10608))|youngsex\.com\/))|n(a(dia(31\.firstmo\.com\/|kristina\.firstmo\.com\/thumbs\/(0(5\.jpg|6\.jpg|8\.jpg)|1(2\.jpg|6\.jpg|7\.jpg|8\.jpg)|2(0\.jpg|1\.jpg)))|ked(\-(girls\-vids\.com\/|nude\.com\/|photos\.org\/)|teenspictures\.com\/str\/thumbs\/7\/7(168\.jpg|485\.jpg))|na\.legalmodolls\.com\/promo\/240x460_02\.jpg|onlinedee\.qipim\.ru\/twocj\.html|rutoepisodeporno\.com\/|sty(\.preteennonude\.com\/(\?ref=preteennonude\.com&x=9622\.8898\.|pic\/pic\.php\?285)|teensdesire\.com\/)|turistteenphotos\.com\/th\/thumb034\.jpg|ughtyteenvids\.com\/\?id=kind\-girls\.net)|c\-mcadd\.org\/model\-girls\-preteen\/tn\/(20952\.jpg|4086\.jpg|7905\.jpg)|e(mo\-glamour\.com\/|odet\.com\/\?id=(nonude\-modelcom|topnonude\-girlsinfo|virginstubenet)|w(\.(18onlygirls\.com\/4234c2e9\/MTE3NjU6NToyNA\/|beataporn\.com\/a31412b5\/Njg1MDo1OjMx\/|nonudeplace\.com\/(aida\.holymodels\.jpg|bella\.holymodels\.jpg|evie\.holymodels\.jpg|g(irls(1\.holymodels\.jpg|3\.holymodels\.jpg)|lenda\.holymodels\.jpg)|kamilla\.holymodels\.jpg|lia\.holymodels\.jpg|scj\/tmp\/0\/(2(29_preteen_model\.jpg|62_nonude_glenda\.jpg)|351_lolita_nude\.jpg|4(02_non_preteen\.jpg|50_prelolitas_prelolitas\.jpg)|511_nonude_pics\.jpg|6(03_lolita_lolita\.jpg|85_prelolitas_prelolitas\.jpg)|7(27_nn_model\.jpg|63_nude_preteen\.jpg)))|younglegalporn\.com\/(3f68aa4a\/Njg1MDo1OjI1\/|ee0598ed\/MzcyODo1OjI1\/))|\-(art\-nude\.net\/streamrotator\/thumbs\/(0\/(234\.jpg|32\.jpg|493\.jpg)|59\/59594\.jpg|65\/65515\.jpg)|content\.info\/cj\/g\/index\.php\?ft=gaybase\.biz|models\.in\/scj\/thumbs\/0\/(576_0053_0053\.jpg|733_From_From\.jpg)|teensex\.com\/(\?facename=indexpdl&updatestat=off)?)|estpornlinks\.com\/|n(nmodels\.com\/scj\/thumbs\/0\/300_Star_Preten\.jpg|ubiles\.net\/)|pornsite(\.org\/(\?x=0058\.4384\.|vids\/x09\/05\/26\/A\-nice\-Blowjob\/index\.html)?|s\.net\/)|yes\.moy\.su\/sixcj\.html)|zemnoykaif\.com\/(\?d=p1)?)|fsx\.com\/(\?x=(0693\.6976\.0182\.9567\.9074\.4120\.5480\.1010\.9410\.5849|2504\.9354\.5283\.5928\.5849\.|4113\.5849\.|5849\.?|7119\.5849nfsx\.com\/\?x=7119\.5849\.))?|i(c(exvideostube\.com\/dtr\/thumbs\/(0(82339\.jpg|92429\.jpg)|3c2af8\.jpg|4c9c39\.jpg|6(2ba29\.jpg|61782\.jpg)|76803f\.jpg|97c126\.jpg|b(46397\.jpg|65702\.jpg)|c(0ab58\.jpg|3d805\.jpg|9065b\.jpg)|d(799f4\.jpg|8b70c\.jpg))|hegalz\.com\/moviesgeneral\.php\?d=etvtube\.com&acc=arkan&in2=)|ghttubes\.com\/|neclips\.com\/(watch\.php\?tag=0glreYaIS9)?|vonog\.zonalibre\.org\/archives\/015324\.html)|n(\-(1\.com\/(ban_468\.jpg)?|art\.info\/(image\/8\.jpg|portal\/main\/index_files\/tn_(1\.jpg|reyna25\.jpg))|b(bs\.info\/(100150\.gif|ban\.gif)|eauties\.com\/)|girls\.biz\/sitesx\/(2\/stella\.jpg|4\/(bella\.jpg|evie\.jpg)|5\/mashamodel\.jpg)|l(ist\.in\/scj\/thumbs\/0\/394_and_5\.jpg|ola\.net\/index\.html\?44)|mods\.com\/index\.html\?36|sites\.com\/(nnmodels(3\.html|4\.html))?|t(een\.thumblogger\.com\/|op\.com\/100x150\.jpg))|100\.in\/(\?id=simply\-models\.net|s(cj\/thumbs\/1\/(3(09_Elba_Pretty\.jpg|31_young_models\.jpg)|68(3_Beautiful_Mara\.jpg|9_Hot_Young\.jpg)|7(03_Model_0071\.jpg|29_Other_Girls\.jpg|57_Model_Girls\.jpg|91_From_Models\.jpg)|8(28_From_Vilma\.jpg|43_Models_Models\.jpg|51_0068_5\.jpg|76_Model_Mara\.jpg))|ites\/(lila\/Lila\-ind\-400x600\.jpg|natasha\/Natasha\-ind\-400x600\.jpg)))|desire\.com\/bannersmall\.gif|elis\.com\/|lopics\.com\/(PC\-tn(alina6b\.jpg|daphne3b\.jpg|roxana2\-2b\.jpg))?|neversleeps\.info\/|preteen\.com\/index\.html\?18|t(eens\.com\/tgp\/my18teens\-arielfuck\/bunny\.htm|op\.org\/(100150\.gif|miniban\/inna\.jpg))|ville\.com\/(\?(ref=n(onudemodel\.net|uvilon\.com)|x=5608\.9251\.9012\.6426\.)|GALL\/art\/trixie\/)?)|o(chesexo\.com\/|n(\-nude\.tv\/non\-nude_room\/151\-3_gb_private_colection_jb_bonus\.html|nudesitescatalog\.com\/banner\.gif|stop\-nn\.info\/(\?ref=newnnmodels\.net|100150\.jpg|galls\/mnrv\/More%20preteen%20nonude%20models%20Dany%20and%20Camy\/)))))
Testing the regex in regex101 it should not match.
My squid was compiles with gnu regex
Squid Cache: Version 3.5.17
Service Name: squid
configure options: '--prefix=/opt/sepia/distro/squid' '--sysconfdir=/var/lib/sepia/' '--disable-auth' '--disable-auto-locale' '--disable-cache-digests' '--disable-cpu-profiling' '--disable-debug-cbdata' '--disable-delay-pools' '--disable-devpoll' '--disable-ecap' '--disable-esi' '--disable-eui' '--disable-external-acl-helpers' '--disable-follow-x-forwarded-for' '--disable-forw-via-db' '--enable-gnuregex' '--disable-htcp' '--enable-icap-client' '--disable-ident-lookups' '--enable-internal-dns' '--disable-ipf-transparent' '--disable-ipfw-transparent' '--disable-ipv6' '--disable-leakfinder' '--disable-pf-transparent' '--disable-poll' '--disable-select' '--disable-snmp' '--with-openssl' '--disable-stacktraces' '--disable-translation' '--disable-url-rewrite-helpers' '--disable-wccp' '--disable-wccpv2' '--disable-win32-service' '--disable-x-accelerator-vary' '--disable-icmp' '--disable-storeid-rewrite-helpers' '--enable-async-io' '--enable-disk-io' '--enable-epoll' '--enable-http-violations' '--enable-inline' '--enable-kill-parent-hack' '--enable-linux-netfilter' '--enable-log-daemon-helpers' '--enable-removal-policies' '--enable-storeio' '--enable-unlinkd' '--enable-x-accelerator-vary' '--enable-zph-qos' '--with-default-user=nobody' '--with-logdir=/var/log/sepia' '--with-pthreads' '--with-included-ltdl' '--with-pidfile=/var/lib/sepia/squid.pid' '--with-netfilter-conntrack' '--disable-arch-native' --enable-ltdl-convenience
Testing with more lines of smaller regexes sometimes leads to the same problem. One regex matching everything
2016-04-27 8:32 GMT-03:00 Alfredo Rezinovsky <alfrenovsky@xxxxxxxxx>:
I saw in debug log that when an ACL has many regexes each one is compared sequentially.If I haveIf will be faster to check just ONE optimized regex like (www\.)?(facebook|google).com than the previous three?I'm really talking about optimizing about 3000 url regexes in one huge regex because comparing each and every url to 3000 regexes is too slow.I know using (www\.facebook\.com)|(facebook\.com)|(www\.google\.com)|(google\.com) with PCRE will produce the same optimized result as (www\.)?(facebook|google)\.com. Squid uses GnuRegex. Does GNURegex lib optimizes this as well ?--Alfrenovsky
Alfrenovsky
_______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
