Hi, On Tue, Feb 18, 2014 at 6:52 PM, Gaudenz Steinlin <gaudenz@xxxxxxxxxx> wrote: > > Hi > > Sage Weil <sage@xxxxxxxxxxx> writes: > >> Dan at CERN noticed that his performance was tanking because updatedb was >> running against /var/lib/ceph. updatedb has a PRUNE line in >> /etc/updatedb.conf that we should presumably be adding ourselves to. One >> user pointed out a package that uses sed to rewrite this line in the init >> script on start. >> >> I have two questions: >> >> - is there no better way than sed to add ourselves to this list? >> - should we do it in the init script, or postinst, or both? >> >> Presumably this is a problem others have solved with other packages. > > At least for Debian neither solution is appropriate. Changing other > packages conffiles in postinst scripts is forbidden by policy. There is > also no way to preserve this reliably over upgrades without user > interaction. I'm not sure if there is an explicit policy for init > scripts, but this seems equally wrong. Also it's unclear how one would > handle the case were an administrator does NOT want to exclude this > directory. > > The only solution I see if you really want to completely exclude the > directory is to convince the mlocate maintainers to either add the > directory to the default configuration or to add something like an > /etc/updatedb.d directory where packages can drop configuration file > snippets. But the latter seems like overkill to me. > > The real question to me is why an updatedb run can drastically impact > ceph performance. At least in Debian updatedb is run with ionice > -c3 in the "Idle" scheduling class. According to the man page this > means: "A program running with idle I/O priority will only get disk time > when no other program has asked for disk I/O for a defined grace period. > The impact of an idle I/O process on normal system activity should be > zero. This scheduling class does not take a priority argument. > Presently, this scheduling class is permitted for an ordinary user > (since kernel 2.6.25)." So it should not have any negative effect. > > Maybe CERN (or the distribution they use) should also run updatedb under > ionice. updatedb _is_ run under ionice on our systems (RHEL), but IO scheduling classes are only implemented by the cfq scheduler. We use deadline, which is recommended for an enterprise disk server, and indeed measured more stable IO latencies with deadline than with cfq. And when you run updatedb on a deadline scheduled drive, there are so many reads queued up that writes can be starved for many 10s of seconds. In our case, we've already added /var/lib/ceph to the PRUNEPATHS via their puppet configuration. Though an upstream solution is probably a good idea since I would assume that most ceph deployments use deadline and this would hit most of them eventually once they have enough files. In addition, as someone else mentioned, you'd better add ceph to the pruned fs types as well, just like /afs is pruned at the moment, lest every client spend all day stat'ing their big cephfs namespace. Cheers, Dan -- Dan van der Ster || Data & Storage Services || CERN IT Department -- > > Gaudenz > > -- > Ever tried. Ever failed. No matter. > Try again. Fail again. Fail better. > ~ Samuel Beckett ~ > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
