On Mon, Jul 03, 2017 at 11:07:22AM -0400, Theodore Ts'o wrote:
> On Mon, Jul 03, 2017 at 01:52:49PM +0200, Jan Tulak wrote:
> > I want to limit the capabilities of this interface to non-interactive
> > only. So, yes, with fsck, JSON would be overkill. But the idea is to
> > have a single format across all the tools, so you don't need a
> > standalone parser for every tool, even if some tools don't need
> > anything more than an exit code and/or one message on stderr. In the
> > case of ext2/3/4 it is more about resize2fs and tune2fs, where the
> > JSON would be much more useful, than fsck.
>
> I'm not sure what sort use cases you have in mind where structured
> output would be useful.
>
> For mke2fs or resize2fs, in general all you care about is "did the
> operation succeed", right? What did you have in mind where this more
> information than "the operation was successful" did you have in mind?
>
> For tune2fs, what options or output in tune2fs are sufficiently file
> system independent that you think it would be worth exporting to your
> infrastructure?
>
> > The generic way would be along the lines: These ten common and
> > frequently used fields are generic and work everywhere, anything else
> > has a prefix (ext4-, xfs-, btrfs-, ...) or is inside of a fs-specific
> > list of extensions "ext4":{"some-option":value}. And during the
> > parsing, all the fields would be mapped to some mkfs arguments,
> > usually 1:1. Similar to what could be done with output, like putting
> > volume identifier into a specific field (e.g. "fsid") no matter what
> > the filesystem is.
>
> It might be useful if you could give some "user stories" that explain
> at a high level what the user might want to do with the ultimate
> user-visible interface that would require this kind of precision.
Yes, please.
> Most users don't know how to use the specialized options to
> mkfs.<FSTYP> and to be honest, most don't need to. The way I've dealt
To reinforce that point, in XFS land we tell people to take the defaults
unless they have verified that a specific problem of theirs is fixed by
changing some tuning parameter (and doesn't cause other problems). Many
of the user complaints I see on the (xfs) list are a result of people
engaging in copy-pasta without understanding the tradeoffs they're
implicitly accepting.
Once users get to the point of having verified their workloads with
non-default options, they often encode a mkfs.xfs command line into
their deployment scripts. So while I can be convinced that there's
value in a(nother) EVMS-like thing encapsulating (generic) mkfs, I'm not
currently persuaded that there's much point in adapting mkfs.$FSTYP to
accept a bunch of geometry/feature options via json.
(This probably changes for things like btrfs that also manage volumes.)
> with this for mke2fs.conf is that when someone has come up with a
> specialized recipe for a unique sort of file system type --- say,
> maybe for a Lustre Metadata server, or the back-end storage for some
> kind of clustre file system like Hadoopfs, or specialized options for
> an Android phone --- someone with wizard-level skills will edit
> /etc/mke2fs.conf, with perhaps something like this:
>
> smr-host-managed = {
> features = extent,huge_file,bigalloc,flex_bg,uninit_bg,dir_nlink,extra_isize,^resize_inode,sparse_super2
> cluster_size = 32768
> hash_alg = half_md4
> reserved_ratio = 0.0
> num_backup_sb = 0
> packed_meta_blocks = 1
> make_hugefiles = 1
> inode_ratio = 4194304
> hugefiles_dir = /smr
> hugefiles_name = smr-file
> hugefiles_digits = 0
> hugefiles_size = 0
> hugefiles_align = 256M
> hugefiles_align_disk = true
> num_hugefiles = 1
> zero_hugefiles = false
> flex_bg_size = 262144
> }
>
> ... and then all the user will have to do is run "mke2fs -t ext4 -T
> smr-host-managed /dev/sdXX". (The intended use case for this is to
> support an SMR-aware user-space application that was going to be
> managing the host-managed SMR zones directly.)
>
> I suspect telling the user that they have to type a whole series of
> parameters of the form:
>
> "ext4":{"some-option":value}
>
> into a GUI would not be a particularly user-friendly suggestion. :-)
>
> The other question I'd ask is how many people really are going to want
> to use your infrastructure? Is it only going to be for the "point and
> click" users who will want a simplified interface? Are you trying to
> make something that will be useful for advanced/expert users? What
> value are you going to be able to add that will convince the
> advanced/export users that they should learn your new
> "fstype":{"some-option":value} syntax when typing at the command line
> will probably be ten times faster, easier, and less rage-inducing than
> trying to reverse-engineer out some interface that was designed not to
> scare the civilians? (There's a reason why many drivers prefer manual
> to automatic transmission on their cars. :-)
I doubt Jan is proposing to eliminate ye olde getopt arguments --
presumably the GUI (or whatever) will generate the json code and the
library stuffs it into mkfs?
Or to put it another way, I would not accept a mkfs.xfs patch ripping
out the classic getopt options, and I suspect Eric Sandeen wouldn't
either.
> > Thanks for pointing out EVMS, I will see what I can learn from that
> > attempt. Starting with screenscraping is certainly an option and might
> > be the only viable one. This raises some other questions, though:
> > given the temporality of the wrapper, I would rather use other
> > languages than C/Bash (e.g. Python) to simplify and speed up the
> > development. But I have doubts about whether you would be willing to
> > adopt this into e2fsprogs, which would, in turn, reduce the usability
> > of this approach.
>
> The reason why you might want to consider C is because:
>
> * It allows the plugin to be imported into many different
> programming languages: Python, Go, Perl, etc., via using
> something like SWIG.
>
> * Different file system maintainers will be willing to accept
> maintenance of your plugin at different times. For some file
> systems, you may have to wrap the command-line tools forever; for
> one thing the file system may no longer be under active
> maintenance (ex: iso9660) but you still might want to be use it
> in your GUI interface. Other file system developers will be
> willing to take over the plugin and support it as a native part
> of their file system tools more quickly.
>
> * The first operations that you might want to make be native
> instead of being screen scraped (getting the file system size,
> the amount of free space, etc.) are things which are most easily
> done in C.
Weird ioctls and other syscalls aren't so hard in python. I've written
a clumsy GETFSMAP wrapper:
https://github.com/djwong/filemapper/blob/master/getfsmap.py
But you're correct to point out that wrapping libext2fs in Python would
be difficult, and even more invasive if you took the interesting parts
of, say, tune2fs and put them into a separate library and made tune2fs
merely an arguments-parsing wrapper around the separate library.
I'd occasionally thought about wrapping libext2fs in Python and
concluded that it was too much work for not enough gain, and Ted would
likely NAK it anyway.
> So if you want ask the file system developers to take over the
> plugin, they are much more likely to be willing to say yes if the
> plugin is already in C, as opposed to asking them to take over
> some Python class where trying to integrate python code to call
> into libext2fs is going to be a pain in the ass. For that
> matter, you might want to implement the plugins to call libext2fs
> and libxfs directly for those basic functions. That's what the
> EVMS developers did, and those interfaces in libext2fs are
> guaranteed to have ABI and API stability. Anyway, if your goal
> is to convince file system developers to eventually take
> ownership of the interface/plugin module, it will be much easier
> to do that if it is in C --- trust me on that.
Yeah, probably. :)
I don't think we'll ever have A[BP]I stability in "libxfs".
For that matter, libxfs isn't even a proper library; we statically link
all the object files directly into the tools. But then Jan is (so far)
only asking to wrap the tools, not reimplement them too.
> * I don't think it's going to be that hard to use C; as I've said,
> I really disbelieve that there are that many places where you
> need to screenscrape. Most of what you will probably need to do
> is to return the exit status of mke2fs, fsck, resize2fs, etc.
> Those programs that you do need to screen scrape will have
> outputs similar to dumpe2fs, which is stupid-easy to parse, and
> are also, as I've noted above, the simplest thing to move to
> being done in native code calling the file system's C library.
>
> Oh, one thing. I'll note that e2fsprogs has progress bar support
> already, and it was designed so it could be easily integrated into a
> GUI. As far as I know Ubuntu was the only distro that used it ---
> progress bars tend not to be high on most distros' product manager's
> feature priority lists --- but it's there. See e2fsck's -C option.
> This support was also plumbed into fsck (see its -C option), so I
> designed it to be something that other file systems could implement.
>
> Also, we're Linux systems programmers, not Web developers writing
> Javascript; why use JSON and require fancy parsing when you can just
> isolate the completion information onto a separate file descriptor? I
I imagine that json makes it easier to export structured information
than a basic flat file, but otoh my experiences with wrapped filesystem
tools is that the good tools report nonzero error status and point you
at the full text logs of what happened. The bad ones, of course, don't
check the status and don't record the output.
> don't know how big and complex your JSON parsing library is, but all
> that was needed to parse *my* completion information is the single
> line of C code:
>
> fscanf(progress_f, "%d %lu %lu %ms\n", &pass, &cur, &max, &text).
FWIW I've been building the same into xfs_scrub if anyone cares... :)
--D
>
> Cheers,
>
> - Ted
