Re: metastore

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Sun, 16 Sep 2007, Daniel Barkalow wrote:

On Sun, 16 Sep 2007, david@xxxxxxx wrote:

On Sun, 16 Sep 2007, Daniel Barkalow wrote:

I however think your idea to have extra "permission information
file" is very interesting.  What would be more palatable, than
mucking with the core level git, would be to have an external
command that takes two tree object names that tells it what the
old and new trees our work tree is switching between, and have
that command to:

 - inspect the diff-tree output to find out what were checked
   out and might need their permission information tweaked;

 - inspect the differences between the "permission information
   file" in these trees to find out what were _not_ checked out,
   but still need their permission information tweaked.

 - tweak whatever external information you are interested in
   expressing in your "permission information file" in the work
   tree for the paths it discovered in the above two steps.
   This step may involve actions specific to projects and call
   hook scripts with <path, info from "permission information
   file" for that path> tuples to carry out the actual tweaking.

Why not have the command also responsible for creating the files that need
to be created (calling back into git to read their contents)? That way,
there's no window where they've been created without their metadata, and
there's more that the core git doesn't have to worry about.

my initial thoughts were to have git do all it's normal work and hook into git
at the point where it's writing the file out (where today it chooses between
writing the data to a file on disk, pipeing to stdout, or pipeing to a pager)
by adding the option to pipe into a different program that would deal with the
permission stuff. this program would only have to write the file and set the
permissions, it wouldn't have to know anything about git other then where to
find the permissions it needs to know.

it sounds like you are suggesting that the hook be much earlier in the
process, and instead of one copy of git running and calling many copies of the
writing program, you would have one copy of the writing program that would
call many copies of git.

A lot of the git commands are actually currently shell scripts  that call
back to git, so that's not too different. The reason to have a single copy
of the writing program is that it would be able to get the whole set of
differences that need to be handled, and first pick out the metadata file,
process it to figure out the writing instructions once, figure out the
changes in the writing instructions, and figure out the changes in the
content, and decide what to do.

I'm still a little unclear on how much work this program would then have to do. it's problably my lack of understanding that's makeing this sound much scarier.

I could see the program getting the index, the target tree, and the
directory to put files in, and being told to do the whole 2-way merge
(except, perhaps, updating the index to match the tree, which git could do
afterwards). As far as git would be concerned, it would mostly be like a
bare repository.

if this functionality does shift to earlier in the process, how much of the
git logic needs to be duplicated in this program?

if this program needs to do the merge, won't it have to duplicate the merge
logic, including the .gitattributes checking for custom merge calls?

This is two-way merge, not three-way merge. The basic concept is that
you're in state A, and you want to be in state B. Rather than writing out
all of state B, you write out all of state B that's different from state
A. Think of taking a diff of two big trees and then applying it as a
patch, instead of copying the new tree onto the old tree; the benefit is
that stuff that doesn't change doesn't get rewritten, and the diff is
blazingly fast, given how we store our information.

so what would this program be given?

it sounds like it would be called once for the entire tree checkout

would it be handed just the start and end commits and query git for everything else it needs?

it sounds like there is more then this, you refer to git fully crafting the new index.

so would this program be accessing an old and new index and do the comparison between the two?

or would git feed it a list of what's changed and then have it query git to find the details of the changes.

3-way merge will be handled by git, and not in a live /etc directory
anyway (that is, you'd want to fix up the metadata files as plain text
files, not as metadata bits on a checked out directory; otherwise, you'll
be trying to put conflict markers in mode bits, and that's clearly not
what you want).

right, we don't want conflict markers on mode bits or other ACL type things, that way lies madness ;)

I have been thinking primarily in terms of doing a complete checkout,
overwriting all files, and secondarily how do do a checkout of just a few
files, but again where all files selected overwrite the existing files.

I wasn't thinking of the fact that git optimizes the checkout and avoids
writing a file that didn't change.

this changes things slightly

prior to this I was thinking that the permission file needed to be handled
differently becouse writing it out needed to avoid doing any circular
refrences where you would need to check the contents of it to write it out.

it now appears as if what really needs to happen is that if the permission
file changes a different program needs to be called when it's written out then
when the other files are written out. by itself this isn't hard as
.gitattributes can have a special entry for this filename and that entry can
specify a different program, and that program fixes all the permissions
(and/or detects that they can't be fixed due to user/filesystem limits,
records the error, checks if the repository is set appropriately, and screams
to the user if it isn't)

While we're at it, you probably don't even want to write the permission
file to the live filesystem. It's just one more thing that could leak
information, and changes to the permissions of files that you record by
committing the live filesystem would presumably be done by changing the
permissions of files in the filesystem, not by changing the text file.

the permissions and ACL's can be queried directly from the filesystem, so I don't see any security problems with writing the permission file to the filesystem.

changing the permissions would be done by changing the files themselves (when you are running as root on a filesystem that supports the changes, otherwise it would need to fall back to writing the file and getting the changes there, but that should be able to be a local config option)

I don't like the idea of having a file that doesn't appear on the local filesystem at any point, it just makes troubleshooting too hard.

(Of course, you could check out the same commits as ordinary source, with
developer-owned 644 files and a 644 "permissions" file, and there you'd
have the permissions file appear in the work tree, and you could edit it
and check it in in a totally mundane way.)

right, and the same thing if the filesystem doesn't support something in the permission file.

it would be a nice optimization to this permission checkout for it to compare
the old and the new permissions so that it only tries to change the
permissions where it needs to, but is that really nessasary? the program can
look at the permissions of the existing files to see what they are and decide
if it needs to change them (this would tromp on local changes that aren't
checked in. how big of a problem is this?) my initial reaction is that having
to know the two commits and do the comparison between them is adding a lot of
logic and git interaction that I'd rather avoid if I could.

You probably want to be able to keep local uncommitted changes. People
like to be able to have things slightly different in their particular
deployment from the way things are in the repository, for stuff that only
applies to one system and isn't "how it should be".

if so this means that the permission changing program definantly needs to operate on the diff of the permisison file, not on the absolute file. this complicates things slightly, but it shouldn't be too bad.

changing topic slightly.

I know git has pre-commit hooks, but I've never needed to use them.

at what point can you hook in?

can you define a hook that runs when you do a git-add? or only when you do a git-commit?

the reason I'm asking is to try and figure out when and how to create the permissions file. when I was thinking in terms of dealing with the permissions as a single bog block it wasn't that bad to say that at git-commit time you have to scan every file and check it's permissions to record them into the file, but with the push for the optimizations that you're talking about this is no longer reasonable and it really should be done when the file is added to the index.

on a related note, if this is implemented as a per-write hook then it makes a lot of sense to have the permission file be per-directory, but if we do a per-checkout hook like you are suggesting then the permission file may make more sense as a single file in the top-level directory.

thoughts?

David Lang
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]

  Powered by Linux