On Tue, 2018-05-15 at 11:44 -0400, David Wysochanski wrote:
> On Tue, 2018-05-15 at 13:59 +0000, Trond Myklebust wrote:
> > On Tue, 2018-05-15 at 09:06 -0400, David Wysochanski wrote:
> > > On Tue, 2018-04-17 at 16:11 -0400, Dave Wysochanski wrote:
> > > > In nfs_idmap_read_and_verify_message there is an unprotected
> > > > sprintf
> > > > that converts the __u32 'im_id' from struct idmap_msg to
> > > > 'id_str'
> > > > that is a stack variable of 'NFS_UINT_MAXLEN' (defined as 11).
> > > > If a uid or gid value is > 2147483647 = 0x7fffffff we corrupt
> > > > kernel memory by one byte and if
> > > > CONFIG_CC_STACKPROTECTOR_STRONG
> > > > is set we see a stack-protector panic as follows:
> > > >
> > > > [11558053.616565] Kernel panic - not syncing: stack-protector:
> > > > Kernel stack is corrupted in: ffffffffa05b8a8c
> > > >
> > > > [11558053.639063] CPU: 6 PID: 9423 Comm: rpc.idmapd Tainted:
> > > > G W ------------ T 3.10.0-514.el7.x86_64 #1
> > > > [11558053.641990] Hardware name: Red Hat OpenStack Compute,
> > > > BIOS
> > > > 1.10.2-3.el7_4.1 04/01/2014
> > > > [11558053.644462] ffffffff818c7bc0 00000000b1f3aec1
> > > > ffff880de0f9bd48 ffffffff81685eac
> > > > [11558053.646430] ffff880de0f9bdc8 ffffffff8167f2b3
> > > > ffffffff00000010 ffff880de0f9bdd8
> > > > [11558053.648313] ffff880de0f9bd78 00000000b1f3aec1
> > > > ffffffff811dcb03 ffffffffa05b8a8c
> > > > [11558053.650107] Call Trace:
> > > > [11558053.651347] [<ffffffff81685eac>] dump_stack+0x19/0x1b
> > > > [11558053.653013] [<ffffffff8167f2b3>] panic+0xe3/0x1f2
> > > > [11558053.666240] [<ffffffff811dcb03>] ? kfree+0x103/0x140
> > > > [11558053.682589] [<ffffffffa05b8a8c>] ?
> > > > idmap_pipe_downcall+0x1cc/0x1e0 [nfsv4]
> > > > [11558053.689710] [<ffffffff810855db>]
> > > > __stack_chk_fail+0x1b/0x30
> > > > [11558053.691619] [<ffffffffa05b8a8c>]
> > > > idmap_pipe_downcall+0x1cc/0x1e0 [nfsv4]
> > > > [11558053.693867] [<ffffffffa00209d6>]
> > > > rpc_pipe_write+0x56/0x70
> > > > [sunrpc]
> > > > [11558053.695763] [<ffffffff811fe12d>] vfs_write+0xbd/0x1e0
> > > > [11558053.702236] [<ffffffff810acccc>] ?
> > > > task_work_run+0xac/0xe0
> > > > [11558053.704215] [<ffffffff811fec4f>] SyS_write+0x7f/0xe0
> > > > [11558053.709674] [<ffffffff816964c9>]
> > > > system_call_fastpath+0x16/0x1b
> > > >
> > > > Fix this by snprintf and a safe length based on sizeof(id_str).
> > > >
> > > > Signed-off-by: Dave Wysochanski <dwysocha@xxxxxxxxxx>
> > > > Reported-by: Stephen Johnston <sjohnsto@xxxxxxxxxx>
> > > > ---
> > > > fs/nfs/nfs4idmap.c | 2 +-
> > > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/fs/nfs/nfs4idmap.c b/fs/nfs/nfs4idmap.c
> > > > index 22dc30a679a0..a8c663f8dd99 100644
> > > > --- a/fs/nfs/nfs4idmap.c
> > > > +++ b/fs/nfs/nfs4idmap.c
> > > > @@ -627,7 +627,7 @@ static int
> > > > nfs_idmap_read_and_verify_message(struct idmap_msg *im,
> > > > if (strcmp(upcall->im_name, im->im_name) != 0)
> > > > break;
> > > > /* Note: here we store the NUL terminator too
> > > > */
> > > > - len = sprintf(id_str, "%d", im->im_id) + 1;
> > > > + len = snprintf(id_str, sizeof(id_str), "%u",
> > > > im-
> > > > > im_id) + 1;
> > > >
> > > > ret = nfs_idmap_instantiate(key, authkey,
> > > > id_str,
> > > > len);
> > > > break;
> > > > case IDMAP_CONV_IDTONAME:
> > >
> > >
> > > I did not see any reply to this and we did have one customer hit
> > > this
> > > which caused a considerable outage of many machines. In essence
> > > once
> > > this happened, it became a DoS on all machines using idmapping
> > > and
> > > they
> > > implemented a temporary workaround.
> > >
> > > Anna / Trond - if you need me to improve the patch header or want
> > > clarification or see a problem with it, please let me know.
> > >
> >
> > If the value of NFS_UINT_MAXLEN is too small, then shouldn't we be
> > increasing it? That would appear to be the real bug here.
> >
>
> Sorry the patch header doesn't explain it well. The %d usage with a
> __u32 is the problem. If we get a large enough value, the '-' sign
> makes it a buffer overflow and the NULL overwrites one byte on the
> stack.
>
> Examples
> crash> p (unsigned) (0x80000000)
> $1 = 2147483648
> crash> p (signed) (0x80000000)
> $2 = -2147483648
>
> So the unsigned max value uses 10 bytes plus a NULL, hence
> NFS_UINT_MAXLEN of 11.
>
>
> > I do agree that the "%d" should be changed to "%u", though. Isn't
> > that
> > sufficient to make the buffer large enough?
> >
> >
>
> Yes you could just change the %d to %u in the sprintf, but the rest
> of
> the code uses snprintf so that's why I chose it.
>
> I'll also note there is nfs_map_numeric_to_string() that is called
> from
> other locations, and we could call from here as well for consistency:
> static int nfs_map_numeric_to_string(__u32 id, char *buf, size_t
> buflen)
> {
> return snprintf(buf, buflen, "%u", id);
> }
>
>
> If you want, I could submit a v2 patch with an improved header and
> this:
>
> diff --git a/fs/nfs/nfs4idmap.c b/fs/nfs/nfs4idmap.c
> index 22dc30a..779411e0 100644
> --- a/fs/nfs/nfs4idmap.c
> +++ b/fs/nfs/nfs4idmap.c
> @@ -343,7 +343,7 @@ static ssize_t nfs_idmap_lookup_name(__u32 id,
> const char *type, char *buf,
> int id_len;
> ssize_t ret;
>
> - id_len = snprintf(id_str, sizeof(id_str), "%u", id);
> + id_len = nfs_map_numeric_to_string(id, id_str,
> sizeof(id_str));
> ret = nfs_idmap_get_key(id_str, id_len, type, buf, buflen,
> idmap);
> if (ret < 0)
> return -EINVAL;
> @@ -627,7 +627,7 @@ static int
> nfs_idmap_read_and_verify_message(struct idmap_msg *im,
> if (strcmp(upcall->im_name, im->im_name) != 0)
> break;
> /* Note: here we store the NUL terminator too */
> - len = sprintf(id_str, "%d", im->im_id) + 1;
> + len = nfs_map_numeric_to_string(im->im_id, id_str,
> sizeof(id_str)) + 1;
>
Yes, I think this makes more sense. Can you please send me a v2 with
this fixup.
Thanks!
Trond
--
Trond Myklebust
CTO, Hammerspace Inc
4300 El Camino Real, Suite 105
Los Altos, CA 94022
www.hammer.space id="-x-evo-selection-end-marker">��.n��������+%������w��{.n�����{��w��� jg��� �����ݢj����G���� ���j:+v���w�m������w�������h�����٥
