On Mon, Aug 30, 2021 at 10:37:31PM +0900, Tetsuo Handa wrote:
> On 2021/08/30 22:00, Dan Carpenter wrote:
> >>> diff --git a/drivers/video/fbdev/vga16fb.c b/drivers/video/fbdev/vga16fb.c
> >>> index e2757ff1c23d..e483a3f5fd47 100644
> >>> --- a/drivers/video/fbdev/vga16fb.c
> >>> +++ b/drivers/video/fbdev/vga16fb.c
> >>> @@ -403,7 +403,7 @@ static int vga16fb_check_var(struct fb_var_screeninfo *var,
> >>>
> >>> if (yres > vyres)
> >>> vyres = yres;
> >>> - if (vxres * vyres > maxmem) {
> >>> + if ((u64) vxres * vyres > (u64) maxmem) {
> >>
> >> Mindlessly changing the sizes is not the solution.
> >> Please use e.g. the array_size() helper from <linux/overflow.h>
> >> instead.
> >
> > On a 64bit system the array_size() macro is going to do the exact same
> > casts? But I do think this code would be easier to understand if the
> > integer overflow check were pull out separately and done first:
> >
> > if (array_size(vxres, vyres) >= UINT_MAX)
> > return -EINVAL;
>
> This is wrong. array_size() returns ULONG_MAX on 64bits upon overflow and
> returns UINT_MAX on 32bits upon overflow. However, UINT_MAX is a valid
> value without overflow (e.g. vxres == UINT_MAX / 15 && vyres == 15).
Huh... I just assumed we didn't allow resolutions that high.
> Comparing like "> (u64) UINT_MAX" is to detect only overflow.
>
Of course, that doesn't work on 32 bit systems. Also the cast isn't
required because of type promotion.
regards,
dan carpenter
