[PATCH 3/3] drm/amdgpu: add gtt_sys_limit

Hongbo.He@xxxxxxx (He, Roger) · Wed, 28 Jun 2017 02:57:02 +0000

Setting limitation for max GTT table size is necessary.
But my only concern is that is it enough for amdgpu_gart_sys_limit 256M. whether it will trigger GTT eviction from GTT to CPU domain easily?
Because GTT size always equals to VRAM size before it scales with system memory size.

Thanks
Roger(Hongbo.He)
-----Original Message-----
From: He, Roger 
Sent: Wednesday, June 28, 2017 9:58 AM
To: 'Felix Kuehling' <felix.kuehling at amd.com>; Christian KÃ¶nig <deathsimple at vodafone.de>; Michel DÃ¤nzer <michel at daenzer.net>; John Brooks <john at fastquake.com>
Cc: amd-gfx at lists.freedesktop.org
Subject: RE: [PATCH 3/3] drm/amdgpu: add gtt_sys_limit

Yeah, recently we indeed encountered an issue as below:
On workstation machine it has 32G system memory and thus the GTT table size takes up about 192M visible VRAM.
Maybe if system has bigger memory, GTT table size will exceed 256M, so Limitation  for max GTT table size is very necessary.

Thanks
Roger(Hongbo.He)
-----Original Message-----
From: amd-gfx [mailto:amd-gfx-bounces@xxxxxxxxxxxxxxxxxxxxx] On Behalf Of Felix Kuehling
Sent: Wednesday, June 28, 2017 3:25 AM
To: Christian KÃ¶nig <deathsimple at vodafone.de>; Michel DÃ¤nzer <michel at daenzer.net>; John Brooks <john at fastquake.com>
Cc: amd-gfx at lists.freedesktop.org
Subject: Re: [PATCH 3/3] drm/amdgpu: add gtt_sys_limit

On 17-06-27 04:01 AM, Christian KÃ¶nig wrote:
> Am 27.06.2017 um 04:57 schrieb Michel DÃ¤nzer:
>> On 27/06/17 08:39 AM, John Brooks wrote:
>>> On Mon, Jun 26, 2017 at 03:39:57PM +0200, Christian KÃ¶nig wrote:
>>>> From: Christian KÃ¶nig <christian.koenig at amd.com>
>>>>
>>>> Limit the size of the GART table for the system domain.
>>>>
>>>> This saves us a bunch of visible VRAM, but also limitates the 
>>>> maximum BO size we can swap out.
>>>>
>>>> Signed-off-by: Christian KÃ¶nig <christian.koenig at amd.com>
>>> Hmm.
>>>
>>> On my system, GTT is 4096MiB (1048576 pages). For this, the table 
>>> takes up
>>> 1048576*8 bytes = 8MiB. Reducing GTT to 256MiB (65536 pages) would 
>>> reduce the size of the table to 512 KiB. A relatively large saving, 
>>> to be sure.
>>> But in the
>>> grander scheme of things, is saving 7.5MiB (3% of visible VRAM @
>>> 256M) worth
>>> cutting GTT memory by a factor of 16?
>
> The amount of GTT memory the application can use through the VM page 
> tables stays the same.
>
> Only the system VM is limited to 256MB and so saves us a whole bunch 
> of space.
>
>> I'm afraid not, especially since it would limit the maximum BO size 
>> to < 256MB, if I understand correctly. Pretty sure that would cause 
>> failures with real world apps.
> Yes, exactly that's the major problem here. I should have put a WIP 
> mark on the patch.

OK, I should adapt this for the KFD branch. Currently we make our GART much bigger. On a system with lots of system memory, we can use up half the visible VRAM for the GART. With something like this we can get it back to something reasonable. But a 256MB limit on single allocation size is definitely too small.

Also, if this breaks S3, we have to make sure the hybrid branches don't pick it up accidentally.

Regards,
  Felix

>
> Christian.

_______________________________________________
amd-gfx mailing list
amd-gfx at lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx