Hi Gordan, On 04/28/2016 05:00 PM, Gordan Bobic wrote: > On 2016-04-28 19:49, Jon Masters wrote: > First of all, Jon, thank you for your thoughts on this matter. No problem :) >> Allow me to add a few thoughts. I have been working with the ARM vendors >> (as well as the ARM Architecture Group) since before the architecture >> was announced, and the issue of page size and 32-bit backward >> compatibility came up in the earliest days. I am speaking from a Red Hat >> perspective and NOT dictating what Fedora should or must do, but I do >> strongly encourage Fedora not to make a change to something like the >> page size simply to support a (relatively) small number of corner cases. > > IMO, the issue of backward compatibility is completely secondary to > the issue of efficiency of memory fragmentation/occupancy when it comes > to 64KB pages. And that isn't a corner case, it is the overwhelmingly > primary case. Let's keep to the memory discussion then, I agree. On the fragmentation argument, I do agree this is an area where server/non-server uses certainly clash. It might well be that we later decide in Fedora that 4K is the right size once there are more 64-bit client devices. >> Having an entire separate several ISAs just for the fairly nonexistent field of >> proprietary non-recompilable third party 32-bit apps doesn't really make >> sense. Sure, running 32-bit via multilib is fun and all, but it's not >> really something that is critical to using ARM systems. > > Except where there's no choice, such as closed source applications > (Plex comes to mind) or libraries without appropriate ARM64 implementation > such as Mono. I'm sure pure aarch64 will be supported by it all at > some point, but the problem is real today. It's definitely true that there are some applications that aren't yet ported to ARMv8, though that list is fairly small (compared with IA32). > But OK, for the sake of this discussion let's completely ignore the > 32-bit support to simplify things. OK :) >> The mandatory page sizes in the v8 architecture are 4K and 64K, with >> various options around the number of bits used for address spaces, huge >> pages (or ginormous pages), and contiguous hinting for smaller "huge" >> pages. There is an option for 16K pages, but it is not mandatory. In the >> server specifications, we don't compel Operating Systems to use 64K, but >> everything is written with that explicitly in mind. By using 64K early >> we ensure that it is possible to do so in a very clean way, and then if >> (over the coming years) the deployment of sufficient real systems proves >> that this was a premature decision, we still have 4K. > > The real question is how much code will bit-rot due to not being > tested with 4KB pages With respect, I think it's the other way around. We have another whole architecture targeting 4K pages by default, and (regretfully perhaps, though that's a personal opinion) it's a pretty popular choice that many people are using in Fedora today. So I don't see any situation in which 4K bitrots over 64K. I did see the opposite being very likely if we didn't start out with 64K as the baseline going in on day one. >> I also asked a few of the chip >> vendors not to implement 32-bit execution (and some of them have indeed >> omitted it after we discussed the needs early on), and am aggressively >> pushing for it to go away over time in all server parts. But there's >> more to it than that. In the (very) many early conversations with >> various performance folks, the feedback was that larger page sizes than >> 4K should generally be adopted for a new arch. Ideally that would have >> been 16K (which other architectures than x86 went with also), but that >> was optional. Optionally necessarily means "does not exist". My advice >> when Red Hat began internal work on ARMv8 was to listen to the experts. > > Linus is not an expert? Note that I never said he isn't an expert. He's one of the smartest guys around, but he's not always right 100% of the time. Folks who run performance numbers were consulted about the merits of 64K (as were a number of chip architects) and they said that was the way to go. We can always later decide (once there's a server market running fully) that this was premature and change to 4K, but it's very hard to go the other way around later if we settle for 4K on day one. The reason is 4K works great out of the box as it's got 30 years of history on that other arch, but for 64K we've only POWER to call on, and its userbase generally aren't stressing the same workloads as on 64-bit ARM. Sometimes they are, and that's been helpful with obscure things like emacs crashing due to a page size assumption or two on arrow presses. >> I am well aware of Linus's views on the topic and I have seen the rants >> on G+ and elsewhere. I am completely willing to be wrong (there is not >> enough data yet) over moving to 64K too soon and ultimately if it was >> premature see things like RHELSA on the Red Hat side switch back to 4K. > > My main concern is around how much code elsewhere will rot and need > attention should this ever happen. I think, once again, that any concern over 4K being a well supported page size is perhaps made moot by the billions of x86 systems out there using that size. Most of the time, it's not the case that applications have assembly code level changes required for 64K. Sure, the toolchain will emit optimized code and it will use adrp and other stuff in v8 to reference pages and offsets, but that compiler code works well. It's not the piece that's got any potential for issue. It's the higher level C code that possibly has assumptions to iron out on a 64K base vs 4K. >> Fedora is its own master, but I strongly encourage retaining the use of >> 64K granules at this time, and letting it play out without responding to >> one or two corner use cases and changing course. There are very many >> design optimizations that can be done when you have a 64K page size, >> from the way one can optimize cache lookups and hardware page table >> walker caches to the reduction of TLB pressure (though I accept that >> huge pages are an answer for this under a 4K granule regime as well). It >> would be nice to blaze a trail rather than take the safe default. > > While I agree with the sentiment, I think something like this is > better decided on carefully considered merit assessed through > empirical measurement. Sure. We had to start with something. Folks now have something that they can use to run numbers on. BUT note that the kind of 64-bit hw that is needed to really answer these questions is only just coming. Again, if 64K was a wrong choice, we can change it. It's only a mistake if we always dogmatically stick to principle in the face of evidence to the contrary. If the evidence says "dude, 64K was at best premature and Linus was right", then that's totally cool with me. We'll meanwhile have a codebase that is even more portable (different arch/pagesz). >> My own opinion is that (in the longer term, beginning with server) we >> should not have a 32-bit legacy of the kind that x86 has to deal with >> forever. We can use virtualization (and later, if it really comes to it, >> containers running 32-bit applications with 4K pages exposed to them - >> an implementation would be a bit like "Clear" containers today) to run >> 32-bit applications on 64-bit without having to do nasty hacks (such as >> multilib) and reduce any potential for confusion on the part of users >> (see also RasPi 3 as an example). It is still early enough in the >> evolution of general purpose aarch64 to try this, and have the pragmatic >> fallback of retreating to 4K if needed. The same approach of running >> under virtualization or within a container model equally applies to >> ILP32, which is another 32-bit ABI that some folks like, in that a third >> party group is welcome to do all of the lifting required. > > This again mashes 32-bit support with page size. If there is no > 32-bit support in the CPU, I am reasonably confident that QEMU > emulation if it will be unusably slow for just about any serious > use case (you might as well run QEMU emulation of ARM32 on x86 > in that case and not even touch upon aarch64). Point noted. If we keep the conversation purely to the relative merits of 64K vs 4K page size upon memory use overhead, fragmentation, and the like, then the previous comment about getting numbers stands. This is absolutely something we intend to gather within the perf team inside Red Hat (and share in some form) as more hardware arrives that can be realistically used to quantify the value. You're welcome to also run numbers and show that there's a definite case for 4K over 64K. >>> 2) Nobody has yet pointed at ARM's own documentation (I did ask >>> earlier) that says that 4KB memory page support is optional >>> rather than mandatory. >> >> Nobody said this was a requirement. I believe you raised this as some >> kind of logical fallacy to reinforce the position that you have taken. Apologies if this wasn't you. > I'm afraid you got that backwards. I believe it was Peter that > said that Seattle didn't support 4KB pages, seemingly implied > as a means of justifying the use of 64KB pages: > https://lists.fedoraproject.org/archives/list/arm@xxxxxxxxxxxxxxxxxxxxxxx/message/ZFARX7PARXUJS5GW32MAAMMRWLVCHB5E/ Seattle is only tested (by us) using 64K pages, the hardware supports 4K pages at an architectural level. I get your argument that this could well mean that if we later drop to 4K pages there could be platforms that have issues. I would counter that I know of at least one other distribution that's fairly popular also which is building with 4K pages, and is being used on some platforms, so the number of platforms that won't be able to handle 4K is probably quite limited. The variety of options out there between distros is a *good* thing for validation. > If Seattle does in fact support the spec mandatory 4KB memory > pages, then that specific SoC is no longer relevant to this > thread. Then we can move on from that. Thanks, Jon. -- Computer Architect | Sent from my Fedora powered laptop _______________________________________________ arm mailing list arm@xxxxxxxxxxxxxxxxxxxxxxx http://lists.fedoraproject.org/admin/lists/arm@xxxxxxxxxxxxxxxxxxxxxxx
