This series something I was hacking on a little while back in an attempt to make our RPC layer more maintainable. There are many aspects I'm unhappy about with current code * When serializing a message we have no clue how big it will be, but xdrmem_create wants a fixed size, so we have to keep trying to serialize in a loop making it bigger each time * We don't control memory allocation/free'ing directly so we can't do a virSecureErase on fields inside the RPC message struct that handle secrets. * The XDR API is generally unpleasant to use as it is outside our virNetMessage object. Ideally we would be reading/writing directly from/to the virNetMessage buffer with APIs on virNetMessage,instead of indirectly via a XDR object. * We want more from XDR than it actually gives us. Our XDR protocol files have annotations to express what we want our code generator todo, or for ACLs. The relationship between the structs and the message numbers is implicit. Essentially we've defined our own language indirectly via comments, and then parse this with regexes which is horrid. * The code rpcgen creates is poor quality which we have to post-process to fix bugs/problems. It also lacks support for modern features like g_auto. Anyway, in a fit of rage I looked at the XDR RFC and thought.. This language is trivial, why do we need to outsource to rpcgen and libtirpc instead of dealing with it directly. This small series moves in that direction. It creates an XDR language lexer and parser, and then a code generator which emits code that is (nearly) identical to what rpcgen would emit. This is sufficient to eliminate rpcgen usage and support g_auto. Since we're still using libtirpc at this stage we can be confident we're still doing the same thing on the wire. I've got some unit tests too with the rpcgen generation to validate stuff. The next step is to change the code generator so that instead of generating code for libtirpc APIs, it will instead directly speak virNetMessage APIs. That would give us full control over our RPC stack guaranteed to be platform portable instead of fighting slight differences in RPC libraries (eg xdr_quad vs xdr_int64 madness). I was going to wait until I had written such code before sending this series, but I've got diverted onto other more important tasks. So here at least is what I have so far. After that foundation is done, we are in a place where we can actually do more innovative things. For example we can directly extend the XDR protocol language if we like, turning our magic comments into properly parsable constructs. With this, we would be in a position to replace our Perl RPC client/server dispatch code generators with something that is more supportable in Python. The python code would work with properly represented objects and formal parsers and not regexes and anonymous complex perl data structures. Daniel P. Berrangé (16): rpcgen: drop type-puning workarounds build-aux: skip E203 and W503 flake8 checks build-aux: introduce 'black' tool for python formatting rpcgen: add an XDR protocol lexer rpcgen: add an XDR protocol abstract syntax tree rpcgen: add an XDR protocol parser rpcgen: define a visitor API for XDR protocol specs rpcgen: add a C code generator for XDR protocol specs rpcgen: add test case for XDR serialization rpcgen: define entrypoint for running new rpcgen impl build: switch over to new rpc generator code rpcgen: add g_auto function support rpc: use g_auto for client RPC return parameters admin: use g_auto for client RPC return parameters remote: use g_auto for client RPC return parameters rpc: add helpers for XDR type serialization build-aux/Makefile.in | 1 + build-aux/meson.build | 5 + build-aux/syntax-check.mk | 42 +- libvirt.spec.in | 2 +- meson.build | 13 +- scripts/meson.build | 2 + scripts/rpcgen/main.py | 90 ++ scripts/rpcgen/meson.build | 16 + scripts/rpcgen/rpcgen/ast.py | 270 ++++++ scripts/rpcgen/rpcgen/generator.py | 509 ++++++++++++ scripts/rpcgen/rpcgen/lexer.py | 213 +++++ scripts/rpcgen/rpcgen/meson.build | 7 + scripts/rpcgen/rpcgen/parser.py | 497 +++++++++++ scripts/rpcgen/rpcgen/visitor.py | 156 ++++ scripts/rpcgen/tests/demo.c | 495 +++++++++++ scripts/rpcgen/tests/demo.h | 264 ++++++ scripts/rpcgen/tests/demo.x | 127 +++ scripts/rpcgen/tests/meson.build | 20 + scripts/rpcgen/tests/simple.x | 35 + scripts/rpcgen/tests/test_demo.c | 782 ++++++++++++++++++ scripts/rpcgen/tests/test_demo_enum.bin | Bin 0 -> 4 bytes .../tests/test_demo_enum_fixed_array.bin | Bin 0 -> 52 bytes .../tests/test_demo_enum_pointer_null.bin | Bin 0 -> 4 bytes .../tests/test_demo_enum_pointer_set.bin | Bin 0 -> 8 bytes .../rpcgen/tests/test_demo_enum_scalar.bin | Bin 0 -> 4 bytes .../test_demo_enum_variable_array_empty.bin | Bin 0 -> 4 bytes .../test_demo_enum_variable_array_set.bin | Bin 0 -> 16 bytes .../tests/test_demo_int_fixed_array.bin | Bin 0 -> 12 bytes .../tests/test_demo_int_pointer_null.bin | Bin 0 -> 4 bytes .../tests/test_demo_int_pointer_set.bin | Bin 0 -> 8 bytes scripts/rpcgen/tests/test_demo_int_scalar.bin | Bin 0 -> 4 bytes .../test_demo_int_variable_array_empty.bin | Bin 0 -> 4 bytes .../test_demo_int_variable_array_set.bin | Bin 0 -> 16 bytes .../tests/test_demo_opaque_fixed_array.bin | Bin 0 -> 12 bytes .../test_demo_opaque_variable_array_empty.bin | Bin 0 -> 4 bytes .../test_demo_opaque_variable_array_set.bin | Bin 0 -> 8 bytes .../test_demo_string_variable_array_empty.bin | Bin 0 -> 4 bytes .../test_demo_string_variable_array_set.bin | Bin 0 -> 12 bytes scripts/rpcgen/tests/test_demo_struct.bin | Bin 0 -> 8 bytes .../tests/test_demo_struct_fixed_array.bin | Bin 0 -> 136 bytes .../tests/test_demo_struct_pointer_null.bin | Bin 0 -> 4 bytes .../tests/test_demo_struct_pointer_set.bin | Bin 0 -> 12 bytes .../rpcgen/tests/test_demo_struct_scalar.bin | 1 + .../test_demo_struct_variable_array_empty.bin | Bin 0 -> 4 bytes .../test_demo_struct_variable_array_set.bin | Bin 0 -> 28 bytes .../tests/test_demo_test_struct_all_types.bin | Bin 0 -> 1752 bytes scripts/rpcgen/tests/test_demo_union_case.bin | Bin 0 -> 8 bytes .../rpcgen/tests/test_demo_union_default.bin | Bin 0 -> 8 bytes .../tests/test_demo_union_fixed_array.bin | Bin 0 -> 168 bytes .../tests/test_demo_union_no_default_case.bin | Bin 0 -> 8 bytes .../tests/test_demo_union_pointer_null.bin | Bin 0 -> 4 bytes .../tests/test_demo_union_pointer_set.bin | Bin 0 -> 12 bytes .../rpcgen/tests/test_demo_union_scalar.bin | Bin 0 -> 8 bytes .../test_demo_union_variable_array_empty.bin | Bin 0 -> 4 bytes .../test_demo_union_variable_array_set.bin | Bin 0 -> 28 bytes .../test_demo_union_void_default_case.bin | Bin 0 -> 8 bytes .../test_demo_union_void_default_default.bin | 1 + scripts/rpcgen/tests/test_generator.py | 60 ++ scripts/rpcgen/tests/test_lexer.py | 116 +++ scripts/rpcgen/tests/test_parser.py | 91 ++ src/admin/admin_remote.c | 50 +- src/admin/meson.build | 8 +- src/locking/meson.build | 8 +- src/logging/meson.build | 8 +- src/lxc/meson.build | 12 +- src/remote/meson.build | 8 +- src/remote/remote_driver.c | 754 ++++++----------- src/rpc/gendispatch.pl | 60 +- src/rpc/genprotocol.pl | 144 ---- src/rpc/meson.build | 9 +- src/rpc/virnetmessage.c | 704 ++++++++++++++++ src/rpc/virnetmessage.h | 88 ++ 72 files changed, 4897 insertions(+), 771 deletions(-) create mode 100755 scripts/rpcgen/main.py create mode 100644 scripts/rpcgen/meson.build create mode 100644 scripts/rpcgen/rpcgen/ast.py create mode 100644 scripts/rpcgen/rpcgen/generator.py create mode 100644 scripts/rpcgen/rpcgen/lexer.py create mode 100644 scripts/rpcgen/rpcgen/meson.build create mode 100644 scripts/rpcgen/rpcgen/parser.py create mode 100644 scripts/rpcgen/rpcgen/visitor.py create mode 100644 scripts/rpcgen/tests/demo.c create mode 100644 scripts/rpcgen/tests/demo.h create mode 100644 scripts/rpcgen/tests/demo.x create mode 100644 scripts/rpcgen/tests/meson.build create mode 100644 scripts/rpcgen/tests/simple.x create mode 100644 scripts/rpcgen/tests/test_demo.c create mode 100644 scripts/rpcgen/tests/test_demo_enum.bin create mode 100644 scripts/rpcgen/tests/test_demo_enum_fixed_array.bin create mode 100644 scripts/rpcgen/tests/test_demo_enum_pointer_null.bin create mode 100644 scripts/rpcgen/tests/test_demo_enum_pointer_set.bin create mode 100644 scripts/rpcgen/tests/test_demo_enum_scalar.bin create mode 100644 scripts/rpcgen/tests/test_demo_enum_variable_array_empty.bin create mode 100644 scripts/rpcgen/tests/test_demo_enum_variable_array_set.bin create mode 100644 scripts/rpcgen/tests/test_demo_int_fixed_array.bin create mode 100644 scripts/rpcgen/tests/test_demo_int_pointer_null.bin create mode 100644 scripts/rpcgen/tests/test_demo_int_pointer_set.bin create mode 100644 scripts/rpcgen/tests/test_demo_int_scalar.bin create mode 100644 scripts/rpcgen/tests/test_demo_int_variable_array_empty.bin create mode 100644 scripts/rpcgen/tests/test_demo_int_variable_array_set.bin create mode 100644 scripts/rpcgen/tests/test_demo_opaque_fixed_array.bin create mode 100644 scripts/rpcgen/tests/test_demo_opaque_variable_array_empty.bin create mode 100644 scripts/rpcgen/tests/test_demo_opaque_variable_array_set.bin create mode 100644 scripts/rpcgen/tests/test_demo_string_variable_array_empty.bin create mode 100644 scripts/rpcgen/tests/test_demo_string_variable_array_set.bin create mode 100644 scripts/rpcgen/tests/test_demo_struct.bin create mode 100644 scripts/rpcgen/tests/test_demo_struct_fixed_array.bin create mode 100644 scripts/rpcgen/tests/test_demo_struct_pointer_null.bin create mode 100644 scripts/rpcgen/tests/test_demo_struct_pointer_set.bin create mode 100644 scripts/rpcgen/tests/test_demo_struct_scalar.bin create mode 100644 scripts/rpcgen/tests/test_demo_struct_variable_array_empty.bin create mode 100644 scripts/rpcgen/tests/test_demo_struct_variable_array_set.bin create mode 100644 scripts/rpcgen/tests/test_demo_test_struct_all_types.bin create mode 100644 scripts/rpcgen/tests/test_demo_union_case.bin create mode 100644 scripts/rpcgen/tests/test_demo_union_default.bin create mode 100644 scripts/rpcgen/tests/test_demo_union_fixed_array.bin create mode 100644 scripts/rpcgen/tests/test_demo_union_no_default_case.bin create mode 100644 scripts/rpcgen/tests/test_demo_union_pointer_null.bin create mode 100644 scripts/rpcgen/tests/test_demo_union_pointer_set.bin create mode 100644 scripts/rpcgen/tests/test_demo_union_scalar.bin create mode 100644 scripts/rpcgen/tests/test_demo_union_variable_array_empty.bin create mode 100644 scripts/rpcgen/tests/test_demo_union_variable_array_set.bin create mode 100644 scripts/rpcgen/tests/test_demo_union_void_default_case.bin create mode 100644 scripts/rpcgen/tests/test_demo_union_void_default_default.bin create mode 100644 scripts/rpcgen/tests/test_generator.py create mode 100644 scripts/rpcgen/tests/test_lexer.py create mode 100644 scripts/rpcgen/tests/test_parser.py delete mode 100755 src/rpc/genprotocol.pl -- 2.39.1
