On Wed, May 19, 2021 at 2:29 PM Jing Zhang <jingzhangos@xxxxxxxxxx> wrote:
>
> Hi David,
>
> On Wed, May 19, 2021 at 11:57 AM David Matlack <dmatlack@xxxxxxxxxx> wrote:
> >
> > On Mon, May 17, 2021 at 9:25 AM Jing Zhang <jingzhangos@xxxxxxxxxx> wrote:
> > >
> > > Update KVM API documentation for binary statistics.
> > >
> > > Signed-off-by: Jing Zhang <jingzhangos@xxxxxxxxxx>
> > > ---
> > > Documentation/virt/kvm/api.rst | 171 +++++++++++++++++++++++++++++++++
> > > 1 file changed, 171 insertions(+)
> > >
> > > diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
> > > index 7fcb2fd38f42..9a6aa9770dfd 100644
> > > --- a/Documentation/virt/kvm/api.rst
> > > +++ b/Documentation/virt/kvm/api.rst
> > > @@ -5034,6 +5034,169 @@ see KVM_XEN_VCPU_SET_ATTR above.
> > > The KVM_XEN_VCPU_ATTR_TYPE_RUNSTATE_ADJUST type may not be used
> > > with the KVM_XEN_VCPU_GET_ATTR ioctl.
> > >
> > > +4.130 KVM_STATS_GETFD
> > > +---------------------
> > > +
> > > +:Capability: KVM_CAP_STATS_BINARY_FD
> > > +:Architectures: all
> > > +:Type: vm ioctl, vcpu ioctl
> > > +:Parameters: none
> > > +:Returns: statistics file descriptor on success, < 0 on error
> > > +
> > > +Errors:
> > > +
> > > + ====== ======================================================
> > > + ENOMEM if the fd could not be created due to lack of memory
> > > + EMFILE if the number of opened files exceeds the limit
> > > + ====== ======================================================
> > > +
> > > +The file descriptor can be used to read VM/vCPU statistics data in binary
> > > +format. The file data is organized into three blocks as below:
> > > ++-------------+
> > > +| Header |
> > > ++-------------+
> > > +| Descriptors |
> > > ++-------------+
> > > +| Stats Data |
> > > ++-------------+
> > > +
> > > +The Header block is always at the start of the file. It is only needed to be
> > > +read one time after a system boot.
> >
> > By system boot do you mean the host or the VM? If the host then it's
> > probably just cleaner to omit that part entirely and just say "It is
> > only needed to be read once.".
> >
> Will change "system boot" to "VM boot".
> > > +It is in the form of ``struct kvm_stats_header`` as below::
> > > +
> > > + #define KVM_STATS_ID_MAXLEN 64
> > > +
> > > + struct kvm_stats_header {
> > > + char id[KVM_STATS_ID_MAXLEN];
> > > + __u32 name_size;
> > > + __u32 count;
> > > + __u32 desc_offset;
> > > + __u32 data_offset;
> > > + };
> > > +
> > > +The ``id`` field is identification for the corresponding KVM statistics. For
> > > +KVM statistics, it is in the form of "kvm-{kvm pid}", like "kvm-12345". For
> >
> > Should this say "For VM statistics, ..." instead?
> >
> Yes, will fix it.
> > > +VCPU statistics, it is in the form of "kvm-{kvm pid}/vcpu-{vcpu id}", like
> > > +"kvm-12345/vcpu-12".
> > > +
> > > +The ``name_size`` field is the size (byte) of the statistics name string
> > > +(including trailing '\0') appended to the end of every statistics descriptor.
> > > +
> > > +The ``count`` field is the number of statistics.
> > > +
> > > +The ``desc_offset`` field is the offset of the Descriptors block from the start
> > > +of the file indicated by the file descriptor.
> > > +
> > > +The ``data_offset`` field is the offset of the Stats Data block from the start
> > > +of the file indicated by the file descriptor.
> > > +
> > > +The Descriptors block is only needed to be read once after a system boot. It is
> >
> > Ditto here about system boot.
> >
> > > +an array of ``struct kvm_stats_desc`` as below::
> >
> > Consider omitting these macros from the documentation, or moving them
> > to later. Readers right here are expecting to see the struct
> > kvm_stats_desc given the previous line.
> >
> How about changing "as below" to "as shown in below code block"?
> > > +
> > > + #define KVM_STATS_TYPE_SHIFT 0
> > > + #define KVM_STATS_TYPE_MASK (0xF << KVM_STATS_TYPE_SHIFT)
> > > + #define KVM_STATS_TYPE_CUMULATIVE (0x0 << KVM_STATS_TYPE_SHIFT)
> > > + #define KVM_STATS_TYPE_INSTANT (0x1 << KVM_STATS_TYPE_SHIFT)
> > > + #define KVM_STATS_TYPE_MAX KVM_STATS_TYPE_INSTANT
> > > +
> > > + #define KVM_STATS_UNIT_SHIFT 4
> > > + #define KVM_STATS_UNIT_MASK (0xF << KVM_STATS_UNIT_SHIFT)
> > > + #define KVM_STATS_UNIT_NONE (0x0 << KVM_STATS_UNIT_SHIFT)
> > > + #define KVM_STATS_UNIT_BYTES (0x1 << KVM_STATS_UNIT_SHIFT)
> > > + #define KVM_STATS_UNIT_SECONDS (0x2 << KVM_STATS_UNIT_SHIFT)
> > > + #define KVM_STATS_UNIT_CYCLES (0x3 << KVM_STATS_UNIT_SHIFT)
> > > + #define KVM_STATS_UNIT_MAX KVM_STATS_UNIT_CYCLES
> > > +
> > > + #define KVM_STATS_SCALE_SHIFT 8
> > > + #define KVM_STATS_SCALE_MASK (0xF << KVM_STATS_SCALE_SHIFT)
> > > + #define KVM_STATS_SCALE_POW10 (0x0 << KVM_STATS_SCALE_SHIFT)
> > > + #define KVM_STATS_SCALE_POW2 (0x1 << KVM_STATS_SCALE_SHIFT)
> > > + #define KVM_STATS_SCALE_MAX KVM_STATS_SCALE_POW2
> >
> > Terminology nit: I think usually this part is called the "base". e.g.
> > when you decompose a number X into N * B^E, B is the "base" and E is
> > the "exponent". I see you're using "exponent" already but it might
> > make sense to change "scale" to "base" throughout this series.
> >
> Will change "SCALE" to "SCALE_BASE".
> > > +
> > > + struct kvm_stats_desc {
> > > + __u32 flags;
> > > + __s16 exponent;
> > > + __u16 size;
> > > + __u32 unused1;
> > > + __u32 unused2;
> > > + char name[0];
> > > + };
> > > +
> > > +The ``flags`` field contains the type and unit of the statistics data described
> > > +by this descriptor. The following flags are supported:
> >
> > nit: Suggest breaking this list out into separate lists so readers can
> > differentiate between the type, unit, and scale. Something like:
> >
> > Bits 0-3 of ``flags`` encode the type:
> >
> > * ``KVM_STATS_TYPE_CUMULATIVE`` ...
> > * ``KVM_STATS_TYPE_INSTANT`` ...
> >
> > Bits 4-7 of ``flags encode the unit:
> >
> > * ``KVM_STATS_UNIT_NONE`` ...
> > ...
> > etc.
> >
> Good suggestion. Will do that.
> > > + * ``KVM_STATS_TYPE_CUMULATIVE``
> > > + The statistics data is cumulative. The value of data can only be increased.
> > > + Most of the counters used in KVM are of this type.
> > > + The corresponding ``count`` filed for this type is always 1.
> > > + * ``KVM_STATS_TYPE_INSTANT``
> > > + The statistics data is instantaneous. Its value can be increased or
> > > + decreased. This type is usually used as a measurement of some resources,
> > > + like the number of dirty pages, the number of large pages, etc.
> > > + The corresponding ``count`` field for this type is always 1.
> > > + * ``KVM_STATS_UNIT_NONE``
> > > + There is no unit for the value of statistics data. This usually means that
> > > + the value is a simple counter of an event.
> > > + * ``KVM_STATS_UNIT_BYTES``
> > > + It indicates that the statistics data is used to measure memory size, in the
> > > + unit of Byte, KiByte, MiByte, GiByte, etc. The unit of the data is
> > > + determined by the ``exponent`` field in the descriptor. The
> > > + ``KVM_STATS_SCALE_POW2`` flag is valid in this case. The unit of the data is
> > > + determined by ``pow(2, exponent)``. For example, if value is 10,
> > > + ``exponent`` is 20, which means the unit of statistics data is MiByte, we
> > > + can get the statistics data in the unit of Byte by
> > > + ``value * pow(2, exponent) = 10 * pow(2, 20) = 10 MiByte`` which is
> > > + 10 * 1024 * 1024 Bytes.
> > > + * ``KVM_STATS_UNIT_SECONDS``
> > > + It indicates that the statistics data is used to measure time/latency, in
> > > + the unit of nanosecond, microsecond, millisecond and second. The unit of the
> > > + data is determined by the ``exponent`` field in the descriptor. The
> > > + ``KVM_STATS_SCALE_POW10`` flag is valid in this case. The unit of the data
> > > + is determined by ``pow(10, exponent)``. For example, if value is 2000000,
> > > + ``exponent`` is -6, which means the unit of statistics data is microsecond,
> > > + we can get the statistics data in the unit of second by
> > > + ``value * pow(10, exponent) = 2000000 * pow(10, -6) = 2 seconds``.
> > > + * ``KVM_STATS_UNIT_CYCLES``
> > > + It indicates that the statistics data is used to measure CPU clock cycles.
> > > + The ``KVM_STATS_SCALE_POW10`` flag is valid in this case. For example, if
> > > + value is 200, ``exponent`` is 4, we can get the number of CPU clock cycles
> > > + by ``value * pow(10, exponent) = 200 * pow(10, 4) = 2000000``.
> > > +
> > > +The ``exponent`` field is the scale of corresponding statistics data. It has two
> > > +values as follows:
> > > + * ``KVM_STATS_SCALE_POW10``
> >
> > I thought the scale was encoded in ``flags`` not ``exponent``? Isn't
> > the exponent the
> >
> The base is encoded in ``flags``, not the exponent.
The description about ``exponent`` is not right here. Will fix it.
> > > + The scale is based on power of 10. It is used for measurement of time and
> > > + CPU clock cycles.
> > > + * ``KVM_STATS_SCALE_POW2``
> > > + The scale is based on power of 2. It is used for measurement of memory size.
> >
> > It might be useful to give an example of how to use the exponent field
> > in practice.
> >
> Those examples where we discuss ``flags`` field also cover the usage
> of exponent field.
Will add an example here too.
> > > +
> > > +The ``size`` field is the number of values of this statistics data. It is in the
> > > +unit of ``unsigned long`` for VCPU or ``__u64`` for VM.
> > > +
> > > +The ``unused1`` and ``unused2`` fields are reserved for future
> > > +support for other types of statistics data, like log/linear histogram.
> > > +
> > > +The ``name`` field points to the name string of the statistics data. The name
> > > +string starts at the end of ``struct kvm_stats_desc``.
> > > +The maximum length (including trailing '\0') is indicated by ``name_size``
> > > +in ``struct kvm_stats_header``.
> > > +
> > > +The Stats Data block contains an array of data values of type ``struct
> > > +kvm_vm_stats_data`` or ``struct kvm_vcpu_stats_data``. It would be read by
> > > +user space periodically to pull statistics data.
> > > +The order of data value in Stats Data block is the same as the order of
> > > +descriptors in Descriptors block.
> > > + * Statistics data for VM::
> > > +
> > > + struct kvm_vm_stats_data {
> > > + unsigned long value[0];
> > > + };
> > > +
> > > + * Statistics data for VCPU::
> > > +
> > > + struct kvm_vcpu_stats_data {
> > > + __u64 value[0];
> > > + };
> > > +
> > > 5. The kvm_run structure
> > > ========================
> > >
> > > @@ -6891,3 +7054,11 @@ This capability is always enabled.
> > > This capability indicates that the KVM virtual PTP service is
> > > supported in the host. A VMM can check whether the service is
> > > available to the guest on migration.
> > > +
> > > +8.33 KVM_CAP_STATS_BINARY_FD
> > > +----------------------------
> > > +
> > > +:Architectures: all
> > > +
> > > +This capability indicates the feature that user space can create get a file
> > > +descriptor for every VM and VCPU to read statistics data in binary format.
> > > --
> > > 2.31.1.751.gd2f1c929bd-goog
> > >
> Thanks,
> Jing
