On Wed, Aug 10, 2016 at 8:52 PM, Hal Rosenstock <hal@xxxxxxxxxxxxxxxxxx> wrote:
> On 8/9/2016 12:26 PM, Jinpu Wang wrote:
>> Hi Sean,
>>
>> I'm testing QoS support for IB. I notice ucmatose has equally
>> performance when set different service type, but set SL in ib_send_bw
>> works well (different SL show different performance base on opensm
>> settings)
>>
>> I capature packats using ibdump, it shows in in LRH the service level
>> fields are all 0 when running traffic with ucmatose.
>>
>> When running ib_send_bw, it carries the right service level I set.
>>
>> Seems in rdma_set_service_type, it sets to tos to id_priv->tos, and
>> lter set to path_rec->qos_class or traffic_class but not to sl
>> directly, what's the consideration here?
>> code snip:
>> switch (cma_family(id_priv)) {
>> case AF_INET:
>> path_rec->qos_class = cpu_to_be16((u16) id_priv->tos);
>> comp_mask |= IB_SA_PATH_REC_QOS_CLASS;
>> break;
>> case AF_INET6:
>> sin6 = (struct sockaddr_in6 *) cma_src_addr(id_priv);
>> path_rec->traffic_class = (u8)
>> (be32_to_cpu(sin6->sin6_flowinfo) >> 20);
>> comp_mask |= IB_SA_PATH_REC_TRAFFIC_CLASS;
>> break;
>> case AF_IB:
>> sib = (struct sockaddr_ib *) cma_src_addr(id_priv);
>> path_rec->traffic_class = (u8)
>> (be32_to_cpu(sib->sib_flowinfo) >> 20);
>>
>>
>> Does it make sense we also set sl here, or service type for ucmatose
>> is totally different with SL for ib_send_bw?
>
> I think this is an OpenSM configuration issue. QoS policy needs to be
> setup to return the proper SL to use for QoS class or TClass in the
> PathRecord response.
>
> -- Hal
>
Thanks Hal,
Configure extra QoS policy seems quite complex.
Do you think patch attached make sense?
--
Mit freundlichen Grüßen,
Best Regards,
Jack Wang
Linux Kernel Developer Storage
ProfitBricks GmbH The IaaS-Company.
ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin
Tel: +49 30 5770083-42
Fax: +49 30 5770085-98
Email: jinpu.wang@xxxxxxxxxxxxxxxx
URL: http://www.profitbricks.de
Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B.
Geschäftsführer: Andreas Gauger, Achim Weiss.
From beec3a2fc25d37d4f735c55231f4e0fbe3f180ac Mon Sep 17 00:00:00 2001
From: Jack Wang <jinpu.wang@xxxxxxxxxxxxxxxx>
Date: Wed, 10 Aug 2016 10:50:53 +0200
Subject: [PATCH] cma: export function to set service level
We want this for isolating network traffic with storage traffic.
So extend cma to allow us to do it for QoS.
Signed-off-by: Jack Wang <jinpu.wang@xxxxxxxxxxxxxxxx>
---
drivers/infiniband/core/cma.c | 14 +++++++++++++-
include/rdma/rdma_cm.h | 13 +++++++++++++
2 files changed, 26 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 66e8516..c464aa7 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -225,6 +225,7 @@ struct rdma_id_private {
u32 options;
u8 srq;
u8 tos;
+ u8 sl;
u8 reuseaddr;
u8 afonly;
enum ib_gid_type gid_type;
@@ -2752,6 +2753,16 @@ static void cma_listen_on_all(struct rdma_id_private *id_priv)
mutex_unlock(&lock);
}
+void rdma_set_service_level(struct rdma_cm_id *id, u8 sl)
+{
+ struct rdma_id_private *id_priv;
+
+ id_priv = container_of(id, struct rdma_id_private, id);
+ id_priv->sl = sl;
+}
+EXPORT_SYMBOL(rdma_set_service_level);
+
+
void rdma_set_service_type(struct rdma_cm_id *id, int tos)
{
struct rdma_id_private *id_priv;
@@ -2838,9 +2849,10 @@ static int cma_query_ib_route(struct rdma_id_private *id_priv, int timeout_ms,
path_rec->pkey = cpu_to_be16(ib_addr_get_pkey(&addr->dev_addr));
path_rec->numb_path = 1;
path_rec->reversible = 1;
+ path_rec->sl = id_priv->sl;
path_rec->service_id = rdma_get_service_id(&id_priv->id, cma_dst_addr(id_priv));
- comp_mask |= IB_SA_PATH_REC_PKEY | IB_SA_PATH_REC_NUMB_PATH |
+ comp_mask |= IB_SA_PATH_REC_PKEY | IB_SA_PATH_REC_NUMB_PATH | IB_SA_PATH_REC_SL |
IB_SA_PATH_REC_REVERSIBLE | IB_SA_PATH_REC_SERVICE_ID;
switch (cma_family(id_priv)) {
diff --git a/include/rdma/rdma_cm.h b/include/rdma/rdma_cm.h
index b34ee4e..df7030e 100644
--- a/include/rdma/rdma_cm.h
+++ b/include/rdma/rdma_cm.h
@@ -374,6 +374,19 @@ int rdma_join_multicast(struct rdma_cm_id *id, struct sockaddr *addr,
void rdma_leave_multicast(struct rdma_cm_id *id, struct sockaddr *addr);
/**
+ * rdma_set_service_level - Set the level of service associated with a
+ * connection identifier.
+ * @id: Communication identifier to associated with service type.
+ * @sl: service level.
+ *
+ * The service level should be specified before
+ * performing route resolution, as existing communication on the
+ * connection identifier may be unaffected. The level of service
+ * requested may not be supported by the network to all destinations.
+ */
+void rdma_set_service_level(struct rdma_cm_id *id, u8 sl);
+
+/**
* rdma_set_service_type - Set the type of service associated with a
* connection identifier.
* @id: Communication identifier to associated with service type.
--
2.7.4
