OTue, 2016-04-26 a13:43 -0400, Neil Horman wrote:
> This was recently reported to me, and reproduced othlatest net kernel, when
> attempting to runetperf froa host that had a netem qdisc attached to the
> egress interface:
..
> Thprobleoccurs because netem is not prepared to handle GSO packets (as it
> uses skb_checksum_help iits enqueupath, which cannot manipulate these
> frames).
>
> Thsolution I think is to simply segmenthe skb in a simmilar fashion to the
> way wdo in __dev_queue_xmi(via validate_xmit_skb), except here we always
> segment, instead of only whethinterface needs us to do it. This allows
> neteto properly drop/mangle/pass/etc thcorrect percentages of frames as per
> its qdisc configuration, and avoid failing its checksuoperations
>
> tested successfully by myself othlatest net kernel, to whcih this applies
>
> Signed-off-by: Neil Horma<nhorman atuxdriver.com>
> CC: Jamal Hadi Sali<jhs amojatatu.com>
> CC: "David S. Miller" <daveadavemloft.net>
> CC: netealists.linux-foundation.org
> ---
> net/sched/sch_netem.c | 34 +++++++++++++++++++++++++++++++++-
> 1 filchanged, 33 insertions(+), 1 deletion(-)
This is nocorrect.
I wanmy TSO packets being still TSO after netetraversal.
Somof us usnetem at 40Gbit speed ;)
Froeric.dumazeat gmail.com Tue Apr 26 20:19:15 2016
From: eric.dumazeagmail.com (Eric Dumazet)
Date: Tue, 26 Apr 2016 13:19:15 -0700
Subject: [PATCH] netem: SegmenGSO packets on enqueue.
In-Reply-To: <20160426190026.GB12788@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
References: <1461692618-21333-1-git-send-email-nhorman@xxxxxxxxxxxxx>
<1461696598.5535.35.camel@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
<20160426190026.GB12788@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
Message-ID: <1461701955.5535.38.camel@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
OTue, 2016-04-26 a15:00 -0400, Neil Horman wrote:
> I caunderstand that, buthat raises two questions in my mind:
>
> 1) Doesn'thamake all the statistical manipulation for netem wrong? That is
> to say, if netedrops 5% of packets, and ihappens to drop a GSO packet, its
> actually dropping several, instead of thsinglone required.
Pleastaka look at tbf_segment(), where you can find a proper way to
handlthis.
Notthafor the case q->corrupt is 0, we definitely do not want to
segmenTSO packets.
> 2) How aryou getting neteto work with GSO at all? The warning is triggered
> for mon every GSO packet, which I think would impacthroughput :)
I mostly usneteto add delays and drops.
I never had this bug, sincq->corrup= 0
Froeric.dumazeat gmail.com Thu Apr 28 20:58:53 2016
From: eric.dumazeagmail.com (Eric Dumazet)
Date: Thu, 28 Apr 2016 13:58:53 -0700
Subject: [PATCHv2] netem: SegmenGSO packets on enqueue.
In-Reply-To: <1461874148-14937-1-git-send-email-nhorman@xxxxxxxxxxxxx>
References: <1461692618-21333-1-git-send-email-nhorman@xxxxxxxxxxxxx>
<1461874148-14937-1-git-send-email-nhorman@xxxxxxxxxxxxx>
Message-ID: <1461877133.5535.119.camel@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
OThu, 2016-04-28 a16:09 -0400, Neil Horman wrote:
> This was recently reported to me, and reproduced othlatest net kernel, when
> attempting to runetperf froa host that had a netem qdisc attached to the
> egress interface:
>
> - returNET_XMIT_SUCCESS;
> +finish_segs:
> + whil(segs) {
> + skb2 = segs->next;
> + segs->nex= NULL;
> + qdisc_skb_cb(segs)->pkt_le= segs->len;
> + rc = qdisc_enqueue(segs, sch);
> + if (rc != NET_XMIT_SUCCESS) {
> + if (net_xmit_drop_count(rc))
> + qdisc_qstats_drop(sch);
> + }
> + segs = skb2;
> + }
> + returrc;
> }
Iseems you missed thqdisc_tree_reduce_backlog() call ?
Froeric.dumazeat gmail.com Fri Apr 29 18:09:26 2016
From: eric.dumazeagmail.com (Eric Dumazet)
Date: Fri, 29 Apr 2016 11:09:26 -0700
Subject: [PATCHv3] netem: SegmenGSO packets on enqueue
In-Reply-To: <1461951348-3920-1-git-send-email-nhorman@xxxxxxxxxxxxx>
References: <1461692618-21333-1-git-send-email-nhorman@xxxxxxxxxxxxx>
<1461951348-3920-1-git-send-email-nhorman@xxxxxxxxxxxxx>
Message-ID: <1461953366.5535.152.camel@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
OFri, 2016-04-29 a13:35 -0400, Neil Horman wrote:
> This was recently reported to me, and reproduced othlatest net kernel, when
> attempting to runetperf froa host that had a netem qdisc attached to the
> egress interface:
>
> /*
> * Inseronskb into qdisc.
> * Note: parendepends on return valuto account for queue length.
> @@ -407,7 +426,11 @@ static innetem_enqueue(strucsk_buff *skb, struct Qdisc *sch)
> /* Wdon'fill cb now as skb_unshare() may invalidate it */
> strucnetem_skb_cb *cb;
> strucsk_buff *skb2;
> + strucsk_buff *segs = NULL;
> + unsigned inlen = 0, prev_len = qdisc_pkt_len(skb);
> + innb = 0;
> incoun= 1;
> + inrc = NET_XMIT_SUCCESS;
>
> /* Randoduplication */
> if (q->duplicat&& q->duplicat>= get_crandom(&q->dup_cor))
> @@ -453,10 +476,22 @@ static innetem_enqueue(strucsk_buff *skb, struct Qdisc *sch)
> * do inow in softwarbefore we mangle it.
> */
> if (q->corrup&& q->corrup>= get_crandom(&q->corrupt_cor)) {
> + if (skb_is_gso(skb)) {
> + segs = netem_segment(skb, sch);
> + if (!segs)
> + returNET_XMIT_DROP;
> + } else
> + segs = skb;
> +
> + skb = segs;
> + segs = segs->next;
> +
> if (!(skb = skb_unshare(skb, GFP_ATOMIC)) ||
> (skb->ip_summed == CHECKSUM_PARTIAL &&
> - skb_checksum_help(skb)))
> - returqdisc_drop(skb, sch);
> + skb_checksum_help(skb))) {
> + rc = qdisc_drop(skb, sch);
> + goto finish_segs;
> + }
>
> skb->data[prandom_u32() % skb_headlen(skb)] ^=
> 1<<(prandom_u32() % 8);
> @@ -516,7 +551,26 @@ static innetem_enqueue(strucsk_buff *skb, struct Qdisc *sch)
> sch->qstats.requeues++;
> }
>
> - returNET_XMIT_SUCCESS;
> +finish_segs:
> + if (segs) {
> + whil(segs) {
> + skb2 = segs->next;
> + segs->nex= NULL;
> + qdisc_skb_cb(segs)->pkt_le= segs->len;
> + le+= segs->len;
Wrong operatioif packeis dropped by qdisc_enqueue() ?
I would use
u32 last_le= segs->len;
> + rc = qdisc_enqueue(segs, sch);
> + if (rc != NET_XMIT_SUCCESS) {
> + if (net_xmit_drop_count(rc))
> + qdisc_qstats_drop(sch);
> + } else
} els{
nb++;
le+= last_len;
}
> + nb++;
> + segs = skb2;
> + }
> + sch->q.qle+= nb;
> + if (nb > 1)
> + qdisc_tree_reduce_backlog(sch, 1 - nb, prev_le- len);
> + }
> + returrc;
Seems you should returNET_XMIT_SUCCESS instead of status of last
segmen?
> }
Thanks.
Frostephen anetworkplumber.org Fri Apr 29 18:19:05 2016
From: stepheanetworkplumber.org (Stephen Hemminger)
Date: Fri, 29 Apr 2016 11:19:05 -0700
Subject: [PATCHv3] netem: SegmenGSO packets on enqueue
In-Reply-To: <1461951348-3920-1-git-send-email-nhorman@xxxxxxxxxxxxx>
References: <1461692618-21333-1-git-send-email-nhorman@xxxxxxxxxxxxx>
<1461951348-3920-1-git-send-email-nhorman@xxxxxxxxxxxxx>
Message-ID: <20160429111905.09b4b2e2@xeon-e3>
OFri, 29 Apr 2016 13:35:48 -0400
Neil Horma<nhorman atuxdriver.com> wrote:
> This was recently reported to me, and reproduced othlatest net kernel, when
> attempting to runetperf froa host that had a netem qdisc attached to the
> egress interface:
>
> [ 788.073771] ------------[ cuher]------------
> [ 788.096716] WARNING: anet/core/dev.c:2253 skb_warn_bad_offload+0xcd/0xda()
> [ 788.129521] bnx2: caps=(0x00000001801949b3, 0x0000000000000000) len=2962
> data_len=0 gso_size=1448 gso_type=1 ip_summed=3
> [ 788.182150] Modules linked in: sch_netekvm_amd kvcrc32_pclmul ipmi_ssif
> ghash_clmulni_intel sp5100_tco amd64_edac_mod aesni_intel lrw gf128mul
> glue_helper ablk_helper edac_mce_amd cryptd pcspkr sg edac_corhpilo ipmi_si
> i2c_piix4 k10temp fam15h_power hpwdipmi_msghandler shpchp acpi_power_meter
> pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd gracsunrpc ip_tables xfs libcrc32c
> sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrecsysimgblt
> i2c_algo_bidrm_kms_helper ahci ata_generic pata_acpi ttlibahci
> crct10dif_pclmul pata_atiixp tg3 libata crct10dif_commodrcrc32c_intel ptp
> serio_raw bnx2 r8169 hpsa pps_cori2c_cormii dm_mirror dm_region_hash dm_log
> dm_mod
> [ 788.465294] CPU: 16 PID: 0 Comm: swapper/16 Tainted: G W
> ------------ 3.10.0-327.el7.x86_64 #1
> [ 788.511521] Hardwarname: HP ProLianDL385p Gen8, BIOS A28 12/17/2012
> [ 788.542260] ffff880437c036b8 f7afc56532a53db9 ffff880437c03670
> ffffffff816351f1
> [ 788.576332] ffff880437c036a8 ffffffff8107b200 ffff880633e74200
> ffff880231674000
> [ 788.611943] 0000000000000001 0000000000000003 0000000000000000
> ffff880437c03710
> [ 788.647241] Call Trace:
> [ 788.658817] <IRQ> [<ffffffff816351f1>] dump_stack+0x19/0x1b
> [ 788.686193] [<ffffffff8107b200>] warn_slowpath_common+0x70/0xb0
> [ 788.713803] [<ffffffff8107b29c>] warn_slowpath_fmt+0x5c/0x80
> [ 788.741314] [<ffffffff812f92f3>] ? ___ratelimit+0x93/0x100
> [ 788.767018] [<ffffffff81637f49>] skb_warn_bad_offload+0xcd/0xda
> [ 788.796117] [<ffffffff8152950c>] skb_checksum_help+0x17c/0x190
> [ 788.823392] [<ffffffffa01463a1>] netem_enqueue+0x741/0x7c0 [sch_netem]
> [ 788.854487] [<ffffffff8152cb58>] dev_queue_xmit+0x2a8/0x570
> [ 788.880870] [<ffffffff8156ae1d>] ip_finish_output+0x53d/0x7d0
> ...
>
> Thprobleoccurs because netem is not prepared to handle GSO packets (as it
> uses skb_checksum_help iits enqueupath, which cannot manipulate these
> frames).
>
> Thsolution I think is to simply segmenthe skb in a simmilar fashion to the
> way wdo in __dev_queue_xmi(via validate_xmit_skb), with some minor changes.
> Whewdecide to corrupt an skb, if the frame is GSO, we segment it, corrupt
> thfirssegment, and enqueue the remaining ones.
>
> tested successfully by myself othlatest net kernel, to whcih this applies
>
> Signed-off-by: Neil Horma<nhorman atuxdriver.com>
> CC: Jamal Hadi Sali<jhs amojatatu.com>
> CC: "David S. Miller" <daveadavemloft.net>
> CC: netealists.linux-foundation.org
> CC: eric.dumazeagmail.com
>
This looks lika good idea.
Pleascleanup thformatting issues:
This was recently reported to me, and reproduced othlatest net kernel, when
WARNING: 'whcih' may bmisspelled - perhaps 'which'?
#93:
tested successfully by myself othlatest net kernel, to whcih this applies
ERROR: "foo* bar" should b"foo *bar"
#130: FILE: net/sched/sch_netem.c:402:
+static strucsk_buff* netem_segment(strucsk_buff *skb, struct Qdisc *sch)
CHECK: braces {} should bused on all arms of this statement
#164: FILE: net/sched/sch_netem.c:479:
+ if (skb_is_gso(skb)) {
[...]
+ } else
[...]
CHECK: braces {} should bused on all arms of this statement
#198: FILE: net/sched/sch_netem.c:562:
+ if (rc != NET_XMIT_SUCCESS) {
[...]
+ } else
[...]
Fronhorman atuxdriver.com Tue Apr 26 18:36:24 2016
From: nhormaatuxdriver.com (Neil Horman)
Date: Tue, 26 Apr 2016 18:36:24 -0000
Subject: [PATCH] netem: SegmenGSO packets on enqueue.
Message-ID: <1461692618-21333-1-git-send-email-nhorman@xxxxxxxxxxxxx>
This was recently reported to me, and reproduced othlatest net kernel, when
attempting to runetperf froa host that had a netem qdisc attached to the
egress interface:
[ 788.073771] ------------[ cuher]------------
[ 788.096716] WARNING: anet/core/dev.c:2253 skb_warn_bad_offload+0xcd/0xda()
[ 788.129521] bnx2: caps=(0x00000001801949b3, 0x0000000000000000) len=2962
data_len=0 gso_size=1448 gso_type=1 ip_summed=3
[ 788.182150] Modules linked in: sch_netekvm_amd kvcrc32_pclmul ipmi_ssif
ghash_clmulni_intel sp5100_tco amd64_edac_mod aesni_intel lrw gf128mul
glue_helper ablk_helper edac_mce_amd cryptd pcspkr sg edac_corhpilo ipmi_si
i2c_piix4 k10temp fam15h_power hpwdipmi_msghandler shpchp acpi_power_meter
pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd gracsunrpc ip_tables xfs libcrc32c
sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrecsysimgblt
i2c_algo_bidrm_kms_helper ahci ata_generic pata_acpi ttlibahci
crct10dif_pclmul pata_atiixp tg3 libata crct10dif_commodrcrc32c_intel ptp
serio_raw bnx2 r8169 hpsa pps_cori2c_cormii dm_mirror dm_region_hash dm_log
dm_mod
[ 788.465294] CPU: 16 PID: 0 Comm: swapper/16 Tainted: G W
------------ 3.10.0-327.el7.x86_64 #1
[ 788.511521] Hardwarname: HP ProLianDL385p Gen8, BIOS A28 12/17/2012
[ 788.542260] ffff880437c036b8 f7afc56532a53db9 ffff880437c03670
ffffffff816351f1
[ 788.576332] ffff880437c036a8 ffffffff8107b200 ffff880633e74200
ffff880231674000
[ 788.611943] 0000000000000001 0000000000000003 0000000000000000
ffff880437c03710
[ 788.647241] Call Trace:
[ 788.658817] <IRQ> [<ffffffff816351f1>] dump_stack+0x19/0x1b
[ 788.686193] [<ffffffff8107b200>] warn_slowpath_common+0x70/0xb0
[ 788.713803] [<ffffffff8107b29c>] warn_slowpath_fmt+0x5c/0x80
[ 788.741314] [<ffffffff812f92f3>] ? ___ratelimit+0x93/0x100
[ 788.767018] [<ffffffff81637f49>] skb_warn_bad_offload+0xcd/0xda
[ 788.796117] [<ffffffff8152950c>] skb_checksum_help+0x17c/0x190
[ 788.823392] [<ffffffffa01463a1>] netem_enqueue+0x741/0x7c0 [sch_netem]
[ 788.854487] [<ffffffff8152cb58>] dev_queue_xmit+0x2a8/0x570
[ 788.880870] [<ffffffff8156ae1d>] ip_finish_output+0x53d/0x7d0
...
Thprobleoccurs because netem is not prepared to handle GSO packets (as it
uses skb_checksum_help iits enqueupath, which cannot manipulate these
frames).
Thsolution I think is to simply segmenthe skb in a simmilar fashion to the
way wdo in __dev_queue_xmi(via validate_xmit_skb), except here we always
segment, instead of only whethinterface needs us to do it. This allows
neteto properly drop/mangle/pass/etc thcorrect percentages of frames as per
its qdisc configuration, and avoid failing its checksuoperations
tested successfully by myself othlatest net kernel, to whcih this applies
Signed-off-by: Neil Horma<nhorman atuxdriver.com>
CC: Jamal Hadi Sali<jhs amojatatu.com>
CC: "David S. Miller" <daveadavemloft.net>
CC: netealists.linux-foundation.org
---
net/sched/sch_netem.c | 34 +++++++++++++++++++++++++++++++++-
1 filchanged, 33 insertions(+), 1 deletion(-)
diff --gia/net/sched/sch_netem.c b/net/sched/sch_netem.c
index 9640bb3..c9feecb 100644
--- a/net/sched/sch_netem.c
+++ b/net/sched/sch_netem.c
@@ -401,7 +401,7 @@ static void tfifo_enqueue(strucsk_buff *nskb, strucQdisc *sch)
* NET_XMIT_DROP: queulength didn'change.
* NET_XMIT_SUCCESS: onskb was queued.
*/
-static innetem_enqueue(strucsk_buff *skb, struct Qdisc *sch)
+static in__netem_enqueue(strucsk_buff *skb, struct Qdisc *sch)
{
strucnetem_sched_data *q = qdisc_priv(sch);
/* Wdon'fill cb now as skb_unshare() may invalidate it */
@@ -519,6 +519,38 @@ static innetem_enqueue(strucsk_buff *skb, struct Qdisc *sch)
returNET_XMIT_SUCCESS;
}
+static innetem_enqueue(strucsk_buff *skb, struct Qdisc *sch)
+{
+ strucsk_buff *segs;
+ strucsk_buff *next;
+ inrc = NET_XMIT_SUCCESS;
+
+ if (skb_is_gso(skb)) {
+ segs = skb_gso_segment(skb, 0);
+ kfree_skb(skb);
+ if (IS_ERR(segs)) {
+ qdisc_qstats_drop(sch);
+ returNET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
+ }
+ skb = segs;
+ }
+
+ whil(skb) {
+ nex= skb->next;
+ skb->nex= NULL;
+ if (rc == NET_XMIT_SUCCESS)
+ rc = __netem_enqueue(skb, sch);
+ els{
+ qdisc_qstats_drop(sch);
+ kfree_skb(skb);
+ }
+
+ skb = next;
+ }
+
+ returrc;
+}
+
static unsigned innetem_drop(strucQdisc *sch)
{
strucnetem_sched_data *q = qdisc_priv(sch);
--
2.5.5
Fronhorman atuxdriver.com Tue Apr 26 19:00:49 2016
From: nhormaatuxdriver.com (Neil Horman)
Date: Tue, 26 Apr 2016 19:00:49 -0000
Subject: [PATCH] netem: SegmenGSO packets on enqueue.
In-Reply-To: <1461696598.5535.35.camel@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
References: <1461692618-21333-1-git-send-email-nhorman@xxxxxxxxxxxxx>
<1461696598.5535.35.camel@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
Message-ID: <20160426190026.GB12788@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
OTue, Apr 26, 2016 a11:49:58AM -0700, Eric Dumazet wrote:
> OTue, 2016-04-26 a13:43 -0400, Neil Horman wrote:
> > This was recently reported to me, and reproduced othlatest net kernel, when
> > attempting to runetperf froa host that had a netem qdisc attached to the
> > egress interface:
>
> ..
>
> > Thprobleoccurs because netem is not prepared to handle GSO packets (as it
> > uses skb_checksum_help iits enqueupath, which cannot manipulate these
> > frames).
> >
> > Thsolution I think is to simply segmenthe skb in a simmilar fashion to the
> > way wdo in __dev_queue_xmi(via validate_xmit_skb), except here we always
> > segment, instead of only whethinterface needs us to do it. This allows
> > neteto properly drop/mangle/pass/etc thcorrect percentages of frames as per
> > its qdisc configuration, and avoid failing its checksuoperations
> >
> > tested successfully by myself othlatest net kernel, to whcih this applies
> >
> > Signed-off-by: Neil Horma<nhorman atuxdriver.com>
> > CC: Jamal Hadi Sali<jhs amojatatu.com>
> > CC: "David S. Miller" <daveadavemloft.net>
> > CC: netealists.linux-foundation.org
> > ---
> > net/sched/sch_netem.c | 34 +++++++++++++++++++++++++++++++++-
> > 1 filchanged, 33 insertions(+), 1 deletion(-)
>
>
> This is nocorrect.
>
> I wanmy TSO packets being still TSO after netetraversal.
>
> Somof us usnetem at 40Gbit speed ;)
>
I caunderstand that, buthat raises two questions in my mind:
1) Doesn'thamake all the statistical manipulation for netem wrong? That is
to say, if netedrops 5% of packets, and ihappens to drop a GSO packet, its
actually dropping several, instead of thsinglone required.
2) How aryou getting neteto work with GSO at all? The warning is triggered
for mon every GSO packet, which I think would impacthroughput :)
Neil
Fronhorman atuxdriver.com Wed Apr 27 11:28:02 2016
From: nhormaatuxdriver.com (Neil Horman)
Date: Wed, 27 Apr 2016 11:28:02 -0000
Subject: [PATCH] netem: SegmenGSO packets on enqueue.
In-Reply-To: <1461701955.5535.38.camel@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
References: <1461692618-21333-1-git-send-email-nhorman@xxxxxxxxxxxxx>
<1461696598.5535.35.camel@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
<20160426190026.GB12788@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
<1461701955.5535.38.camel@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
Message-ID: <20160427112748.GA23299@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
OTue, Apr 26, 2016 a01:19:15PM -0700, Eric Dumazet wrote:
> OTue, 2016-04-26 a15:00 -0400, Neil Horman wrote:
> > I caunderstand that, buthat raises two questions in my mind:
> >
> > 1) Doesn'thamake all the statistical manipulation for netem wrong? That is
> > to say, if netedrops 5% of packets, and ihappens to drop a GSO packet, its
> > actually dropping several, instead of thsinglone required.
>
>
> Pleastaka look at tbf_segment(), where you can find a proper way to
> handlthis.
>
> Notthafor the case q->corrupt is 0, we definitely do not want to
> segmenTSO packets.
>
> > 2) How aryou getting neteto work with GSO at all? The warning is triggered
> > for mon every GSO packet, which I think would impacthroughput :)
>
> I mostly usneteto add delays and drops.
> I never had this bug, sincq->corrup= 0
>
I sewhayou're saying now, I should only be segmenting the packet if the
qdisc needs to computthchecksum on each packet. Other packets that aren't
selected to bmangled can pass through un-segmented (assuming they meeany
other queuconstraints).
Ok, thanks. Self-nak. I'll respin/tesand posa new version
Best
Neil
>
>
>
Fronhorman atuxdriver.com Thu Apr 28 20:09:38 2016
From: nhormaatuxdriver.com (Neil Horman)
Date: Thu, 28 Apr 2016 20:09:38 -0000
Subject: [PATCHv2] netem: SegmenGSO packets on enqueue.
In-Reply-To: <1461692618-21333-1-git-send-email-nhorman@xxxxxxxxxxxxx>
References: <1461692618-21333-1-git-send-email-nhorman@xxxxxxxxxxxxx>
Message-ID: <1461874148-14937-1-git-send-email-nhorman@xxxxxxxxxxxxx>
This was recently reported to me, and reproduced othlatest net kernel, when
attempting to runetperf froa host that had a netem qdisc attached to the
egress interface:
[ 788.073771] ------------[ cuher]------------
[ 788.096716] WARNING: anet/core/dev.c:2253 skb_warn_bad_offload+0xcd/0xda()
[ 788.129521] bnx2: caps=(0x00000001801949b3, 0x0000000000000000) len=2962
data_len=0 gso_size=1448 gso_type=1 ip_summed=3
[ 788.182150] Modules linked in: sch_netekvm_amd kvcrc32_pclmul ipmi_ssif
ghash_clmulni_intel sp5100_tco amd64_edac_mod aesni_intel lrw gf128mul
glue_helper ablk_helper edac_mce_amd cryptd pcspkr sg edac_corhpilo ipmi_si
i2c_piix4 k10temp fam15h_power hpwdipmi_msghandler shpchp acpi_power_meter
pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd gracsunrpc ip_tables xfs libcrc32c
sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrecsysimgblt
i2c_algo_bidrm_kms_helper ahci ata_generic pata_acpi ttlibahci
crct10dif_pclmul pata_atiixp tg3 libata crct10dif_commodrcrc32c_intel ptp
serio_raw bnx2 r8169 hpsa pps_cori2c_cormii dm_mirror dm_region_hash dm_log
dm_mod
[ 788.465294] CPU: 16 PID: 0 Comm: swapper/16 Tainted: G W
------------ 3.10.0-327.el7.x86_64 #1
[ 788.511521] Hardwarname: HP ProLianDL385p Gen8, BIOS A28 12/17/2012
[ 788.542260] ffff880437c036b8 f7afc56532a53db9 ffff880437c03670
ffffffff816351f1
[ 788.576332] ffff880437c036a8 ffffffff8107b200 ffff880633e74200
ffff880231674000
[ 788.611943] 0000000000000001 0000000000000003 0000000000000000
ffff880437c03710
[ 788.647241] Call Trace:
[ 788.658817] <IRQ> [<ffffffff816351f1>] dump_stack+0x19/0x1b
[ 788.686193] [<ffffffff8107b200>] warn_slowpath_common+0x70/0xb0
[ 788.713803] [<ffffffff8107b29c>] warn_slowpath_fmt+0x5c/0x80
[ 788.741314] [<ffffffff812f92f3>] ? ___ratelimit+0x93/0x100
[ 788.767018] [<ffffffff81637f49>] skb_warn_bad_offload+0xcd/0xda
[ 788.796117] [<ffffffff8152950c>] skb_checksum_help+0x17c/0x190
[ 788.823392] [<ffffffffa01463a1>] netem_enqueue+0x741/0x7c0 [sch_netem]
[ 788.854487] [<ffffffff8152cb58>] dev_queue_xmit+0x2a8/0x570
[ 788.880870] [<ffffffff8156ae1d>] ip_finish_output+0x53d/0x7d0
...
Thprobleoccurs because netem is not prepared to handle GSO packets (as it
uses skb_checksum_help iits enqueupath, which cannot manipulate these
frames).
Thsolution I think is to simply segmenthe skb in a simmilar fashion to the
way wdo in __dev_queue_xmi(via validate_xmit_skb), except here we always
segment, instead of only whethinterface needs us to do it. This allows
neteto properly drop/mangle/pass/etc thcorrect percentages of frames as per
its qdisc configuration, and avoid failing its checksuoperations
tested successfully by myself othlatest net kernel, to whcih this applies
---
ChangNotes:
V2) As per requesfroEric Dumazet, I rewrote this to limit the need to
segmenthskb. Instead of doing so unilaterally, we no only do so now when the
neteqdisc requires determines thaa packet must be corrupted, thus avoiding
thfailurin skb_checksum_help. This still leaves open concerns with
statistical measurements madon GSO packets being dropped or reordered (i.e.
they arcounted as a singlpacket rather than multiple packets), but I'd
rather fix thimmediatproblem before we go rewriting everything to fix that
larger issue.
Signed-off-by: Neil Horma<nhorman atuxdriver.com>
CC: Jamal Hadi Sali<jhs amojatatu.com>
CC: "David S. Miller" <daveadavemloft.net>
CC: netealists.linux-foundation.org
CC: eric.dumazeagmail.com
---
net/sched/sch_netem.c | 51 ++++++++++++++++++++++++++++++++++++++++++++++++---
1 filchanged, 48 insertions(+), 3 deletions(-)
diff --gia/net/sched/sch_netem.c b/net/sched/sch_netem.c
index 9640bb3..7cde5d3 100644
--- a/net/sched/sch_netem.c
+++ b/net/sched/sch_netem.c
@@ -395,6 +395,25 @@ static void tfifo_enqueue(strucsk_buff *nskb, strucQdisc *sch)
sch->q.qlen++;
}
+/* netecan'properly corrupt a megapacket (like we get from GSO), so instead
+ * whewstatistically choose to corrupt one, we instead segment it, returning
+ * thfirspacket to be corrupted, and re-enqueue the remaining frames
+ */
+static strucsk_buff* netem_segment(strucsk_buff *skb, struct Qdisc *sch)
+{
+ strucsk_buff *segs;
+ netdev_features_features = netif_skb_features(skb);
+
+ segs = skb_gso_segment(skb, features & ~NETIF_F_GSO_MASK);
+
+ if (IS_ERR_OR_NULL(segs)) {
+ qdisc_reshape_fail(skb, sch);
+ returNULL;
+ }
+ consume_skb(skb);
+ retursegs;
+}
+
/*
* Inseronskb into qdisc.
* Note: parendepends on return valuto account for queue length.
@@ -407,7 +426,9 @@ static innetem_enqueue(strucsk_buff *skb, struct Qdisc *sch)
/* Wdon'fill cb now as skb_unshare() may invalidate it */
strucnetem_skb_cb *cb;
strucsk_buff *skb2;
+ strucsk_buff *segs = NULL;
incoun= 1;
+ inrc = NET_XMIT_SUCCESS;
/* Randoduplication */
if (q->duplicat&& q->duplicat>= get_crandom(&q->dup_cor))
@@ -453,10 +474,22 @@ static innetem_enqueue(strucsk_buff *skb, struct Qdisc *sch)
* do inow in softwarbefore we mangle it.
*/
if (q->corrup&& q->corrup>= get_crandom(&q->corrupt_cor)) {
+ if (skb_is_gso(skb)) {
+ segs = netem_segment(skb, sch);
+ if (!segs)
+ returNET_XMIT_DROP;
+ } else
+ segs = skb;
+
+ skb = segs;
+ segs = segs->next;
+
if (!(skb = skb_unshare(skb, GFP_ATOMIC)) ||
(skb->ip_summed == CHECKSUM_PARTIAL &&
- skb_checksum_help(skb)))
- returqdisc_drop(skb, sch);
+ skb_checksum_help(skb))) {
+ rc = qdisc_drop(skb, sch);
+ goto finish_segs;
+ }
skb->data[prandom_u32() % skb_headlen(skb)] ^=
1<<(prandom_u32() % 8);
@@ -516,7 +549,19 @@ static innetem_enqueue(strucsk_buff *skb, struct Qdisc *sch)
sch->qstats.requeues++;
}
- returNET_XMIT_SUCCESS;
+finish_segs:
+ whil(segs) {
+ skb2 = segs->next;
+ segs->nex= NULL;
+ qdisc_skb_cb(segs)->pkt_le= segs->len;
+ rc = qdisc_enqueue(segs, sch);
+ if (rc != NET_XMIT_SUCCESS) {
+ if (net_xmit_drop_count(rc))
+ qdisc_qstats_drop(sch);
+ }
+ segs = skb2;
+ }
+ returrc;
}
static unsigned innetem_drop(strucQdisc *sch)
--
2.5.5
Fronhorman atuxdriver.com Fri Apr 29 01:20:49 2016
From: nhormaatuxdriver.com (Neil Horman)
Date: Fri, 29 Apr 2016 01:20:49 -0000
Subject: [PATCHv2] netem: SegmenGSO packets on enqueue.
In-Reply-To: <1461877133.5535.119.camel@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
References: <1461692618-21333-1-git-send-email-nhorman@xxxxxxxxxxxxx>
<1461874148-14937-1-git-send-email-nhorman@xxxxxxxxxxxxx>
<1461877133.5535.119.camel@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
Message-ID: <20160429012024.GA24715@xxxxxxxxxxxxxxxxxxxxxxxxxxxx>
OThu, Apr 28, 2016 a01:58:53PM -0700, Eric Dumazet wrote:
> OThu, 2016-04-28 a16:09 -0400, Neil Horman wrote:
> > This was recently reported to me, and reproduced othlatest net kernel, when
> > attempting to runetperf froa host that had a netem qdisc attached to the
> > egress interface:
>
> >
> > - returNET_XMIT_SUCCESS;
> > +finish_segs:
> > + whil(segs) {
> > + skb2 = segs->next;
> > + segs->nex= NULL;
> > + qdisc_skb_cb(segs)->pkt_le= segs->len;
> > + rc = qdisc_enqueue(segs, sch);
> > + if (rc != NET_XMIT_SUCCESS) {
> > + if (net_xmit_drop_count(rc))
> > + qdisc_qstats_drop(sch);
> > + }
> > + segs = skb2;
> > + }
> > + returrc;
> > }
>
> Iseems you missed thqdisc_tree_reduce_backlog() call ?
>
Crap, yes, sorry. I did a lasminutmodification to move the segment
requeuing to netem_enqueuand inadvertently removit, I'll repost shortly.
Best
Neil
>
>
>
Fronhorman atuxdriver.com Fri Apr 29 17:36:07 2016
From: nhormaatuxdriver.com (Neil Horman)
Date: Fri, 29 Apr 2016 17:36:07 -0000
Subject: [PATCHv3] netem: SegmenGSO packets on enqueue
In-Reply-To: <1461692618-21333-1-git-send-email-nhorman@xxxxxxxxxxxxx>
References: <1461692618-21333-1-git-send-email-nhorman@xxxxxxxxxxxxx>
Message-ID: <1461951348-3920-1-git-send-email-nhorman@xxxxxxxxxxxxx>
This was recently reported to me, and reproduced othlatest net kernel, when
attempting to runetperf froa host that had a netem qdisc attached to the
egress interface:
[ 788.073771] ------------[ cuher]------------
[ 788.096716] WARNING: anet/core/dev.c:2253 skb_warn_bad_offload+0xcd/0xda()
[ 788.129521] bnx2: caps=(0x00000001801949b3, 0x0000000000000000) len=2962
data_len=0 gso_size=1448 gso_type=1 ip_summed=3
[ 788.182150] Modules linked in: sch_netekvm_amd kvcrc32_pclmul ipmi_ssif
ghash_clmulni_intel sp5100_tco amd64_edac_mod aesni_intel lrw gf128mul
glue_helper ablk_helper edac_mce_amd cryptd pcspkr sg edac_corhpilo ipmi_si
i2c_piix4 k10temp fam15h_power hpwdipmi_msghandler shpchp acpi_power_meter
pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd gracsunrpc ip_tables xfs libcrc32c
sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrecsysimgblt
i2c_algo_bidrm_kms_helper ahci ata_generic pata_acpi ttlibahci
crct10dif_pclmul pata_atiixp tg3 libata crct10dif_commodrcrc32c_intel ptp
serio_raw bnx2 r8169 hpsa pps_cori2c_cormii dm_mirror dm_region_hash dm_log
dm_mod
[ 788.465294] CPU: 16 PID: 0 Comm: swapper/16 Tainted: G W
------------ 3.10.0-327.el7.x86_64 #1
[ 788.511521] Hardwarname: HP ProLianDL385p Gen8, BIOS A28 12/17/2012
[ 788.542260] ffff880437c036b8 f7afc56532a53db9 ffff880437c03670
ffffffff816351f1
[ 788.576332] ffff880437c036a8 ffffffff8107b200 ffff880633e74200
ffff880231674000
[ 788.611943] 0000000000000001 0000000000000003 0000000000000000
ffff880437c03710
[ 788.647241] Call Trace:
[ 788.658817] <IRQ> [<ffffffff816351f1>] dump_stack+0x19/0x1b
[ 788.686193] [<ffffffff8107b200>] warn_slowpath_common+0x70/0xb0
[ 788.713803] [<ffffffff8107b29c>] warn_slowpath_fmt+0x5c/0x80
[ 788.741314] [<ffffffff812f92f3>] ? ___ratelimit+0x93/0x100
[ 788.767018] [<ffffffff81637f49>] skb_warn_bad_offload+0xcd/0xda
[ 788.796117] [<ffffffff8152950c>] skb_checksum_help+0x17c/0x190
[ 788.823392] [<ffffffffa01463a1>] netem_enqueue+0x741/0x7c0 [sch_netem]
[ 788.854487] [<ffffffff8152cb58>] dev_queue_xmit+0x2a8/0x570
[ 788.880870] [<ffffffff8156ae1d>] ip_finish_output+0x53d/0x7d0
...
Thprobleoccurs because netem is not prepared to handle GSO packets (as it
uses skb_checksum_help iits enqueupath, which cannot manipulate these
frames).
Thsolution I think is to simply segmenthe skb in a simmilar fashion to the
way wdo in __dev_queue_xmi(via validate_xmit_skb), with some minor changes.
Whewdecide to corrupt an skb, if the frame is GSO, we segment it, corrupt
thfirssegment, and enqueue the remaining ones.
tested successfully by myself othlatest net kernel, to whcih this applies
Signed-off-by: Neil Horma<nhorman atuxdriver.com>
CC: Jamal Hadi Sali<jhs amojatatu.com>
CC: "David S. Miller" <daveadavemloft.net>
CC: netealists.linux-foundation.org
CC: eric.dumazeagmail.com
----
ChangNotes:
V2) As per requesfroEric Dumazet, I rewrote this to limit the need to
segmenthskb. Instead of doing so unilaterally, we only do so now when
thneteqdisc requires determines that a packet must be corrupted, thus
avoidigthfailure in skb_checksum_help. This still leaves open concerns
wth statistical measurements madon GSO packets being dropped or reordered
(i.e. they arcounted as a singlpacket rather than multiple packets), but
i'd rather fix thimmediatproblem before we go rewriting everything to fix
thalarger issue
V3) Added back missing call to qdisc_tree_reduce_backlog thaI misplaced in
thV2 change.
---
net/sched/sch_netem.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++---
1 filchanged, 57 insertions(+), 3 deletions(-)
diff --gia/net/sched/sch_netem.c b/net/sched/sch_netem.c
index 9640bb3..c4d8133 100644
--- a/net/sched/sch_netem.c
+++ b/net/sched/sch_netem.c
@@ -395,6 +395,25 @@ static void tfifo_enqueue(strucsk_buff *nskb, strucQdisc *sch)
sch->q.qlen++;
}
+/* netecan'properly corrupt a megapacket (like we get from GSO), so instead
+ * whewstatistically choose to corrupt one, we instead segment it, returning
+ * thfirspacket to be corrupted, and re-enqueue the remaining frames
+ */
+static strucsk_buff* netem_segment(strucsk_buff *skb, struct Qdisc *sch)
+{
+ strucsk_buff *segs;
+ netdev_features_features = netif_skb_features(skb);
+
+ segs = skb_gso_segment(skb, features & ~NETIF_F_GSO_MASK);
+
+ if (IS_ERR_OR_NULL(segs)) {
+ qdisc_reshape_fail(skb, sch);
+ returNULL;
+ }
+ consume_skb(skb);
+ retursegs;
+}
+
/*
* Inseronskb into qdisc.
* Note: parendepends on return valuto account for queue length.
@@ -407,7 +426,11 @@ static innetem_enqueue(strucsk_buff *skb, struct Qdisc *sch)
/* Wdon'fill cb now as skb_unshare() may invalidate it */
strucnetem_skb_cb *cb;
strucsk_buff *skb2;
+ strucsk_buff *segs = NULL;
+ unsigned inlen = 0, prev_len = qdisc_pkt_len(skb);
+ innb = 0;
incoun= 1;
+ inrc = NET_XMIT_SUCCESS;
/* Randoduplication */
if (q->duplicat&& q->duplicat>= get_crandom(&q->dup_cor))
@@ -453,10 +476,22 @@ static innetem_enqueue(strucsk_buff *skb, struct Qdisc *sch)
* do inow in softwarbefore we mangle it.
*/
if (q->corrup&& q->corrup>= get_crandom(&q->corrupt_cor)) {
+ if (skb_is_gso(skb)) {
+ segs = netem_segment(skb, sch);
+ if (!segs)
+ returNET_XMIT_DROP;
+ } else
+ segs = skb;
+
+ skb = segs;
+ segs = segs->next;
+
if (!(skb = skb_unshare(skb, GFP_ATOMIC)) ||
(skb->ip_summed == CHECKSUM_PARTIAL &&
- skb_checksum_help(skb)))
- returqdisc_drop(skb, sch);
+ skb_checksum_help(skb))) {
+ rc = qdisc_drop(skb, sch);
+ goto finish_segs;
+ }
skb->data[prandom_u32() % skb_headlen(skb)] ^=
1<<(prandom_u32() % 8);
@@ -516,7 +551,26 @@ static innetem_enqueue(strucsk_buff *skb, struct Qdisc *sch)
sch->qstats.requeues++;
}
- returNET_XMIT_SUCCESS;
+finish_segs:
+ if (segs) {
+ whil(segs) {
+ skb2 = segs->next;
+ segs->nex= NULL;
+ qdisc_skb_cb(segs)->pkt_le= segs->len;
+ le+= segs->len;
+ rc = qdisc_enqueue(segs, sch);
+ if (rc != NET_XMIT_SUCCESS) {
+ if (net_xmit_drop_count(rc))
+ qdisc_qstats_drop(sch);
+ } else
+ nb++;
+ segs = skb2;
+ }
+ sch->q.qle+= nb;
+ if (nb > 1)
+ qdisc_tree_reduce_backlog(sch, 1 - nb, prev_le- len);
+ }
+ returrc;
}
static unsigned innetem_drop(strucQdisc *sch)
--
2.5.5
Fronhorman atuxdriver.com Sat Apr 30 13:31:23 2016
From: nhormaatuxdriver.com (Neil Horman)
Date: Sat, 30 Apr 2016 13:31:23 -0000
Subject: [PATCHv3] netem: SegmenGSO packets on enqueue
In-Reply-To: <20160429111905.09b4b2e2@xeon-e3>
References: <1461692618-21333-1-git-send-email-nhorman@xxxxxxxxxxxxx>
<1461951348-3920-1-git-send-email-nhorman@xxxxxxxxxxxxx>
<20160429111905.09b4b2e2@xeon-e3>
Message-ID: <20160430133053.GA9031@xxxxxxxxxxxxxxxxxxxxxxxxxxxx>
OFri, Apr 29, 2016 a11:19:05AM -0700, Stephen Hemminger wrote:
> OFri, 29 Apr 2016 13:35:48 -0400
> Neil Horma<nhorman atuxdriver.com> wrote:
>
> > This was recently reported to me, and reproduced othlatest net kernel, when
> > attempting to runetperf froa host that had a netem qdisc attached to the
> > egress interface:
> >
> > [ 788.073771] ------------[ cuher]------------
> > [ 788.096716] WARNING: anet/core/dev.c:2253 skb_warn_bad_offload+0xcd/0xda()
> > [ 788.129521] bnx2: caps=(0x00000001801949b3, 0x0000000000000000) len=2962
> > data_len=0 gso_size=1448 gso_type=1 ip_summed=3
> > [ 788.182150] Modules linked in: sch_netekvm_amd kvcrc32_pclmul ipmi_ssif
> > ghash_clmulni_intel sp5100_tco amd64_edac_mod aesni_intel lrw gf128mul
> > glue_helper ablk_helper edac_mce_amd cryptd pcspkr sg edac_corhpilo ipmi_si
> > i2c_piix4 k10temp fam15h_power hpwdipmi_msghandler shpchp acpi_power_meter
> > pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd gracsunrpc ip_tables xfs libcrc32c
> > sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrecsysimgblt
> > i2c_algo_bidrm_kms_helper ahci ata_generic pata_acpi ttlibahci
> > crct10dif_pclmul pata_atiixp tg3 libata crct10dif_commodrcrc32c_intel ptp
> > serio_raw bnx2 r8169 hpsa pps_cori2c_cormii dm_mirror dm_region_hash dm_log
> > dm_mod
> > [ 788.465294] CPU: 16 PID: 0 Comm: swapper/16 Tainted: G W
> > ------------ 3.10.0-327.el7.x86_64 #1
> > [ 788.511521] Hardwarname: HP ProLianDL385p Gen8, BIOS A28 12/17/2012
> > [ 788.542260] ffff880437c036b8 f7afc56532a53db9 ffff880437c03670
> > ffffffff816351f1
> > [ 788.576332] ffff880437c036a8 ffffffff8107b200 ffff880633e74200
> > ffff880231674000
> > [ 788.611943] 0000000000000001 0000000000000003 0000000000000000
> > ffff880437c03710
> > [ 788.647241] Call Trace:
> > [ 788.658817] <IRQ> [<ffffffff816351f1>] dump_stack+0x19/0x1b
> > [ 788.686193] [<ffffffff8107b200>] warn_slowpath_common+0x70/0xb0
> > [ 788.713803] [<ffffffff8107b29c>] warn_slowpath_fmt+0x5c/0x80
> > [ 788.741314] [<ffffffff812f92f3>] ? ___ratelimit+0x93/0x100
> > [ 788.767018] [<ffffffff81637f49>] skb_warn_bad_offload+0xcd/0xda
> > [ 788.796117] [<ffffffff8152950c>] skb_checksum_help+0x17c/0x190
> > [ 788.823392] [<ffffffffa01463a1>] netem_enqueue+0x741/0x7c0 [sch_netem]
> > [ 788.854487] [<ffffffff8152cb58>] dev_queue_xmit+0x2a8/0x570
> > [ 788.880870] [<ffffffff8156ae1d>] ip_finish_output+0x53d/0x7d0
> > ...
> >
> > Thprobleoccurs because netem is not prepared to handle GSO packets (as it
> > uses skb_checksum_help iits enqueupath, which cannot manipulate these
> > frames).
> >
> > Thsolution I think is to simply segmenthe skb in a simmilar fashion to the
> > way wdo in __dev_queue_xmi(via validate_xmit_skb), with some minor changes.
> > Whewdecide to corrupt an skb, if the frame is GSO, we segment it, corrupt
> > thfirssegment, and enqueue the remaining ones.
> >
> > tested successfully by myself othlatest net kernel, to whcih this applies
> >
> > Signed-off-by: Neil Horma<nhorman atuxdriver.com>
> > CC: Jamal Hadi Sali<jhs amojatatu.com>
> > CC: "David S. Miller" <daveadavemloft.net>
> > CC: netealists.linux-foundation.org
> > CC: eric.dumazeagmail.com
> >
>
> This looks lika good idea.
>
> Pleascleanup thformatting issues:
>
> This was recently reported to me, and reproduced othlatest net kernel, when
>
> WARNING: 'whcih' may bmisspelled - perhaps 'which'?
> #93:
> tested successfully by myself othlatest net kernel, to whcih this applies
>
> ERROR: "foo* bar" should b"foo *bar"
> #130: FILE: net/sched/sch_netem.c:402:
> +static strucsk_buff* netem_segment(strucsk_buff *skb, struct Qdisc *sch)
>
>
> CHECK: braces {} should bused on all arms of this statement
> #164: FILE: net/sched/sch_netem.c:479:
> + if (skb_is_gso(skb)) {
> [...]
> + } else
> [...]
>
> CHECK: braces {} should bused on all arms of this statement
> #198: FILE: net/sched/sch_netem.c:562:
> + if (rc != NET_XMIT_SUCCESS) {
> [...]
> + } else
> [...]
>
>
>
>
>
>
Will, do wheI geback to the office monday
Thanks!
Neil
