*Coder Blog

Life, Technology, and Meteorology

Fixing BufferBloat

For years I’ve had issues with with my internet connection crapping out whenever I’m uploading data. The connection will be fine when browsing the web and downloading data at the rated speed of 60 mbps. However, whenever I tried to upload a large file, it would saturate my upload link and slow every other network connection to a crawl. Websites wouldn’t load, DNS wouldn’t resolve, and ping times would be measured in seconds.

My theory for a long time was that the upload bandwidth was becoming so saturated, that TCP ACKs for incoming data wouldn’t get sent out in a reasonable amount of time. So for a long time I was looking for a way to prioritize TCP SYN/ACK packets. However, I never ended up figuring out how to do this.

A few nights ago while looking for a different solution, I stumbled across the idea of BufferBloat causing issues when saturating an internet link. Apparently, modern networking equipment has a lot more connection buffer memory than older equipment. This seems like a good thing, with memory prices being so cheap it makes sense to include a large buffer to help keep the network link saturated. The increased buffer could be in your router, cable modem, or any number of components at your ISP. Unfortunately, this can cause problems when your link is slow enough to fill the buffer quickly.

When TCP connections are created, the networking hardware starts spooling up the connection speed by gauging the TCP ACK packet response times. The faster the ACK packets arrive, the faster the network connection appears to be to the network interface, so the interface tries to send data even faster until the link is saturated.

The problem is that networking equipment between the network interface on your computer and your destination may be buffering a lot of network packets. In my case, I have a 5 mbps upload link, so either my modem or router is buffering enough data while I’m uploading a large file that TCP ACK packets are taking several seconds to arrive back. During that time, the packets are just sitting in the buffer waiting to be sent. Once the bandwidth to send the packets is available, they transmit relatively quickly, but from the standpoint of my computer the response time is very slow. This kills the connection.

The fix is to limit the amount of outgoing bandwidth on your router using QoS. What you want to do is to limit your bandwidth to about 0.5 – 1 mbps less than your connection can handle. On the ZyXel ZyWALL 110, this is done through the BWM (bandwidth management) screen in the configuration. First, enable BWM with the checkbox at the top of the page. Then add a new rule:

Configuration
Enable: checked
Description: Outgoing Bandwidth Limit
BWM Type: Shared

Criteria
User: any
Schedule: none
Incoming Interface: lan1
Outgoing Interface: wan1
Source: any
Destination: any
DSCP Code: any
Service Type: Service Object
Service Object: any

DSCP Marking
Inbound Marking: preserve
Outbound Marking: preserve

Bandwidth Shaping
Inbound: 0 kbps (disabled), Priority 4, Maximum 0 kbps (disabled)
Outbound: 4000 kbps, Priority 4, Maximum 4000 kbps

802.1P Marking
Priority Code: 0
Interface: none

Log: no

The key above is in the Bandwidth Shaping section. Set your outbound guaranteed bandwidth and bandwidth limit to 0.5 – 1 mbps below your maximum upload speed. Here I set mine to 4000 kbps, which is a megabit less than my internet upload speed of 5 mbps.

Once I did this, my connection improved dramatically. I take a slight hit in upload rate for the single upload connection, but overall my internet connection is a lot more responsive for other connections and devices while uploading. If you think you might be experiencing this same issue, try running a speed test over at DSL Reports on your internet connection. Their speed test will check and give you a report on BufferBloat. Before the fix, my upload BufferBloat was around 600 ms. After the fix the BufferBloat is down to only 20 ms.

3 Comments

  1. Great diagnosis! And I’m glad your ZyWall had the ability to tweak the QoS to make the difference.

    I don’t know if you’re terminally interested, but the team at http://www.bufferbloat.net has devised the fq_codel algorithm that typically only shaves off a few percent from your upload and download bandwidth to control bufferbloat (instead of the ~20% decrease you used.) I don’t know if it’s available in the ZyWall, but you could also contact their support line to see if they’re going to embrace it. (See http://www.bufferbloat.net/projects/cerowrt/wiki/What_to_do_about_Bufferbloat for details.)

  2. Ross Williams

    June 6, 2016 at 4:55 pm

    Thank you, this worked very well at minimizing buffer bloat. After setting my limits to 12000 kbps, I’m seeing upload bloat of 5-10 ms, which is excellent, of course.

    One thing to mention for others struggling with upload speeds on a Zywall 110. I first noticed that my upload speeds were really slow (2-3Mbps on a 10 Mbps plan) when I had the firewall disabled because I was trying to figure out voip phone settings. With the firewall disabled, the BWM settings had no impact on buffer bloat. As soon as I re-enabled the firewall, my upload speeds went up to appx 11.5 – 11.7 Mbps, with buffer bloat while uploading of 150-250 ms. With the BWM settings at 12000, I’m still getting 11.25-11.5 Mbps up, with virtually no bloat.

    So the firewall has to be enabled for optimal upload speeds. This is counter-intuitive to me, but I don’t really know much about routers and networks.

  3. I just ran the speedtest on dslreports.com today and got an “F” on bufferbloat! Now I get a “B” by setting my Zywall outgoing to be my purchased upload speed (.768 meh). Bloat went from ~1500 to ~80 while uploading. I can tweak it a little bit but with a < 1024 kbps speed there isn't much wiggle room.
    Thanks for the tip!

Leave a Reply

Your email address will not be published.

*

© 2017 *Coder Blog

Theme by Anders NorenUp ↑