The short version
Usage graphs from different layers of the stack disagreed, so none could be trusted for reporting. Lining the counters up showed that the VLAN-side eth.11 interfaces are the right place to measure what users actually consume.
Walkthrough
Click through the counters. Use Next and Back to follow the investigation.
Overview
Bandwidth was visible at several layers of one gateway and one router: tunnel interfaces, VLAN interfaces, the MikroTik Local bridge and the WAN. The numbers did not match.
Problem statement
I needed one trustworthy number for how much bandwidth users consume, and several graphs gave several answers.
Investigation
- Lined up the counters from all four layers for the same period.
- Asked what each layer actually sees, instead of asking which one was wrong.
- Chose the layer closest to the user.
ip -s link show <interface>
cat /proc/net/devRoot cause
Nothing was broken. Each layer counts a different slice of traffic. Tunnel interfaces count encapsulated traffic, the bridge sees everything that crosses it, and the WAN counts everything that leaves the building, not only user traffic.
Resolution
Measure user-side bandwidth on the VLAN-side eth.11 interfaces. That is one source of truth for reporting, and the other counters stay useful for troubleshooting.
Lessons learned
- Before declaring a counter wrong, work out what it measures.
- Pick the measurement point on purpose, and write down why.
Skills demonstrated
Confidentiality: all identifying information, credentials, customer data, internal addresses and proprietary configurations have been removed or replaced with illustrative examples.