# QoS Manager - Troubleshooting and Debugging Guide

This document covers troubleshooting procedures and platform-specific debugging commands for qosmngr. For configuration, deployment scenarios, and API reference, see [QoS_Manager.md](QoS_Manager.md).

## Table of Contents

1. [General Troubleshooting](#1-general-troubleshooting)
2. [Airoha Platform Debugging](#2-airoha-platform-debugging)
3. [Linux/Generic Platform Debugging](#3-linuxgeneric-platform-debugging)
4. [Broadcom Platform Debugging](#4-broadcom-platform-debugging)

---

## 1. General Troubleshooting

### QoS Not Taking Effect After Configuration Change

```bash
# Ensure the service is running
/etc/init.d/qos status

# Force a full restart
/etc/init.d/qos restart

# Check for configuration errors in syslog
logread | grep qos
```

### Classification Rules Not Matching

```bash
# Verify ebtables rules are installed
ebtables -t broute -L --Lc

# Verify iptables mangle rules
iptables -t mangle -L -n -v

# Check rule order (lower order = higher priority)
uci show qos | grep order
```

### Queue Stats Showing Zero

```bash
# Verify queues exist and are enabled
uci show qos | grep queue

# Check if the interface exists
ip link show ae_wan

# Get stats directly
ubus call qos queue_stats '{"ifname":"ae_wan"}'
```

### Checking Active QoS Configuration

```bash
# Dump entire QoS UCI config
uci show qos

# Dump TR-181 data model
obuspa -c get Device.QoS.

# Check board.json for available ports
cat /etc/board.json | jsonfilter -e '@.network.lan.ports' -e '@.network.wan.ports'
```

### Common Issues

| Issue | Cause | Solution |
|---|---|---|
| No queues in stats | Board not detected | Check `/etc/board.json` for port definitions |
| Classification ignored | Rule disabled or wrong order | Verify `enable '1'` and check `order` value |
| Shaping not working | Rate value wrong | Rate is in **bits per second**, not bytes |
| Policer not applied | Name mismatch | Ensure `classify->policer` matches `policer->name` |
| VLAN translation one-way | Missing reverse rule | Add both directions (ingress + egress) |
| P-bit marking not visible | Wrong interface | Use VLAN sub-interface (e.g., `eth0.1`, not `ae_wan`) |

---

## 2. Airoha Platform Debugging

On Airoha (Econet) platforms, QoS is managed by hardware engines accessed through dedicated userspace tools and proc interfaces. This section covers the low-level debugging commands available on the device.

### 2.1 Tools Overview

| Tool | Path | Purpose |
|---|---|---|
| `qdmamgr_wan` | `/userfs/bin/qdmamgr_wan` | WAN QDMA engine management (egress queues, shaping, TRTCM) |
| `qdmamgr_lan` | `/userfs/bin/qdmamgr_lan` | LAN QDMA engine management (ingress rate control) |
| `qosrule` | `/userfs/bin/qosrule` | Traffic classification rules and QoS discipline |
| `fe` | `/userfs/bin/fe` | Forwarding Engine (FE) - rate limiting, counters, traffic classification |
| `qosmgr` | `/userfs/bin/qosmgr` | High-level QoS discipline wrapper |
| `switchmgr` | `/userfs/bin/switchmgr` | Switch port rate control |

### 2.2 Inspecting Queue Scheduling Configuration

The scheduling discipline and queue weights are written to proc entries during `hw_commit_all()`.

```bash
# Check the active QoS scheduling mode
# Values: 8QPQ (Strict Priority), 8QWRR (Weighted Round Robin), HW8QPQWRR (SP+WRR hybrid)
cat /proc/qdma_wan/mac_qos_flag

# Check the active QoS mode and weights
# Format: <mode> <w7> <w6> <w5> <w4> <w3> <w2> <w1> <w0>
# mode: 0=SP, 1=WRR, 2=SPWRR
cat /proc/qdma_wan/mac_qos

# Check which queues are enabled (bitmask, 255 = all 8 queues)
cat /proc/qdma_wan/mac_default_queuemask

# Check TCP ACK prioritization (0=disabled, 1=enabled)
cat /proc/qdma_wan/tcp_ack_flag
cat /proc/qdma_lan/tcp_ack_flag
```

### 2.3 Inspecting QDMA QoS State

Use `qdmamgr_wan get` and `qdmamgr_lan get` to query the hardware state.

```bash
# Get the overall QoS configuration
/userfs/bin/qdmamgr_wan get qos
/userfs/bin/qdmamgr_lan get qos

# Get downstream QoS information
/userfs/bin/qdmamgr_wan get downstream_qos
```

### 2.4 Inspecting Egress Shaping (WAN TX TRTCM)

WAN egress shaping uses the TX TRTCM (Two-Rate Three-Color Meter) engine.

```bash
# Check TX TRTCM initialization state
/userfs/bin/qdmamgr_wan get general_tx_init

# Get TX TRTCM configuration for a specific channel
# Channel 0 is typically the WAN channel
/userfs/bin/qdmamgr_wan get general_tx_trtcm

# Get TX rate limiting information
/userfs/bin/qdmamgr_wan get general_tx_ratelimit

# Get per-queue rate limiting (EN7581+ only, not EN7523)
/userfs/bin/qdmamgr_wan get general_tx_q_ratelimit
```

To identify the WAN channel number:

```bash
# First entry is the WAN channel (-1 means not a switch port)
cat /proc/tc3162/eth_portmap
```

### 2.5 Inspecting Ingress Rate Limiting (RX TRTCM)

LAN-side and WAN-side ingress policing uses the RX TRTCM engine.

```bash
# Check RX TRTCM initialization state
/userfs/bin/qdmamgr_lan get general_rx_init
/userfs/bin/qdmamgr_wan get general_rx_init

# Get RX TRTCM configuration (shows per-TSID settings)
/userfs/bin/qdmamgr_lan get general_rx_trtcm
/userfs/bin/qdmamgr_wan get general_rx_trtcm

# Get RX rate limiting information
/userfs/bin/qdmamgr_lan get general_rx_ratelimit
/userfs/bin/qdmamgr_wan get general_rx_ratelimit
```

For WAN ingress rate limiting via the Forwarding Engine:

```bash
# Check FE rate limit register (WAN port)
cat /proc/tc3162/fe_ratelimit

# Check the WAN port index from FE registers
cat /proc/tc3162/fe_reg | grep FE_WAN_PORT
```

### 2.6 Inspecting Global Rate Limiting

```bash
# Get global rate limit initialization
/userfs/bin/qdmamgr_wan get general_glb_init
/userfs/bin/qdmamgr_lan get general_glb_init

# Get global rate limit values
/userfs/bin/qdmamgr_wan get general_glb_ratelimit
/userfs/bin/qdmamgr_lan get general_glb_ratelimit
```

### 2.7 Inspecting Traffic Classification (qosrule)

```bash
# Check if QoS is enabled (per IP version)
# Outputs the current enable/disable state
/userfs/bin/qosrule switch

# Check the active QoS discipline
# PQ=Priority Queue, WRR=Weighted Round Robin, PQWRR=hybrid, CAR=policer
/userfs/bin/qosrule discpline
```

### 2.8 Checking Congestion and Buffer State

```bash
# Get LMGR (Link Manager) status: free descriptors, used descriptors, used buffers
/userfs/bin/qdmamgr_wan get lmgr_status
/userfs/bin/qdmamgr_lan get lmgr_status

# Get TX queue congestion configuration
/userfs/bin/qdmamgr_wan get txq_cngst
/userfs/bin/qdmamgr_wan get txq_cngst_mode

# Get TX dynamic congestion status
/userfs/bin/qdmamgr_wan get tx_dyn_cngst

# Get TX buffer thresholds
/userfs/bin/qdmamgr_wan get tx_buf_thrh

# Get WRED (Weighted Random Early Detection) configuration
/userfs/bin/qdmamgr_wan get txwred

# Get RX RED configuration
/userfs/bin/qdmamgr_wan get rxred
```

### 2.9 Checking Flow Counters and Statistics

```bash
# Get flow counter values (per-flow accounting)
# Groups: 0 (63 entries), 1 (31 entries), 2 (127 entries/meters)
/userfs/bin/qdmamgr_wan get flowcnt
/userfs/bin/qdmamgr_lan get flowcnt

# Get SLA (Service Level Agreement) configuration
/userfs/bin/qdmamgr_wan get sla_cfg

# Get SLA TRTCM values
/userfs/bin/qdmamgr_wan get general_sla_trtcm
/userfs/bin/qdmamgr_wan get general_sla_ratelimit
```

### 2.10 Forwarding Engine (FE) Debugging

```bash
# Get FE rate limiting information for specific meters
/userfs/bin/fe ratelimit

# Get FE counter values
/userfs/bin/fe cnt

# Check PCP (Priority Code Point) configuration
/userfs/bin/fe pcp

# Check VLAN configuration in FE
/userfs/bin/fe get ext_tpid
/userfs/bin/fe get mc_vlan

# Check FE accounting meter mapping
cat /proc/tc3162/fe_acnt_mapping
```

### 2.11 Proc Filesystem Debug Points

The Airoha kernel modules expose debug information via proc:

```bash
# QDMA buffer manager counters
cat /proc/qdma_wan/bm_counters
cat /proc/qdma_lan/bm_counters

# IFC (Intelligent Flow Control) debug
cat /proc/ifc_debug

# Check if packets are being sent to PPE (hardware acceleration)
cat /proc/ifc_send_to_ppe

# FE registers
cat /proc/tc3162/fe_reg

# Ethernet port mapping
cat /proc/tc3162/eth_portmap
```

### 2.12 Checking WAN Link Speed and Shaping State

qosmngr stores runtime state in `/tmp/qos/`:

```bash
# Current WAN link speed (auto-detected on link-up)
cat /tmp/qos/wan_link_speed

# Applied shaping rate (if explicit shaper configured)
cat /tmp/qos/wan_link_shape_rate

# WAN ingress policing rate
cat /tmp/qos/wan_ingress_shape_speed

# Generated ebtables/iptables rule scripts
ls -la /tmp/qos/
```

### 2.13 Airoha Debugging Checklist

When QoS is not working as expected on an Airoha device:

1. **Verify the service started correctly:**
   ```bash
   /etc/init.d/qos status
   logread | grep -E 'qos|qdma'
   ```

2. **Check scheduling mode is applied:**
   ```bash
   cat /proc/qdma_wan/mac_qos_flag    # Should be 8QPQ, 8QWRR, or HW8QPQWRR
   cat /proc/qdma_wan/mac_qos         # Check mode and weight values
   ```

3. **Check shaping is applied:**
   ```bash
   /userfs/bin/qdmamgr_wan get general_tx_trtcm   # WAN egress
   /userfs/bin/qdmamgr_lan get general_rx_trtcm    # LAN ingress
   ```

4. **Check classification rules are installed:**
   ```bash
   ebtables -t broute -L --Lc         # L2 rules with hit counters
   iptables -t mangle -L -n -v        # L3/L4 rules with counters
   ```

5. **Check queue stats are incrementing:**
   ```bash
   ubus call qos queue_stats '{"ifname":"ae_wan"}'
   # Wait, then re-run to see if counters change
   ```

6. **Check buffer/congestion state:**
   ```bash
   /userfs/bin/qdmamgr_wan get lmgr_status         # Free vs used descriptors
   /userfs/bin/qdmamgr_wan get tx_dyn_cngst         # Congestion state
   ```

7. **Check the WAN channel mapping:**
   ```bash
   cat /proc/tc3162/eth_portmap        # WAN channel index
   cat /proc/tc3162/fe_reg | grep FE_WAN_PORT   # FE WAN port
   ```

8. **Check platform variant (affects available features):**
   ```bash
   strings /proc/device-tree/compatible
   # EN7523: no per-queue shaping, no SPWRR
   # EN7581+: per-queue shaping supported
   ```

---

## 3. Linux/Generic Platform Debugging

On the generic Linux platform, QoS is implemented using standard kernel traffic control (`tc`) with HTB qdiscs, `iptables`/`ip6tables` mangle rules, and `ebtables` broute rules. This section covers how to inspect and debug each layer.

### 3.1 Traffic Control (tc) Hierarchy

The generic platform builds the following tc hierarchy per interface:

```
Root: TBF (Token Bucket Filter) - port-level shaping
 └── HTB (Hierarchical Token Bucket) - queue hierarchy
      └── Parent class (1:1) - port bandwidth ceiling
           ├── Queue class (1:8) - highest priority (precedence 1)
           │    └── [SFQ qdisc if WRR]
           ├── Queue class (1:7) - ...
           ├── ...
           └── Queue class (1:1) - lowest priority (precedence 8)
                └── [SFQ qdisc if WRR]
```

Packets are steered to queue classes by `tc filter` rules that match on fwmark values set by iptables/ebtables.

### 3.2 Inspecting tc Qdiscs, Classes, and Filters

```bash
# Show all qdiscs on an interface (with statistics)
tc -s qdisc show dev eth0

# Show all HTB classes (with statistics: rate, ceil, sent bytes/packets)
tc -s class show dev eth0

# Show all filters (fwmark-based classification to queues)
tc -s filter show dev eth0

# Show the root qdisc details
tc -d qdisc show dev eth0 root

# Show ingress qdisc (if policer is configured)
tc -s qdisc show dev eth0 ingress
tc -s filter show dev eth0 parent ffff:
```

**What to look for:**

- `tc -s class show dev eth0` - Check `Sent` counters per class to verify traffic is reaching the correct queue. Each class corresponds to a queue priority level.
- `tc -s filter show dev eth0` - Verify that `match mark` filters exist and map to the correct `flowid` (class).
- `tc -s qdisc show dev eth0` - Check `dropped` and `overlimits` counters on the root TBF to confirm shaping is active.

**Example output (healthy):**

```
# tc -s class show dev eth0
class htb 1:1 root rate 100Mbit ceil 100Mbit burst 1500b cburst 1500b
 Sent 1234567 bytes 8901 pkt (dropped 0, overlimits 0 requeues 0)

class htb 1:8 parent 1:1 prio 0 rate 100Mbit ceil 100Mbit burst 1500b
 Sent 500000 bytes 3000 pkt (dropped 0, overlimits 0 requeues 0)

class htb 1:1 parent 1:1 prio 7 rate 10Mbit ceil 100Mbit burst 1500b
 Sent 734567 bytes 5901 pkt (dropped 12, overlimits 45 requeues 0)
```

### 3.3 Inspecting iptables/ip6tables Mangle Rules

Classification rules that match on L3/L4 criteria (IP, port, protocol, DSCP, packet length) are installed in the `mangle` table.

```bash
# View all QoS mangle chains with packet/byte counters
iptables -t mangle -L -n -v
iptables -t mangle -L qos_forward -n -v --line-numbers
iptables -t mangle -L qos_prerouting -n -v --line-numbers
iptables -t mangle -L qos_output -n -v --line-numbers

# Same for IPv6
ip6tables -t mangle -L -n -v
ip6tables -t mangle -L qos_forward -n -v --line-numbers
ip6tables -t mangle -L qos_prerouting -n -v --line-numbers
ip6tables -t mangle -L qos_output -n -v --line-numbers

# Zero counters and re-check to see live traffic matching
iptables -t mangle -Z qos_forward
# ... generate traffic ...
iptables -t mangle -L qos_forward -n -v
```

**Chain purposes:**

| Chain | Purpose |
|---|---|
| `qos_prerouting` | Marks packets entering from a specific interface (LAN->WAN) |
| `qos_forward` | Marks packets transiting the router (through-traffic) |
| `qos_output` | Marks packets originating from the router itself (including loopback) |

**Mark format:** Decimal integer set via `-j MARK --set-mark <traffic_class>`. The mark value (0-7) maps 1:1 to the tc filter that routes packets to the corresponding HTB class.

### 3.4 Inspecting ebtables Broute Rules

L2 classification (MAC, VLAN, P-bit, DHCP options) and packet marking/rewriting are handled by ebtables in the `broute` table.

```bash
# View all broute chains with counters
ebtables -t broute -L --Lc

# View the main QoS chain
ebtables -t broute -L qos --Lc --Ln

# View VLAN translation rules
ebtables -t broute -L prevlanxlate --Lc --Ln

# View DSCP-to-P-bit mapping rules
ebtables -t broute -L dscp2pbits --Lc --Ln

# View NAT table (locally originated L2 traffic)
ebtables -t nat -L qos_output --Lc --Ln

# Zero counters to monitor live matching
ebtables -t broute -Z qos
# ... generate traffic ...
ebtables -t broute -L qos --Lc
```

**Chain purposes:**

| Table | Chain | Purpose |
|---|---|---|
| broute | `qos` | Main L2 classification: MAC, VLAN, PCP, ethertype, IP, port matching. Sets fwmark via `-j mark --mark-or 0x<traffic_class>` |
| broute | `prevlanxlate` | VLAN ID and P-bit rewriting via `-j vlantranslation` |
| broute | `dscp2pbits` | DSCP-to-P-bit mapping via `-j dscp2pbit --dscp2pbit-mapping <dscp>=<pbit>,...` |
| nat | `qos_output` | Marks locally originated bridged traffic |

### 3.5 Inspecting IP Policy Routing Rules

When `forwarding_policy` is set on a classification rule, IP routing rules are created instead of fwmark-based queue assignment.

```bash
# View all policy routing rules (priority ordered)
ip rule show

# View rules for a specific routing table
ip rule show table 256

# Check which routes exist in the target table
ip route show table 256

# Test route resolution for a specific flow
ip route get 1.1.1.1 from 192.168.1.100

# View IPv6 policy rules
ip -6 rule show
```

**TOS/fwmark handling:** When a classification rule has `dscp_filter` set and the resulting TOS value exceeds `0x10`, the DSCP cannot be matched directly in `ip rule`. In this case, an iptables rule sets a fwmark, and the ip rule matches on `fwmark` instead of `tos`.

### 3.6 Inspecting Ingress Policing

On the generic Linux platform, ingress policing uses the `tc ingress` qdisc with `police` filters.

```bash
# Check if ingress qdisc is attached
tc qdisc show dev eth0 | grep ingress

# View police filters and their counters
tc -s filter show dev eth0 parent ffff:

# Check the number of configured policers
cat /tmp/qos/max_policer_inst
```

**Example output:**

```
# tc -s filter show dev eth0 parent ffff:
filter protocol ip pref 1 u32
  match ip src 0.0.0.0/0
  police rate 5000Kbit burst 15Kb
    Sent 12345678 bytes 9876 pkt (dropped 543, overlimits 1234)
```

### 3.7 Temporary Files and Runtime State

qosmngr generates rule scripts and state files in `/tmp/qos/`:

```bash
# List all runtime files
ls -la /tmp/qos/

# View generated iptables rules (before execution)
cat /tmp/qos/classify.iptables
cat /tmp/qos/classify.ip6tables

# View generated ebtables rules
cat /tmp/qos/classify.ebtables

# View generated IP policy routing rules
cat /tmp/qos/classify.iprule

# View per-interface queue ordering
cat /tmp/qos/<ifname>/q_order
cat /tmp/qos/<ifname>/q_idx

# View the QoS config snapshot used for reload diffing
cat /tmp/qos/qos
```

These files are regenerated on each `/etc/init.d/qos reload`.

### 3.8 Linux/Generic Debugging Checklist

When QoS is not working as expected on a generic Linux device:

1. **Verify the service started and rules are loaded:**
   ```bash
   /etc/init.d/qos status
   logread | grep qos
   ```

2. **Check tc hierarchy is built:**
   ```bash
   tc qdisc show dev eth0          # Should show TBF root + HTB child
   tc class show dev eth0          # Should show parent + per-queue classes
   tc filter show dev eth0         # Should show mark-based filters
   ```

3. **Check iptables marking rules exist:**
   ```bash
   iptables -t mangle -L qos_forward -n -v      # Should have rules with MARK targets
   iptables -t mangle -L qos_prerouting -n -v
   ```

4. **Check ebtables L2 rules exist:**
   ```bash
   ebtables -t broute -L qos --Lc                # Should have rules with mark targets
   ```

5. **Verify packets are being marked (zero counters, generate traffic, re-check):**
   ```bash
   iptables -t mangle -Z qos_forward
   # ... generate matching traffic ...
   iptables -t mangle -L qos_forward -n -v       # Check pkts/bytes columns
   ```

6. **Verify packets reach the correct tc class:**
   ```bash
   tc -s class show dev eth0       # Check Sent counters per class
   ```

7. **Verify shaping is active (check for drops/overlimits):**
   ```bash
   tc -s qdisc show dev eth0       # Check dropped/overlimits on TBF root
   ```

8. **Check ingress policing (if configured):**
   ```bash
   tc -s filter show dev eth0 parent ffff:    # Check police counters
   ```

9. **Inspect generated rule scripts for errors:**
   ```bash
   cat /tmp/qos/classify.iptables
   cat /tmp/qos/classify.ebtables
   ```

---

## 4. Broadcom Platform Debugging

On Broadcom platforms, QoS is managed by hardware through `tmctl` (Traffic Management Control), `ethswctl` (Ethernet Switch Control), `bdmf_shell` (Broadcom Device Management Framework), and `fcctl` (Flow Cache Control). This section covers the low-level debugging commands available on the device.

### 4.1 Tools Overview

| Tool | Purpose |
|---|---|
| `tmctl` | Primary QoS tool: queue config, port/queue shaping, policers, drop algorithms, statistics, DSCP/P-bit mapping |
| `ethswctl` | Ethernet switch port control: ingress rate limiting, port-to-interface mapping |
| `bdmf_shell` | Broadcom Device Management Framework: attach policers to ports, low-level HW configuration |
| `fcctl` | Flow cache control: flush learned flows after QoS changes |
| `ethtool` | Query interface link speed (used for percentage-based rate calculation) |

### 4.2 Inspecting Queue Configuration

```bash
# Get configuration for a specific queue (scheduling, priority, weight, shaper)
tmctl getqcfg --devtype 0 --if eth0 --qid 0
tmctl getqcfg --devtype 0 --if eth0 --qid 7

# Query all 8 queues on an interface
for qid in 0 1 2 3 4 5 6 7; do
    echo "=== Queue $qid ==="
    tmctl getqcfg --devtype 0 --if eth0 --qid $qid
done
```

**Output fields:**
- `qid` - Queue ID (0-7)
- `priority` - Priority value (7=highest for SP, 0 for WRR queues)
- `qsize` - Queue size in packets (default 1024)
- `weight` - WRR/WDRR/WFQ weight (1-63)
- `schedMode` - 1=SP, 2=WRR, 3=WDRR, 4=WFQ
- `shapingRate` - Per-queue shaping rate in kbps (0=disabled, -1=unsupported)
- `burstSize` - Burst size in bytes
- `minRate` - Minimum guaranteed rate in kbps

### 4.3 Inspecting Queue Statistics

```bash
# Get TX and drop counters for a specific queue
tmctl getqstats --devtype 0 --if eth0 --qid 0

# Query all queues
for qid in 0 1 2 3 4 5 6 7; do
    echo "=== Queue $qid ==="
    tmctl getqstats --devtype 0 --if eth0 --qid $qid
done
```

**Output fields:**
- `txPackets` - Total transmitted packets
- `txBytes` - Total transmitted bytes
- `droppedPackets` - Total dropped packets
- `droppedBytes` - Total dropped bytes

A value of `0xffffffff` means the counter is not available.

### 4.4 Inspecting Port TM Capabilities

```bash
# Get port traffic management capabilities
tmctl getporttmparms --devtype 0 --if eth0
```

**Output fields:**
- `schedCaps` - Supported scheduling algorithms (bitmap: SP, WRR, WDRR, WFQ, SP+WRR)
- `maxQueues` - Maximum number of queues the port supports
- `maxSpQueues` - Max SP queues when coexisting with WRR
- `portShaper` - Port-level shaping supported (0/1)
- `queueShaper` - Per-queue shaping supported (0/1)
- `cfgFlags` - Active scheduling configuration
- `numQueues` - Currently configured queue count
- `dualRate` - Dual-rate shaping supported (0/1)

### 4.5 Inspecting Port Shaper

```bash
# Get port-level shaper configuration
tmctl getportshaper --devtype 0 --if eth0

# Get port RX (ingress) rate configuration
tmctl getportrxrate --devtype 0 --if eth0

# Get Overall Rate Limiter (ORL) shaper
tmctl getorlshaper --devtype 0
```

**Output fields:**
- `shapingRate` - Rate in kbps
- `shapingBurstSize` - Burst size in bytes
- `minRate` - Minimum guaranteed rate in kbps

### 4.6 Inspecting Drop Algorithm and Flow Control

```bash
# Get the drop algorithm for a queue (DT, RED, WRED, CoDel, PI2)
tmctl getqdropalg --devtype 0 --if eth0 --qid 0

# Get flow control settings for a queue
tmctl getqflowctrl --devtype 0 --if eth0 --qid 0
```

**Drop algorithm types:**
- `0` = Drop Tail (DT) - simple tail drop
- `1` = RED - Random Early Detection
- `2` = WRED - Weighted RED (two priority classes with separate thresholds)
- `3` = CoDel - Controlled Delay
- `4` = PI2 - Proportional Integral controller

**Flow control output:**
- `enable` - Flow control enabled (0/1)
- `pauseThr` - Pause threshold (stop transmitting)
- `hystThr` - Hysteresis threshold (resume transmitting)

### 4.7 Inspecting DSCP and P-bit Mappings

```bash
# Get DSCP to P-bit mapping table (all 64 DSCP values)
tmctl getdscptopbit --devtype 0 --if eth0

# Get P-bit to queue mapping (all 8 P-bit values)
tmctl getpbittoq --devtype 0 --if eth0

# Get forced DSCP setting (downstream/upstream)
tmctl getforcedscp --dir 0    # Downstream
tmctl getforcedscp --dir 1    # Upstream

# Get packet-based QoS settings
tmctl getpktbasedqos --dir 0 --qostype 0    # Flow Cache, downstream
tmctl getpktbasedqos --dir 1 --qostype 1    # Ingress Classifier, upstream
```

### 4.8 Inspecting Port Mirroring (for Packet Capture)

Port mirroring can be used to capture traffic for debugging:

```bash
# Get current RX mirror configuration
tmctl getmirrorrx

# Get current TX mirror configuration
tmctl getmirrortx

# Set up RX mirror (mirror eth0 RX traffic to eth1)
tmctl addmirrorrx --devtype 0 --if eth0 --destif eth1

# Set up TX mirror
tmctl addmirrortx --devtype 0 --if eth0 --destif eth1

# Remove mirrors when done
tmctl delmirrorrx --devtype 0 --if eth0
tmctl delmirrortx --devtype 0 --if eth0
```

### 4.9 Enabling tmctl Trace Logging

tmctl has a built-in trace facility for debugging API calls:

```bash
# Enable trace logging to console
tmctl trace --enable 1 --printtofile 0

# Enable trace logging to file
tmctl trace --enable 1 --printtofile 1

# Disable trace logging
tmctl trace --enable 0 --printtofile 0
```

When enabled, all tmctl API calls are logged with their parameters and return values.

### 4.10 Inspecting Ethernet Switch (ethswctl)

```bash
# Get the interface name for a physical switch port
ethswctl getifname <unit> <port>
# Example: ethswctl getifname 0 0  -> returns "eth0"

# Check ingress rate control on a port
# (use the unit and port numbers from board.json)
ethswctl -c rxratectrl -n <unit> -p <port>
```

### 4.11 Inspecting BDMF and Flow Cache

```bash
# Get the BDMF shell session ID
cat /var/bdmf_sh_id

# Check flow cache status
fcctl status

# Flush flow cache (forces re-classification of all flows)
fcctl flush
```

**Note:** After any change to traffic class mapping, `fcctl flush` must be called to force existing flows to be re-classified with the new QoS settings. qosmngr does this automatically during reload.

### 4.12 Broadcom Scheduling Mode Reference

When configuring queues via tmctl, the scheduling mode and priority interact as follows:

| schedMode | Value | Priority Field | Weight Field |
|---|---|---|---|
| SP (Strict Priority) | 1 | 0-7 (7=highest). On some platforms, priority must equal qid | Ignored |
| WRR | 2 | Must be 0 | 1-63 |
| WDRR | 3 | 0-7 | 1-63 |
| WFQ | 4 | 0-7 | 1-63 |

Mixed SP+WRR is supported: higher-priority queues use SP (priority > 0), lower-priority queues use WRR (priority = 0, weight > 0).

### 4.13 Broadcom Debugging Checklist

When QoS is not working as expected on a Broadcom device:

1. **Verify the service started correctly:**
   ```bash
   /etc/init.d/qos status
   logread | grep qos
   ```

2. **Check port TM is initialized and queues exist:**
   ```bash
   tmctl getporttmparms --devtype 0 --if eth0
   # numQueues should be 8, schedCaps should list supported algorithms
   ```

3. **Check queue configuration is correct:**
   ```bash
   for qid in 0 1 2 3 4 5 6 7; do
       tmctl getqcfg --devtype 0 --if eth0 --qid $qid
   done
   # Verify schedMode, priority, weight values match UCI config
   ```

4. **Check port shaper is applied:**
   ```bash
   tmctl getportshaper --devtype 0 --if eth0
   # shapingRate should match UCI shaper->rate (in kbps)
   ```

5. **Check queue stats are incrementing:**
   ```bash
   tmctl getqstats --devtype 0 --if eth0 --qid 7
   # Run twice with traffic to see counters change
   ```

6. **Check classification rules are installed:**
   ```bash
   ebtables -t broute -L --Lc         # L2 rules with hit counters
   iptables -t mangle -L -n -v        # L3/L4 rules with counters
   ```

7. **Check flow cache is up to date:**
   ```bash
   fcctl status
   # If flows are stale after a QoS change, flush them:
   fcctl flush
   ```

8. **Check ingress policing (ethswctl):**
   ```bash
   # Get the unit/port for the interface from board.json
   ethswctl -c rxratectrl -n 0 -p 0
   ```

9. **Check drop algorithm for queue congestion:**
   ```bash
   tmctl getqdropalg --devtype 0 --if eth0 --qid 0
   # If droppedPackets is high, check RED/WRED thresholds
   ```

10. **Enable tmctl tracing for detailed API logging:**
    ```bash
    tmctl trace --enable 1 --printtofile 0
    # Reproduce the issue, then check output
    tmctl trace --enable 0 --printtofile 0
    ```

---

## Statistics and Monitoring

### ubus Queue Statistics

The primary method for real-time queue monitoring:

```bash
# Get all stats for all interfaces
ubus call qos queue_stats

# Get all queues on WAN
ubus call qos queue_stats '{"ifname":"ae_wan"}'

# Get specific queue
ubus call qos queue_stats '{"ifname":"ae_wan", "qid":7}'
```

**Example output (showing active VoIP queue):**

```json
{
  "queues": [
    {
      "qid": 7,
      "iface": "ae_wan",
      "tx_packets": 15234,
      "tx_bytes": 4876800,
      "tx_dropped_packets": 0,
      "tx_dropped_bytes": 0
    }
  ]
}
```

### TR-181 QueueStats

For TR-181 based management (USP, CWMP):

```bash
# Enable stats collection for a specific queue
obuspa -c add Device.QoS.QueueStats.
obuspa -c set Device.QoS.QueueStats.1.Queue Device.QoS.Queue.1.
obuspa -c set Device.QoS.QueueStats.1.Interface Device.Ethernet.Interface.6.
obuspa -c set Device.QoS.QueueStats.1.Enable true

# Read statistics
obuspa -c get Device.QoS.QueueStats.1.OutputPackets
obuspa -c get Device.QoS.QueueStats.1.OutputBytes
obuspa -c get Device.QoS.QueueStats.1.DroppedPackets
obuspa -c get Device.QoS.QueueStats.1.DroppedBytes

# Or get all stats at once
obuspa -c get Device.QoS.QueueStats.1.
```

### USP Agent

```bash
# Get all QueueStats instances
obuspa -c get Device.QoS.QueueStats.

# Get specific stat
obuspa -c get Device.QoS.QueueStats.1.OutputPackets
```

### JSONRPC (Remote Access)

For remote management via HTTP/JSON-RPC:

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "call",
  "params": ["<SESSION_ID>", "qos", "queue_stats", {"ifname": "ae_wan"}]
}
```

### Monitoring ebtables/iptables Rules

To inspect the active classification rules applied by qosmngr:

```bash
# View L2 classification rules (ebtables broute table)
ebtables -t broute -L --Lc

# View DSCP-to-P-bit rules
ebtables -t broute -L dscp2pbit --Lc

# View VLAN translation rules
ebtables -t broute -L prevlanxlate --Lc

# View L3/L4 classification rules (iptables mangle table)
iptables -t mangle -L qos_forward -n -v
iptables -t mangle -L qos_prerouting -n -v

# View IP policy routing rules
ip rule show
```
