Diagnosing PRTG UDP Errors on a Syslog Server: A Packet Capture Story with Claude Code and OpenClaw
Once again, I gave Claude Code an old PCAP from an issue I troubleshot a while ago, and gave it this prompt:
Our SNMP PRTG tool is showing a large number of UDP errors on the syslog servers NIC, we took a traffic capture demo/net-dump-udp-errors.pcap, please analyze and report what could be the issue, add explanation of each command executed to obtain data for the analysis, and recommendations.
(BTW, the fix was a combination of the /etc/sysctl.conf settings and reducing noise from the sources)
This is Claude Code with Sonnet 4.6 narrative (I asked it to tell the story):
This is the story of that investigation — every command I ran, what the output told me, and where it led.
The Setup
The capture file was net-dump-udp-errors.pcap (57 MB). The syslog server's IP is 10.20.0.17 listening on UDP port 514.
Step 1: Get the lay of the land — capinfos
Before doing anything else, I always start with capinfos to understand what I'm dealing with:
capinfos net-dump-udp-errors.pcapOutput told me: 259,891 packets captured over 99.6 seconds. File encapsulation is Linux cooked-mode capture v1 (SLL). Average packet size 211 bytes, average rate 2,609 packets/sec.
Two things immediately caught my eye: the Linux cooked-mode encapsulation (more on that later) and the packet rate — nearly 2,600 pps sustained for 100 seconds is non-trivial for a syslog listener.
Step 2: Protocol breakdown — tshark -z ptype,tree
tshark -r net-dump-udp-errors.pcap -q -z ptype,treeOutput told me: 99.99% of all traffic is UDP. There are exactly 23 non-UDP frames (NONE/other). This is almost purely a UDP story — no TCP, no ICMP at this stage.
Step 3: Traffic over time — tshark -z io,stat,10
tshark -r net-dump-udp-errors.pcap -q -z io,stat,10Output told me: Traffic is remarkably flat across all 10-second intervals — ranging from 22,392 to 29,067 frames per interval. This is not a spike or a storm; it's a sustained, continuous flood. Whatever is causing the UDP errors isn't a one-time event.
Step 4: Who's talking to whom — tshark -z conv,udp
tshark -r net-dump-udp-errors.pcap -q -z conv,udpOutput told me: Every single conversation is one-directional: various sources → 10.20.0.17:514. Zero return traffic. Classic syslog behavior. The top sender is 10.20.254.4 with 33,578 frames, followed by many devices in the 172.XXX.x.x range — all sending to port 514. Critically, I notice multiple source ports from the same IPs (e.g., :45454 and another port from 10.20.254.4). That's a flag worth remembering.
Step 5: Rank the senders — custom tshark + awk
tshark -r net-dump-udp-errors.pcap -Y "udp.dstport == 514" \
-T fields -e ip.src -e udp.length \
| awk '{sum[$1]+=$2; count[$1]++} END \
{for(ip in sum) printf "%s\t%d frames\t%d bytes\n", ip, count[ip], sum[ip]}' \
| sort -k2 -rn | head -20Output told me: 208 unique sources are sending syslog to this server. The top sender — 10.20.254.4 — contributes 34,434 frames. Looking at the actual syslog message content (next step), I'd learn what it is.
Step 6: What is the top sender sending? — tshark -e syslog.msg
tshark -r net-dump-udp-errors.pcap \
-Y "ip.src == 10.20.254.4 and udp.dstport == 514" \
-T fields -e syslog.msg -e syslog.level -e syslog.facility \
| head -10Output told me: The messages look like this:
AzureFW01 flows src=192.168.XXX.29 dst=10.20.0.125 protocol=udp sport=61636 dport=53 pattern: allow all
AzureFW01 flows src=10.20.0.13 dst=192.168.XXX.65 mac=B4:DE:XX:XX:XX:4F protocol=tcp sport=51639 dport=3030 pattern: allow (src 10.20.0.0/16)The top sender is the Azure Firewall (AzureFW01), logging every single allowed flow. Firewalls logging all traffic are notoriously chatty — this immediately explains why it's the heaviest sender.
Step 7: Check for ICMP errors — tshark -Y "icmp"
tshark -r net-dump-udp-errors.pcap -Y "icmp" \
-T fields -e ip.src -e ip.dst -e icmp.type -e icmp.codeOutput told me: Nothing. No ICMP Port Unreachable messages anywhere. This means the server isn't actively rejecting packets — port 514 is open and the syslog daemon is running. The drops are happening silently, inside the kernel, before the application ever sees the packet.
Step 8: Payload size profile — custom awk on udp.length
tshark -r net-dump-udp-errors.pcap -Y "udp.dstport == 514" \
-T fields -e udp.length \
| awk '{sum+=$1; count++; if($1>max)max=$1; if(min==""||$1<min)min=$1} \
END {printf "Total frames: %d\nAvg: %.1f bytes\nMin: %d\nMax: %d\n", count, sum/count, min, max}'Output told me: Average UDP payload is 175.6 bytes. Min 93 bytes, max 1,408 bytes (well under the 1,472-byte Ethernet limit — no fragmentation issues). No malformed or oversized messages here.
Step 9: Is the top sender bursting? — inter-arrival time analysis
tshark -r net-dump-udp-errors.pcap \
-Y "ip.src == 10.20.254.4 and udp.dstport == 514" \
-T fields -e frame.time_relative -e udp.length \
| awk 'NR>1{gap=$1-prev; if(gap<0.001) fast++; total++} \
{prev=$1} END {printf "Total: %d\nSub-1ms gaps: %d (%.1f%%)\n", total, fast, fast/total*100}'Output told me: 61.1% of the Azure Firewall's packets arrive less than 1ms after the previous one. It's not sending at a steady rate — it's micro-bursting continuously. Even if the average rate looks manageable, these bursts can overflow a kernel socket queue in microseconds.
Step 10: Peak rate of the top sender
tshark -r net-dump-udp-errors.pcap \
-Y "ip.src == 10.20.254.4 and udp.dstport == 514" \
-T fields -e frame.time_relative \
| awk '{sec=int($1); count[sec]++} \
END {max=0; for(s in count){if(count[s]>max)max=count[s]}; \
total=0; for(s in count)total+=count[s]; \
printf "Avg pps: %.0f\nPeak pps: %d\n", total/length(count), max}'Output told me: Azure Firewall averages 344 pps and peaks at 520 pps — alone, before counting the other 207 senders. At peak, combined server load approaches 1,784 pps of real traffic.
Step 11: Something feels off — duplicate detection
At this point I had a nagging feeling. The raw packet count (259,891) was high, but the senders I'd seen individually didn't seem to add up to that. I ran a duplicate detection pass using IP ID + UDP length + message content as a unique key:
tshark -r net-dump-udp-errors.pcap -Y "udp.dstport == 514" \
-T fields -e ip.src -e ip.id -e udp.length -e syslog.msg \
| awk '{key=$1"_"$2"_"$3"_"$4; if(key in seen){dup[$1]++} \
else {seen[key]=1; total[$1]++}} \
END {for(ip in total) printf "%s total=%d dups=%d dup_pct=%.0f%%\n", \
ip, total[ip]+dup[ip], dup[ip], dup[ip]/(total[ip]+dup[ip])*100}' \
| sort -k4 -t= -rn | head -10Output told me: Every. Single. Source IP shows exactly 50% duplication. Not approximately — exactly. 10.20.254.4: 50%. 172.XXX.XXX.1: 50%. 192.168.XX.99: 50%. All 208 of them. This is too uniform to be a network problem. Something systematic is happening.
Step 12: Same source port or different? — confirm the duplicate type
tshark -r net-dump-udp-errors.pcap \
-Y "ip.src == 10.20.254.4 and udp.dstport == 514" \
-T fields -e frame.time_relative -e ip.id -e udp.srcport -e udp.length -e syslog.msg \
| awk '{key=$2"_"$4"_"$5; if(key in seen){ \
printf "ORIG_PORT=%s DUP_PORT=%s GAP=%.6f\n", seen_port[key], $3, $1-seen_time[key]} \
else {seen[key]=1; seen_port[key]=$3; seen_time[key]=$1}}' | head -5Output told me:
ORIG_PORT=45454 DUP_PORT=45454 GAP=0.000000
ORIG_PORT=45454 DUP_PORT=45454 GAP=0.000000Same source port. Zero time gap. This is not two syslog agents configured to send to the same destination (which would use different ephemeral ports). This is the exact same packet appearing twice. My attention snapped back to the encapsulation type from Step 1: Linux cooked-mode capture v1.
Step 13: The smoking gun — frame number delta
tshark -r net-dump-udp-errors.pcap \
-Y "ip.src == 10.20.254.4 and udp.dstport == 514" \
-T fields -e frame.number -e ip.id -e udp.length \
| awk '{key=$2"_"$3; if(key in seen){ \
printf "FRAME1=%s FRAME2=%s DELTA=%d\n", seen[key], $1, $1-seen_fn[key]} \
else {seen[key]=$1; seen_fn[key]=$1}}' | head -10Output told me:
FRAME1=9 FRAME2=10 DELTA=1
FRAME1=13 FRAME2=14 DELTA=1
FRAME1=23 FRAME2=24 DELTA=1Every duplicate pair is consecutive frames — one frame apart, zero time difference. This is the definitive signature of tcpdump -i any.
When you capture on Linux's any pseudo-interface, libpcap sees each packet twice: once as it arrives at the NIC (ingress) and once as it's delivered to the socket (post-routing). The result is exactly what we see: every packet doubled, in consecutive frames, with no timestamp difference.
Step 14: Quantify the capture artifact vs real traffic
tshark -r net-dump-udp-errors.pcap -Y "udp.dstport == 514" \
-T fields -e frame.len -e frame.time_relative \
| awk '{sec=int($2); bytes[sec]+=$1; pkts[sec]++} \
END {max_b=0; max_p=0; for(s in bytes){ \
if(bytes[s]>max_b)max_b=bytes[s]; if(pkts[s]>max_p)max_p=pkts[s]}; \
total_b=0; total_p=0; for(s in bytes){total_b+=bytes[s]; total_p+=pkts[s]}; \
printf "Avg: %.2f Mbps / %.0f pps\nPeak: %.2f Mbps / %d pps\n", \
(total_b*8/1000000)/length(bytes), total_p/length(bytes), max_b*8/1000000, max_p}'Output told me:
Avg throughput: 4.40 Mbps / 2,599 pps
Peak throughput: 6.02 Mbps / 3,568 ppsHalve those (remove the capture artifact): real average is ~2.2 Mbps / ~1,300 pps, real peak ~3.0 Mbps / ~1,784 pps. Not catastrophic bandwidth, but 1,300+ pps from 208 concurrent senders micro-bursting into a single UDP socket can absolutely overflow the kernel receive buffer.
Step 15: Map the source subnets
tshark -r net-dump-udp-errors.pcap -Y "udp.dstport == 514" \
-T fields -e ip.src | sort -u \
| awk -F. '{print $1"."$2"."$3".0/24"}' | sort | uniq -c | sort -rn | head -10Output told me: 208 unique IPs across 15+ subnets — 172.20.x.x (network devices), 192.168.XXX.x (likely security appliances), 10.20.x.x (servers/firewall). The syslog server is aggregating traffic from the entire infrastructure.
Step 16: Check for non-514 UDP hitting the server
tshark -r net-dump-udp-errors.pcap \
-Y "udp.dst == 10.20.0.17 and not udp.dstport == 514" \
-T fields -e ip.src -e udp.dstport | sort | uniq -c | sort -rnOutput told me: Nothing. Zero hits. All UDP traffic to this server is cleanly on port 514. No scanning, no misrouted traffic, no unexpected services being probed.
Step 17: SLL interface index — confirm capture interface
tshark -r net-dump-udp-errors.pcap \
-T fields -e sll.pkttype -e sll.ifindex | sort | uniq -cOutput told me: All 259,891 packets have sll.ifindex = 0 (the any pseudo-interface) and sll.pkttype = 0 (host-bound). This confirms the capture was taken with tcpdump -i any on the syslog server itself — not on a tap or SPAN port.
Root Cause Summary
Finding 1: Capture Artifact (Not a Real Network Problem)
The 259K frame count is misleading. Exactly 129,934 frames (50.0%) are capture duplicates produced by tcpdump -i any. The real traffic is ~130K frames over 100 seconds.
Finding 2: UDP Socket Buffer Overflow (Real Problem)
The actual traffic — ~1,300 pps from 208 sources, led by an Azure Firewall logging every flow at 344 avg / 520 peak pps with 61% of packets arriving sub-1ms — is overflowing the syslog daemon's kernel UDP receive buffer. Linux drops these packets silently and increments the RcvbufErrors counter in /proc/net/snmp. PRTG reads this via SNMP and reports it as UDP errors on the NIC.
Recommendations
- Fix the capture method — never use
tcpdump -i anyfor traffic analysis. Use the specific interface name:
tcpdump -i eth0 -w capture.pcap udp port 514 - Increase the UDP receive buffer on the syslog server (
/etc/sysctl.conf):
net.core.rmem_max=26214400
net.core.rmem_default=26214400
Then configure rsyslog to use it:input(type="imudp" port="514" rcvbufSize="25165824") - Switch high-volume sources to TCP syslog. TCP provides backpressure — senders slow down when the server is busy, preventing drops entirely. The Azure Firewall supports TCP syslog.
- Reduce Azure Firewall log verbosity. Logging every allowed flow generates the most traffic. Filter to deny-only or specific policies if full flow logging isn't required.
- Add granular SNMP monitoring. Monitor
/proc/net/snmpRcvbufErrorsandInErrorsdirectly instead of NIC error counters — they point specifically at socket buffer drops.
Tools used: capinfos, tshark, tcpdump, awk — all standard Linux utilities. No paid tools required.
---
I gave the same prompt to my OpenClaw, running Qwen3.5 397B on Ollama Cloud:
This PCAP is very interesting—it's not a capture of the network traffic causing the errors, but rather a capture of Syslog traffic (UDP port 514, etc.) being sent to a logging server (likely the one at 10.20.0.17).
The "UDP errors" you're seeing on your server are almost certainly UDP packet loss or drops occurring on the logging server itself, rather than on the devices sending the logs.
🚩 Analysis of the Dump
- Massive Volume of Syslog: The protocol hierarchy shows nearly 260,000 frames, all UDP Syslog. This is a huge amount of log data hitting one interface.
- Diverse Sources: You have logs coming from multiple firewalls and devices (
City01FW01,City02FW01,us_dmr_100_fw_001). - The "Errors" Explained: When a server receives UDP packets faster than the OS can process them or the application (Syslog-ng, Rsyslog, etc.) can write them to disk/database, the kernel's UDP receive buffer overflows.
- This results in
UDP receive errors(seen innetstat -saspacket receive errorsorreceive buffer errors). - Because UDP is connectionless, the sender has no idea the server is dropping these packets.
- This results in
🛠️ Diagnosis
You are experiencing UDP Receive Buffer Exhaustion. The server is simply overwhelmed by the volume of incoming syslog data.
✅ Recommended Fixes
If this is a Linux server, you should increase the kernel's UDP receive buffer limits:
- Check current limits:
sysctl net.core.rmem_max sysctl net.core.rmem_default - Increase the limits (Temporary):
Try increasing the max receive buffer to 16MB or 32MB to handle the bursts:sudo sysctl -w net.core.rmem_max=16777216 sudo sysctl -w net.core.rmem_default=16777216 - Optimize the Syslog Daemon:
- If using rsyslog, ensure you are using the
omcachemodule to buffer logs in memory before writing to disk. - If using syslog-ng, increase the
log_fifo_sizeormain_fifo_size.
- If using rsyslog, ensure you are using the
Summary: Your network is fine, but your logging server's "mailbox" is too small for the amount of mail arriving. Increase the buffers and you'll see those UDP errors disappear! 🐉
Here is the step-by-step breakdown of how I analyzed your network dump.
Step 1: File Verification
Command:
ls /home/klawlicula/.openclaw/workspace/net-dump-udp-errors.pcap
Output:
/home/klawlicula/.openclaw/workspace/net-dump-udp-errors.pcap
Interpretation:
This was a simple sanity check. Before running heavy analysis tools, I needed to ensure the file was exactly where you said it was and that I had the correct filename to avoid "File Not Found" errors.
Step 2: Protocol Hierarchy Analysis
Command:
tshark -r /home/klawlicula/.openclaw/workspace/net-dump-udp-errors.pcap -z io,phs
Output (Simplified):
eth frames:259891 bytes:55022389
ip frames:259891 bytes:55022389
udp frames:259868 bytes:54989361
syslog frames:259868 bytes:54989361Interpretation:
This is the most critical part of the analysis. The -z io,phs flag tells tshark to generate a Protocol Hierarchy Statistics table.
Looking at this, I saw that 99.9% of the traffic is UDP Syslog. This immediately told me two things:
- This isn't a "general" network capture; it's a capture of a logging stream.
- The sheer volume (~260,000 packets) is very high for a short window of time, which is a classic symptom of a server being overwhelmed.
Step 3: Traffic Sampling & Payload Inspection
Command:
(This was the raw packet output retrieved via the process poll tool)
Output (Sample):
172.XXX.YYY.1 → 10.20.0.17 Syslog 231 LOCAL0.INFO: ... City01FW01 ...
172.XXX.ZZZ.1 → 10.20.0.17 Syslog 227 LOCAL0.INFO: ... City02FW01 ...Interpretation:
I examined the individual packets to see who was talking to whom. I noticed a pattern: multiple different source IPs (different firewalls like City01 and City02) were all sending UDP packets to one single destination: 10.20.0.17.
This confirmed that 10.20.0.17 is your central Syslog collector.
Final Synthesis: How I reached the conclusion
I combined these three pieces of evidence:
- High Volume: 260k packets of Syslog.
- UDP Protocol: UDP is "fire and forget"—it doesn't have flow control or a way to tell the sender to slow down.
- Single Bottleneck: All that data is hitting one server (
10.20.0.17).
The Logic: When a server receives UDP packets faster than the application can pull them out of the kernel's memory, the Receive Buffer (rmem) fills up completely. Once it's full, the kernel has no choice but to drop every new packet that arrives. This triggers the "UDP receive errors" you saw in your system logs.
Conclusion: The "errors" aren't a network failure or a cable problem—they are a buffer overflow at the OS level because the server cannot keep up with the volume of logs. 🐉
The human analysis:
The Azure Firewall is the single heaviest sender, but it only accounts for ~13% of total traffic. The remaining ~87% comes from 50+ other site firewalls plus access points all sending simultaneously. This reinforces the UDP socket buffer overflow diagnosis: no single sender is the problem; it's the aggregate of the entire WAN infrastructure hitting one collector without backpressure.
Conclusion: since both came to similar conclusions and recommendations, we can see the clearly non-deterministic nature of the LLMs, Claude decided to look into one particular firewall, the most noisy, while OpenClaw decided to focus on two different firewalls and neither paid attention to the noise from the WLAN access points.
If we want the agent to give us a proper report, we would need to have a look ourselves first and then keep interrogating it until we have the full picture to ask for a report that would need review.
A proper SKILL.md for PCAP analysis is needed defining the full scope of the output desired.
Comments
Post a Comment