diff --git a/src/content/docs/docs/network-monitoring/discover-devices.mdx b/src/content/docs/docs/network-monitoring/discover-devices.mdx index 27febb1c..978492a8 100644 --- a/src/content/docs/docs/network-monitoring/discover-devices.mdx +++ b/src/content/docs/docs/network-monitoring/discover-devices.mdx @@ -6,43 +6,76 @@ sidebar: order: 2 --- -Discovery is the quickest way to onboard a site. You give the agent a few IP ranges and the community strings to try, and it scans them, finds the devices that answer SNMP, and starts monitoring them automatically. For a handful of known addresses, [Add SNMP devices](/network-monitoring/monitoring-snmp/) is the more direct route. +Discovery is the quickest way to onboard a site. You give the agent a few IP ranges and the credentials to try, and it scans them, finds the devices that answer SNMP, and starts monitoring them automatically. For a handful of known addresses, [Add SNMP devices](/network-monitoring/monitoring-snmp/) is the more direct route. ## Before you start - A Linux or Docker KloudMate Agent with network access to the ranges you want to scan. Kubernetes agents can't run discovery. -- The v1/v2c community strings the devices use. Discovery tries these against every address in range. - -:::caution -Discovery credentials are v1/v2c only. To monitor a device that requires SNMPv3, add it directly with the [SNMP wizard](/network-monitoring/monitoring-snmp/). -::: +- The v1/v2c community strings or the SNMPv3 user the devices use. Discovery tries them against every address in range. +- For SNMPv3 discovery, exclusions, and **Follow neighbours**, agent 2.11.0 or later. See [Agent version](#agent-version). ## Add a scan In **Network Monitoring**, open the **Add** menu and choose **Discover devices**. Fill in the scan: -- **Ranges to scan**: one CIDR or range per line (or comma-separated), for example `10.0.0.0/24` or `10.0.1.1-50`. -- **Candidate communities**: the v1/v2c community strings to try, such as `public, private`. These are sent as plain text and used only for discovery. -- **Version**: v2c or v1. -- **Scan interval**: how often to re-scan, for example `1h`. +- **Ranges to scan**: one CIDR or range per line (or comma-separated), for example `10.0.0.0/24`, `10.0.1.1-50`, or `10.0.1.1-10.0.3.254`. Discovery scans IPv4 only, and the ranges in one scan can cover at most 65,536 addresses in total, the size of a `/16`. +- **Exclude (optional)**: addresses the scan never probes, one IP, CIDR, or range per line. See [Exclude addresses](#exclude-addresses). +- **Version**: v2c, v1, or v3. +- **Candidate communities**: for v1 and v2c, the community strings to try, such as `public, private`. +- **SNMPv3 user**: for v3, a **Username** and **Security level** (`noAuthNoPriv`, `authNoPriv`, or `authPriv`), plus the **Auth protocol** and password and the **Priv protocol** and password that the level needs. A scan tries one v3 user, so add a scan per user if your devices use several. +- **Scan interval**: how often to re-scan, for example `1h`. The minimum is `10m`, and a shorter interval is raised to it. - **Port**: the SNMP port to probe, `161` by default. +- **Follow neighbours**: also scan the LLDP and CDP neighbours of the devices the scan finds. See [Follow neighbours](#follow-neighbours). -You can add several scans on one agent, each with different ranges, ports, or communities, and the agent runs each on its own interval and combines the results. +Communities and v3 passwords are sent to the agent as plain text and used only for discovery. + +You can add several scans on one agent, each with different ranges, ports, or credentials, and the agent runs each on its own interval and combines the results. ![The Discover devices form with a collector agent, ranges to scan, candidate communities, version, scan interval, and port fields](./images/discover-add-scan.png) +## Exclude addresses + +Use **Exclude** for addresses a scan must never touch, such as a fragile appliance or a network that belongs to someone else. For example, scanning `10.0.0.0/24` with `10.0.0.128/25` excluded probes only `10.0.0.1` to `10.0.0.127`. + +The agent checks each address against the exclusions instead of listing every excluded address, so a wide prefix such as `10.0.0.0/8` works and has no size limit. An excluded CIDR also covers its network and broadcast addresses. Exclusions apply to neighbours that **Follow neighbours** finds, too. + +## Follow neighbours + +Turn on **Follow neighbours** to reach devices outside your ranges through the LLDP and CDP tables of the devices you've already found. The scan reads each found device's neighbour management addresses with the credential that answered it, probes those addresses with the scan's credentials, and repeats this for every new device, until no new addresses turn up. + +The crawl stays contained: + +- It follows only private IPv4 addresses, in `10.0.0.0/8`, `172.16.0.0/12`, and `192.168.0.0/16`. +- It never probes an excluded address. +- It stops once the scan has found 256 devices, the same cap that limits how many devices one scan monitors. + ## What happens next -Every device a scan finds is monitored automatically, using the community that answered it. The agent applies its updated configuration to start polling, which briefly restarts data collection on that agent. It only restarts when the set of monitored devices actually changes, so a steady scan that keeps finding the same devices does nothing. +Every device a scan finds is monitored automatically, using the credential that answered it. Results appear after the next scan interval. + +On agent 2.11.0 or later, newly found devices are added to the running agent without restarting data collection, unless they're the first SNMP devices on that agent. Older agents briefly restart data collection whenever the set of monitored devices changes. Either way, a steady scan that keeps finding the same devices changes nothing. :::caution A single scan monitors up to 256 devices. Point scans at focused ranges rather than an entire `/16`, or a large range will leave some devices unmonitored. ::: -The **Discovered devices** panel lists what the scans found: address, name, vendor, description, and whether each is reachable. Results appear after the next scan interval, and **Refresh** reloads them. Monitored devices also show up on the **All devices** tab like any other SNMP device. +The **Discovered devices** panel lists what the scans found: address, name, vendor, description, and whether each is reachable. **Refresh** reloads it. Monitored devices also show up on the **All devices** tab like any other SNMP device. ![The Discovered devices table with address, name, vendor, description, and reachable columns](./images/discovered-devices.png) +## When one device answers on several addresses + +Routers and layer-3 switches often answer SNMP on more than one of their addresses. A scan treats two addresses as the same device only when they report the same LLDP chassis ID (`lldpLocChassisId`) or, on SNMPv3, the same engine ID. It then keeps one address for that device and monitors it once. + +A device that reports neither, such as a v2c device without LLDP, can't be matched, so every address it answers on is listed and monitored as a separate device. To avoid the duplicates, exclude the extra addresses. + ## Adjust what's monitored Discovery is sticky. A device stays monitored even if a later scan misses it once, so a brief blip doesn't make it vanish. Instead it reports **Down**, which is the accurate state. To stop monitoring devices, narrow or remove the range that covers them (or delete the scan) from **Configure**. The agent then re-accumulates from the ranges that remain. See [Manage and troubleshoot](/network-monitoring/manage-troubleshoot/). + +## Agent version + +SNMPv3 discovery, **Exclude**, and **Follow neighbours** need agent 2.11.0 or later. On an older agent, or one whose version isn't known, the form disables these options, apart from clearing a value a scan already has. If a scan uses them and its agent is older: + +- A scan with exclusions or a v3 user doesn't run at all on that agent, so it never probes addresses you excluded. It runs once the agent is on 2.11.0 or later. +- **Follow neighbours** is dropped, and the scan runs without it. diff --git a/src/content/docs/docs/network-monitoring/flow-monitoring.mdx b/src/content/docs/docs/network-monitoring/flow-monitoring.mdx index 49045cfa..c2a57189 100644 --- a/src/content/docs/docs/network-monitoring/flow-monitoring.mdx +++ b/src/content/docs/docs/network-monitoring/flow-monitoring.mdx @@ -1,14 +1,16 @@ --- title: "Flow Monitoring (NetFlow & sFlow)" -description: "See the application traffic mix on your network links with NetFlow, IPFIX, and sFlow in KloudMate." +description: "See the application traffic mix, top talkers, and the services behind the traffic on your network links with NetFlow, IPFIX, and sFlow in KloudMate." sidebar: label: "Flow monitoring" order: 4 --- -SNMP shows how much traffic crosses an interface, but not what that traffic is. Flow monitoring fills that gap. It breaks the load down by application, across HTTPS, streaming, DNS, database, and the rest. You add a listener on the agent, point your devices' flow export at it, and the mix shows up on the **Traffic** tab. +import { Tabs, TabItem } from '@astrojs/starlight/components'; -The agent listens for NetFlow, IPFIX, and sFlow. It classifies each flow by its well-known port and rolls the flows into per-application byte and packet counts. +SNMP shows how much traffic crosses an interface, but not what that traffic is. Flow monitoring fills that gap. It breaks the load down by application, across HTTPS, streaming, DNS, database, and the rest, ranks the subnets and hosts that send and receive the most, and names the services and CDNs behind the traffic. You add a listener on the agent, point your devices' flow export at it, and the results show up on the **Traffic** tab. + +The agent listens for NetFlow, IPFIX, and sFlow. It classifies each flow by its well-known port and rolls the flows into per-application byte and packet counts. Top talkers and service names need agent 2.11.0 or later. ## Add a listener @@ -17,7 +19,9 @@ In **Network Monitoring**, open the **Add** menu and choose **Flow monitoring**: - **Flow protocol**: **NetFlow / IPFIX** or **sFlow**. - **Listener port**: the UDP port the agent listens on. It defaults to `2055` for NetFlow and `6343` for sFlow. -Add one listener per protocol your devices export. NetFlow and sFlow can run at the same time on different ports, and the agent combines their results. Then configure each device to send flow records to `agent-ip:port`. +Add one listener per protocol your devices export. NetFlow and sFlow can run at the same time on different ports, and the agent combines their results. Every listener on an agent needs its own port, and that includes [trap and syslog receivers](/network-monitoring/traps-and-syslog/). If two listeners share a port, only one of them is set up on the agent. + +Then configure each device to send flow records to `agent-ip:port`. The form shows an **Exporter setup** snippet for Cisco, Juniper, MikroTik, and Fortinet, filled in with the listener's port. ![The Flow monitoring form with a flow protocol selector, a listener port, a Ship raw flow logs toggle, and a sample-rate field](./images/flow-monitoring-drawer.png) @@ -25,6 +29,136 @@ Add one listener per protocol your devices export. NetFlow and sFlow can run at Flow volume can be high, so turn on sampling at the sending device rather than exporting every flow. Applications are identified by well-known port, not by deep packet inspection. All traffic on port 443 is labeled `https`, so an app on a non-standard port shows up as `other`. ::: +## Configure the exporting device + +These are starting configurations for the default ports. Replace `` with the agent's address, use your listener port if you changed it, and replace the interface names with your own. + + + + +NetFlow v9 on IOS XE, with Flexible NetFlow. Apply the monitor on each interface where traffic enters: + +```text +flow record KM-RECORD + match ipv4 source address + match ipv4 destination address + match ipv4 protocol + match transport source-port + match transport destination-port + match interface input + collect interface output + collect counter bytes long + collect counter packets long + collect timestamp absolute first + collect timestamp absolute last +! +flow exporter KM-EXPORTER + destination + transport udp 2055 + export-protocol netflow-v9 + template data timeout 60 +! +flow monitor KM-MONITOR + record KM-RECORD + exporter KM-EXPORTER + cache timeout active 60 +! +interface GigabitEthernet0/0/0 + ip flow monitor KM-MONITOR input +``` + +sFlow on NX-OS: + +```text +feature sflow +sflow agent-ip +sflow collector-ip vrf default +sflow collector-port 6343 +sflow sampling-rate 4096 +sflow data-source interface ethernet1/1 +``` + + + + +IPFIX with inline J-Flow on MX routers: + +```text +set services flow-monitoring version-ipfix template km-ipv4 ipv4-template +set services flow-monitoring version-ipfix template km-ipv4 flow-active-timeout 60 +set services flow-monitoring version-ipfix template km-ipv4 flow-inactive-timeout 15 +set forwarding-options sampling instance km-sample input rate 1000 +set forwarding-options sampling instance km-sample family inet output flow-server port 2055 +set forwarding-options sampling instance km-sample family inet output flow-server version-ipfix template km-ipv4 +set forwarding-options sampling instance km-sample family inet output inline-jflow source-address +set chassis fpc 0 sampling-instance km-sample +set interfaces ge-0/0/0 unit 0 family inet sampling input +``` + +sFlow on EX and QFX switches: + +```text +set protocols sflow collector udp-port 6343 +set protocols sflow sample-rate ingress 2048 +set protocols sflow polling-interval 20 +set protocols sflow interfaces ge-0/0/0 +``` + + + + +NetFlow v9 on RouterOS. MikroTik devices send NetFlow or IPFIX, so use a NetFlow listener for them: + +```text +/ip traffic-flow set enabled=yes interfaces=all active-flow-timeout=1m inactive-flow-timeout=15s +/ip traffic-flow target add dst-address= port=2055 version=9 +``` + + + + +NetFlow on FortiOS 7.2.8 and later. Earlier versions set `collector-ip` and `collector-port` directly under `config system netflow`: + +```text +config system netflow + config collectors + edit 1 + set collector-ip + set collector-port 2055 + next + end + set active-flow-timeout 1 + set inactive-flow-timeout 15 +end +config system interface + edit "port1" + set netflow-sampler rx + next +end +``` + +sFlow on FortiOS 7.2.8 and later. Earlier versions set `collector-ip` and `collector-port` directly under `config system sflow`: + +```text +config system sflow + config collectors + edit 1 + set collector-ip + set collector-port 6343 + next + end +end +config system interface + edit "port1" + set sflow-sampler enable + set sample-rate 2000 + next +end +``` + + + + ## What you get Both metrics are byte and packet totals per application, per interval: @@ -34,17 +168,80 @@ Both metrics are byte and packet totals per application, per interval: Each carries three attributes: `app` (`https`, `dns`, `voip_sip`, `database`, and so on, plus `other`), `transport` (`tcp` or `udp`), and `exporter`, the management IP of the device that sent the flow. Because `exporter` matches a device's address, KloudMate ties the traffic mix back to the SNMP device that carries it, so a saturated interface and its application breakdown sit in one place. +Agent 2.11.0 or later also emits: + +- `netflow_talker_src_bytes` and `netflow_talker_dst_bytes`, by `src_subnet` or `dst_subnet` (a `/24` for IPv4, a `/48` for IPv6). +- `netflow_talker_src_ip_bytes` and `netflow_talker_dst_ip_bytes`, by `src_ip` or `dst_ip`. +- `netflow_service_bytes`, by `service`, `category`, and `classified_by`. +- `netflow_cdn_bytes`, by `cdn`, `category`, and `classified_by`. + +Each has a matching `_packets` metric, and each carries `direction` and `exporter`. + ## Where results show up -Open the **Traffic** tab for the workspace-wide view: top applications, top transports, and totals over the selected window. Each device drawer also has a **Traffic** tab, scoped to the flow that device exports. +Open the **Traffic** tab for the workspace-wide view: top applications, top transports, top talkers, services and CDNs, and flow health over the selected window. Each device drawer also has a **Traffic** tab, scoped to the flow that device exports. ![The Traffic tab showing total traffic, total packets, and average throughput tiles above a stacked traffic-over-time-by-application chart](./images/traffic-tab.png) ## Find the top talkers -The per-application metric answers "what apps are on this link," but it drops the source and destination addresses, so it can't tell you which host is the bandwidth hog. To answer that, turn on raw flow logs when you add or edit a listener: +The **Top talkers** panel ranks the subnets and hosts that move the most traffic. Switch between **Source subnets**, **Destination subnets**, **Source hosts**, and **Destination hosts**. A subnet is the `/24` (IPv4) or `/48` (IPv6) that an address falls in, so you can spot a busy office network first and then drill down to its hosts. + +Each agent reports only its busiest talkers every minute and groups the rest, and the panel leaves that group out. When the exporter samples flows, the agent scales the sampled volumes up, so the numbers are estimates rather than exact byte counts. + +### Compare two time windows + +To see what changed, pick a baseline: **Previous period**, **Same time yesterday**, or **Same time last week**. The table shows each talker's bytes in the selected window and in the baseline window, plus the change as a percentage. A talker with no traffic in the baseline is marked **New**, and one with no traffic now is marked **Gone**. + +For example, with the last hour selected and **Same time last week** as the baseline, a host that started a large transfer this week appears on **Source hosts** with a large increase, or as **New**. + +## Name services and CDNs + +On agent 2.11.0 or later, the agent names the service at the remote end of each flow, such as `youtube`, `netflix`, `microsoft`, or `zoom`, with a category such as `video`, `cloud`, or `conferencing`. It checks, in order: + +1. **Address prefix**: a curated list of prefixes, such as the public DNS resolvers of Google, Cloudflare, and Quad9, and Amazon CloudFront's published ranges. +2. **Network (ASN)**: the remote network's autonomous system number, matched against a curated list of the networks that carry one operator's traffic. The ASN comes from the router's flow export, or from the IP-to-ASN dataset when the router sends none. +3. **Port**: a flow that matches neither gets no service name. It keeps its application from the well-known port. + +Delivery networks such as Cloudflare, Akamai, Fastly, and CloudFront carry traffic for many different customers, so flows to them are named only by CDN, with no service. Operators that deliver their own content, such as Google, Netflix, Meta, and Apple, get both a service and a CDN name. + +The **Services and CDNs** section of the Traffic tab shows: + +- **Top services**: named services and their category. Traffic without a service name is left out. +- **Top CDNs**: delivery networks, including operators that deliver their own content. +- **How services were named**: the share of bytes named **By address prefix**, **By network (ASN)**, or **By port**, plus the traffic left **Not named** and the traffic **Grouped past the top list**. + +Check **How services were named** before you rely on the rankings. A large **Not named** or **By port** share means most of your traffic goes to networks the curated lists don't cover, so the top services only describe the part that was named. + +### The IP-to-ASN dataset + +When a router doesn't export ASNs, which is common without a full BGP table, the agent looks them up in the [IPtoASN](https://iptoasn.com/) dataset. It downloads `https://iptoasn.com/data/ip2asn-combined.tsv.gz` about once a week and keeps it in its data directory, `/var/lib/kmagent` on Linux. It downloads the dataset only when the agent has a flow listener. + +The agent host needs outbound HTTPS access to `iptoasn.com`. If a download fails, the agent keeps using the last dataset it has and tries again 6 hours later. Until the first download succeeds, flows from routers that don't export ASNs can't be named by network. + +## Check flow health + +The **Flow health** section of the Traffic tab tells you whether flow is arriving and how much of the interface traffic it covers. + +**Flow listeners** shows one row per agent over the selected window: + +| Column | What it counts | +|---|---| +| **Records** | Flow records the agent decoded. | +| **Dropped** | Flow packets the agent received but never decoded, for example because its queue was full. | +| **Kernel drops** | Flow packets the kernel dropped at the listener's socket because the agent didn't read them in time. Linux only. | +| **Not matched to SNMP** | Records whose interface matched no SNMP-polled interface, so they have no interface name. | +| **Sampling unknown** | Records with no sampling rate the agent could resolve. Their volumes undercount. | + +An agent shows **Dropping** when any packets were dropped or kernel-dropped, and **No records** when nothing arrived. + +**Flow against SNMP** compares the flow bytes on each interface with the SNMP in and out octets for the same interface. An interface that exports all of its traffic with the right sampling rate comes out near 100%. A share well under 100% usually means the interface doesn't export flow, or the sampling rate is wrong. The panel only lists interfaces on devices that both export flow and are polled over SNMP from the same address. It also appears on each device's **Traffic** tab. + +## Search individual flows with raw flow logs + +The metrics above don't keep individual flows. To search single flows by address and port, turn on raw flow logs when you add or edit a listener: -- **Ship raw flow logs**: off by default. When on, the agent also stores each flow as a log with its source and destination address and port, so you can rank the top talkers and search individual flows. +- **Ship raw flow logs**: off by default. When on, the agent also stores each flow as a log with its source and destination address and port. - **Sample rate (%)**: the percentage of flows to keep as logs, from 1 to 100. Leave it blank to keep all of them. -Raw flow logs carry more volume than the per-app metrics, so sample them on a busy link. With them enabled, a **Top talkers** view appears on the Traffic tab, ranking sources, destinations, and conversations. +Raw flow logs carry more volume than the metrics, so sample them on a busy link. With them enabled, the Traffic tab's **Top hosts from raw flow logs** section ranks top sources, destinations, and conversations from those logs. diff --git a/src/content/docs/docs/network-monitoring/index.mdx b/src/content/docs/docs/network-monitoring/index.mdx index 31476595..b82d8858 100644 --- a/src/content/docs/docs/network-monitoring/index.mdx +++ b/src/content/docs/docs/network-monitoring/index.mdx @@ -1,25 +1,26 @@ --- title: "Network Monitoring" -description: "Monitor SNMP devices, site reachability, and traffic flow across your network with the KloudMate Agent." +description: "Monitor SNMP devices, site reachability, traffic flow, and SNMP traps and syslog across your network with the KloudMate Agent." sidebar: label: "Overview" order: 0 --- -Use **Network Monitoring** to watch the routers, switches, firewalls, access points, and PDUs that carry your traffic. You poll them over SNMP for health and interface metrics, check reachability for the sites that can't speak SNMP, and read the application traffic mix from NetFlow or sFlow. +Use **Network Monitoring** to watch the routers, switches, firewalls, access points, and PDUs that carry your traffic. You poll them over SNMP for health and interface metrics, check reachability for the sites that can't speak SNMP, read the traffic mix from NetFlow or sFlow, and collect the traps and syslog your devices send. A Linux or Docker [KloudMate Agent](/kloudmate-agent/) does the collecting. It polls devices over the network, so one agent can watch an entire site, and nothing gets installed on the devices themselves. :::note -Network monitoring runs on Linux and Docker agents only. Kubernetes agents can't poll SNMP, run reachability checks, or listen for flow data. +Network monitoring runs on Linux and Docker agents only. Kubernetes agents can't poll SNMP, run reachability checks, or listen for flow, traps, or syslog. ::: ## What you can monitor - **SNMP devices**: interface throughput and errors, CPU, memory, uptime, and vendor-specific health. Built-in profiles cover Cisco, Juniper, Arista, HPE, Aruba, Dell, Fortinet, Palo Alto, MikroTik, F5, APC, and many more. -- **Sites without SNMP**: a gateway-and-internet ping pair that shows whether the local network is up and whether the ISP link is working. -- **Traffic flow**: the per-application byte and packet mix on a link, from NetFlow, IPFIX, or sFlow exporters. -- **Routing and links**: BGP peers, MPLS L3VPN VRFs, IP SLA probes, and LLDP/CDP topology, once you turn those capabilities on. +- **Sites without SNMP**: a gateway-and-internet ping pair that shows whether the local network is up and whether the ISP link is working, with jitter and a hop-by-hop path to the internet. +- **Traffic flow**: the per-application byte and packet mix on a link, the top talkers, and the services and CDNs behind the traffic, from NetFlow, IPFIX, or sFlow exporters. +- **Device events**: SNMP traps and syslog, matched to the device that sent them. +- **Routing and links**: LLDP and CDP topology from every device, plus BGP peers, MPLS L3VPN VRFs, and IP SLA probes once you turn those capabilities on. ## The Network Monitoring tabs @@ -27,11 +28,12 @@ Open **Network Monitoring**. You land on the **Overview** tab. | Tab | What it shows | |---|---| -| **Overview** | Network-wide health: problem tiles, a live problems feed, and a status board grouped by site, role, or vendor. | +| **Overview** | Network-wide health: overall status and uptime, the busiest ports, the probable cause of an outage, current problems, and every device grouped by site, role, or vendor. | | **All devices** | The full, searchable table of SNMP devices with their current status and last-seen time. | -| **Reachability** | Ping results for sites monitored without SNMP. | -| **Traffic** | The application traffic mix from your flow listeners. | -| **Topology** | A link map drawn from LLDP and CDP neighbour data. | +| **Reachability** | Ping results, jitter, and traced paths for sites monitored without SNMP. | +| **Traffic** | The application mix, top talkers, services, and flow health from your flow listeners. | +| **Topology** | A link map drawn from LLDP and CDP neighbour data, with the links that were lost. | +| **SLA and capacity** | Availability per site and device, and p95 utilization per port, over a period. | For how to read each tab, see [Using Network Monitoring](/network-monitoring/network-page/). @@ -45,6 +47,7 @@ Everything is configured in **Network Monitoring**. There are no collector files - [Discover devices](/network-monitoring/discover-devices/): scan IP ranges and monitor what answers. - [Monitor sites without SNMP](/network-monitoring/reachability-checks/): add gateway and internet ping checks. - [Flow monitoring](/network-monitoring/flow-monitoring/): listen for NetFlow, IPFIX, or sFlow. +- [Traps and syslog](/network-monitoring/traps-and-syslog/): receive SNMP traps and syslog from your devices. To change or remove anything later, open **Configure**. See [Manage and troubleshoot](/network-monitoring/manage-troubleshoot/). diff --git a/src/content/docs/docs/network-monitoring/manage-troubleshoot.mdx b/src/content/docs/docs/network-monitoring/manage-troubleshoot.mdx index 7330eb32..7f573384 100644 --- a/src/content/docs/docs/network-monitoring/manage-troubleshoot.mdx +++ b/src/content/docs/docs/network-monitoring/manage-troubleshoot.mdx @@ -1,24 +1,24 @@ --- title: "Manage and Troubleshoot Network Monitoring" -description: "Edit, disable, or remove network monitoring, and fix devices that aren't reporting in KloudMate." +description: "Edit, disable, or remove network monitoring, and fix devices, listeners, and receivers that aren't reporting in KloudMate." sidebar: label: "Manage & troubleshoot" - order: 6 + order: 7 --- -SNMP devices, ping probes, flow listeners, and discovery scans are all managed from one place. This page covers changing what you've configured and fixing the common reasons a device isn't reporting. +SNMP devices, ping probes, flow listeners, discovery scans, and trap and syslog receivers are all managed from one place. This page covers changing what you've configured and fixing the common reasons something isn't reporting. ## Manage what's configured -In **Network Monitoring**, select **Configure**. Pick a collector agent to see everything it's set to monitor, grouped by kind: **SNMP device**, **Ping probe**, **Flow monitoring**, and **Discovery**. For any entry you can: +In **Network Monitoring**, select **Configure**. Pick a collector agent to see everything it's set to monitor, grouped by kind: **SNMP device**, **Ping probe**, **Flow monitoring**, **Discovery**, **SNMP traps**, and **Syslog**. For any entry you can: - **Enable or disable** it. Disabling stops the polling without deleting the configuration, so you can turn it back on later. -- **Edit** it to change credentials, targets, capabilities, ports, or scan ranges. +- **Edit** it to change credentials, targets, capabilities, polling intervals, ports, or scan ranges. - **Remove** it. ![The Configure drawer, Configured network monitoring, listing ping, discovery, and flow entries with per-row enable toggles and edit and delete controls](./images/configure-integrations.png) -Enabling and disabling take effect on their own. Any edit that changes what the agent polls, and any add or remove, briefly restarts data collection on that agent, so expect a short gap in that poller's telemetry. +Enabling and disabling take effect on their own. Any edit that changes what the agent collects, and any add or remove, briefly restarts data collection on that agent, so expect a short gap in that poller's telemetry. The exception is on agent 2.11.0 or later: a change that only affects SNMP devices is applied without a restart, unless it adds the agent's first SNMP device or removes its last one. ## An SNMP device won't report @@ -27,7 +27,7 @@ If a device stays **Down** or never appears on the **All devices** tab, work dow 1. **Reachability**: the agent host must have a network path to the device on UDP port 161. Check firewalls and routing between them. 2. **Credentials**: confirm the community string (v1/v2c) or v3 user, and that the version matches what the device expects. A v2c community won't authenticate against a v3-only device. 3. **SNMP access list**: many devices restrict SNMP by source IP. Add the agent host's address to the device's allowed list, or no metrics come back even when everything else is correct. -4. **Agent placement**: network monitoring runs on Linux and Docker agents only. A Kubernetes agent is rejected for SNMP, ping, flow, and discovery. +4. **Agent placement**: network monitoring runs on Linux and Docker agents only. A Kubernetes agent is rejected for SNMP, ping, flow, discovery, traps, and syslog. 5. **SNMP enabled**: confirm the SNMP service is actually running on the device. ## A device shows "Not reporting" @@ -37,16 +37,58 @@ If a device stays **Down** or never appears on the **All devices** tab, work dow - The **poller** is offline. Check the Overview for a poller-not-reporting banner. If the agent stopped sending its heartbeat, every device it watches goes stale until it comes back. - The device genuinely stopped responding and its last data aged out. Confirm reachability and credentials as above. +A device turns **Not reporting** once its last data is more than three polling intervals old, and at least 5 minutes old. A device polled every 15 minutes therefore takes 45 minutes to be flagged. + Discovery is sticky, so a device found by a scan stays listed even after it stops answering. That's deliberate, so you don't lose sight of it. ## A reachability check won't run The gateway probe uses ICMP, which needs `CAP_NET_RAW` or root on the agent host. Running the agent in Docker without `--cap-add=NET_RAW` is the usual cause of a gateway probe that never reports. The internet probe uses TCP and doesn't need that permission, so if only the gateway probe is failing, check ICMP privilege first. See [Monitor sites without SNMP](/network-monitoring/reachability-checks/). +If the **Path** panel stays empty with **Trace path** on, check that the agent is on 2.11.0 or later, runs on Linux, and can open raw sockets. Without raw socket rights, the agent logs `km_ping: no path metrics from this host`. On a Linux host, run `journalctl -u kmagent | grep "no path metrics"`. + ## Flow monitoring shows no traffic - Confirm each device's flow export points at the agent's IP and the **listener port** you configured (`2055` for NetFlow, `6343` for sFlow by default). - Check that the device is actually exporting flows and that sampling isn't set so aggressively that little arrives. - NetFlow and sFlow need separate listeners on separate ports. A device exporting sFlow to a NetFlow listener won't be understood. +- Check the **Flow listeners** panel on the **Traffic** tab. **No records** means the agent decoded no flow. **Dropped** and **Kernel drops** mean flow arrived but some of it was lost. See [Check flow health](/network-monitoring/flow-monitoring/#check-flow-health). + +If traffic arrives but most of it isn't named by service, and the router doesn't export ASNs, check that the agent host can reach `https://iptoasn.com` over HTTPS. See [The IP-to-ASN dataset](/network-monitoring/flow-monitoring/#the-ip-to-asn-dataset). + +If a device is reachable but a specific metric is missing, check that the matching capability is on. BGP, MPLS L3VPN, and IP SLA metrics only appear when you enable those capabilities on the device in the [SNMP wizard](/network-monitoring/monitoring-snmp/). LLDP and CDP neighbours need no capability on agent 2.11.0 or later. + +## Listener ports clash + +Every flow listener, trap receiver, and syslog receiver on an agent needs its own port, whatever its protocol. The form won't save a port that another listener of the same kind already uses, or a UDP port that another UDP listener uses. If two listeners still end up on one port, for example a TCP syslog receiver on a flow listener's port, only one of them is set up on the agent. + +A port can also be held by something outside KloudMate, such as a syslog daemon on the agent host that's set to listen on `514`. The agent then leaves that receiver out and logs `network events: receiver left out` with the reason. Stop the other process or pick a different port. On a Linux host, run `journalctl -u kmagent | grep "network events"`. In Docker, run `docker logs km-agent 2>&1 | grep "network events"`. + +## Traps or syslog don't arrive + +Work down this list: + +1. **Agent version**: receivers need agent 2.11.0 or later. An older agent ignores them. +2. **Destination**: the device must send to the agent host's address and the receiver's port, over the right protocol. Open that port in any firewall between them. +3. **Community**: for traps, the device's community must be on the receiver's **Communities** list, or the list must be blank. Traps with other communities are rejected without being counted. +4. **Binding**: check the agent's log for `network events: receiver left out`, as described in [Listener ports clash](#listener-ports-clash). + +If events show up in [Logs](/logs/log-explorer/) but not on the device's **Events** tab, the agent couldn't match the sender's IP to an SNMP device. Poll the device over SNMP from the same agent that receives its events, and check that the address it sends from is one it reports. See [How events map to devices](/network-monitoring/traps-and-syslog/#how-events-map-to-devices). + +## Events are being dropped + +When an agent lost traps or syslog messages in the last hour, the **Traps and syslog** form shows a warning with the counts for that agent. The reason tells you where they were lost: `queue_full` when the trap queue was full during a burst, `refused` when the agent's logs pipeline didn't accept a trap, and `socket` when the kernel dropped packets before the agent read them. See [Dropped events](/network-monitoring/traps-and-syslog/#dropped-events). + +## Options are disabled on older agents + +These options need agent 2.11.0 or later. An agent whose version isn't known counts as older. + +| Option | On an older agent | +|---|---| +| **Polling interval** of 5 or 15 minutes | The field stays at 60 seconds. A slower interval saved earlier is dropped, and the device is polled every 60 seconds. | +| **Trace path** on a reachability check | The switch is disabled. Saved path settings are dropped, and the probe runs without tracing. | +| SNMPv3 and **Exclude** in discovery | Disabled in the form. A scan that already uses them doesn't run on that agent. | +| **Follow neighbours** in discovery | The switch is disabled. A saved setting is dropped, and the scan runs without it. | +| Trap and syslog receivers | The form won't add one. A saved receiver is ignored until the agent is upgraded. | -If a device is reachable but a specific metric is missing, check that the matching capability is on. BGP, MPLS L3VPN, LLDP, and IP SLA metrics only appear when you enable those capabilities on the device in the [SNMP wizard](/network-monitoring/monitoring-snmp/). +Data that older agents don't collect, such as CDP neighbours, per-VLAN MAC tables, jitter, top talkers, and service names, stays empty until you [upgrade the agent](/kloudmate-agent/upgrade/). diff --git a/src/content/docs/docs/network-monitoring/monitoring-snmp.mdx b/src/content/docs/docs/network-monitoring/monitoring-snmp.mdx index 02343704..f0969366 100644 --- a/src/content/docs/docs/network-monitoring/monitoring-snmp.mdx +++ b/src/content/docs/docs/network-monitoring/monitoring-snmp.mdx @@ -6,7 +6,7 @@ sidebar: order: 1 --- -Add SNMP devices when you already know the addresses you want to poll, such as a core switch, an edge router, a firewall, or a rack PDU. The **Add SNMP devices** wizard walks you through picking an agent, listing targets, entering credentials, and choosing a device profile. From there the agent polls each device on a schedule and reports its metrics to KloudMate. +Add SNMP devices when you already know the addresses you want to poll, such as a core switch, an edge router, a firewall, or a rack PDU. The **Add SNMP devices** wizard walks you through picking an agent, listing targets, entering credentials, and choosing a device profile and polling interval. From there the agent polls each device on a schedule and reports its metrics to KloudMate. To sweep a subnet and monitor whatever answers instead, use [Discover devices](/network-monitoring/discover-devices/). @@ -61,6 +61,14 @@ A device profile decides which tables the agent walks and what metrics it emits. ![The Profile step with the Auto-detect, Pick from catalog, and Custom options, and the optional Extra capabilities checkboxes for BGP peers, MPLS L3VPN, LLDP neighbours, and IP SLA](./images/snmp-wizard-profile.png) +#### Polling interval + +**Polling interval** sets how often the agent polls the devices in this batch: **Every 60 seconds** (the default), **Every 5 minutes**, or **Every 15 minutes**. A slower interval suits devices that don't need minute-level detail, such as access switches and PDUs. Every metric from the device, including `km_snmp_up`, arrives at that pace. MAC address tables are read every 5 minutes, or at the device's interval when it's longer. + +Device status follows the interval. A device shows as **Not reporting** only after its last data is more than three polling intervals old, and never sooner than 5 minutes, so a device polled every 15 minutes isn't flagged between polls. + +The 5- and 15-minute options need agent 2.11.0 or later. On an older agent the field stays at 60 seconds. To change the interval of devices you've already added, edit them from **Configure**. + #### Extra capabilities Under **Extra capabilities (optional)**, turn on tables that only matter for routers and layer-3 devices. Leave them off for plain hosts and switches, where they add polling work with no benefit. @@ -69,16 +77,17 @@ Under **Extra capabilities (optional)**, turn on tables that only matter for rou |---|---| | **BGP peers** | Peer state per neighbour, for edge and ISP routers. | | **MPLS L3VPN** | VRF operational status and route counts. | -| **LLDP neighbours** | Link-layer neighbour and port data, which feeds the [Topology](/network-monitoring/network-page/) map. | | **IP SLA** | Cisco IP SLA reachability probes, for per-WAN ISP verdicts. | +LLDP and CDP neighbours aren't on this list because the agent collects them from every device. See [Neighbours and MAC address tables](#neighbours-and-mac-address-tables). + ### 5. Tags -Add key-value tags such as `site`, `group`, or `role`, or anything you use to organize devices. They apply to every device in this batch and drive the filtering and grouping in Network Monitoring. Grouping the Overview status board by `site` or `role`, for example, only works if your devices carry that tag. +Add key-value tags such as `site`, `group`, or `role`, or anything you use to organize devices. They apply to every device in this batch and drive the filtering and grouping in Network Monitoring. Grouping the Overview's **Devices** table by `site` or `role`, for example, only works if your devices carry that tag. ### 6. Review & apply -Check the staged devices, then select **Apply configuration**. Saving briefly restarts data collection on that agent, so the wizard batches every device in the run and applies them together to keep the interruption short. +Check the staged devices, then select **Apply configuration**. On agent 2.11.0 or later, a change that only adds, edits, or removes SNMP devices reaches the running agent without restarting data collection. The exception is the first SNMP device on an agent, or removing the last one. Older agents briefly restart data collection on every save, so the wizard applies every device in the run together to keep the interruption short. ## What you get @@ -93,3 +102,24 @@ Within a few minutes the devices appear on the **All devices** tab with an **Up* Per device, you get `snmp_device_uptime`, `snmp_cpu_usage`, and `snmp_memory_usage` (plus `snmp_memory_used`, `_free`, and `_total`). The agent's own reachability check is `km_snmp_up`, which reads `1` when the device answers and `0` when it doesn't. Turn on capabilities and you also get `snmp_bgp_peer_state`, `snmp_mpls_l3vpn_vrf_oper_status`, `snmp_isp_reachability`, and related metrics. If a device stays **Down** or never appears, work through the checklist in [Manage and troubleshoot](/network-monitoring/manage-troubleshoot/). + +## Neighbours and MAC address tables + +On agent 2.11.0 or later, the agent reads the LLDP and CDP neighbour tables of every SNMP device, including devices found by [discovery](/network-monitoring/discover-devices/). There's nothing to turn on. Each neighbour becomes a `snmp_topology_link` series with `neighbor_device`, `neighbor_port`, `local_port_num`, and `discovery_protocol` (`lldp` or `cdp`), and these series draw the [Topology](/network-monitoring/network-page/#topology-tab) map. A neighbour that speaks both protocols is reported once by each, and the map shows them as one link. + +Agents older than 2.11.0 don't collect CDP, and they collect LLDP only for devices that were set up with the **LLDP neighbours** capability. + +Switches also report their MAC address tables, every 5 minutes or at the device's polling interval when that's longer: + +- `snmp_interface_mac_count`: how many addresses each port has learned, across all VLANs. +- `snmp_fdb_entry`: one series per learned address, with `mac_address`, `interface_name`, `vlan`, and `ip_address` when the device's own ARP table has it. + +On agent 2.11.0 or later, the agent reads every VLAN's addresses. It uses the Q-BRIDGE-MIB table where the switch has one. On a Cisco switch without it, the agent walks each active VLAN with `community@` on v1 and v2c, or the `vlan-` context on v3. Other switches report the plain BRIDGE-MIB table, which has no `vlan`. Older agents read only the plain table, which on Cisco covers only the default VLAN. + +Large tables have limits. A device with more than 5,000 learned addresses reports no `snmp_fdb_entry` series, and per-port counts stop past 20,000 table rows. These tables feed the port and host views described in [Using Network Monitoring](/network-monitoring/network-page/#hosts-on-switch-ports). + +## Sensors + +The device drawer's **Overview** tab has a **Sensors** panel with the device's temperature, fan, power, and voltage readings. Sensors in a fault state, such as a failed fan or a power supply that's off, are flagged and listed first within their type. + +The panel reads the sensor metrics that the device's profile already collects: Cisco environmental and FRU status, ENTITY-SENSOR-MIB sensors, and the Huawei, Aruba, MikroTik, and Dell OS10 sensor tables. If the profile collects none of these, the panel stays empty. Readings whose unit isn't fixed by the MIB, such as volts and watts, are shown as the device reports them. diff --git a/src/content/docs/docs/network-monitoring/network-page.mdx b/src/content/docs/docs/network-monitoring/network-page.mdx index aded6146..936e496e 100644 --- a/src/content/docs/docs/network-monitoring/network-page.mdx +++ b/src/content/docs/docs/network-monitoring/network-page.mdx @@ -1,33 +1,38 @@ --- title: "Using Network Monitoring" -description: "Read the Overview, drill into devices, and understand device statuses in KloudMate Network Monitoring." +description: "Read the Overview, find the probable cause of an outage, follow topology and link changes, place hosts on switch ports, and report availability and capacity in KloudMate Network Monitoring." sidebar: label: "Using Network Monitoring" - order: 5 + order: 6 --- -Once devices are reporting, you watch them in **Network Monitoring**. It opens on the **Overview** tab, built to answer one question fast: is anything wrong, and where? From there you can drill into any device. This page explains how to read each surface. +Once devices are reporting, you watch them in **Network Monitoring**. It opens on the **Overview** tab, built to answer one question fast: is anything wrong, and where? From there you can drill into any device or port. This page explains how to read each surface. ## Overview tab -The Overview leads with problems, not a wall of devices. +The Overview leads with problems, not a wall of devices. It covers the time range you pick, refreshes itself, and shows how long ago it updated. -**Health tiles** across the top count the current problems by kind: +The **health strip** across the top gives the overall state (**Healthy**, **Degraded**, or **Critical**), what's wrong in a few words, and fleet uptime over the range as a bar. Below it, counts summarize your devices by status, your ports by use, the active problems by severity, and the average traffic in and out. -- **Devices down** -- **Uplinks / ISP down** -- **BGP / VRF down** -- **Interfaces errored** +![The Overview tab with the Devices down / Uplinks / BGP-VRF / Interfaces errored tiles, a Current problems row marked Critical, and status-board cards grouped by site](./images/network-overview.png) -Selecting a tile filters the feed below to that category. +The rest of the page shows: -The **Current problems** feed lists active issues, worst first, one row per affected device. A device with several faults collapses into a single row with its most severe problem as the headline. Each row links to the device drawer. +- **Highest inbound traffic** and **Highest outbound traffic**: the busiest ports across all devices. Select a port to open it. +- **Probable cause**: when devices are down, where the outage most likely starts. See [Probable cause](#probable-cause). +- **Current problems**: active issues, worst first, one row per affected device. A device with several faults collapses into a single row with its most severe problem as the headline. Filter the feed by **Devices down**, **Uplinks / ISP down**, **Links down**, **BGP / VRF down**, or **Interfaces errored**. +- **Top applications**: the application mix from your flow listeners. +- **Devices**: every device grouped by **Site**, **Role**, or **Vendor**, with its status, availability over the range, ports, ports in use, and traffic. Devices that need attention come first in each group. -[Reachability-check sites](/network-monitoring/reachability-checks/) appear here too. A failed gateway probe shows as the gateway being unreachable; a failed internet probe as an uplink problem. A site that stops reporting entirely shows as **Site not reporting** — for a site with one internet link, that's how an outage surfaces, because the agent can't send a failure over the link that's down. +[Reachability-check sites](/network-monitoring/reachability-checks/) appear in the problems feed too. A failed gateway probe shows as the gateway being unreachable, and a failed internet probe as an uplink problem. A site that stops reporting entirely shows as **Site not reporting**. For a site with one internet link, that's how an outage surfaces, because the agent can't send a failure over the link that's down. -The **status board** groups your devices into cards. Switch the grouping between **Site**, **Role**, and **Vendor**. Each card takes the worst status among its devices and shows a `healthy / total` count, so you see both the state and how much is affected at a glance. Problem groups float to the top, and healthy groups collapse. +### Probable cause -![The Overview tab with the Devices down / Uplinks / BGP-VRF / Interfaces errored tiles, a Current problems row marked Critical, and status-board cards grouped by site](./images/network-overview.png) +When one or more SNMP devices are down, the **Probable cause** panel names the down device furthest upstream on the topology map. If every device a down device connects up through is also down, the outage is put down to that upstream device. For example, if a core switch goes down and takes three access switches with it, the panel names the core switch and lists the three access switches as also down below it. + +Under each cause, the panel lists evidence from the selected time range, newest first: links that stopped being reported, and [traps and syslog](/network-monitoring/traps-and-syslog/) from the affected devices. + +The panel is labelled **Rule-based**. It works from the topology map and the device states only, so treat the device it names as a place to start, not a confirmed root cause. Without LLDP or CDP links between your devices, it can't tell upstream from downstream, so each down device counts as its own cause. The panel appears only when a device is down and there's something to show: other down devices below it, or evidence. ## Device statuses @@ -37,45 +42,76 @@ Status appears as a color, a label, and an icon throughout the page: |---|---| | **Up** | The device is reachable and reporting. | | **Down** | A reachability check failed; the device didn't answer. | -| **Not reporting** | The device stopped sending data. It's still listed, with "last seen X ago." | +| **Not reporting** | The device's last data is more than three polling intervals old, and at least 5 minutes old. It's still listed, with "last seen X ago." | | **Warning** | The device is up but has an active issue, such as an errored interface. | | **Unknown** | Status can't be determined. | -**Not reporting** and **Down** are different on purpose. Down means the agent polled the device and got no answer. Not reporting means the data itself stopped arriving, often because the agent that polls it is offline rather than because the device is. +**Not reporting** and **Down** are different on purpose. Down means the agent polled the device and got no answer. Not reporting means the data itself stopped arriving, often because the agent that polls it is offline rather than because the device is. Because the threshold follows each device's [polling interval](/network-monitoring/monitoring-snmp/#polling-interval), a device polled every 15 minutes isn't flagged between polls. That distinction drives the **poller-not-reporting banner**. If a collector agent stops sending its heartbeat, the Overview shows a banner naming the affected poller instead of marking every device it watched as down. Device status under that agent may be stale until it reports again. ## All devices tab -**All devices** is the flat, searchable table, the drill view for power users. Filter by vendor, site, or status, sort by any column, and read each device's current status and **Last seen** time. A device that stopped reporting stays in the table as **Not reporting** rather than disappearing. +**All devices** is the flat, searchable list of every SNMP device, as cards or as a table. Filter by vendor, site, or status, and read each device's current status and **Last seen** time. A device that stopped reporting stays in the list as **Not reporting** rather than disappearing. + +To find where a machine is plugged in, enter its MAC address in **Find a device by MAC address**. The search covers the MAC tables that switches reported in the last 7 days. An address seen on several ports is usually plugged into the port with the fewest devices, and that port is listed first. ![The All devices tab: summary tiles for Devices, Up, Down, Not reporting, and Sites above a table of devices with vendor, site, address, status, and last-seen columns](./images/all-devices.png) ## Reachability and Traffic tabs -- **Reachability** shows the ping results for sites monitored without SNMP. See [Monitor sites without SNMP](/network-monitoring/reachability-checks/). -- **Traffic** shows the application traffic mix from your flow listeners. See [Flow monitoring](/network-monitoring/flow-monitoring/). +- **Reachability** shows the ping results, jitter, and traced paths for sites monitored without SNMP. See [Monitor sites without SNMP](/network-monitoring/reachability-checks/). +- **Traffic** shows the application mix, top talkers, services and CDNs, and flow health from your flow listeners. See [Flow monitoring](/network-monitoring/flow-monitoring/). ## Topology tab -The **Topology** tab draws a link map from LLDP and CDP neighbour data. Devices are nodes, neighbour links are edges, and both use the same status colors as the Overview. A down endpoint reddens its links so the fault path stands out. +The **Topology** tab draws a link map from LLDP and CDP neighbour data. Devices are nodes, neighbour links are edges, and both use the same status colors as the Overview. Gateways and routers sit at the top and access points at the bottom. A link whose port is down is drawn in red, and a lost link is dashed. Select a link to open its port. ![The Topology tab showing a monitored Chennai device linked to two unmonitored neighbour nodes, with a status legend for up, down, not reporting, and unmonitored](./images/topology.png) -Topology needs the **LLDP neighbours** capability turned on for a device, either in the [SNMP wizard](/network-monitoring/monitoring-snmp/) or by editing the device under **Configure**. Until then the map is empty. It's a secondary view for locating a fault, not the place you start. +On agent 2.11.0 or later, the agent collects LLDP and CDP neighbours from every SNMP device, so the map fills in without any setup. Older agents collect only LLDP, and only from devices that were set up with the **LLDP neighbours** capability. A neighbour that isn't polled over SNMP still appears on the map as an **Unmonitored** node. + +### Link changes + +The map includes every link reported in the last 7 days. The **Link changes** panel below it lists the links that have been lost: links that neither end has reported for more than three polling intervals of the slower device, and at least 15 minutes. For each lost link, it shows the ports at both ends, whether LLDP or CDP found it, whether either device is down or not reporting, and when the link was last seen. + +A lost link next to a device that's down or not reporting usually says more about the device than the link. A lost link between two healthy devices points at the link itself, such as a moved cable, a disabled port, or a neighbour protocol that was turned off. New links aren't listed, because only the time each link was last seen is kept. + +## SLA and capacity tab + +The **SLA and capacity** tab reports availability and port capacity over a period, the last 7 days by default. + +- **Availability by site**: the average availability of each site's devices. +- **Availability by device**: each device's availability, worst first, with a history bar. Select a device to open it. +- **Interface capacity**: the 95th percentile (p95) of each physical port's traffic rate over the period, in and out, against the port speed. **p95 utilization** uses the busier direction. Select a port to open it. + +A device's availability is the share of its polls that it answered over the period: the average of `km_snmp_up` across every poll. Time when nothing polled the device, such as while its agent was offline, is left out rather than counted as downtime. Availability below 99.9% is highlighted as a warning, and below 99% as an error. The history bar samples the period, so it can miss a short outage that the percentage still counts. + +Utilization above 60% is highlighted as a warning, and above 80% as an error. Because p95 ignores the busiest 5% of the period, a port with occasional bursts can show a low p95 while still saturating briefly. ## Device drawer -Selecting a device opens its drawer, with tabs that depend on what the device reports: +Selecting a device opens its drawer. The header shows the device's status, and the tabs depend on what the device reports: -- **Overview**: status, uptime, CPU and memory, and the device's active issues. +- **Overview**: the device's details (manufacturer, model, version, serial, address, uptime, how often it's polled, role, and site), its ports at a glance, active issues, the busiest ports, CPU, memory, errors and discards, top applications, [sensors](/network-monitoring/monitoring-snmp/#sensors), and the monitored hosts on its ports. - **Interfaces**: every interface with its operational state, sorted so down ports surface first. Admin-disabled ports are marked as disabled, not counted as faults. - **Routing**: BGP peers, MPLS L3VPN VRFs, and IP SLA results. This tab appears only when the device has those capabilities enabled. -- **Traffic**: the application mix this device exports, when flow monitoring is set up. -- **Tags**: the site, role, and other tags on the device. +- **Traffic**: the flow this device exports, when flow monitoring is set up. +- **Events**: the [traps and syslog](/network-monitoring/traps-and-syslog/#the-events-tab) this device sent. ![The device drawer Overview tab: an interfaces up/down/disabled summary, an active-issues list, and stat tiles for uptime, CPU, memory, and traffic](./images/device-drawer-overview.png) ![The device drawer Interfaces tab: a table of interfaces with status, IP, speed, throughput, utilization, and errors, sorted with down ports first](./images/device-drawer-interfaces.png) -The page auto-refreshes and stamps how long ago it updated, so you're always reading a recent picture. +Select a port anywhere on the page to open the port drawer. It shows the port's details, what it's connected to, link uptime, traffic and utilization, errors, the flow through it, and who's on it. + +## Hosts on switch ports + +Switches report the MAC addresses they've learned on each port (see [Neighbours and MAC address tables](/network-monitoring/monitoring-snmp/#neighbours-and-mac-address-tables)). KloudMate matches those addresses to the hosts that run a KloudMate Agent, so you can see which server is on which switch port. + +- **Monitored hosts on this device**, on a switch's **Overview** tab, lists the monitored hosts in its MAC table over the last 24 hours, with the port each is on. Select a host to open that port. +- **Who's on this port**, in the port drawer, lists every address the switch learned on the port in the last 24 hours, with the manufacturer, the IP address when known, the monitored host when it matches, and when it was last seen. + +A host is matched by MAC address first, then by IP address. MAC matching needs the host's agent on 2.11.0 or later, which reports the host's MAC addresses. IP matching only works when the switch's own ARP table has the host's address, which a layer-2 switch usually doesn't. An address that two hosts share, such as a Docker bridge address, matches neither host. + +A port with five or more learned addresses usually leads to other equipment, such as another switch or a hypervisor. Hosts found on such a port are marked **Behind this port**, because they're connected further down the line rather than plugged in directly. diff --git a/src/content/docs/docs/network-monitoring/reachability-checks.mdx b/src/content/docs/docs/network-monitoring/reachability-checks.mdx index 193d12b4..170ebd3b 100644 --- a/src/content/docs/docs/network-monitoring/reachability-checks.mdx +++ b/src/content/docs/docs/network-monitoring/reachability-checks.mdx @@ -1,12 +1,12 @@ --- title: "Monitor Sites Without SNMP" -description: "Watch sites where the router can't speak SNMP using gateway and internet reachability checks." +description: "Watch sites where the router can't speak SNMP using gateway and internet reachability checks, with jitter and a hop-by-hop path to the internet." sidebar: label: "Sites without SNMP" order: 3 --- -Not every site has a device you can poll. A branch office might sit behind a consumer router or a cheap gateway that doesn't support SNMP. For those sites, add a **reachability check**: a pair of ping probes that tell you whether the local network is up and whether the internet connection is working. +Not every site has a device you can poll. A branch office might sit behind a consumer router or a cheap gateway that doesn't support SNMP. For those sites, add a **reachability check**: a pair of ping probes that tell you whether the local network is up and whether the internet connection is working. You can also track jitter and trace the internet probe's path hop by hop. ## How the two probes split the problem @@ -33,6 +33,7 @@ In **Network Monitoring**, open the **Add** menu and choose **Reachability check - **Agent**: pick the agent running **at this site**. The gateway probe uses this agent's own gateway. - **Site name**: a label such as `hotel-blr-01`. It applies to both probes so they group under one site. - **Public host:port**: a reliable public endpoint to connect to, for example `1.1.1.1:443`. A successful connection means the internet path is working. +- **Trace path** (optional): record each hop between the agent and the public host. See [Trace the path to the internet](#trace-the-path-to-the-internet). The agent discovers and pings its own default gateway. Saving adds both probes and briefly restarts data collection on the agent. @@ -40,14 +41,51 @@ The agent discovers and pings its own default gateway. Saving adds both probes a ## Where results show up -Open **Network Monitoring → Reachability**. Each probe is a row with its endpoint, site, role (gateway or internet), protocol, round-trip time, packet loss, status, and when it was last seen. Summary tiles at the top count how many checks are up, down, and not reporting. +Open **Network Monitoring → Reachability**. Each probe is a row with its endpoint, site, role (gateway or internet), protocol, round-trip time, jitter, packet loss, status, and when it was last seen. Summary tiles at the top count how many checks are up, down, and not reporting. Select a row to open the site's drawer, with charts for reachability, round-trip time, packet loss, and jitter, and the traced path. ![The Reachability tab listing gateway and internet probes with their site, role, protocol, RTT, packet loss, and status](./images/reachability-tab.png) -The agent emits three metrics per probe: +The agent emits these metrics per probe: - `km_ping_up`: `1` when the target is reachable, `0` when it isn't - `km_ping_rtt`: round-trip time in milliseconds - `km_ping_packet_loss`: percentage lost (ICMP probes only) +- `km_ping_jitter`: jitter in milliseconds (ICMP probes only) Each carries the `site` and `role` tags you set, so you can build [Dashboards](/visualize-data/dashboards/) and [Alerts](/alerts/create-alerts/) that group by site or fire only when an internet probe drops. + +## Jitter + +Jitter tells you how steady the round-trip time is, which matters more than the average for voice and video calls. Each ICMP probe sends several echoes, and jitter is the mean difference between consecutive round trips that were answered. For example, round trips of 20, 30, and 25 ms give a jitter of 7.5 ms. + +Jitter is measured on ICMP probes, which in a reachability check is the gateway probe. A probe needs at least two answered echoes to report it. It needs agent 2.11.0 or later, and the column stays empty on older agents. + +## Trace the path to the internet + +When the internet probe fails or slows down, the traced path shows where along the way it happens: on the site's own link, at the ISP, or further out. Turn on **Trace path** when you add the check, then pick how often to trace under **Trace every**: every minute, every 5 minutes (the default), every 15 minutes, or every hour. To add tracing to an existing check, open **Configure**, edit the site's internet probe, and turn on **Trace path**. + +Each trace sends three TCP SYN packets per hop to the public host and port, raising the TTL one hop at a time, and reads the ICMP replies from the routers on the way. It stops when the public host answers, when a router reports it unreachable, or after 30 hops. Because a trace can send up to 90 packets, the agent won't trace more often than once a minute. + +Trace path only works on the internet probe, since it uses TCP. It needs: + +- Agent 2.11.0 or later. On an older agent, or one whose version isn't known, the switch is disabled. +- An agent running on Linux. +- Permission to open raw sockets to read the ICMP replies: `CAP_NET_RAW` or root on a host, or `--cap-add=NET_RAW` in Docker. The standard Docker install runs the agent with `--privileged`, which includes it. + +Without raw socket rights, the agent logs `km_ping: no path metrics from this host` once and sends no path data. The probe itself keeps running. + +### Read the hop view + +Open the site's drawer from the **Reachability** tab. The **Path** panel at the top shows the latest trace to each traced endpoint, marked **Reached** or **Not reached**, with one row per hop: + +- **Address**: the router that answered, or **No answer** when none of the hop's probes did. +- **RTT**: the mean round trip of the hop's answered probes. +- **Loss**: the share of the hop's probes that got no answer. + +A hop listed twice answered from two addresses, which usually means a load-balanced route. A router that rate-limits its own replies can show loss on its hop even though traffic through it, and to the target, isn't lost. Hops after the last one that answered are left out. + +The path is stored as metrics on the internet probe: + +- `km_ping_path_hop_rtt` and `km_ping_path_hop_loss`, per hop, with `hop_index` and `hop_address` (`*` when the hop didn't answer) +- `km_ping_path_hops`: the number of hops to the target, or to the last hop that answered +- `km_ping_path_reached`: `1` when the trace reached the target, `0` when it didn't diff --git a/src/content/docs/docs/network-monitoring/traps-and-syslog.mdx b/src/content/docs/docs/network-monitoring/traps-and-syslog.mdx new file mode 100644 index 00000000..11ce9ec2 --- /dev/null +++ b/src/content/docs/docs/network-monitoring/traps-and-syslog.mdx @@ -0,0 +1,79 @@ +--- +title: "Receive SNMP Traps and Syslog" +description: "Collect SNMP traps and syslog from network devices with the KloudMate Agent, and see each device's events on its page." +sidebar: + label: "Traps and syslog" + order: 5 +--- + +Polling tells you how a device is doing every minute or so. Traps and syslog tell you what happened in between: a link that flapped, a fan that failed, a configuration change, a login. Add a receiver on an agent and point your devices at it. Every event lands in [Logs](/logs/log-explorer/), and events from devices that the same agent polls over SNMP also appear on that device's **Events** tab. + +Receivers need a Linux or Docker KloudMate Agent on version 2.11.0 or later. Kubernetes agents can't run them. + +## Add a receiver + +In **Network Monitoring**, open the **Add** menu and choose **Traps and syslog**. Pick the collector agent, then fill in the receiver: + +- **Receive**: **SNMP traps** or **Syslog**. +- **Port**: the port to listen on. It defaults to `162` for traps and `514` for syslog. +- **Communities** (traps): the community strings to accept, comma-separated. Leave it blank to accept traps with any community. See [Limit traps by community](#limit-traps-by-community). +- **Protocol** (syslog): **UDP** or **TCP**. +- **Message format** (syslog): **RFC 3164 (BSD)** or **RFC 5424**. Pick the format your devices send. +- **Site (optional)**: a label such as `hotel-blr-01`, added to every event from this receiver. + +Select **Add receiver**. The receiver starts within a minute. Then configure each device to send traps or syslog to the agent's address and the port you chose. + +You can run several receivers on one agent, for example traps on `162`, syslog over UDP on `514`, and syslog over TCP on `1514` for devices that send RFC 5424. Each receiver needs its own port, and that includes [flow listeners](/network-monitoring/flow-monitoring/). If two listeners on an agent share a port, only one of them is set up. + +To change or remove a receiver, open **Traps and syslog** again and pick the agent, or edit the **SNMP traps** or **Syslog** entry from **Configure**. + +## Ports, privileges, and Docker + +Ports `162` and `514` are below 1024, so the agent needs root, or the `CAP_NET_BIND_SERVICE` capability, to bind them. The Linux agent runs as root, and the standard Docker install runs the agent with `--privileged --network host`, so both can use the default ports. With host networking, the receivers listen on the Docker host's own addresses, and devices send to the host's IP. + +Receivers listen on every IPv4 address of the agent host. Open the ports in any firewall between your devices and the agent: UDP for traps and UDP syslog, and TCP for TCP syslog. + +If the agent can't bind a port, because another process holds it or the agent lacks the rights, it leaves that receiver out and logs `network events: receiver left out` with the reason. The agent's other receivers and integrations keep running. On a Linux host, run `journalctl -u kmagent | grep "network events"`. In Docker, run `docker logs km-agent 2>&1 | grep "network events"`. + +## Limit traps by community + +Receivers accept SNMPv1 and SNMPv2c traps. When **Communities** lists one or more strings, the receiver rejects any trap whose community isn't on the list, before the trap is queued. Rejected traps aren't stored, and they don't count as dropped events. Leave the field blank to accept every community. + +## How events map to devices + +For every trap and syslog message, the agent takes the sender's IP address and looks it up in the SNMP devices that the same agent polls, matching their management address and every interface address they report. When it finds a match, it names the event after that device. When it doesn't, it uses the sender's IP as the device name. + +So to see a device's events on its page, poll the device over SNMP from the same agent that receives its traps and syslog. For example, if a router polled as `edge-rtr-01` at `10.0.0.1` sends syslog from its loopback `10.255.0.1`, the event is still named `edge-rtr-01`, as long as the router reports that loopback address over SNMP. + +Each event keeps its severity and message, and carries these attributes: + +| Attribute | Value | +|---|---| +| `device.name` | The matched SNMP device's name, or the sender's IP when nothing matched. | +| `device.address` | The sender's IP address. | +| `network.event.source` | `snmp_trap` or `syslog`. | +| `site` | The receiver's **Site**, when you set one. | + +To find events from devices that aren't polled over SNMP, search [Logs](/logs/log-explorer/) for records that have `network.event.source`, and filter `device.name` by the sender's IP. + +## The Events tab + +Open a device from **Network Monitoring** and select the **Events** tab. It lists the traps and syslog messages that device sent in the selected time range, newest first, with the time, source, severity, and message, up to the latest 100. Select **Open in logs** to see every event in [Logs](/logs/log-explorer/) with the same filters. + +Events also feed the **Probable cause** panel on the Overview. When a device is down, its recent traps and syslog messages are listed as evidence. See [Using Network Monitoring](/network-monitoring/network-page/#probable-cause). + +## Dropped events + +A burst of traps or syslog can arrive faster than the agent forwards it. The agent counts the events it loses in `kmagent_network_events_dropped`, by `source` (`snmp_trap` or `syslog`) and `reason`. When you open **Traps and syslog** and pick an agent that lost events in the last hour, the form shows a warning with the counts. + +| Reason | Source | What happened | +|---|---|---| +| `queue_full` | Traps | The trap receiver's in-memory queue, which holds 16,384 traps, was full. | +| `refused` | Traps | The agent's logs pipeline didn't accept the trap. | +| `socket` | Traps and UDP syslog | The kernel dropped packets at the receiver's socket because the agent didn't read them in time. Counted on Linux only. | + +If drops keep coming back, spread the sending devices across more agents, or reduce what the devices send, for example by raising their syslog severity threshold. + +## Agent version + +Receivers need agent 2.11.0 or later. On an older agent, or one whose version isn't known, the form won't add a receiver. A receiver that's already configured for an older agent is ignored by that agent and starts once you upgrade it.