Educational ICT Virtualisation Specialist

Twitter LinkedIn E-mail
Precedence Technologies Ltd
Technology House, 36a Union Lane
Cambridge, CB4 1QB, United Kingdom
T: +44 (0)8456 446 800 / +44 (0)1223 359900
F: +44 (0)8456 446 899 / +44 (0)1223 359459
E: enquiries@precedence.co.uk
Network-Troubleshooting

Jump To: Support > KB > NetManager > Network > Troubleshooting

Network troubleshooting (Advanced)

The following commands must be run as root either at the console or by connecting in with ssh.

View network buffers in use

netmanager 1# netstat -m
4941 mbufs in use:
        3638 mbufs allocated to data
        1300 mbufs allocated to packet headers
        3 mbufs allocated to socket names and addresses
106 calls to protocol drain routines
netmanager 2# nmbclusters
32768

Of 32768 network buffers in total, 4941 are in use. High amounts in use may indicate network problems, broken packets, viruses or simply extremely high use.

Connections using NAT

Viewing total number of NATted connections~:

netmanager 1# ipnat -l | awk '/^MAP/{print}' | wc -l
    2506

2506 connections have direct connections (i.e. not through the proxy) to the Internet

Viewing list of connections page by page:

netmanager 70# ipnat -l | awk '/^MAP/{print}' | more
MAP 10.28.248.119   35198 <- -> a.b.c.d     46912 [173.194.78.188 5228]
MAP 10.28.248.58    57086 <- -> a.b.c.d     46048 [62.128.100.35 443]
MAP 10.28.248.119   36718 <- -> a.b.c.d     44210 [54.246.86.223 5223]
MAP 10.28.248.58    57085 <- -> a.b.c.d     46142 [62.128.100.35 443]
MAP 10.28.248.58    57084 <- -> a.b.c.d     50823 [62.128.100.35 443]
MAP 10.13.0.62      59245 <- -> a.b.c.d     42551 [130.117.190.174 443]
MAP 10.28.248.58    57082 <- -> a.b.c.d     52576 [62.128.100.35 443]
MAP 10.28.248.58    57081 <- -> a.b.c.d     45495 [62.128.100.35 443]
MAP 10.28.248.58    57078 <- -> a.b.c.d     57140 [62.128.100.106 443]
MAP 10.28.248.58    57077 <- -> a.b.c.d     57915 [62.128.100.106 443]
MAP 10.28.248.58    57076 <- -> a.b.c.d     59654 [62.128.100.106 443]
MAP 10.28.248.58    57075 <- -> a.b.c.d     40989 [62.128.100.106 443]
MAP 10.28.248.58    57074 <- -> a.b.c.d     52588 [62.128.100.106 443]
MAP 10.28.248.58    57073 <- -> a.b.c.d     46319 [62.128.100.106 443]
MAP 10.28.248.58    57072 <- -> a.b.c.d     56227 [62.128.100.106 443]
MAP 10.28.248.58    57071 <- -> a.b.c.d     59353 [62.128.100.106 443]
MAP 10.28.248.58    57070 <- -> a.b.c.d     53804 [62.128.100.106 443]
MAP 10.28.248.58    57069 <- -> a.b.c.d     54368 [62.128.100.35 443]
MAP 10.28.248.58    57068 <- -> a.b.c.d     54755 [62.128.100.106 443]
MAP 10.28.248.58    57067 <- -> a.b.c.d     57311 [62.128.100.106 443]
MAP 10.28.248.58    57066 <- -> a.b.c.d     40304 [62.128.100.106 443]
MAP 10.28.248.58    57065 <- -> a.b.c.d     47047 [62.128.100.35 443]
MAP 10.28.248.58    57064 <- -> a.b.c.d     54242 [62.128.100.106 443]
MAP 10.28.248.58    57063 <- -> a.b.c.d     59199 [62.128.100.106 443]
MAP 10.28.248.58    57062 <- -> a.b.c.d     40287 [62.128.100.106 443]
MAP 10.13.0.62      59244 <- -> a.b.c.d     49711 [130.117.190.171 443]
MAP 10.28.248.40    61822 <- -> a.b.c.d     52529 [194.72.9.34 53]
MAP 10.28.248.194   55285 <- -> a.b.c.d     40914 [173.194.34.129 443]
MAP 10.28.248.40    61598 <- -> a.b.c.d     46318 [194.72.9.34 53]
MAP 10.28.248.40    63028 <- -> a.b.c.d     52367 [194.72.9.34 53]
MAP 10.28.248.40    61904 <- -> a.b.c.d     42137 [194.72.9.34 53]
MAP 10.13.0.40      51413 <- -> a.b.c.d     43467 [178.184.49.239 6881]

We can see that 10.28.248.58 is using https (port 443) directly (not via the proxy). 10.28.248.40 is using DNS (port 53) directly, not via the local DNS servers

Getting a list of which clients are using port 53:

netmanager 1# ipnat -l | awk '/^MAP.* 53]/{print $2}' | sort | uniq
10.28.246.131
10.28.246.134
10.28.246.22
10.28.246.58
10.28.246.71
10.28.248.39
10.28.248.40

Getting number of DNS connections by IP address:

netmanager 1# ipnat -l | awk '/^MAP.* 53]/{ip[$2]++} END{for(i in ip) printf "%s %d\n", i, ip[i]}'
10.28.246.131 3
10.28.246.134 1
10.28.248.40 283
10.28.246.22 2
10.28.246.71 14
10.28.248.39 272

We can see that the two major culprits are 10.28.248.39 and 10.28.248.40

Checking basic web access

Default proxy:

netmanager 1# lynx http://www.precedence.co.uk/

Other proxy:

netmanager 1# env http_proxy=http://10.10.10.10:8080 lynx http://www.precedence.co.uk/

Viewing processes running with top

netmanager 1# top
load averages:  2.86,  2.75,  2.17;               up 3+23:24:36                                                                                                                                       15:34:33
66 processes: 64 sleeping, 2 on CPU
CPU0 states:  0.0% user,  0.0% nice,  0.2% system,  5.8% interrupt, 94.0% idle
CPU1 states:  0.0% user,  0.0% nice,  0.2% system,  0.0% interrupt, 99.8% idle
CPU2 states:  1.8% user,  0.0% nice,  1.6% system,  0.0% interrupt, 96.6% idle
CPU3 states:  0.2% user,  0.0% nice,  0.2% system,  0.0% interrupt, 99.6% idle
Memory: 1212M Act, 595M Inact, 1416K Wired, 37M Exec, 1325M File, 15M Free
Swap: 8064M Total, 343M Used, 7721M Free

  PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
  217 root      85    0   402M  389M biowai/2   2:02  0.24%  0.24% squid
 3527 nobody    85    0  2136K 1360K biowai/0   0:03  0.24%  0.24% unlinkd
  958 mysql     85    0    87M   15M sigwai/1 136:21  0.00%  0.00% mysqld
    0 root       0    0     0K   23M CPU/3     11:31  0.00%  0.00% [system]
  334 root      43    0    62M   13M parked/3   1:44  0.00%  0.00% named
  161 root      85    0    10M 2912K select/3   0:20  0.00%  0.00% nmbd
17981 root      85    0    66M   11M kqueue/2   0:13  0.00%  0.00% squid
17074 root      85    0    66M   11M kqueue/1   0:13  0.00%  0.00% squid

View traffic per second on a given network interface

netmanager 104# netstat -bw1 -I wm1
    wm1 in             wm1 out          total in           total out
     bytes               bytes             bytes               bytes
74356661825          9438122506        84577627762          86685889796
    864952               82854            922420              791044
     25146                3572             31573                7450
    727525              104989            834401              922554
    725963               76883            848967              791453
    647634               80429            711378              794802
    413032               50913            488503              403792
^C
© Copyright Precedence Technologies 1999-2017
Page last modified on May 14, 2013, at 03:38 PM by sborrill