3/23/2019

netstat 의 기본이용법 및 network 문제파악

1. netstat의 사용법

netstat를 이용하여 전체 Network 의 상태를 파악이 가능하지만, 세부적인 통신내용을 알기는  어려운 것 같다. 

$ netstat --help
usage: netstat [-vWeenNcCF] [] -r         netstat {-V|--version|-h|--help}
       netstat [-vWnNcaeol] [ ...]
       netstat { [-vWeenNac] -i | [-cWnNe] -M | -s }

        -r, --route              display routing table
        -i, --interfaces         display interface table
        -g, --groups             display multicast group memberships
        -s, --statistics         display networking statistics (like SNMP)
        -M, --masquerade         display masqueraded connections

        -v, --verbose            be verbose
        -W, --wide               don't truncate IP addresses
        -n, --numeric            don't resolve names
        --numeric-hosts          don't resolve host names
        --numeric-ports          don't resolve port names
        --numeric-users          don't resolve user names
        -N, --symbolic           resolve hardware names
        -e, --extend             display other/more information
        -p, --programs           display PID/Program name for sockets
        -c, --continuous         continuous listing

        -l, --listening          display listening server sockets
        -a, --all, --listening   display all sockets (default: connected)
        -o, --timers             display timers
        -F, --fib                display Forwarding Information Base (default)
        -C, --cache              display routing cache instead of FIB

  ={-t|--tcp} {-u|--udp} {-w|--raw} {-x|--unix} --ax25 --ipx --netrom
  =Use '-6|-4' or '-A ' or '--'; default: inet
  List of possible address families (which support routing):
    inet (DARPA Internet) inet6 (IPv6) ax25 (AMPR AX.25) 
    netrom (AMPR NET/ROM) ipx (Novell IPX) ddp (Appletalk DDP) 
    x25 (CCITT X.25) 


1.1  netstat  기본사용법 

사용방법은 Network의 전체 구성인 Routing Table 정보부터 보는 것이 맞을 것 같으며,  관련된 Multicast 와 점점 세부 네트워크로 들어가 분석하는 것이 맞을 것 같다.

  • Routing Table 정보 확인
현재 연결된 Network 의 구성을 알아보자

$ netstat -re  // extened 
or 
$ netstat -r   
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
default         router.asus.com 0.0.0.0         UG        0 0          0 eth0
10.0.0.0        *               255.255.255.0   U         0 0          0 eth0
link-local      *               255.255.0.0     U         0 0          0 usb0
172.17.0.0      *               255.255.0.0     U         0 0          0 docker0
192.168.55.0    *               255.255.255.0   U         0 0          0 l4tbr0
192.168.55.0    *               255.255.255.0   U         0 0          0 usb0


  • Multicast 관련 정보 확인 
  --groups , -g  Display multicast group membership information for IPv4 and IPv6.

$ netstat -g
IPv6/IPv4 Group Memberships
Interface       RefCnt Group
--------------- ------ ---------------------
lo              1      all-systems.mcast.net
eth0            1      224.0.0.251
eth0            1      all-systems.mcast.net
usb0            1      224.0.0.251
usb0            1      all-systems.mcast.net
usb1            1      all-systems.mcast.net
l4tbr0          1      224.0.0.251
l4tbr0          1      all-systems.mcast.net
wlan0           1      all-systems.mcast.net
docker0         1      224.0.0.251
docker0         1      all-systems.mcast.net
lo              1      ff02::1
lo              1      ff01::1
dummy0          1      ff02::1
dummy0          1      ff01::1
eth0            1      ff02::fb
eth0            1      ff02::1:fff4:9ad0
eth0            1      ff02::1
eth0            1      ff01::1
tunl0           1      ff02::1
tunl0           1      ff01::1
usb0            1      ff02::fb
usb0            1      ff02::1:ffe6:405f
usb0            1      ff02::1
usb0            1      ff01::1
usb1            1      ff02::fb
usb1            1      ff02::1:ff72:214
usb1            1      ff02::1
usb1            1      ff01::1
l4tbr0          1      ff02::fb
l4tbr0          1      ff02::1:ff2a:6c1f
l4tbr0          1      ff02::1
l4tbr0          1      ff01::1
wlan0           1      ff02::1
wlan0           1      ff01::1
docker0         1      ff02::1
docker0         1      ff01::1


  • TCP/UDP 관련된 모든 Socket  상태정보 표시
TCP/UDP 관련 Socket 의 정보들 및 상태 정보 와 PID를 확인도 가능하다.
옵션에따라 점점 추가 되며 아래와 같이 파악 가능하다.

$ netstat -atn           # all socket and numeric For tcp
$ netstat -aun           # all socket and numeric For udp
$ netstat -atun          # all socket and numeric For both (TCP,UDP)

// 각 State 와 PID 정보 및 Protocol 과  Recv Q/Send Q 정보 
$ netstat -tulnp          # listening server and numeric and PID for both (TCP,UDP)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.1.1:53            0.0.0.0:*               LISTEN      -               
tcp        0      0 192.168.55.2:53         0.0.0.0:*               LISTEN      -               
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      -               
tcp        0      0 192.168.55.1:53         0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -               
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      -               
tcp        0      0 127.0.0.1:6011          0.0.0.0:*               LISTEN      -               
tcp        0      0 127.0.0.1:6012          0.0.0.0:*               LISTEN      -               
tcp        0      0 127.0.0.1:2947          0.0.0.0:*               LISTEN      -               
tcp6       0      0 fe80::c03:7dff:fea1::53 :::*                    LISTEN      -               
tcp6       0      0 ::1:53                  :::*                    LISTEN      -               
tcp6       0      0 :::22                   :::*                    LISTEN      -               
tcp6       0      0 ::1:631                 :::*                    LISTEN      -               
tcp6       0      0 ::1:6011                :::*                    LISTEN      -               
tcp6       0      0 ::1:6012                :::*                    LISTEN      -               
tcp6       0      0 ::1:2947                :::*                    LISTEN      -               
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           -               
udp   12585600      0 0.0.0.0:1334            0.0.0.0:*                           1695/python     
udp        0      0 0.0.0.0:52316           0.0.0.0:*                           -               
udp        0      0 127.0.1.1:53            0.0.0.0:*                           -               
udp        0      0 192.168.55.2:53         0.0.0.0:*                           -               
udp        0      0 127.0.0.1:53            0.0.0.0:*                           -               
udp        0      0 192.168.55.1:53         0.0.0.0:*                           -               
udp        0      0 0.0.0.0:68              0.0.0.0:*                           -               
udp        0      0 0.0.0.0:631             0.0.0.0:*                           -               
udp        0      0 0.0.0.0:53992           0.0.0.0:*                           -               
udp6       0      0 :::5353                 :::*                                -               
udp6       0      0 :::51705                :::*                                -               
udp6       0      0 fe80::c03:7dff:fea1::53 :::*                                -               
udp6       0      0 ::1:53                  :::*                                -  

  https://askubuntu.com/questions/538208/how-to-check-opened-closed-ports-on-my-computer


  • network Interface의 정보확인  ( Layer 2  분석) 
실제로 Network Driver, Layer 2의  오류파악가능하며, 보통 Ether phy chip(MII) 연결 후, 문제가 있을 경우 
Network TEST를 통해서 기본확인가능하며, Kernel의 Driver에 따라 에러감지가 제대로 되지 않을 수 있으므로 주의하자
만약 에러가 발생하면, ifconfig에서도 쉽게 발견가능

netstat  -i, --interfaces    Display a table of all network interfaces.

$ netstat -i   or netstat -ic // Interface TX/RX Status 
Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
docker0    1500 0         0      0      0 0             0      0      0      0 BMU
eth0       1500 0   2529153      0      0 0        829553      0      0      0 BMRU
l4tbr0     1500 0    127916      0      0 0        103336      0      0      0 BMRU
lo        65536 0      3462      0      0 0          3462      0      0      0 LRU
usb0       1500 0     33527  33527      0 0         32753      0      0      0 BMRU
usb1       1500 0     94389      0      0 0         70958      0      0      0 BMRU
wlan0      1500 0         0      0      0 0             0      0      0      0 BMU



  1. MTU(Maximum Transfer Unit): default 1500  // ethernet 
  2. Met: 메트릭
  3. RX-OK : 받은 것. 정상 수신 패킷
  4. RX-ERR: 파손 수신 패킷 (상위 usb0은 100% 에러)
  5. RX-DRP: 유실 수신 패킷
  6. RX-OVR: packet overrun(커널에서 처리 하는 인터럽트보다 패킷이 빨리 도착했을때 패킷 유실)
  7. TX-OK : 보낸 것. 정상 송신 패킷.
  8. TX-ERR: 파손 송신 패킷.
  9. TX-DRP: 유실 송신 패킷.
  10. TX-OVR: packet overrun



  • Flg:
  1. B: 브로드케스트 주소 지정
  2. L: 루프백 인터페이스
  3. M: 모든 패킷 수집(promiscuous mode)
  4. O: 인터페이스에 대한 ARP가 off 되어 있음.
  5. P: point-to-point 연결
  6. R: 인터페이스 동작중(Running)
  7. U: 인터페이스 up 인 상태.



2. Network 문제사항 및 에러확인 

Network 관련설정은 처음 booting /etc/sysctrl.conf에 의해서 설정되며 없다면, kernel의 default 값으로 설정된다.

Network 정보 (TCP / UDP / IP / ICMP) 의 관련된사항들을 기록하였기때문에, 분석이 가능하며, 문제사항을 쉽게 파악이 가능하다.

netstat -s 옵션이외에도 proc 정보로도 확인가능
sysctl -a 설정정보확인가능

netstat -s , --statistics   Display summary statistics for each protocol.

$ netstat -s  // UDP / TCP / ICMP  사용통계 및 문제에러사항 확인 
Ip:
    1807814 total packets received
    0 forwarded
    0 incoming packets discarded
    1188470 incoming packets delivered
    955549 requests sent out
    40 outgoing packets dropped
Icmp:
    100 ICMP messages received
    2 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 100
    102 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 102
IcmpMsg:
        InType3: 100
        OutType3: 102
Tcp:
    5894 active connections openings
    15 passive connection openings
    4 failed connection attempts
    5748 connection resets received
    6 connections established
    965515 segments received
    952551 segments send out
    146 segments retransmited
    0 bad segments received.
    34 resets sent
Udp:
    59434 packets received
    82 packets to unknown port received.
    0 packet receive errors
    1454 packets sent
    IgnoredMulti: 139426
UdpLite:
TcpExt:
    102 TCP sockets finished time wait in fast timer
    12122 delayed acks sent
    38 delayed acks further delayed because of locked socket
    Quick ack mode was activated 8 times
    404422 packets directly queued to recvmsg prequeue.
    36435935 bytes directly in process context from backlog
    434921871 bytes directly received in process context from prequeue
    152208 packet headers predicted
    344744 packets header predicted and directly queued to user
    1419 acknowledgments not containing data payload received
    284057 predicted acknowledgments
    41 congestion windows recovered without slow start after partial ack
    36 timeouts in loss state
    1 retransmits in slow start
    46 other TCP timeouts
    TCPLossProbes: 2
    4 DSACKs sent for old packets
    12 connections reset due to unexpected data
    3 connections reset due to early user close
    4 connections aborted due to timeout
    IPReversePathFilter: 194
    TCPRcvCoalesce: 104075
    TCPOFOQueue: 2614
    TCPOFOMerge: 3
    TCPChallengeACK: 27
    TCPSpuriousRtxHostQueues: 2
    TCPAutoCorking: 68999
    TCPFromZeroWindowAdv: 15
    TCPToZeroWindowAdv: 22
    TCPWantZeroWindowAdv: 122
    TCPSynRetrans: 4
    TCPOrigDataSent: 561796
    TCPHystartTrainDetect: 3
    TCPHystartTrainCwnd: 50
    TCPACKSkippedChallenge: 31
    TCPKeepAlive: 2
IpExt:
    InNoRoutes: 1
    InMcastPkts: 82733
    OutMcastPkts: 2070
    InBcastPkts: 139441
    OutBcastPkts: 18
    InOctets: 976384465
    OutOctets: 295103139
    InMcastOctets: 8631056
    OutMcastOctets: 99130
    InBcastOctets: 17773399
    OutBcastOctets: 812
    InNoECTPkts: 1819528


  • UDP 만 별도 문제사항확인  
UDP Packet 의  RecvBufffer 의 문제를 알게되었는데, 이 문제를 net.core.rmem_max 변경과 socket option의 RecvBuff Size 올려주면 이를 에러율을 줄이는 것은 가능하다.
혹은 net.core.rmem_maxnet.core.rmem_default 변경을 해주는 것인데, 이것보다는 위의 것이 더 나은 방법인 것 같다.
(/etc/sysctrl.conf 에서 변경가능)

$ netstat -su    //UDP
IcmpMsg:
    InType3: 84
    OutType3: 1411
Udp:
    119049 packets received
    2339998 packets to unknown port received.
    6006884 packet receive errors
    2618 packets sent
    RcvbufErrors: 6006884                    //Recv Buffer Error , cat /proc/net/snmp | grep Udp 값 동일 
    IgnoredMulti: 494148
UdpLite:
IpExt:
    InNoRoutes: 1
    InMcastPkts: 170389
    OutMcastPkts: 6945
    InBcastPkts: 425852
    OutBcastPkts: 18
    InOctets: 14776499147
    OutOctets: 3391504334
    InMcastOctets: 11684052
    OutMcastOctets: 279654
    InBcastOctets: 39906199
    OutBcastOctets: 812
    InNoECTPkts: 12363330


UDP Packet Drop 현상 ( Test1, Test2 의 에러율 확인)
  https://stackoverflow.com/questions/21664950/udp-packet-drop-inerrors-vs-rcvbuferrors


  • Socket Buffer 설정정보확인 (proc)
OS마다 Socket에 할당하는 Memory는 다르며, 각 설정역시 다르기때문에 이 부분은 확인을 해야하며, 이부분도 부팅시 설정이 가능하다.

$ ls /proc/sys/net/core/
busy_poll      dev_weight             max_skb_frags  netdev_budget       netdev_tstamp_prequeue  rmem_max               tstamp_allow_data  wmem_max           xfrm_aevent_rseqth
busy_read      flow_limit_cpu_bitmap  message_burst  netdev_max_backlog  optmem_max              rps_sock_flow_entries  warnings           xfrm_acq_expires   xfrm_larval_drop
default_qdisc  flow_limit_table_len   message_cost   netdev_rss_key      rmem_default            somaxconn              wmem_default       xfrm_aevent_etime

//Socket Recv Buffer (Memory)
$ cat /proc/sys/net/core/rmem_default   //Socket Recv Buffer Default 
229376
$ cat /proc/sys/net/core/rmem_max   // Socket Recv Buffer Max 
16777216
//Socket Write Buffer (Memory)
$ cat /proc/sys/net/core/wmem_default   //Socket Write Buffer Default 
$ cat /proc/sys/net/core/wmem_max   // Socket Write Buffer Max 

  • IP기반의 protocol 정보
proc 정보를 이용하여 network 정보 확인

$ls /proc/sys/net/ipv4/
conf/                              ipfrag_high_thresh                 tcp_fin_timeout                    tcp_reordering
fwmark_reflect                     ipfrag_low_thresh                  tcp_frto                           tcp_retrans_collapse
icmp_echo_ignore_all               ipfrag_max_dist                    tcp_fwmark_accept                  tcp_retries1
icmp_echo_ignore_broadcasts        ipfrag_secret_interval             tcp_invalid_ratelimit              tcp_retries2
icmp_errors_use_inbound_ifaddr     ipfrag_time                        tcp_keepalive_intvl                tcp_rfc1337
icmp_ignore_bogus_error_responses  neigh/                             tcp_keepalive_probes               tcp_rmem
icmp_msgs_burst                    netfilter/                         tcp_keepalive_time                 tcp_sack
icmp_msgs_per_sec                  ping_group_range                   tcp_limit_output_bytes             tcp_slow_start_after_idle
icmp_ratelimit                     route/                             tcp_low_latency                    tcp_stdurg
icmp_ratemask                      tcp_abort_on_overflow              tcp_max_orphans                    tcp_syn_retries
igmp_link_local_mcast_reports      tcp_adv_win_scale                  tcp_max_reordering                 tcp_synack_retries
igmp_max_memberships               tcp_allowed_congestion_control     tcp_max_syn_backlog                tcp_thin_dupack
igmp_max_msf                       tcp_app_win                        tcp_max_tw_buckets                 tcp_thin_linear_timeouts
igmp_qrv                           tcp_autocorking                    tcp_mem                            tcp_timestamps
inet_peer_maxttl                   tcp_available_congestion_control   tcp_min_rtt_wlen                   tcp_tso_win_divisor
inet_peer_minttl                   tcp_base_mss                       tcp_min_tso_segs                   tcp_tw_recycle
inet_peer_threshold                tcp_challenge_ack_limit            tcp_moderate_rcvbuf                tcp_tw_reuse
ip_default_ttl                     tcp_congestion_control             tcp_mtu_probing                    tcp_window_scaling
ip_dynaddr                         tcp_default_init_rwnd              tcp_no_metrics_save                tcp_wmem
ip_early_demux                     tcp_dsack                          tcp_notsent_lowat                  tcp_workaround_signed_windows
ip_forward                         tcp_early_retrans                  tcp_orphan_retries                 udp_mem
ip_forward_use_pmtu                tcp_ecn                            tcp_pacing_ca_ratio                udp_rmem_min
ip_local_port_range                tcp_ecn_fallback                   tcp_pacing_ss_ratio                udp_wmem_min
ip_local_reserved_ports            tcp_fack                           tcp_probe_interval                 vs/
ip_no_pmtu_disc                    tcp_fastopen                       tcp_probe_threshold                xfrm4_gc_thresh

$ cat /proc/sys/net/ipv4/tcp_mem 
94236 125648 188472 


  • SNMP 관련정보 
netstat에서 보는 통계의 값과 유사하며, 네트워크 관리목적으로 사용이 되기 때문에, 추후 세부관리까지 한다면, SNMP를 이용하여 MIB로 정보를 제공하여 관리를 하도록 하자.

- TCP 관련부분 분석

$ cat /proc/net/snmp 
Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests OutDiscards OutNoRoutes ReasmTimeout ReasmReqds ReasmOKs ReasmFails FragOKs FragFails FragCreates
Ip: 1 64 12843660 0 0 0 0 0 12843660 22324680 0 0 0 0 0 0 0 0 0
Icmp: InMsgs InErrors InCsumErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps
Icmp: 1 0 0 1 0 0 0 0 0 0 0 0 0 0 4 0 4 0 0 0 0 0 0 0 0 0 0
IcmpMsg: InType3 OutType3
IcmpMsg: 1 4
Tcp: RtoAlgorithm RtoMin RtoMax MaxConn ActiveOpens PassiveOpens AttemptFails EstabResets CurrEstab InSegs OutSegs RetransSegs InErrs OutRsts InCsumErrors
Tcp: 1 200 120000 -1 224248 11 2 3 3 12799177 22311554 11 0 15 0
Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors IgnoredMulti
Udp: 8452 3 0 3600 0 0 0 70
UdpLite: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors IgnoredMulti
UdpLite: 0 0 0 0 0 0 0 0 

- UDP RcvBuffErrors 분석

RecvbufErrors가 지속적으로 갑자기 증가되는 문제발생하여 관련된 부분 디버깅

$ netstat -su
IcmpMsg:
    InType3: 88
    OutType3: 88
Udp:
    6404859 packets received
    88 packets to unknown port received.
    2295229 packet receive errors
    1148 packets sent
    RcvbufErrors: 2295229
    IgnoredMulti: 281547
UdpLite:
IpExt:
    InMcastPkts: 10936
    OutMcastPkts: 2477
    InBcastPkts: 275312
    OutBcastPkts: 18
    InOctets: 12773054136
    OutOctets: 240440435
    InMcastOctets: 379933
    OutMcastOctets: 102955
    InBcastOctets: 23165236
    OutBcastOctets: 812
    InNoECTPkts: 9232445

$ cat /proc/net/snmp 
Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests OutDiscards OutNoRoutes ReasmTimeout ReasmReqds ReasmOKs ReasmFails FragOKs FragFails FragCreates
Ip: 1 64 7026190 0 1 0 0 0 7025803 300352 40 0 0 0 0 0 0 0 0
Icmp: InMsgs InErrors InCsumErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps
Icmp: 88 0 0 88 0 0 0 0 0 0 0 0 0 0 88 0 88 0 0 0 0 0 0 0 0 0 0
IcmpMsg: InType3 OutType3
IcmpMsg: 88 88
Tcp: RtoAlgorithm RtoMin RtoMax MaxConn ActiveOpens PassiveOpens AttemptFails EstabResets CurrEstab InSegs OutSegs RetransSegs InErrs OutRsts InCsumErrors
Tcp: 1 200 120000 -1 37 5 2 6 3 190165 374801 39 0 25 0
Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors IgnoredMulti
Udp: 6155446 88 466455 1007 466455 0 0 206305
UdpLite: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors IgnoredMulti
UdpLite: 0 0 0 0 0 0 0 0

$ cat /proc/1695/net/udp   // RX QUEUE를 Socket으로 6x1024x1024로 설정했지만, 아래와 같이 12x1024x1024로 증가되었음 
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode ref pointer drops             
  229: 00000000:14E9 00000000:0000 07 00000000:00000000 00:00000000 00000000   110        0 33308 2 0000000000000000 0         
  306: 00000000:0536 00000000:0000 07 00000000:00C00300 00:00000000 00000000  1001        0 37343 2 0000000000000000 2295417   //RcvbufErrors 동일값 
 2136: 00000000:CC5C 00000000:0000 07 00000000:00000000 00:00000000 00000000 65534        0 34055 2 0000000000000000 0         
 3121: 0101007F:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 34043 2 0000000000000000 0         
 3121: 0237A8C0:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000   111        0 24293 2 0000000000000000 0         
 3121: 0100007F:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 29242 2 0000000000000000 0         
 3121: 0137A8C0:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 29240 2 0000000000000000 0         
 3136: 00000000:0044 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 300750 2 0000000000000000 0        
 3699: 00000000:0277 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 321449 2 0000000000000000 0        
 3812: 00000000:D2E8 00000000:0000 07 00000000:00000000 00:00000000 00000000   110        0 33310 2 0000000000000000 0    

$ cat /proc/1695/oom_score 
193

$ netstat -aun
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
udp        0      0 0.0.0.0:5353            0.0.0.0:*                          
udp   12585728      0 0.0.0.0:1334            0.0.0.0:*                 //00C00300 : 12583680 과 동일한 수치지만, 지속적으로 증가           
udp        0      0 0.0.0.0:52316           0.0.0.0:*                          
udp        0      0 127.0.1.1:53            0.0.0.0:*                          
udp        0      0 192.168.55.2:53         0.0.0.0:*                          
udp        0      0 127.0.0.1:53            0.0.0.0:*                          
udp        0      0 192.168.55.1:53         0.0.0.0:*                          
udp        0      0 0.0.0.0:68              0.0.0.0:*                          
udp        0      0 0.0.0.0:631             0.0.0.0:*                          
udp        0      0 0.0.0.0:53992           0.0.0.0:*                          
udp6       0      0 :::5353                 :::*                               
udp6       0      0 :::51705                :::*                               
udp6       0      0 fe80::c03:7dff:fea1::53 :::*                               
udp6       0      0 ::1:53                  :::* 


SOCKET의 SO_RCVBUF 설정
  https://serverfault.com/questions/758885/rx-queue-value-in-proc-net-udp-grows-to-twice-the-so-rcvbuf-value

2.1 OS 및 Socket 관련 세부설정  

상위 proc에서 확인된 socket 내부 buffer 설정을 linux에서는 sysctl 명령어로 변경이 가능하며, 이외 buffer 설정이외에도, 다양한 설정이 가능하다. 이부분은 별도의 Manual을 참조하자.
  • Socket 관련된 memory 설정값 확인 
$ sysctl -a | grep mem
sysctl: permission denied on key 'fs.protected_hardlinks'
sysctl: permission denied on key 'fs.protected_symlinks'
sysctl: permission denied on key 'kernel.cad_pid'
sysctl: permission denied on key 'kernel.usermodehelper.bset'
.....
//229376 =  224 * 1024
//6291456 = 6 * 1024 * 1024
//3145728 = 3 * 1024 * 1024
//16777216 = 16 * 1024 *1024
//12583680 = 12 * 1024 * 1024
//26214400 = 25 * 1024*1024
 
net.core.optmem_max = 20480
net.core.rmem_default = 229376         // Byte (UDP/TCP)
net.core.rmem_max = 229376             // Byte (UDP/TCP)
net.core.wmem_default = 229376        // Byte  (UDP/TCP)        
net.core.wmem_max = 229376            // Byte  (UDP/TCP)
net.ipv4.igmp_max_memberships = 20
sysctl: permission denied on key 'net.ipv4.tcp_fastopen_key'
sysctl: permission denied on key 'net.ipv6.conf.all.stable_secret'
net.ipv4.tcp_mem = 94236 125648 188472     // Page(4KB) , min / pressure / max  
net.ipv4.tcp_rmem = 4096 87380 6291456      // Byte         , min / pressure / max     
net.ipv4.tcp_wmem = 4096 16384 4194304     // Byte        , min / pressure / max
net.ipv4.udp_mem = 188472 251297 376944  // Page(4KB)    min / pressure / max
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
net.ipv4.vs.amemthresh = 1024
sysctl: permission denied on key 'net.ipv6.conf.default.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.docker0.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.dummy0.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.eth0.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.l4tbr0.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.lo.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.tunl0.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.usb0.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.usb1.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.wlan0.stable_secret'
sysctl: permission denied on key 'vm.mmap_rnd_bits'
sysctl: permission denied on key 'vm.mmap_rnd_compat_bits'
vm.lowmem_reserve_ratio = 256 32
vm.overcommit_memory = 0

  • Kernel의 내부 Socket Buffer 변경
실시간으로 network memory  변경

$ sysctl -w net.core.rmem_default="229376"
$ sysctl -w net.core.wmem_default="229376"
$ sysctl -w net.core.rmem_max="16777216"
$ sysctl -w net.core.wmem_max="16777216"
$ sysctl -w net.ipv4.tcp_rmem="229376 229376 16777216"
$ sysctl -w net.ipv4.tcp_wmem="229376 229376 16777216"

Recvbuffer 만 변경하여 테스트 진행

$ sudo sysctl -w net.core.rmem_default=26214400
$ sudo sysctl -w net.core.rmem_max=26214400

  • /etc/sysctl.conf 에서 기본설정 
상위 설정을 테스트로 하여 초기 부팅시 아래의 설정에서 기본으로 설정으로 정하자.

$ sudo vi /etc/sysctl.conf
........
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 229376 16777216
net.ipv4.tcp_wmem = 4096 229376 16777216
........
$ ls /etc/sysctl.d/     // 확인 

man sysctl.conf 참조

/proc 및 /sys 정보관련정보
  https://www.kernel.org/doc/Documentation/sysctl/net.txt


  http://www.ubuntugeek.com/performance-tuning-with-system-control-sysctl-in-ubuntu.html
  https://kensei.co.kr/289
  https://rocksea.tistory.com/64
  https://meetup.toast.com/posts/53
  https://wwwx.cs.unc.edu/~sparkst/howto/network_tuning.php
  https://www.systutorials.com/241303/how-to-enlarge-linux-udp-buffer-size/
  https://m.blog.naver.com/PostView.nhn?blogId=parkjy76&logNo=220874390930&proxyReferer=https%3A%2F%2Fwww.google.com%2F

  https://stackoverflow.com/questions/31546835/tcp-receiving-window-size-higher-than-net-core-rmem-max

  https://m.blog.naver.com/PostView.nhn?blogId=kwoncharlie&logNo=10109682872&proxyReferer=https%3A%2F%2Fwww.google.com%2F



    • watch or shell script 를 이용하여 감시 

    $ sudo watch -d -n0 "netstat -atnp | grep ESTA"     // -d, difference , -n interval second , grep ESTABLISHED 된것을 확인 
    $ man watch // 세부명령어 확인 
    



    TCP 의 Socket 상태도
      https://upload.wikimedia.org/wikipedia/commons/e/e4/Tcp_state_diagram.svg


    댓글 없음 :