1. netstat의 사용법
netstat를 이용하여 전체 Network 의 상태를 파악이 가능하지만, 세부적인 통신내용을 알기는 어려운 것 같다.
$ netstat --help
usage: netstat [-vWeenNcCF] [] -r netstat {-V|--version|-h|--help}
netstat [-vWnNcaeol] [ ...]
netstat { [-vWeenNac] -i | [-cWnNe] -M | -s }
-r, --route display routing table
-i, --interfaces display interface table
-g, --groups display multicast group memberships
-s, --statistics display networking statistics (like SNMP)
-M, --masquerade display masqueraded connections
-v, --verbose be verbose
-W, --wide don't truncate IP addresses
-n, --numeric don't resolve names
--numeric-hosts don't resolve host names
--numeric-ports don't resolve port names
--numeric-users don't resolve user names
-N, --symbolic resolve hardware names
-e, --extend display other/more information
-p, --programs display PID/Program name for sockets
-c, --continuous continuous listing
-l, --listening display listening server sockets
-a, --all, --listening display all sockets (default: connected)
-o, --timers display timers
-F, --fib display Forwarding Information Base (default)
-C, --cache display routing cache instead of FIB
={-t|--tcp} {-u|--udp} {-w|--raw} {-x|--unix} --ax25 --ipx --netrom
=Use '-6|-4' or '-A ' or '--'; default: inet
List of possible address families (which support routing):
inet (DARPA Internet) inet6 (IPv6) ax25 (AMPR AX.25)
netrom (AMPR NET/ROM) ipx (Novell IPX) ddp (Appletalk DDP)
x25 (CCITT X.25)
1.1 netstat 기본사용법
사용방법은 Network의 전체 구성인 Routing Table 정보부터 보는 것이 맞을 것 같으며, 관련된 Multicast 와 점점 세부 네트워크로 들어가 분석하는 것이 맞을 것 같다.
현재 연결된 Network 의 구성을 알아보자
$ netstat -re // extened
or
$ netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
default router.asus.com 0.0.0.0 UG 0 0 0 eth0
10.0.0.0 * 255.255.255.0 U 0 0 0 eth0
link-local * 255.255.0.0 U 0 0 0 usb0
172.17.0.0 * 255.255.0.0 U 0 0 0 docker0
192.168.55.0 * 255.255.255.0 U 0 0 0 l4tbr0
192.168.55.0 * 255.255.255.0 U 0 0 0 usb0
--groups , -g Display multicast group membership information for IPv4 and IPv6.
$ netstat -g
IPv6/IPv4 Group Memberships
Interface RefCnt Group
--------------- ------ ---------------------
lo 1 all-systems.mcast.net
eth0 1 224.0.0.251
eth0 1 all-systems.mcast.net
usb0 1 224.0.0.251
usb0 1 all-systems.mcast.net
usb1 1 all-systems.mcast.net
l4tbr0 1 224.0.0.251
l4tbr0 1 all-systems.mcast.net
wlan0 1 all-systems.mcast.net
docker0 1 224.0.0.251
docker0 1 all-systems.mcast.net
lo 1 ff02::1
lo 1 ff01::1
dummy0 1 ff02::1
dummy0 1 ff01::1
eth0 1 ff02::fb
eth0 1 ff02::1:fff4:9ad0
eth0 1 ff02::1
eth0 1 ff01::1
tunl0 1 ff02::1
tunl0 1 ff01::1
usb0 1 ff02::fb
usb0 1 ff02::1:ffe6:405f
usb0 1 ff02::1
usb0 1 ff01::1
usb1 1 ff02::fb
usb1 1 ff02::1:ff72:214
usb1 1 ff02::1
usb1 1 ff01::1
l4tbr0 1 ff02::fb
l4tbr0 1 ff02::1:ff2a:6c1f
l4tbr0 1 ff02::1
l4tbr0 1 ff01::1
wlan0 1 ff02::1
wlan0 1 ff01::1
docker0 1 ff02::1
docker0 1 ff01::1
- TCP/UDP 관련된 모든 Socket 상태정보 표시
TCP/UDP 관련 Socket 의 정보들 및 상태 정보 와 PID를 확인도 가능하다.
옵션에따라 점점 추가 되며 아래와 같이 파악 가능하다.
$ netstat -atn # all socket and numeric For tcp
$ netstat -aun # all socket and numeric For udp
$ netstat -atun # all socket and numeric For both (TCP,UDP)
// 각 State 와 PID 정보 및 Protocol 과 Recv Q/Send Q 정보
$ netstat -tulnp # listening server and numeric and PID for both (TCP,UDP)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.1.1:53 0.0.0.0:* LISTEN -
tcp 0 0 192.168.55.2:53 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN -
tcp 0 0 192.168.55.1:53 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:6011 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:6012 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:2947 0.0.0.0:* LISTEN -
tcp6 0 0 fe80::c03:7dff:fea1::53 :::* LISTEN -
tcp6 0 0 ::1:53 :::* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 ::1:631 :::* LISTEN -
tcp6 0 0 ::1:6011 :::* LISTEN -
tcp6 0 0 ::1:6012 :::* LISTEN -
tcp6 0 0 ::1:2947 :::* LISTEN -
udp 0 0 0.0.0.0:5353 0.0.0.0:* -
udp 12585600 0 0.0.0.0:1334 0.0.0.0:* 1695/python
udp 0 0 0.0.0.0:52316 0.0.0.0:* -
udp 0 0 127.0.1.1:53 0.0.0.0:* -
udp 0 0 192.168.55.2:53 0.0.0.0:* -
udp 0 0 127.0.0.1:53 0.0.0.0:* -
udp 0 0 192.168.55.1:53 0.0.0.0:* -
udp 0 0 0.0.0.0:68 0.0.0.0:* -
udp 0 0 0.0.0.0:631 0.0.0.0:* -
udp 0 0 0.0.0.0:53992 0.0.0.0:* -
udp6 0 0 :::5353 :::* -
udp6 0 0 :::51705 :::* -
udp6 0 0 fe80::c03:7dff:fea1::53 :::* -
udp6 0 0 ::1:53 :::* -
https://askubuntu.com/questions/538208/how-to-check-opened-closed-ports-on-my-computer
- network Interface의 정보확인 ( Layer 2 분석)
실제로 Network Driver, Layer 2의 오류파악가능하며, 보통 Ether phy chip(MII) 연결 후,
문제가 있을 경우
Network TEST를 통해서 기본확인가능하며, Kernel의 Driver에 따라 에러감지가 제대로 되지 않을 수 있으므로 주의하자
만약 에러가 발생하면,
ifconfig에서도 쉽게 발견가능
netstat -i, --interfaces Display a table of all network interfaces.
$ netstat -i or netstat -ic // Interface TX/RX Status
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
docker0 1500 0 0 0 0 0 0 0 0 0 BMU
eth0 1500 0 2529153 0 0 0 829553 0 0 0 BMRU
l4tbr0 1500 0 127916 0 0 0 103336 0 0 0 BMRU
lo 65536 0 3462 0 0 0 3462 0 0 0 LRU
usb0 1500 0 33527 33527 0 0 32753 0 0 0 BMRU
usb1 1500 0 94389 0 0 0 70958 0 0 0 BMRU
wlan0 1500 0 0 0 0 0 0 0 0 0 BMU
- MTU(Maximum Transfer Unit): default 1500 // ethernet
- Met: 메트릭
- RX-OK : 받은 것. 정상 수신 패킷
- RX-ERR: 파손 수신 패킷 (상위 usb0은 100% 에러)
- RX-DRP: 유실 수신 패킷
- RX-OVR: packet overrun(커널에서 처리 하는 인터럽트보다 패킷이 빨리 도착했을때 패킷 유실)
- TX-OK : 보낸 것. 정상 송신 패킷.
- TX-ERR: 파손 송신 패킷.
- TX-DRP: 유실 송신 패킷.
- TX-OVR: packet overrun
- B: 브로드케스트 주소 지정
- L: 루프백 인터페이스
- M: 모든 패킷 수집(promiscuous mode)
- O: 인터페이스에 대한 ARP가 off 되어 있음.
- P: point-to-point 연결
- R: 인터페이스 동작중(Running)
- U: 인터페이스 up 인 상태.
2. Network 문제사항 및 에러확인
Network 관련설정은 처음 booting /etc/sysctrl.conf에 의해서 설정되며 없다면, kernel의 default 값으로 설정된다.
Network 정보 (TCP / UDP / IP / ICMP) 의 관련된사항들을 기록하였기때문에, 분석이 가능하며, 문제사항을 쉽게 파악이 가능하다.
netstat -s 옵션이외에도 proc 정보로도 확인가능
sysctl -a 설정정보확인가능
netstat -s , --statistics Display summary statistics for each protocol.
$ netstat -s // UDP / TCP / ICMP 사용통계 및 문제에러사항 확인
Ip:
1807814 total packets received
0 forwarded
0 incoming packets discarded
1188470 incoming packets delivered
955549 requests sent out
40 outgoing packets dropped
Icmp:
100 ICMP messages received
2 input ICMP message failed.
ICMP input histogram:
destination unreachable: 100
102 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 102
IcmpMsg:
InType3: 100
OutType3: 102
Tcp:
5894 active connections openings
15 passive connection openings
4 failed connection attempts
5748 connection resets received
6 connections established
965515 segments received
952551 segments send out
146 segments retransmited
0 bad segments received.
34 resets sent
Udp:
59434 packets received
82 packets to unknown port received.
0 packet receive errors
1454 packets sent
IgnoredMulti: 139426
UdpLite:
TcpExt:
102 TCP sockets finished time wait in fast timer
12122 delayed acks sent
38 delayed acks further delayed because of locked socket
Quick ack mode was activated 8 times
404422 packets directly queued to recvmsg prequeue.
36435935 bytes directly in process context from backlog
434921871 bytes directly received in process context from prequeue
152208 packet headers predicted
344744 packets header predicted and directly queued to user
1419 acknowledgments not containing data payload received
284057 predicted acknowledgments
41 congestion windows recovered without slow start after partial ack
36 timeouts in loss state
1 retransmits in slow start
46 other TCP timeouts
TCPLossProbes: 2
4 DSACKs sent for old packets
12 connections reset due to unexpected data
3 connections reset due to early user close
4 connections aborted due to timeout
IPReversePathFilter: 194
TCPRcvCoalesce: 104075
TCPOFOQueue: 2614
TCPOFOMerge: 3
TCPChallengeACK: 27
TCPSpuriousRtxHostQueues: 2
TCPAutoCorking: 68999
TCPFromZeroWindowAdv: 15
TCPToZeroWindowAdv: 22
TCPWantZeroWindowAdv: 122
TCPSynRetrans: 4
TCPOrigDataSent: 561796
TCPHystartTrainDetect: 3
TCPHystartTrainCwnd: 50
TCPACKSkippedChallenge: 31
TCPKeepAlive: 2
IpExt:
InNoRoutes: 1
InMcastPkts: 82733
OutMcastPkts: 2070
InBcastPkts: 139441
OutBcastPkts: 18
InOctets: 976384465
OutOctets: 295103139
InMcastOctets: 8631056
OutMcastOctets: 99130
InBcastOctets: 17773399
OutBcastOctets: 812
InNoECTPkts: 1819528
UDP Packet 의 RecvBufffer 의 문제를 알게되었는데, 이 문제를
net.core.rmem_max 변경과
socket option의 RecvBuff Size 올려주면 이를 에러율을 줄이는 것은 가능하다.
혹은
net.core.rmem_max 와
net.core.rmem_default 변경을 해주는 것인데, 이것보다는 위의 것이 더 나은 방법인 것 같다.
(/etc/sysctrl.conf 에서 변경가능)
$ netstat -su //UDP
IcmpMsg:
InType3: 84
OutType3: 1411
Udp:
119049 packets received
2339998 packets to unknown port received.
6006884 packet receive errors
2618 packets sent
RcvbufErrors: 6006884 //Recv Buffer Error , cat /proc/net/snmp | grep Udp 값 동일
IgnoredMulti: 494148
UdpLite:
IpExt:
InNoRoutes: 1
InMcastPkts: 170389
OutMcastPkts: 6945
InBcastPkts: 425852
OutBcastPkts: 18
InOctets: 14776499147
OutOctets: 3391504334
InMcastOctets: 11684052
OutMcastOctets: 279654
InBcastOctets: 39906199
OutBcastOctets: 812
InNoECTPkts: 12363330
UDP Packet Drop 현상 ( Test1, Test2 의 에러율 확인)
https://stackoverflow.com/questions/21664950/udp-packet-drop-inerrors-vs-rcvbuferrors
- Socket Buffer 설정정보확인 (proc)
OS마다 Socket에 할당하는 Memory는 다르며, 각 설정역시 다르기때문에 이 부분은 확인을 해야하며, 이부분도 부팅시 설정이 가능하다.
$ ls /proc/sys/net/core/
busy_poll dev_weight max_skb_frags netdev_budget netdev_tstamp_prequeue rmem_max tstamp_allow_data wmem_max xfrm_aevent_rseqth
busy_read flow_limit_cpu_bitmap message_burst netdev_max_backlog optmem_max rps_sock_flow_entries warnings xfrm_acq_expires xfrm_larval_drop
default_qdisc flow_limit_table_len message_cost netdev_rss_key rmem_default somaxconn wmem_default xfrm_aevent_etime
//Socket Recv Buffer (Memory)
$ cat /proc/sys/net/core/rmem_default //Socket Recv Buffer Default
229376
$ cat /proc/sys/net/core/rmem_max // Socket Recv Buffer Max
16777216
//Socket Write Buffer (Memory)
$ cat /proc/sys/net/core/wmem_default //Socket Write Buffer Default
$ cat /proc/sys/net/core/wmem_max // Socket Write Buffer Max
proc 정보를 이용하여 network 정보 확인
$ls /proc/sys/net/ipv4/
conf/ ipfrag_high_thresh tcp_fin_timeout tcp_reordering
fwmark_reflect ipfrag_low_thresh tcp_frto tcp_retrans_collapse
icmp_echo_ignore_all ipfrag_max_dist tcp_fwmark_accept tcp_retries1
icmp_echo_ignore_broadcasts ipfrag_secret_interval tcp_invalid_ratelimit tcp_retries2
icmp_errors_use_inbound_ifaddr ipfrag_time tcp_keepalive_intvl tcp_rfc1337
icmp_ignore_bogus_error_responses neigh/ tcp_keepalive_probes tcp_rmem
icmp_msgs_burst netfilter/ tcp_keepalive_time tcp_sack
icmp_msgs_per_sec ping_group_range tcp_limit_output_bytes tcp_slow_start_after_idle
icmp_ratelimit route/ tcp_low_latency tcp_stdurg
icmp_ratemask tcp_abort_on_overflow tcp_max_orphans tcp_syn_retries
igmp_link_local_mcast_reports tcp_adv_win_scale tcp_max_reordering tcp_synack_retries
igmp_max_memberships tcp_allowed_congestion_control tcp_max_syn_backlog tcp_thin_dupack
igmp_max_msf tcp_app_win tcp_max_tw_buckets tcp_thin_linear_timeouts
igmp_qrv tcp_autocorking tcp_mem tcp_timestamps
inet_peer_maxttl tcp_available_congestion_control tcp_min_rtt_wlen tcp_tso_win_divisor
inet_peer_minttl tcp_base_mss tcp_min_tso_segs tcp_tw_recycle
inet_peer_threshold tcp_challenge_ack_limit tcp_moderate_rcvbuf tcp_tw_reuse
ip_default_ttl tcp_congestion_control tcp_mtu_probing tcp_window_scaling
ip_dynaddr tcp_default_init_rwnd tcp_no_metrics_save tcp_wmem
ip_early_demux tcp_dsack tcp_notsent_lowat tcp_workaround_signed_windows
ip_forward tcp_early_retrans tcp_orphan_retries udp_mem
ip_forward_use_pmtu tcp_ecn tcp_pacing_ca_ratio udp_rmem_min
ip_local_port_range tcp_ecn_fallback tcp_pacing_ss_ratio udp_wmem_min
ip_local_reserved_ports tcp_fack tcp_probe_interval vs/
ip_no_pmtu_disc tcp_fastopen tcp_probe_threshold xfrm4_gc_thresh
$ cat /proc/sys/net/ipv4/tcp_mem
94236 125648 188472
netstat에서 보는 통계의 값과 유사하며, 네트워크 관리목적으로 사용이 되기 때문에, 추후 세부관리까지 한다면, SNMP를 이용하여 MIB로 정보를 제공하여 관리를 하도록 하자.
- TCP 관련부분 분석
$ cat /proc/net/snmp
Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests OutDiscards OutNoRoutes ReasmTimeout ReasmReqds ReasmOKs ReasmFails FragOKs FragFails FragCreates
Ip: 1 64 12843660 0 0 0 0 0 12843660 22324680 0 0 0 0 0 0 0 0 0
Icmp: InMsgs InErrors InCsumErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps
Icmp: 1 0 0 1 0 0 0 0 0 0 0 0 0 0 4 0 4 0 0 0 0 0 0 0 0 0 0
IcmpMsg: InType3 OutType3
IcmpMsg: 1 4
Tcp: RtoAlgorithm RtoMin RtoMax MaxConn ActiveOpens PassiveOpens AttemptFails EstabResets CurrEstab InSegs OutSegs RetransSegs InErrs OutRsts InCsumErrors
Tcp: 1 200 120000 -1 224248 11 2 3 3 12799177 22311554 11 0 15 0
Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors IgnoredMulti
Udp: 8452 3 0 3600 0 0 0 70
UdpLite: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors IgnoredMulti
UdpLite: 0 0 0 0 0 0 0 0
- UDP RcvBuffErrors 분석
RecvbufErrors가 지속적으로 갑자기 증가되는 문제발생하여 관련된 부분 디버깅
$ netstat -su
IcmpMsg:
InType3: 88
OutType3: 88
Udp:
6404859 packets received
88 packets to unknown port received.
2295229 packet receive errors
1148 packets sent
RcvbufErrors: 2295229
IgnoredMulti: 281547
UdpLite:
IpExt:
InMcastPkts: 10936
OutMcastPkts: 2477
InBcastPkts: 275312
OutBcastPkts: 18
InOctets: 12773054136
OutOctets: 240440435
InMcastOctets: 379933
OutMcastOctets: 102955
InBcastOctets: 23165236
OutBcastOctets: 812
InNoECTPkts: 9232445
$ cat /proc/net/snmp
Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests OutDiscards OutNoRoutes ReasmTimeout ReasmReqds ReasmOKs ReasmFails FragOKs FragFails FragCreates
Ip: 1 64 7026190 0 1 0 0 0 7025803 300352 40 0 0 0 0 0 0 0 0
Icmp: InMsgs InErrors InCsumErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps
Icmp: 88 0 0 88 0 0 0 0 0 0 0 0 0 0 88 0 88 0 0 0 0 0 0 0 0 0 0
IcmpMsg: InType3 OutType3
IcmpMsg: 88 88
Tcp: RtoAlgorithm RtoMin RtoMax MaxConn ActiveOpens PassiveOpens AttemptFails EstabResets CurrEstab InSegs OutSegs RetransSegs InErrs OutRsts InCsumErrors
Tcp: 1 200 120000 -1 37 5 2 6 3 190165 374801 39 0 25 0
Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors IgnoredMulti
Udp: 6155446 88 466455 1007 466455 0 0 206305
UdpLite: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors IgnoredMulti
UdpLite: 0 0 0 0 0 0 0 0
$ cat /proc/1695/net/udp // RX QUEUE를 Socket으로 6x1024x1024로 설정했지만, 아래와 같이 12x1024x1024로 증가되었음
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
229: 00000000:14E9 00000000:0000 07 00000000:00000000 00:00000000 00000000 110 0 33308 2 0000000000000000 0
306: 00000000:0536 00000000:0000 07 00000000:00C00300 00:00000000 00000000 1001 0 37343 2 0000000000000000 2295417 //RcvbufErrors 동일값
2136: 00000000:CC5C 00000000:0000 07 00000000:00000000 00:00000000 00000000 65534 0 34055 2 0000000000000000 0
3121: 0101007F:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 34043 2 0000000000000000 0
3121: 0237A8C0:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000 111 0 24293 2 0000000000000000 0
3121: 0100007F:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 29242 2 0000000000000000 0
3121: 0137A8C0:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 29240 2 0000000000000000 0
3136: 00000000:0044 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 300750 2 0000000000000000 0
3699: 00000000:0277 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 321449 2 0000000000000000 0
3812: 00000000:D2E8 00000000:0000 07 00000000:00000000 00:00000000 00000000 110 0 33310 2 0000000000000000 0
$ cat /proc/1695/oom_score
193
$ netstat -aun
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
udp 0 0 0.0.0.0:5353 0.0.0.0:*
udp 12585728 0 0.0.0.0:1334 0.0.0.0:* //00C00300 : 12583680 과 동일한 수치지만, 지속적으로 증가
udp 0 0 0.0.0.0:52316 0.0.0.0:*
udp 0 0 127.0.1.1:53 0.0.0.0:*
udp 0 0 192.168.55.2:53 0.0.0.0:*
udp 0 0 127.0.0.1:53 0.0.0.0:*
udp 0 0 192.168.55.1:53 0.0.0.0:*
udp 0 0 0.0.0.0:68 0.0.0.0:*
udp 0 0 0.0.0.0:631 0.0.0.0:*
udp 0 0 0.0.0.0:53992 0.0.0.0:*
udp6 0 0 :::5353 :::*
udp6 0 0 :::51705 :::*
udp6 0 0 fe80::c03:7dff:fea1::53 :::*
udp6 0 0 ::1:53 :::*
SOCKET의 SO_RCVBUF 설정
https://serverfault.com/questions/758885/rx-queue-value-in-proc-net-udp-grows-to-twice-the-so-rcvbuf-value
2.1 OS 및 Socket 관련 세부설정
상위 proc에서 확인된 socket 내부 buffer 설정을 linux에서는 sysctl 명령어로 변경이 가능하며, 이외 buffer 설정이외에도, 다양한 설정이 가능하다.
이부분은 별도의 Manual을 참조하자.
$ sysctl -a | grep mem
sysctl: permission denied on key 'fs.protected_hardlinks'
sysctl: permission denied on key 'fs.protected_symlinks'
sysctl: permission denied on key 'kernel.cad_pid'
sysctl: permission denied on key 'kernel.usermodehelper.bset'
.....
//229376 = 224 * 1024
//6291456 = 6 * 1024 * 1024
//3145728 = 3 * 1024 * 1024
//16777216 = 16 * 1024 *1024
//12583680 = 12 * 1024 * 1024
//26214400 = 25 * 1024*1024
net.core.optmem_max = 20480
net.core.rmem_default = 229376 // Byte (UDP/TCP)
net.core.rmem_max = 229376 // Byte (UDP/TCP)
net.core.wmem_default = 229376 // Byte (UDP/TCP)
net.core.wmem_max = 229376 // Byte (UDP/TCP)
net.ipv4.igmp_max_memberships = 20
sysctl: permission denied on key 'net.ipv4.tcp_fastopen_key'
sysctl: permission denied on key 'net.ipv6.conf.all.stable_secret'
net.ipv4.tcp_mem = 94236 125648 188472 // Page(4KB) , min / pressure / max
net.ipv4.tcp_rmem = 4096 87380 6291456 // Byte , min / pressure / max
net.ipv4.tcp_wmem = 4096 16384 4194304 // Byte , min / pressure / max
net.ipv4.udp_mem = 188472 251297 376944 // Page(4KB) min / pressure / max
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
net.ipv4.vs.amemthresh = 1024
sysctl: permission denied on key 'net.ipv6.conf.default.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.docker0.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.dummy0.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.eth0.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.l4tbr0.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.lo.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.tunl0.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.usb0.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.usb1.stable_secret'
sysctl: permission denied on key 'net.ipv6.conf.wlan0.stable_secret'
sysctl: permission denied on key 'vm.mmap_rnd_bits'
sysctl: permission denied on key 'vm.mmap_rnd_compat_bits'
vm.lowmem_reserve_ratio = 256 32
vm.overcommit_memory = 0
- Kernel의 내부 Socket Buffer 변경
실시간으로 network memory 변경
$ sysctl -w net.core.rmem_default="229376"
$ sysctl -w net.core.wmem_default="229376"
$ sysctl -w net.core.rmem_max="16777216"
$ sysctl -w net.core.wmem_max="16777216"
$ sysctl -w net.ipv4.tcp_rmem="229376 229376 16777216"
$ sysctl -w net.ipv4.tcp_wmem="229376 229376 16777216"
Recvbuffer 만 변경하여 테스트 진행
$ sudo sysctl -w net.core.rmem_default=26214400
$ sudo sysctl -w net.core.rmem_max=26214400
상위 설정을 테스트로 하여 초기 부팅시 아래의 설정에서 기본으로 설정으로 정하자.
$ sudo vi /etc/sysctl.conf
........
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 229376 16777216
net.ipv4.tcp_wmem = 4096 229376 16777216
........
$ ls /etc/sysctl.d/ // 확인
man sysctl.conf 참조
/proc 및 /sys 정보관련정보
https://www.kernel.org/doc/Documentation/sysctl/net.txt
http://www.ubuntugeek.com/performance-tuning-with-system-control-sysctl-in-ubuntu.html
https://kensei.co.kr/289
https://rocksea.tistory.com/64
https://meetup.toast.com/posts/53
https://wwwx.cs.unc.edu/~sparkst/howto/network_tuning.php
https://www.systutorials.com/241303/how-to-enlarge-linux-udp-buffer-size/
https://m.blog.naver.com/PostView.nhn?blogId=parkjy76&logNo=220874390930&proxyReferer=https%3A%2F%2Fwww.google.com%2F
https://stackoverflow.com/questions/31546835/tcp-receiving-window-size-higher-than-net-core-rmem-max
https://m.blog.naver.com/PostView.nhn?blogId=kwoncharlie&logNo=10109682872&proxyReferer=https%3A%2F%2Fwww.google.com%2F
- watch or shell script 를 이용하여 감시
$ sudo watch -d -n0 "netstat -atnp | grep ESTA" // -d, difference , -n interval second , grep ESTABLISHED 된것을 확인
$ man watch // 세부명령어 확인
TCP 의 Socket 상태도
https://upload.wikimedia.org/wikipedia/commons/e/e4/Tcp_state_diagram.svg
