Jeonghun (James) Lee: Debug-Trace

레이블이 Debug-Trace인 게시물을 표시합니다. 모든 게시물 표시

5/02/2021

ESP32 Backtrace 와 addr2line

1. ESP32의 Monitor 기능

esp32 개발환경에서 제공해주는 idf.py -p /dev/ttyUSBx monitor , 즉 monitor 기능은 기본적으로 gdb option을 포함하고 있다.

그래서, abort 시 backtrace가 가능한데, 이때 사용하는 기능이 addr2line 기능이며, 이는 각 Stack의 Address를 Code의 위치를 찾는다.

기본기능으로 아래와 같이 addr2line을 이용하여 위치를 찾아주며, 이 정보는 stderr로 표시해주고 있다.

ESP32가 Exception 이 발생할때 아래와 같이 Dump와 Backtrace 정보출력

Guru Meditation Error of type StoreProhibited occurred on core  0. Exception was unhandled.
Register dump:
PC      : 0x400f360d  PS      : 0x00060330  A0      : 0x800dbf56  A1      : 0x3ffb7e00
A2      : 0x3ffb136c  A3      : 0x00000005  A4      : 0x00000000  A5      : 0x00000000
A6      : 0x00000000  A7      : 0x00000080  A8      : 0x00000000  A9      : 0x3ffb7dd0
A10     : 0x00000003  A11     : 0x00060f23  A12     : 0x00060f20  A13     : 0x3ffba6d0
A14     : 0x00000047  A15     : 0x0000000f  SAR     : 0x00000019  EXCCAUSE: 0x0000001d
EXCVADDR: 0x00000000  LBEG    : 0x4000c46c  LEND    : 0x4000c477  LCOUNT  : 0x00000000

Backtrace: 0x400f360d:0x3ffb7e00 0x400dbf56:0x3ffb7e20 0x400dbf5e:0x3ffb7e40 0x400dbf82:0x3ffb7e60 0x400d071d:0x3ffb7e90

addr2line이용하여 추적가능

Guru Meditation Error of type StoreProhibited occurred on core  0. Exception was unhandled.
Register dump:
PC      : 0x400f360d  PS      : 0x00060330  A0      : 0x800dbf56  A1      : 0x3ffb7e00
0x400f360d: do_something_to_crash at /home/gus/esp/32/idf/examples/get-started/hello_world/main/./hello_world_main.c:57
(inlined by) inner_dont_crash at /home/gus/esp/32/idf/examples/get-started/hello_world/main/./hello_world_main.c:52
A2      : 0x3ffb136c  A3      : 0x00000005  A4      : 0x00000000  A5      : 0x00000000
A6      : 0x00000000  A7      : 0x00000080  A8      : 0x00000000  A9      : 0x3ffb7dd0
A10     : 0x00000003  A11     : 0x00060f23  A12     : 0x00060f20  A13     : 0x3ffba6d0
A14     : 0x00000047  A15     : 0x0000000f  SAR     : 0x00000019  EXCCAUSE: 0x0000001d
EXCVADDR: 0x00000000  LBEG    : 0x4000c46c  LEND    : 0x4000c477  LCOUNT  : 0x00000000

Backtrace: 0x400f360d:0x3ffb7e00 0x400dbf56:0x3ffb7e20 0x400dbf5e:0x3ffb7e40 0x400dbf82:0x3ffb7e60 0x400d071d:0x3ffb7e90
0x400f360d: do_something_to_crash at /home/gus/esp/32/idf/examples/get-started/hello_world/main/./hello_world_main.c:57
(inlined by) inner_dont_crash at /home/gus/esp/32/idf/examples/get-started/hello_world/main/./hello_world_main.c:52
0x400dbf56: still_dont_crash at /home/gus/esp/32/idf/examples/get-started/hello_world/main/./hello_world_main.c:47
0x400dbf5e: dont_crash at /home/gus/esp/32/idf/examples/get-started/hello_world/main/./hello_world_main.c:42
0x400dbf82: app_main at /home/gus/esp/32/idf/examples/get-started/hello_world/main/./hello_world_main.c:33
0x400d071d: main_task at /home/gus/esp/32/idf/components/esp32/./cpu_start.c:254

Backtrace는 2개의 Address로 구성되며, PC와 A1, 즉 , Arg라고 짐작 (세부내용은 ABI 참조해야함)

addr2line 이용하여 소스위치확인

$ xtensa-esp32-elf-addr2line -pfiaC -e build/PROJECT.elf ADDRESS  // 상위 Address 입력하면 각 소스위치파악

ESP32 Exception이 발생할 경우 순서도

GDB-Stub 사용

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/fatal-errors.html#gdb-stub

ESP32 Exception, Fatal Errors

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/fatal-errors.html#fatal-errors

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/fatal-errors.html#guru-meditation-errors

ESP32용 Backtrace Shell Script

https://github.com/tve/esp32-backtrace

1.1 backtrace 와 addr2line 기본사용법

backtrace는 Stack Trace라고도 하며, 기본적으로 함수의 Call Stack을 추적하는 방법을 말하며, 주로 사용되어지는 함수가 backtrace기능이다.

이때 보조적으로 Code의 위치를 찾아주는 기능이 addr2line tool 이며, 이를 이용하여 쉽게 어디서 호출되어지는 분석이 가능하다.

https://en.wikipedia.org/wiki/Stack_trace

https://www.gnu.org/software/libc/manual/html_node/Backtraces.html

기본설정

GCC 로 Compile 시 반드시 -g 옵션 과 -rdynamic (동적 Library)

backtrace 함수와 addr2line 사용법

모든 예제를 설명을 아래 사이트로 참고

https://m.blog.naver.com/PostView.nhn?blogId=hermet&logNo=198926090&proxyReferer=https:%2F%2Fwww.google.com%2F

https://jybaek.tistory.com/469

https://sjp38.github.io/ko/post/backtrace_c_language/

1.2 addr2line 이용

ESP32는 기존의 Backtrace처럼 사용하지 않는 구조로 보이며, Backtrace가 아래와 같이 나오면 addr2line을 이용하여 쉽게 Source 위치 파악은 가능하다

Backtrace:0x40090677:0x3ffe3710 0x40090c89:0x3ffe3730 0x400955fa:0x3ffe3750 0x400da8f2:0x3ffe37c0 0x400da9f0:0x3ffe37e0 0x400ec484:0x3ffe3800 0x400ec516:0x3ffe3840 0x400ec651:0x3ffe3870 0x400ec6e4:0x3ffe38a0 0x400daa13:0x3ffe38c0 0x400db275:0x3ffe38f0 0x400db4fa:0x3ffe3920 0x400d8deb:0x3ffe3a70 0x400d978c:0x3ffe3a90 0x40090cf9:0x3ffe3ac0

$ xtensa-esp32-elf-addr2line 0x40090677 -e ./build/demos  
$ xtensa-esp32-elf-addr2line 0x40090c89 -e ./build/demos

1.3 이외 Tool 사용

Linux Kernel 처럼 map 분석

Symbol Table 과 Linker Script Section 과 Library 비롯하여 모든 기본정보 확인

$ cat ./build/demos.map    
// Discarded input sections     (Linker Scirpt 의 Section 과 상대주소 파악 ) 
// Memory Configuration         (실제 Memory 영역과 Attribute로 x/r/w 확인 ) 
// Linker script and memory map (Linker Scirpt 와 같이 절대주소 파악 )
// Cross Reference Table        (Symbol Table로 Symbol의 사용여부 확인가능)

readelf 로 Library 분석 및 ELF 분석

각 Library 및 ELF Format 관련정보 확인

$ xtensa-esp32-elf-readelf -a ./build/esp-idf/app_trace/libapp_trace.a    //Build 내에서 사용하는 ESP-IDF Library 분석 (-a 전체정보)  
$ xtensa-esp32-elf-readelf -h ./build/esp-idf/app_trace/libapp_trace.a                           //ELF 기본정보확인 (Endian/Entry Point/Machine)   
$ xtensa-esp32-elf-readelf -s ./build/esp-idf/app_trace/libapp_trace.a | grep function           //library Symbol Table 확인   
$ xtensa-esp32-elf-readelf -S ./build/demo                          //Linker Script Section 확인

5/19/2020

ftrace 와 systrace (자료수집)

각 Ftrace 와 systrace 와 기타 자료수집을 하며, 별도의 설명을 하지 않겠다.

NVIDIA-IDE Nsight(Ecplise 기반)
https://ahyuo79.blogspot.com/2019/05/nvidia-nsight-tools-jetpack-42.html

systrace는 Google Android팀과 Google Chrome팀의 소유

systrace 의 이해

https://source.android.com/devices/tech/debug/systrace
https://developer.android.com/studio/profile/systrace.html

Android systrace 관련자료

  https://source.android.com/devices/tech/perf/boot-times#systrace
  https://developer.android.com/studio/profile/systrace
  https://google-developer-training.github.io/android-developer-advanced-course-practicals/unit-2-make-your-apps-fast-and-small/lesson-4-performance/4-1c-p-systrace-and-dumpsys/4-1c-p-systrace-and-dumpsys.html

Trace compass IDE (Ecplise 기반)

https://archive.eclipse.org/tracecompass.incubator/doc/org.eclipse.tracecompass.incubator.ftrace.doc.user/User-Guide.html

Jetpack

https://developer.android.com/jetpack

Ftrace

  https://source.android.com/devices/tech/debug/ftrace
  https://source.android.google.cn/devices/tech/debug/ftrace?hl=ko
  https://www.kernel.org/doc/Documentation/trace/ftrace.txt

GDB (VSCode)

https://source.android.google.cn/devices/tech/debug/gdb?hl=ko
CONFIG_HZ
https://source.android.google.cn/devices/tech/debug/jank_jitter?hl=ko

4/22/2020

Yocto User Config 의 Profiling and Tracing

1. Yocto 의 Profile 기능 추가 및 사용

Yocto 기반으로 Profile 기능과 Trace Debug기능을 추가하여 사용하는 방법을 알아보기 위해서 아래와 같이 설정변경 후,
이를 기반으로 Profile 과 Tracing 사용을 해본다.

Yocto에서 Profile 과 Tracing은 소스를 세부적으로 분석하는 Debug 기술이자, 성능측정도 가능하므로 중요한 기술이다.

이를 제대로 사용하려면 각 IDE Tool 도 알아두어야 한다

대부분 Ecplise 기반으로 제공되어지며, 각 Chip 제조사 Application들도 보면 기반을 다 Ecplise 기반으로 되어있다.

Ecplise/Ecplise-CDT

https://ahyuo79.blogspot.com/search/label/IDE-Eclipse

https://ahyuo79.blogspot.com/search/label/IDE-Eclipse-CDT

Trace Compass
https://ahyuo79.blogspot.com/search/label/IDE-Trace%20Compass

TI CCS (TI사 제공) 이외 각 제조사 IDE 참조

https://ahyuo79.blogspot.com/search/label/IDE-CCSv6

1.1 Yocto 의 기본문법 및 User Config 이해

Yocto의 기본적인 이해

Yocto 기본 사용
https://ahyuo79.blogspot.com/2015/10/yocto.html
https://ahyuo79.blogspot.com/2015/10/imx6.html

Yocto 기본문법 과 정리 (필독)
https://ahyuo79.blogspot.com/2020/01/raspberry-pi3-linux-yocto.html

Yocto Recipe 작성방법 (반드시 확인 및 이해)

https://ahyuo79.blogspot.com/2020/02/yocto-recipe.html

Yocto User Configuration 수정방법
https://ahyuo79.blogspot.com/2020/02/yocto-user-configuration.html

기본 Yocto Recipe 및 Kernel Recipe 수정방법
https://ahyuo79.blogspot.com/2020/02/yocto-recipe.html

Yocto sysLog 구성

https://ahyuo79.blogspot.com/2020/04/yocto-syslogd.html

1.2 Yocto User Configuration 설정변경

Yocto 의 User Configuration 수정

$ cat conf/local.conf  //일반적인 설정

## Yocto Version에따라 설정변경될 수 있음 아래 Manual 참조
## 일반적인 EXTRA_IMAGE_FEATURES 설정으로 debug 기본정보포함 (기본 /sys/kernel/debug는 동작됨)
EXTRA_IMAGE_FEATURES ?= "debug-tweaks"

별도로 추가하여 설정

## Profile 과 Tracing 을 위해 별도로 추가되는 명령들 
## IMAGE의 Profile Tool을 추가 
EXTRA_IMAGE_FEATURES += "tools-profile"

## Package들에 Debug 정보를 추가 (bitbake를 이용하여 xxx-dbg Manual로 해도됨)
EXTRA_IMAGE_FEATURES += "dbg-pkgs"

## Yocto는 default로 strip을 하여 debug정보 및 삭제하므로 이를 방지 
INHIBIT_PACKAGE_STRIP = "1"

##  *-dbg packages 만들 때 Bin 와 Debug 정보를 어떻게 분리할지를 결정 (bin 에 .debug 생성) 
PACKAGE_DEBUG_SPLIT_STYLE = 'debug-file-directory'

** EXTRA_IMAGE_FEATURES 는 항상 Yocto Version에 따라 반드시 참조

gdb 와 같이 사용하고자 하면 아래와 같이 설정

EXTRA_IMAGE_FEATURES += "dbg-pkgs"

EXTRA_IMAGE_FEATURES += "tools-debug"

Yocto에서 deb 방식으로 package 관리할 경우 (apt-get)

https://imxdev.gitlab.io/tutorial/How_to_apt-get_to_the_Yocto_Project_image/

EXTRA_IMAGE_FEATURES
https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#ref-features-image

1.3 Yocto 의 Kernel Config 수정

WORKDIR 에서 현재 Kernel Config 확인

Yocto의 Build space에서 직접 빌드된 Kernel Source의 Config 확인

$ vi ./tmp/work/imx6sxsabresd-poky-linux-gnueabi/linux-imx/4.19.35-r0/build/.config  //직접 찾아서 확인 
CONFIG_BUILD_LTTNG       //확인했으나 없음
CONFIG_FUNCTION_TRACER   //확인했으나 없음 

$ vi ../sources/meta-fsl-bsp-release/imx/meta-bsp/recipes-kernel/linux/linux-imx_4.19.35.bbappend // Kernel Recipe가 문제가 있음 
# 부분 막음 
do_preconfigure_append()

https://en.wikipedia.org/wiki/Ftrace

Kernel 의 CONFIG 설정변경

$ bitbake linux-imx  -c configme    // patch 후 *cfg를 적용을 한다고하는데, 동작안됨
$ bitbake linux-imx  -c menuconfig    // 설정변경
//각 메뉴 선택 후 HELP에서 CONFIG 확인가능

//이미 설정되어있음 
> Kernel hacking > Compile-time checks and compiler options > Debug Filesystem (CONFIG_DEBUG_FS)

//새로설정 
> Kernel hacking > Tracers                                            // (CONFIG_FTRACE)
> Kernel hacking > Tracers > Kernel Function Tracer                  // (CONFIG_FUNCTION_TRACER)
> Kernel hacking > Tracers > Kernel Function Graph Tracer            // (CONFIG_FUNCTION_GRAPH_TRACER)
> Kernel hacking > Tracers > Interrupts-off Latency Tracer           // (CONFIG_IRQSOFF_TRACER)
> Kernel hacking > Tracers > Preemption-off Latency Tracer           // (CONFIG_PREEMPT_TRACER)
> Kernel hacking > Tracers > Scheduling Latency Tracer               // (CONFIG_SCHED_TRACER)
> Kernel hacking > Tracers > Trace max stack                         // (CONFIG_STACK_TRACER)
> Kernel hacking > Tracers > Support for tracing block IO actions    // (CONFIG_BLK_DEV_IO_TRACE)
> Kernel hacking > Tracers > Kernel function profiler                // (CONFIG_FUNCTION_PROFILER)
CONFIG_DYNAMIC_FTRACE="y"                                              // 찾지못함 (상위 설정시 자동설정됨)
CONFIG_FTRACE_MCOUNT_RECORD="y"                                        // 찾지못함 (상위 설정시 자동설정됨)


//추후 별도로 추가 
> Kernel hacking > Tracers > Trace syscalls                          // (CONFIG_FTRACE_SYSCALLS)

fragment가 생성되었으나 이전 .config 의 나의 설정과 다름 (주의)
추후 필요한 것만 직접 적용하여 넣도록하자

$ bitbake linux-imx  -c diffconfig    //*.cfg 생성 후 이를 Kernel Recipe에 추가 
....
......../fragment.cfg

$ cat fragment.cfg  // 상위와 다르게 나옴 자세히 보면 선택한 것도 미선택으로 됨 
CONFIG_TRACEPOINTS=y
CONFIG_UPROBES=y
# CONFIG_NET_DROP_MONITOR is not set
# CONFIG_ATH6KL_TRACING is not set
# CONFIG_MXC_GPU_VIV is not set
CONFIG_BINARY_PRINTF=y
# CONFIG_DEBUG_PAGE_REF is not set
CONFIG_TRACE_IRQFLAGS=y
CONFIG_STACKTRACE=y
CONFIG_NOP_TRACER=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_TRACE_CLOCK=y
CONFIG_RING_BUFFER=y
CONFIG_EVENT_TRACING=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_RING_BUFFER_ALLOW_SWAP=y
CONFIG_PREEMPTIRQ_TRACEPOINTS=y
CONFIG_TRACING=y
CONFIG_GENERIC_TRACER=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
# CONFIG_FUNCTION_GRAPH_TRACER is not set
CONFIG_TRACE_PREEMPT_TOGGLE=y
# CONFIG_PREEMPTIRQ_EVENTS is not set
CONFIG_IRQSOFF_TRACER=y
CONFIG_PREEMPT_TRACER=y
CONFIG_SCHED_TRACER=y
# CONFIG_HWLAT_TRACER is not set
# CONFIG_FTRACE_SYSCALLS is not set
CONFIG_TRACER_SNAPSHOT=y
CONFIG_TRACER_SNAPSHOT_PER_CPU_SWAP=y
CONFIG_BRANCH_PROFILE_NONE=y
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
# CONFIG_PROFILE_ALL_BRANCHES is not set
CONFIG_STACK_TRACER=y
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_UPROBE_EVENTS=y
CONFIG_PROBE_EVENTS=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_FUNCTION_PROFILER=y
CONFIG_FTRACE_MCOUNT_RECORD=y
# CONFIG_FTRACE_STARTUP_TEST is not set
# CONFIG_TRACEPOINT_BENCHMARK is not set
# CONFIG_RING_BUFFER_BENCHMARK is not set
# CONFIG_RING_BUFFER_STARTUP_TEST is not set
# CONFIG_PREEMPTIRQ_DELAY_TEST is not set
# CONFIG_TRACE_EVAL_MAP_FILE is not set
CONFIG_TRACING_EVENTS_GPIO=y

$ cp ./build/tmp/work/imx6sxsabresd-poky-linux-gnueabi/linux-imx/4.19.35-r0/fragment.cfg ../sources/meta-fsl-bsp-release/imx/meta-bsp/recipes-kernel/linux/files/kernel_ftrace.cfg
//적용 Kernel Recipe 수정

ftrace 의 Kernel config 설정
  https://stackoverflow.com/questions/41238386/how-to-enable-or-configure-ftrace-module
  https://docs.google.com/presentation/d/13zIFUTTdChr7JNpZnb1K56aUsv_E1cOWausjQpsqcGc/htmlpresent
  https://docs.windriver.com/bundle/Wind_River_Linux_Tutorial_Dynamic_Kernel_Debugging_with_ftrace_and_kprobes_LTS_1/page/zjd1552591139310.html

Kernel 만 다시빌드 및 배포

$ bitbake linux-imx  -c compile -f    // -f force로 강제로 빌드 후 .config 설정확인 
$ bitbake linux-imx  -c deploy        // 이를 적용 

$ bitbake linux-imx  -c clean        // 전체삭제 
$ bitbake linux-imx          // 전체동작 (오동작) 
$ bitbake linux-imx  -c configure         // 전체동작 (오동작)

https://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#var-KBUILD_DEFCONFIG
https://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#kernel-related-tasks

bitbake로 image 생성

$ bitbake core-image-base

Yocto Profile manual
Zeus(3.0)
  https://yoctoproject.org/docs/3.0/profile-manual/profile-manual.html
Warrior(2.7)
  https://yoctoproject.org/docs/2.7/profile-manual/profile-manual.html
Fido(1.8)
  https://yoctoproject.org/docs/1.8/profile-manual/profile-manual.html

1.4 Yocto 설정 변경 시 문제사항

처음 발생하는 문제

Parition을 고정 크기를 했을 경우, Debug 정보가 들어가기 때문에 나의 경우는 약 4배 좀 더 용량이 더 커져서 문제발생
이 문제는 아래의 WIC 부분 참조

두번째 발생하는 문제

빌드가 잘 된 후 아래와 같이 Image를 적용 후 Booting 하며 문제발생
아래문제도 역시 /var/log 와 kernel size 문제를 제거후 해결됨

.......
         Starting Flush Journal to Persistent Storage...
systemd-journald[164]: Received request to flush runtime journal from PID 1
[  OK  ] Started Flush Journal to Persistent Storage.
[    **] A start job is running for dev-mmcblk3p5.device (24s / 1min 30s)random: crng init done
random: 7 urandom warning(s) missed due to ratelimiting
[ TIME ] Timed out waiting for device dev-mmcblk3p5.device.

https://www.linode.com/community/questions/17915/why-did-my-server-miss-urandom-warnings-due-to-rate-limiting

Yocto 의 WIC Partition (해결방법)

Partition Size도 가변사이즈로 변경(고정사이즈일 경우)
Partition 도 삭제했다면 sources에서도 fstab도 같이 수정
/var/log 부분기능삭제

https://ahyuo79.blogspot.com/2020/02/yocto-partition.html

2. Yocto의 Profile 및 Trace 확인 및 사용

현재 설정된 기본기능확인 (EXTRA_IMAGE_FEATURES 확인)

$ cat /etc/fstab    // /sys/debug/kernel 별도로 없으며, 상위 debug-tweaks 참조  

$ ls /sys/kernel/debug  // 상위 (EXTRA_IMAGE_FEATURES ?= "debug-tweaks") 적용 부터 동작 
2100000.caam/       extfrag/            pm_genpd/
bdi/                fault_around_bytes  pm_qos/
block/              gpio                pwm
ci_hdrc.0/          hid/                regmap/
ci_hdrc.1/          iio/                regulator/
clear_warn_once     memblock/           sleep_time
clk/                mmc2/               suspend_stats
device_component/   mmc3/               ubi/
devices_deferred    mtd/                ubifs/
dma_buf/            opp/                usb/
dri/                pinctrl/            wakeup_sources

$ cat /sys/kernel/debug/gpio  // GPIO Debug   (sysfs도 이용하지만, device tree에서 직접 모듈로 연결가능) 
....
 gpio-112 (                    |peri_3v3            ) out hi  // sys file이 아닌 device tree에서 gpio 연결 
 ...
 gpio-114 (                    |sysfs               ) out lo         
 ...
 gpio-160 (                    |sysfs               ) out hi  // sys file에서 export 한 다음 direction 설정 후 값 설정
 gpio-161 (                    |sysfs               ) out hi
 gpio-170 (                    |spi_imx             ) out hi

// PinCtrl 관련설정값을 쉽게 확인 (device tree의 설정값과 비교) 
$ cat /sys/kernel/debug/pinctrl/pinctrl-handles 

// PinCtrl 관련설정값을 세부적으로 확인 (device tree의 설정값과 비교) 
$ cat /sys/kernel/debug/pinctrl/pinctrl-maps 

// USB 정보 lsusb 
$ cat /sys/kernel/debug/usb/devices

T:  Bus=01 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#=  1 Spd=480  MxCh= 1
B:  Alloc=  0/800 us ( 0%), #Int=  1, #Iso=  0
D:  Ver= 2.00 Cls=09(hub  ) Sub=00 Prot=01 MxPS=64 #Cfgs=  1
P:  Vendor=1d6b ProdID=0002 Rev= 4.19
S:  Manufacturer=Linux 4.19.35-1.1.0+g0f9917c ehci_hcd
S:  Product=EHCI Host Controller
S:  SerialNumber=ci_hdrc.1
C:* #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr=  0mA
I:* If#= 0 Alt= 0 #EPs= 1 Cls=09(hub  ) Sub=00 Prot=00 Driver=hub
E:  Ad=81(I) Atr=03(Int.) MxPS=   4 Ivl=256ms 
......

// PMIC Regulator 정보 
$ ls /sys/kernel/debug/regulator/   
0-0008-COIN
0-0008-SW1AB
0-0008-SW2
0-0008-SW3A
0-0008-SW3B
0-0008-SWBST
0-0008-VGEN1
0-0008-VGEN2
0-0008-VGEN3
0-0008-VGEN4
0-0008-VGEN5
0-0008-VGEN6
0-0008-VREFDDR
0-0008-VSNVS
.......

$ perf  // EXTRA_IMAGE_FEATURES 추가설정 후 존재확인

$ lttng           // lttng          확인
$ lttng-crash     // lttng-crash    확인
$ lttng-relayd    // lttng-relayd   확인
$ lttng-sessiond  // lttng-sessiond 확인

$ babeltrace      // babeltrace 확인
$ babeltrace-log  // babeltrace-log

$ blktrace  //blktrace  존재확인  (Block Device Trace) 
$ blkparse  //blkparse  존재확인   
$ iowatcher  //blktrace  visual tool   

$ dtrace  //dtrace

$ ls /sys/kernel/debug/tracing  // 미존재하면 Kernel Config 수정 필요 
README                      set_ftrace_filter
available_events            set_ftrace_notrace
available_filter_functions  set_ftrace_pid
available_tracers           snapshot
buffer_size_kb              stack_max_size
buffer_total_size_kb        stack_trace
current_tracer              stack_trace_filter
dyn_ftrace_total_info       timestamp_mode
enabled_functions           trace
events                      trace_clock
free_buffer                 trace_marker
function_profile_enabled    trace_marker_raw
instances                   trace_options
options                     trace_pipe
per_cpu                     trace_stat
printk_formats              tracing_cpumask
saved_cmdlines              tracing_max_latency
saved_cmdlines_size         tracing_on
saved_tgids                 tracing_thresh
set_event                   uprobe_events
set_event_pid               uprobe_profile

2.1 Ftrace의 기본동작 확인

$ mount -t debugfs nodev /sys/kernel/debug

$ echo 1 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable
$ echo 1 > /sys/kernel/debug/tracing/events/sched/sched_switch/enable
$ echo 1 > /sys/kernel/debug/tracing/events/syscalls/enable

## Start Recording 
$ echo 1 > /sys/kernel/debug/tracing/tracing_on

$ cat /sys/kernel/debug/tracing/trace > myFtraceFile.txt


$ cat myFtraceFile.txt  | head -n 20
# tracer: nop
#
# entries-in-buffer/entries-written: 6606/6606   #P:1
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
          <idle>-0     [000] dns4  2163.718052: sched_wakeup: comm=rcu_preempt pid=10 prio=120 target_cpu=000
          <idle>-0     [000] dns4  2163.718080: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000
          <idle>-0     [000] dns4  2163.728019: sched_wakeup: comm=rcu_preempt pid=10 prio=120 target_cpu=000
          <idle>-0     [000] dnH5  2163.728046: sched_wakeup: comm=kworker/0:0 pid=342 prio=120 target_cpu=000
          <idle>-0     [000] dns4  2163.728068: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000
          <idle>-0     [000] dns4  2163.738009: sched_wakeup: comm=rcu_preempt pid=10 prio=120 target_cpu=000
          <idle>-0     [000] dns4  2163.738035: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000
          <idle>-0     [000] dns4  2163.758001: sched_wakeup: comm=kworker/0:0 pid=342 prio=120 target_cpu=000
          <idle>-0     [000] dns4  2163.768017: sched_wakeup: comm=rcu_preempt pid=10 prio=120 target_cpu=000

  https://archive.eclipse.org/tracecompass.incubator/doc/org.eclipse.tracecompass.incubator.ftrace.doc.user/User-Guide.html
  https://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/Trace-Compass-Main-Features.html
  https://archive.eclipse.org/tracecompass.incubator/doc/org.eclipse.tracecompass.incubator.kernel.doc.user/User-Guide.html

2.2 LTTng 사용방법

아래와 같이 Ecplise를 이용하여 LTTng와 함께 쉽게 디버깅가능

LTTng Command 사용법 및 Trace Compass or Ecplise에서 분석

$ lttng create mySession  // Trace Compass or Ecplise 의 LTTng의 Control의 SSH로 접속가능 

$ lttng enable-event sched_switch -k   // Trace Compass의 Control의 Provider의 Kernel에서도 설정가능 

$ lttng start // Trace Compass의 Control의 Session의 mySession에서 Start 가능 

$ lttng stop // Trace Compass의 Control의 Session의 mySession에서 Stop 가능 

$ lttng view // Consol에서 정보보기 

$ lttng destroy mySession  // 본인이 필요 없다면 삭제 , 분석을 한다면 미삭제 

$ tar cvzf lttngtest.tar.gz lttng-traces   // 이안에 상위에서 만든 각각의 Session들이 존재

Ecplise 에서 LTTng 분석

LTTng->Control에서는 SSH로 Target에 연결하여 Control로 가능
lttng create Session만 진행 후 모든 것을 Remote로 진행가능
상위에서 저장된 lttngtest.tar.gz 정보를 Trace Project에서 import하여 분석가능도 가능

관련참조자료
https://www.nxp.com/docs/en/application-note/AN5172.pdf

3. EXTRA_IMAGE_FEATURE의 확장 설정

User Config의 local config 부분에 확장추가

$ cat conf/local.conf
## Yocto Profile Manual 부분의 참고 (미테스트)
## Profile Tool (perf, systemtap, and LTTng ) 설치  (Warrior 사용가능)
EXTRA_IMAGE_FEATURES += "perf"

## Debugging Tool 설치 ( strace and gdb)  (Warrior 사용가능)
EXTRA_IMAGE_FEATURES += "tools-debug"

https://wiki.yoctoproject.org/wiki/Tracing_and_Profiling#General_Setup
https://wiki.yoctoproject.org/wiki/Tracing_and_Profiling#Collecting_and_viewing_a_trace_in_Eclipse

IMAGE_FEATURE 기능의 변화

사용되는 Yocto의 Version에 따라 IMAGE FEATURE의 기능이 다르므로 주의

Zeus(3.0)
  https://www.yoctoproject.org/docs/3.0/ref-manual/ref-manual.html#ref-features-image
Warrior(2.7)
  https://www.yoctoproject.org/docs/2.7/ref-manual/ref-manual.html#ref-features-image
Fido(1.8)
  https://www.yoctoproject.org/docs/1.8/ref-manual/ref-manual.html#ref-features-image

Yocto SDK Manual
  https://www.yoctoproject.org/docs/2.7/sdk-manual/sdk-manual.html

Intel Vtune
  https://software.intel.com/content/www/us/en/develop/documentation/vtune-help/top/set-up-analysis-target/embedded-linux-targets/configure-yocto-project-with-linux-target-package.html
  https://software.intel.com/content/www/us/en/develop/documentation/vtune-help/top.html

4/12/2019

strace

1. STRACE란?

strace는 SunOS에서 System Call을 Trace를 하기위해서 만들어진 Debug Tool이며, 이를 이용하여 실행중인 Program도 Trace가 가능하다.

$ sudo apt-get install strace

https://en.wikipedia.org/wiki/Strace

1.1 STRACE 기본사용법

$ strace -h
usage: strace [-CdffhiqrtttTvVwxxy] [-I n] [-e expr]...
              [-a column] [-o file] [-s strsize] [-P path]...
              -p pid... / [-D] [-E var=val]... [-u username] PROG [ARGS]
   or: strace -c[dfw] [-I n] [-e expr]... [-O overhead] [-S sortby]
              -p pid... / [-D] [-E var=val]... [-u username] PROG [ARGS]

Output format:
  -a column      alignment COLUMN for printing syscall results (default 40)
  -i             print instruction pointer at time of syscall
  -o file        send trace output to FILE instead of stderr
  -q             suppress messages about attaching, detaching, etc.
  -r             print relative timestamp
  -s strsize     limit length of print strings to STRSIZE chars (default 32)
  -t             print absolute timestamp
  -tt            print absolute timestamp with usecs
  -T             print time spent in each syscall
  -x             print non-ascii strings in hex
  -xx            print all strings in hex
  -y             print paths associated with file descriptor arguments
  -yy            print ip:port pairs associated with socket file descriptors

Statistics:
  -c             count time, calls, and errors for each syscall and report summary
  -C             like -c but also print regular output
  -O overhead    set overhead for tracing syscalls to OVERHEAD usecs
  -S sortby      sort syscall counts by: time, calls, name, nothing (default time)
  -w             summarise syscall latency (default is system time)

Filtering:
  -e expr        a qualifying expression: option=[!]all or option=[!]val1[,val2]...
     options:    trace, abbrev, verbose, raw, signal, read, write
  -P path        trace accesses to path

Tracing:
  -b execve      detach on execve syscall
  -D             run tracer process as a detached grandchild, not as parent
  -f             follow forks
  -ff            follow forks with output into separate files
  -I interruptible
     1:          no signals are blocked
     2:          fatal signals are blocked while decoding syscall (default)
     3:          fatal signals are always blocked (default if '-o FILE PROG')
     4:          fatal signals and SIGTSTP (^Z) are always blocked
                 (useful to make 'strace -o FILE PROG' not stop on ^Z)

Startup:
  -E var         remove var from the environment for command
  -E var=val     put var=val in the environment for command
  -p pid         trace process with process id PID, may be repeated
  -u username    run command as username handling setuid and/or setgid

Miscellaneous:
  -d             enable debug output to stderr
  -v             verbose mode: print unabbreviated argv, stat, termios, etc. args
  -h             print help message
  -V             print version

http://man7.org/linux/man-pages/man1/strace.1.html

1.2 /proc를 이용하여 동작중인 Process 확인

$ sudo cat /proc/550/maps  // 실행과 관련된 공유라이브러리 확인 
$ sudo cat /proc/550/status  // Process 상태확인 
$ readelf -h /proc/550/exe    // 실행파일 확인 
$ readelf -s /proc/550/exe    // symbol table 확인

1.3 PID 기반으로 STRACE 사용

동작 중이 Process 기반으로 아래와 같이 PID를 넣어 직접 Profile을 해보고 Debug을 해봅니다.

상위 동작 중인 PID 기반으로 Debug

$ strace -p 550 -f -t -s 65535 // -p PID -f fork된 밑에 process들도 같이 분석, -t timestamp 출력  -s string 사이즈 변경 
...........
[pid   994] 11:25:13 pselect6(0, NULL, NULL, NULL, {0, 10000000}, NULL) = 0 (Timeout)
[pid   994] 11:25:13 fstat(31, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
[pid   994] 11:25:13 fstat(31, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
[pid   994] 11:25:13 read(31, "count 180\nunhandled 0\nlast_unhan"..., 1024) = 42
...............

$ strace -p 550 -f -tt  // -p PID -f fork된 밑에 process들도 같이 분석, -t timestamp 출력 
[pid   994] 11:30:52.563464 fstat(31, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
[pid   994] 11:30:52.563761 fstat(31, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
[pid   994] 11:30:52.564047 read(31, "count 180\nunhandled 0\nlast_unhan"..., 1024) = 42

System Call 함수의 호출의 통계

간단히 본인이 원하는 System Call 함수의 Profile 가능하며, 통계를 내어 분석도 가능

$ strace -cp 550 -f  // -p PID -f fork된 밑에 process들도 같이 분석, -c count call
strace: Process 550 attached with 4 threads
^Cstrace: Process 550 detached
strace: Process 743 detached
strace: Process 994 detached
strace: Process 995 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 98.67    0.488000        1132       431           pselect6
  0.94    0.004653          12       382           sendto
  0.13    0.000666           2       382           openat
  0.10    0.000495           1       382           close
  0.09    0.000457           1       764           fstat
  0.06    0.000310           1       382           read
  0.00    0.000000           0         4         2 futex
------ ----------- ----------- --------- --------- ----------------
100.00    0.494581                  2727         2 total

1.4 STRACE 사용

보통 아래와 같이 본인이 trace하고 싶은 app과 함께 같이 사용을 하며, 현재 test program을 없기에, ps command를 분석한다.

$ strace -c >/dev/null ps                     // ps program 내부 system call 분석 ,  ps의 내용은 /dev/null 출력 
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 38.53    0.012719          46       279           read
 24.46    0.008074          31       260         6 open
 13.92    0.004595          18       254           close
  9.01    0.002973          25       119         3 stat64
  3.72    0.001227          29        43           mmap2
  2.82    0.000932          29        32           mprotect
  1.75    0.000577         289         2           getdents64
  1.25    0.000412          16        25           fstat64
  1.17    0.000385          14        28           lseek
  0.99    0.000328          14        24           rt_sigaction
  0.99    0.000327          20        16        15 access
  0.31    0.000102          51         2         2 statfs
  0.31    0.000101          51         2           munmap
  0.18    0.000061          15         4         3 ioctl
  0.11    0.000035          12         3           brk
  0.09    0.000031          16         2           _llseek
  0.06    0.000021          21         1           cacheflush
  0.05    0.000018          18         1           futex
  0.05    0.000016          16         1           ugetrlimit
  0.05    0.000015          15         1           set_tid_address
  0.04    0.000014          14         1           rt_sigprocmask
  0.04    0.000014          14         1           set_tls
  0.04    0.000013          13         1           geteuid32
  0.04    0.000013          13         1           set_robust_list
  0.03    0.000009           9         1           write
  0.00    0.000000           0         1           execve
  0.00    0.000000           0         1           uname
------ ----------- ----------- --------- --------- ----------------
100.00    0.033012                  1106        29 total

$ strace -e open >/dev/null ps       // ps 프로그램에서 open만 검색 
open("/etc/ld.so.preload", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/arm-linux-gnueabihf/libarmmem.so", O_RDONLY|O_CLOEXEC) = 3
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/arm-linux-gnueabihf/libprocps.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/lib/arm-linux-gnueabihf/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/lib/arm-linux-gnueabihf/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/lib/arm-linux-gnueabihf/libsystemd.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/lib/arm-linux-gnueabihf/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/lib/arm-linux-gnueabihf/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/lib/arm-linux-gnueabihf/liblzma.so.5", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/arm-linux-gnueabihf/liblz4.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/lib/arm-linux-gnueabihf/libgcrypt.so.20", O_RDONLY|O_CLOEXEC) = 3
open("/lib/arm-linux-gnueabihf/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/lib/arm-linux-gnueabihf/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/lib/arm-linux-gnueabihf/libpcre.so.3", O_RDONLY|O_CLOEXEC) = 3
open("/lib/arm-linux-gnueabihf/libgpg-error.so.0", O_RDONLY|O_CLOEXEC) = 3
.......

https://hokstad.com/5-simple-ways-to-troubleshoot-using-strace
http://www.brendangregg.com/perf.html
https://brunch.co.kr/@alden/12
http://linuxspot.tistory.com/253

12/02/2018

ltrace

1. ltrace 의 기능

대부분 Ubuntu에 기본으로 설치가 되어있는 것으로 보이며, 동적라이브러리를 추적할때 사용한다고 한다.
주목적은 shared library 추적이며, dlopen시 마다 이를 hooking해서 추적한다.

ltrace의 기능소개
https://en.wikipedia.org/wiki/Ltrace

https://linux.die.net/man/1/ltrace

기본설치

$ sudo apt install ltrace

기본사용법

$ ltrace --help
Usage: ltrace [option ...] [command [arg ...]]
Trace library calls of a given program.

  -a, --align=COLUMN  align return values in a secific column.
  -A MAXELTS          maximum number of array elements to print.
  -b, --no-signals    don't print signals.
  -c                  count time and calls, and report a summary on exit.
  -C, --demangle      decode low-level symbol names into user-level names.
  -D, --debug=MASK    enable debugging (see -Dh or --debug=help).
  -Dh, --debug=help   show help on debugging.
  -e FILTER           modify which library calls to trace.
  -f                  trace children (fork() and clone()).
  -F, --config=FILE   load alternate configuration file (may be repeated).
  -h, --help          display this help and exit.
  -i                  print instruction pointer at time of library call.
  -l, --library=LIBRARY_PATTERN only trace symbols implemented by this library.
  -L                  do NOT display library calls.
  -n, --indent=NR     indent output by NR spaces for each call level nesting.
  -o, --output=FILENAME write the trace output to file with given name.
  -p PID              attach to the process with the process ID pid.
  -r                  print relative timestamps.
  -s STRSIZE          specify the maximum string size to print.
  -S                  trace system calls as well as library calls.
  -t, -tt, -ttt       print absolute timestamps.
  -T                  show the time spent inside each call.
  -u USERNAME         run command with the userid, groupid of username.
  -V, --version       output version information and exit.
  -x FILTER           modify which static functions to trace.

Report bugs to ltrace-devel@lists.alioth.debian.org

현재 동작 중인 Process 기본분석

/proc의 기본정보를 이용하여 ELF 정보를 비롯하여 Map 정보등 세세한 정보를 얻을 수 있다

$ ps -aux   //현재 동작 중인 Process 확인 
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
....
systemd+     713  0.0  0.0  90456  6336 ?        Ssl   5월21   0:01 /lib/systemd/systemd-timesyncd
...
$ sudo cat /proc/713/maps  // 보고자하는 PID의 MAP 확인 (사용되어지는 공유 Library 확인)
...
7fe6e9dfa000-7fe6e9dfc000 rw-p 00000000 00:00 0
7fe6e9dfc000-7fe6e9e24000 r--p 00000000 08:15 2367775                    /usr/lib/x86_64-linux-gnu/libseccomp.so.2.4.3
7fe6e9e24000-7fe6e9e2f000 r-xp 00028000 08:15 2367775                    /usr/lib/x86_64-linux-gnu/libseccomp.so.2.4.3
7fe6e9e2f000-7fe6e9e33000 r--p 00033000 08:15 2367775                    /usr/lib/x86_64-linux-gnu/libseccomp.so.2.4.3
7fe6e9e33000-7fe6e9e4e000 r--p 00036000 08:15 2367775                    /usr/lib/x86_64-linux-gnu/libseccomp.so.2.4.3
7fe6e9e4e000-7fe6e9e4f000 rw-p 00051000 08:15 2367775                    /usr/lib/x86_64-linux-gnu/libseccomp.so.2.4.3
7fe6e9e4f000-7fe6e9e52000 r--p 00000000 08:15 2361499                    /usr/lib/x86_64-linux-gnu/librt-2.31.so
7fe6e9e52000-7fe6e9e56000 r-xp 00003000 08:15 2361499                    /usr/lib/x86_64-linux-gnu/librt-2.31.so
......

$ sudo cat /proc/713/status  // Process 상태확인 
Name:   systemd-timesyn
Umask:  0022
State:  S (sleeping)
Tgid:   713
Ngid:   0
Pid:    713
PPid:   1
TracerPid:      0
Uid:    102     102     102     102
Gid:    104     104     104     104
FDSize: 64
Groups: 104
NStgid: 713
NSpid:  713
NSpgid: 713

$ sudo readelf -h /proc/713/exe    // 실행중인 파일 ELF의 포맷확인 및 기본정보확인
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              DYN (Shared object file)
.....

$ sudo readelf -s /proc/713/exe    // 실행중인 파일의 ELF에서 symbol table 확인   

Symbol table '.dynsym' contains 108 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND getegid@GLIBC_2.2.5 (2)
     2: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __errno_location@GLIBC_2.2.5 (2)
     3: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND config_parse_sec@SD_SHARED (3)
     4: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND sendto@GLIBC_2.2.5 (2)
     5: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND bind@GLIBC_2.2.5 (2)
     6: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND sd_event_loop@SD_SHARED (3)
     7: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND socket@GLIBC_2.2.5 (2)
     8: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND recvmsg@GLIBC_2.2.5 (2)
     9: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND log_assert_failed_unreach@SD_SHARED (3)
..........

ltrace 기본사용법

우선 ps로 CPU를 가장 많이 사용하는 PID를 사용하여 각 상태를 확인을 하자.

아래와 같이 너무 많이 호출되어지기 때문에 특정함수만 검색하고 싶다면 상위 -e 옵션추가

$ ps -aux    //CPU를 가장 많이 이용하는 PID 찾기 
......

$ sudo ltrace -p 2025 -i     //ltrace로 pid 추적 timestamp 이용 
[0x138e9a5] __errno_location()                                                                                                   = 0x7afe3648
[0x138e9a5] __errno_location( unfinished ...
[0x138e31c] clock_gettime(1, 0x7afdfd30, 0x7afdfd30, 0x7b10b5ce)                                                                 = 0
[0x138e9a5] ... __errno_location resumed )                                                                                     = 0x7fbd7afe46c8
......

$ sudo ltrace -p 7026 -t     //ltrace로 pid 추적 timestamp 이용 
13:42:24 __errno_location( unfinished ...
13:42:24 __errno_location( unfinished ...
13:42:24 ... __errno_location resumed )                                                                                        = 0x482ea648
13:42:24 ... __errno_location resumed )                                                                                        = 0x7fd2482eb6c8
13:42:24 clock_gettime(1, 0x482e6d30, 0x482e6d30, 0x484125ce unfinished ...
13:42:24 clock_gettime(1, 0x7ffec722d690, 0x7ffec722d690, 0x7fd2484125ce unfinished ...
13:42:24 ... clock_gettime resumed )                                                                                           = 0
13:42:24 ... clock_gettime resumed )
..............

참고자료

https://jiming.tistory.com/237

debugfs
https://m.blog.naver.com/PostView.nhn?blogId=dudwo567890&logNo=130156521971&proxyReferer=https%3A%2F%2Fwww.google.com%2F

10/20/2014

Linux debug Strace 사용 (추후 정리후 삭제)

Strace Debug

$ strace -h
usage: strace [-CdffhiqrtttTvVwxxy] [-I n] [-e expr]...
              [-a column] [-o file] [-s strsize] [-P path]...
              -p pid... / [-D] [-E var=val]... [-u username] PROG [ARGS]
   or: strace -c[dfw] [-I n] [-e expr]... [-O overhead] [-S sortby]
              -p pid... / [-D] [-E var=val]... [-u username] PROG [ARGS]

Output format:
  -a column      alignment COLUMN for printing syscall results (default 40)
  -i             print instruction pointer at time of syscall
  -o file        send trace output to FILE instead of stderr
  -q             suppress messages about attaching, detaching, etc.
  -r             print relative timestamp
  -s strsize     limit length of print strings to STRSIZE chars (default 32)
  -t             print absolute timestamp
  -tt            print absolute timestamp with usecs
  -T             print time spent in each syscall
  -x             print non-ascii strings in hex
  -xx            print all strings in hex
  -y             print paths associated with file descriptor arguments
  -yy            print ip:port pairs associated with socket file descriptors

Statistics:
  -c             count time, calls, and errors for each syscall and report summary
  -C             like -c but also print regular output
  -O overhead    set overhead for tracing syscalls to OVERHEAD usecs
  -S sortby      sort syscall counts by: time, calls, name, nothing (default time)
  -w             summarise syscall latency (default is system time)

Filtering:
  -e expr        a qualifying expression: option=[!]all or option=[!]val1[,val2]...
     options:    trace, abbrev, verbose, raw, signal, read, write
  -P path        trace accesses to path

Tracing:
  -b execve      detach on execve syscall
  -D             run tracer process as a detached grandchild, not as parent
  -f             follow forks
  -ff            follow forks with output into separate files
  -I interruptible
     1:          no signals are blocked
     2:          fatal signals are blocked while decoding syscall (default)
     3:          fatal signals are always blocked (default if '-o FILE PROG')
     4:          fatal signals and SIGTSTP (^Z) are always blocked
                 (useful to make 'strace -o FILE PROG' not stop on ^Z)

Startup:
  -E var         remove var from the environment for command
  -E var=val     put var=val in the environment for command
  -p pid         trace process with process id PID, may be repeated
  -u username    run command as username handling setuid and/or setgid

Miscellaneous:
  -d             enable debug output to stderr
  -v             verbose mode: print unabbreviated argv, stat, termios, etc. args
  -h             print help message
  -V             print version

$ sudo cat /proc/550/maps  // 실행과 관련된 공유라이브러리 확인 
$ sudo cat /proc/550/status  // Process 상태확인 
$ readelf -h /proc/550/exe    // 실행파일 확인 
$ readelf -s /proc/550/exe    // symbol table 확인     


$ strace -p 550 -f -t
...........
[pid   994] 11:25:13 pselect6(0, NULL, NULL, NULL, {0, 10000000}, NULL) = 0 (Timeout)
[pid   994] 11:25:13 openat(AT_FDCWD, "/proc/irq/366/spurious", O_RDONLY) = 31
[pid   994] 11:25:13 fstat(31, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
[pid   994] 11:25:13 fstat(31, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
[pid   994] 11:25:13 read(31, "count 180\nunhandled 0\nlast_unhan"..., 1024) = 42
...............

$ strace -p 550 -f -tt
[pid   994] 11:30:52.563464 fstat(31, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
[pid   994] 11:30:52.563761 fstat(31, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
[pid   994] 11:30:52.564047 read(31, "count 180\nunhandled 0\nlast_unhan"..., 1024) = 42

$ strace -cp 550 -f
strace: Process 550 attached with 4 threads
^Cstrace: Process 550 detached
strace: Process 743 detached
strace: Process 994 detached
strace: Process 995 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 98.67    0.488000        1132       431           pselect6
  0.94    0.004653          12       382           sendto
  0.13    0.000666           2       382           openat
  0.10    0.000495           1       382           close
  0.09    0.000457           1       764           fstat
  0.06    0.000310           1       382           read
  0.00    0.000000           0         4         2 futex
------ ----------- ----------- --------- --------- ----------------
100.00    0.494581                  2727         2 total

http://www.brendangregg.com/perf.html
https://brunch.co.kr/@alden/12

http://linuxspot.tistory.com/253

http://man7.org/linux/man-pages/man1/strace.1.html

피드 구독하기: 글 ( Atom )