Jetson AGX Xavier에서 Jetapck 4.1.1를 설치하고 DeepStream SDK 3.0를 설치 진행한 후 , NVIDIA에서 제공하는 DeepStream Referece Apps들을 실행해보자.
NVIDIA에서 제공하는 Deepstream Reference App
https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps
- Deepstream Reference Apps 설치 및 구조파악
$ cd ~
$ git clone https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps
$ tree -L 3 deepstream_reference_apps
deepstream_reference_apps
├── anomaly
│ ├── apps
│ │ └── deepstream-anomaly-detection
│ ├── config
│ │ └── ds_pgie_config.txt
│ ├── plugins
│ │ ├── gst-dsdirection
│ │ └── gst-dsopticalflow
│ └── README.md
├── CaffeMNIST
│ ├── config_infer_primary_CaffeMNIST.txt
│ ├── data
│ │ └── mnist3d.prototxt
│ ├── deepstream_app_config_CaffeMNIST.txt
│ ├── labels.txt
│ ├── nvdsinfer_custom_impl_CaffeMNIST
│ │ ├── CMakeLists.txt
│ │ ├── factoryCaffeMNISTLegacy.h
│ │ ├── nvdsiplugin_CaffeMNIST.cpp
│ │ └── nvdsparseoutput_CaffeMNIST.cpp
│ └── README.md
├── LICENSE
├── Makefile.config
├── README.md
├── senet
│ ├── apps
│ │ ├── deepstream-senet
│ │ └── trt-senet
│ ├── config
│ │ ├── config_infer_primary_resnet10.txt
│ │ └── config_infer_secondary_senet.txt
│ ├── data
│ │ ├── imagenet_labels.txt
│ │ └── resnet10_labels.txt
│ ├── lib
│ │ ├── calibrator.cpp
│ │ ├── calibrator.h
│ │ ├── CMakeLists.txt
│ │ ├── ds_image.cpp
│ │ ├── ds_image.h
│ │ ├── network_config.cpp
│ │ ├── network_config.h
│ │ ├── se_resnet50.cpp
│ │ ├── se_resnet50.h
│ │ ├── trt_utils.cpp
│ │ └── trt_utils.h
│ ├── LICENSE
│ ├── README.md
│ └── Revised_Scripts
│ ├── base_revised.py
│ └── dumpTFWts_revised.py
└── yolo
├── apps
│ ├── deepstream-yolo
│ └── trt-yolo
├── config
│ ├── yolov2-tiny.txt
│ ├── yolov2.txt
│ ├── yolov3-tiny.txt
│ └── yolov3.txt
├── data
│ ├── calibration_images.txt
│ ├── labels.txt
│ ├── test_images.txt
│ ├── yolov2-calibration.table
│ ├── yolov2-tiny-calibration.table
│ ├── yolov3-calibration.table
│ └── yolov3-tiny-calibration.table
├── lib // 내부를 보면 이것도 IPlugin 을 사용하여 Custom Layer
│ ├── calibrator.cpp
│ ├── calibrator.h
│ ├── CMakeLists.txt
│ ├── ds_image.cpp
│ ├── ds_image.h
│ ├── kernels.cu
│ ├── plugin_factory.cpp // IPlugIn
│ ├── plugin_factory.h
│ ├── trt_utils.cpp
│ ├── trt_utils.h
│ ├── yolo_config_parser.cpp
│ ├── yolo_config_parser.h
│ ├── yolo.cpp
│ ├── yolo.h
│ ├── yoloplugin_lib.cpp
│ ├── yoloplugin_lib.h
│ ├── yolov2.cpp
│ ├── yolov2.h
│ ├── yolov3.cpp
│ └── yolov3.h
├── plugins
│ ├── gst-yoloplugin-tegra
│ └── gst-yoloplugin-tesla
├── prebuild.sh
├── README.md
└── samples
└── objectDetector_YoloV3
Deepstream 기반의 Example이며, Deepstream SDK를 설치하고 이를 설치하여 동작해보도록 하자.
DeepStream Reference Apps
https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps
2. DeepStream Yolo 관련내용
Yolo Readme를 읽어보면, Jetson Xavier가 지원이 가능하며, Jetson TX1/2도 지원이 가능한것을 확인가능.
- Yolo Deepstream를 위해 Plugin 설치
$ cd deepstream_reference_apps/yolo // yolo main $ sh prebuild.sh $ cd plugins/gst-yoloplugin-tegra/ $ mkdir build && cd build $ cmake -D DS_SDK_ROOT=~/deepstream_sdk_on_jetson -D CMAKE_BUILD_TYPE=Release .. $ make && sudo make install .... [100%] Built target gstnvyolo [ 84%] Built target yolo-lib [100%] Built target gstnvyolo Install the project... -- Install configuration: "Release" -- Installing: /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvyolo.so -- Set runtime path of "/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvyolo.so" to "/usr/local/cuda-10.0/lib64:/usr/lib/aarch64-linux-gnu/tegra"
2.1 deepstream-yolo-app 설치 및 실행
- deepstream-yolo-app 설치
$ cd ~/deepstream_reference_apps/yolo // yolo main $ cd apps/deepstream-yolo $ mkdir build && cd build $ cmake -D DS_SDK_ROOT=~/deepstream_sdk_on_jetson -D CMAKE_BUILD_TYPE=Release .. $ make && sudo make install ... [100%] Linking CXX executable deepstream-yolo-app [100%] Built target deepstream-yolo-app [100%] Built target deepstream-yolo-app Install the project... -- Install configuration: "Release" -- Installing: /usr/local/bin/deepstream-yolo-app $ cd ../../../
- deepstream-yolo-app 실행
$ pwd /home/nvidia/deepstream_reference_apps/yolo //config의 변경해보면서 각 개별 테스트 $ deepstream-yolo-app Tegra ~/deepstream_sdk_on_jetson/samples/streams/sample_720p.h264 config/yolov3.txt //느리지만, 잘 인식 , Frame별로 인식확인가능 ........ yolo_83 yolo_95 yolo_107 Using previously generated plan file located at data/yolov3-kFLOAT-kGPU-batch1.engine Loading TRT Engine... Loading Complete! Running... NvMMLiteOpen : Block : BlockType = 261 NvMMLiteBlockCreate : Block : BlockType = 261 Allocating new output: 1280x720 (x 12), ThumbnailMode = 0 OPENMAX: HandleNewStreamFormat: 3528: Send OMX_EventPortSettingsChanged: nFrameWidth = 1280, nFrameHeight = 720 Frame Number = 0 Number of objects = 10 Car Count = 3 Person Count = 5 Bicycle Count = 0 Truck Count = 0 Frame Number = 1 Number of objects = 10 Car Count = 3 Person Count = 5 Bicycle Count = 0 Truck Count = 0 Frame Number = 2 Number of objects = 10 Car Count = 4 Person Count = 4 Bicycle Count = 0 Truck Count = 0 Frame Number = 3 Number of objects = 10 Car Count = 4 Person Count = 4 Bicycle Count = 0 Truck Count = 0 Frame Number = 4 Number of objects = 10 Car Count = 5 Person Count = 4 Bicycle Count = 0 Truck Count = 0 Frame Number = 5 Number of objects = 11 Car Count = 5 Person Count = 4 Bicycle Count = 0 Truck Count = 0 Frame Number = 6 Number of objects = 9 Car Count = 6 Person Count = 2 Bicycle Count = 0 Truck Count = 0 Frame Number = 7 Number of objects = 11 Car Count = 7 Person Count = 3 Bicycle Count = 0 Truck Count = 0 Frame Number = 8 Number of objects = 13 Car Count = 8 Person Count = 3 Bicycle Count = 0 Truck Count = 0 Frame Number = 9 Number of objects = 13 Car Count = 8 Person Count = 3 Bicycle Count = 0 Truck Count = 0 ....... $ deepstream-yolo-app Tegra ~/deepstream_sdk_on_jetson/samples/streams/sample_720p.h264 config/yolov3-tiny.txt //빠르지만, 잘 인식못함. Output blob names : yolo_17 yolo_24 Using previously generated plan file located at data/yolov3-tiny-kFLOAT-kGPU-batch1.engine Loading TRT Engine... Loading Complete! Running... NvMMLiteOpen : Block : BlockType = 261 NvMMLiteBlockCreate : Block : BlockType = 261 Allocating new output: 1280x720 (x 12), ThumbnailMode = 0 OPENMAX: HandleNewStreamFormat: 3528: Send OMX_EventPortSettingsChanged: nFrameWidth = 1280, nFrameHeight = 720 Frame Number = 0 Number of objects = 3 Car Count = 0 Person Count = 3 Bicycle Count = 0 Truck Count = 0 Frame Number = 1 Number of objects = 4 Car Count = 1 Person Count = 3 Bicycle Count = 0 Truck Count = 0 Frame Number = 2 Number of objects = 4 Car Count = 1 Person Count = 3 Bicycle Count = 0 Truck Count = 0 Frame Number = 3 Number of objects = 4 Car Count = 1 Person Count = 3 Bicycle Count = 0 Truck Count = 0 Frame Number = 4 Number of objects = 3 Car Count = 0 Person Count = 3 Bicycle Count = 0 Truck Count = 0 Frame Number = 5 Number of objects = 4 Car Count = 1 Person Count = 3 Bicycle Count = 0 Truck Count = 0 Frame Number = 6 Number of objects = 3 Car Count = 0 Person Count = 3 Bicycle Count = 0 Truck Count = 0 Frame Number = 7 Number of objects = 4 Car Count = 1 Person Count = 3 Bicycle Count = 0 Truck Count = 0 Frame Number = 8 Number of objects = 3 Car Count = 0 Person Count = 3 Bicycle Count = 0 Truck Count = 0 Frame Number = 9 Number of objects = 4 Car Count = 1 Person Count = 3 Bicycle Count = 0 Truck Count = 0 ...
현재 제공해주는 deepstream-app과 차이는Network도 차이도 있겠지만, Yolo는 Track 부분을 별도로 연결하지 않아 상위와 같이 Frame 당 인식한 것을 분석하여 보여준다.
이곳도 분석을 보면 osd_sink_pad_buffer_probe 의 Callback Function을 보면 대충 알겠다.
이 소스는 기존의 구성된 소스와 다르게 yolo에서 한번에 모든것을 분석 Detection 및 분류를 다한다.
- trt-yolo-app 설치 및 실행
현재 필요한 data/test_images.txt의 test Image들이 없어서 동작이 안되지만, 아래와 같이 넣어주고 동작확인만 하자.
정확하게 확인할 경우는 yolo가 구분할 수 있는 그림을 넣어주면 될 것 같다.
$ cd ~/deepstream_reference_apps/yolo // yolo main $ cd apps/trt-yolo $ mkdir build && cd build $ cmake -D CMAKE_BUILD_TYPE=Release .. $ make && sudo make install ... -- Found TensorRT headers at /usr/include/aarch64-linux-gnu -- Found TensorRT libs at /usr/lib/aarch64-linux-gnu/libnvinfer.so;/usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so -- Configuring done -- Generating done -- Build files have been written to: /home/nvidia/nvyolo/deepstream_reference_apps/yolo/apps/trt-yolo/build [ 7%] Linking CXX static library libyolo-lib.a [ 84%] Built target yolo-lib [ 92%] Linking CXX executable trt-yolo-app [100%] Built target trt-yolo-app Install the project... -- Install configuration: "Release" -- Installing: /usr/local/bin/trt-yolo-app -- Set runtime path of "/usr/local/bin/trt-yolo-app" to "" $ cd ../../../ $ trt-yolo-app --flagfile=config/yolov3.txt $ find / -name *.jpg 2> /dev/null > data/test_images.txt // jpg 파일이 없어서 에러발생, $ trt-yolo-app --flagfile=config/yolov3.txt // 상위와 같이 만들어주면, inference 가능하며, 실제 image파일 넣어 테스트 하자
https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps/tree/master/yolo
3. CaffeMNIST 관련내용
추후 사용할 일이 있다면, 그때 사용
https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps/blob/master/CaffeMNIST/README.md
4. Anomaly Detection Reference Apps (Xavier 미지원)
Deepstream의 또 따른 예제이며, 이것도 진행을 했지만, 아래의 링크를 보면, Xavier에서는 미지원이다.
혹시나 해서 관련부분을 수정하면서 해보려고 했지만, DeepStream SDK에서 지원을 해줘야 가능한 소스이다.
너무 좋은 예제라서 한번 돌려보고 싶지만, 이번에 말고 다음기회에 하자 (반드시 기억하자)
이상감지 (Anomaly Detection)
http://intothedata.com/02.scholar_category/anomaly_detection/
- OpenCV 4.0 설치진행 (용량이 부족하므로 확장해야함)
(나의 경우는 확장 Storage가 없어서 NFS로 진행)
$ cd ~
$ wget https://github.com/opencv/opencv/archive/4.0.1.zip
$ wget https://github.com/opencv/opencv_contrib/archive/4.0.1.zip
$ ls // 상위 이름이 동일함
4.0.1.zip 4.0.1.zip.1
$ unzip 4.0.1.zip
$ unzip 4.0.1.zip.1
$ ls
... 4.0.1.zip 4.0.1.zip.1 opencv-4.0.1 opencv_contrib-4.0.1
$ mv opencv-4.0.1 opencv // Build 시 두 Directory의 용량이 너무 커서 NFS or 확장 Storage를 사용해야함
$ mv opencv_contrib-4.0.1 opencv_contrib
$ cd ~/opencv/
$ mkdir build
$ cd build
$ cmake -D CMAKE_BUILD_TYPE=RELEASE \
-DCMAKE_INSTALL_PREFIX=/usr/local \
-DWITH_CUDA=ON \
-DINSTALL_PYTHON_EXAMPLES=ON \
-DOPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
-DBUILD_EXAMPLES=ON \
-DBUILD_opencv_cudacodec=OFF \
-DCMAKE_CXX_FLAGS="" ..
........ 로그를 보면, Caffe 도 찾고 tessract 도 찾고 freetype2도 찾고 많이 찾는데, 추후 필요사용할 경우, 미리설치해놓고 설치해보자.
-- Disabled: cudacodec world
-- Disabled by dependency: -
-- Unavailable: cnn_3dobj cvv freetype hdf java js matlab ovis python3 sfm viz
-- Applications: tests perf_tests examples apps
-- Documentation: NO
-- Non-free algorithms: NO
--
-- GUI:
-- GTK+: NO
-- VTK support: NO
--
-- Media I/O:
-- ZLib: /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.11)
-- JPEG: libjpeg-turbo (ver 1.5.3-62)
-- WEBP: build (ver encoder: 0x020e)
-- PNG: build (ver 1.6.35)
-- TIFF: build (ver 42 - 4.0.9)
-- JPEG 2000: build (ver 1.900.1)
-- OpenEXR: build (ver 1.7.1)
-- HDR: YES
-- SUNRASTER: YES
-- PXM: YES
-- PFM: YES
--
-- Video I/O:
-- DC1394: NO
-- FFMPEG: NO
-- avcodec: NO
-- avformat: NO
-- avutil: NO
-- swscale: NO
-- avresample: NO
-- GStreamer:
-- base: YES (ver 1.14.4)
-- video: YES (ver 1.14.4)
-- app: YES (ver 1.14.4)
-- riff: YES (ver 1.14.4)
-- pbutils: YES (ver 1.14.4)
-- v4l/v4l2: linux/videodev2.h
--
-- Parallel framework: pthreads
--
-- Trace: YES (built-in)
--
-- Other third-party libraries:
-- Lapack: NO
-- Eigen: YES (ver 3.3.4)
-- Custom HAL: YES (carotene (ver 0.0.1))
-- Protobuf: build (3.5.1)
--
-- NVIDIA CUDA: YES (ver 10.0, CUFFT CUBLAS)
-- NVIDIA GPU arch: 72
-- NVIDIA PTX archs:
--
-- OpenCL: YES (no extra features)
-- Include path: /home/nvidia/opencv/3rdparty/include/opencl/1.2
-- Link libraries: Dynamic load
--
-- Python 2:
-- Interpreter: /usr/bin/python2.7 (ver 2.7.15)
-- Libraries: /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.15+)
-- numpy: /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.13.3)
-- install path: lib/python2.7/dist-packages/cv2/python-2.7
--
-- Python (for build): /usr/bin/python2.7
--
-- Java:
-- ant: NO
-- JNI: NO
-- Java wrappers: NO
-- Java tests: NO
--
-- Install to: /usr/local
$ sudo make //시간이 많이 걸리며, Jetson Xavier의 용량부족발생 (50%까지도 못감) (NFS로 연결하여 용량부분 해결)
$ sudo make install
$ sudo ldconfig
- Deepstream PlugIn 부분 빌드 ( Xavier에서 미지원)
$ pwd
/home/nvidia/opencv/build
$ export DS_SDK_ROOT=/home/nvidia/deepstream_sdk_on_jetson
$ export OPENCV_ROOT=/home/nvidia/opencv/build
$ echo $DS_SDK_ROOT
/home/nvidia/deepstream_sdk_on_jetson
$ echo $OPENCV_ROOT
$ cd ~/deepstream_reference_apps/anomaly/plugins/gst-dsopticalflow
$ mkdir build
$ cd build
$ cmake -D CMAKE_BUILD_TYPE=RELEASE -D DS_SDK_ROOT=$DS_SDK_ROOT -D OPENCV_ROOT=$OPENCV_ROOT ..
.....
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda-10.0 (found suitable version "10.0", minimum required is "10.0")
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
-- Found OpenCV: /home/nvidia/opencv/build (found version "4.0.1") found components: core imgproc imgcodecs highgui dnn
-- Checking for modules 'gstreamer-1.0>=1.4;gstreamer-video-1.0>=1.4;gstreamer-app-1.0>=1.4'
-- Found gstreamer-1.0, version 1.14.4
-- Found gstreamer-video-1.0, version 1.14.4
-- Found gstreamer-app-1.0, version 1.14.4
-- Found OpenCV: /home/nvidia/opencv/build (found suitable version "4.0.1", minimum required is "4.0") found components: opencv_calib3d opencv_core opencv_dnn opencv_features2d opencv_flann opencv_gapi opencv_highgui opencv_imgcodecs opencv_imgproc opencv_ml opencv_objdetect opencv_photo opencv_stitching opencv_video opencv_videoio opencv_aruco opencv_bgsegm opencv_bioinspired opencv_ccalib opencv_cudaarithm opencv_cudabgsegm opencv_cudafeatures2d opencv_cudafilters opencv_cudaimgproc opencv_cudalegacy opencv_cudaobjdetect opencv_cudaoptflow opencv_cudastereo opencv_cudawarping opencv_cudev opencv_datasets opencv_dnn_objdetect opencv_dpm opencv_face opencv_fuzzy opencv_hfs opencv_img_hash opencv_line_descriptor opencv_optflow opencv_phase_unwrapping opencv_plot opencv_reg opencv_rgbd opencv_saliency opencv_shape opencv_stereo opencv_structured_light opencv_superres opencv_surface_matching opencv_text opencv_tracking opencv_videostab opencv_xfeatures2d opencv_ximgproc opencv_xobjdetect opencv_xphoto
.......
$ make // Deepstream SDK에서 아래를 지원해줘야 하는데, 미지원
/home/nvidia/deepstream_reference_apps/anomaly/plugins/gst-dsopticalflow/gstdsopticalflow.cpp:488:13: error: ‘CHECK_NVDS_MEMORY_AND_GPUID’ was not declared in this scope
if (CHECK_NVDS_MEMORY_AND_GPUID (dsopticalflow, surface))
^~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/nvidia/deepstream_reference_apps/anomaly/plugins/gst-dsopticalflow/gstdsopticalflow.cpp:506:80: error: ‘NvBufSurface {aka struct _NvBufSurface}’ has no member named ‘buf_data’; did you mean ‘buf_fds’?
if (DsOpticalFlowProcess(dsopticalflow->dsopticalflowlib_ctx, surface->buf_data[0],
^~~~~~~~
$ sudo make install
$ cd ../../../
- 상위 Make Error 관련사항 ( Deepstream SDK가 지원해줘함)
https://devtalk.nvidia.com/default/topic/1049374/deepstream-sdk-on-jetson/ai-iot-deepstream_reference_apps-gst_dsoptical-flow-build-error/
- 이상감지 관련소스