Jeonghun (James) Lee: AI-DataSet

1. NVIDIA Docker에서 사용하는 DataSET

각 DataSet 사용법 및 관련부분 링크

NVIDIA Tranfer Learning (COCO2017)
  https://github.com/NVIDIA/DeepLearningExamples/blob/master/TensorFlow/Detection/SSD/download_all.sh

COCOAPI (Coco Image /annotations)
  https://github.com/cocodataset/cocoapi

Google DATA SET 검색
   https://toolbox.google.com/datasetsearch

DATASET

아래 소스에서 사용하는 Dataset을 보도록하자
https://github.com/tensorflow/models/tree/master/research/slim

COCO DATASET 2017 Download

  http://images.cocodataset.org/zips/train2017.zip
  http://images.cocodataset.org/annotations/annotations_trainval2017.zip
  http://images.cocodataset.org/zips/val2017.zip

http://images.cocodataset.org/zips/test2017.zip

COCOSET 사용방법

https://patrickwasp.com/create-your-own-coco-style-dataset/

  https://ukayzm.github.io/cocodataset/
  https://wordbe.tistory.com/entry/COCO-dataset
  https://eehoeskrap.tistory.com/353

COCO Annotation Tool
https://eehoeskrap.tistory.com/353

상위 파일들이 필요한 파일인데, gsutil로 download를 할수가 없다.
하지만 다른 파일들은 gsutil로 Download가 가능하므로 gsutil을 설치하고 빨리 download하자

https://jungwoon.github.io/bigquery/2017/10/28/gcs-gsutil/

Train SET , Validation SET , TEST SET 의 역할 및 기능
  https://3months.tistory.com/118
  https://rfriend.tistory.com/188
  https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets
  https://towardsdatascience.com/train-validation-and-test-sets-72cb40cba9e7
  https://machinelearningmastery.com/difference-test-validation-datasets/
  https://dbrang.tistory.com/1204

gsutil 설치

$ sudo apt install curl
$ curl https://sdk.cloud.google.com | bash
google-cloud-sdk  // $HOME 설치됨 
$ source ~/.bashrc

$ gsutil ls gs://images.cocodataset.org/zips/     // 상위 *.zip file ls만 되고 download가 안되지만 다른 directory는 동작가능 
gs://images.cocodataset.org/zips/
gs://images.cocodataset.org/zips/test-stuff2017.zip
gs://images.cocodataset.org/zips/test2014.zip
gs://images.cocodataset.org/zips/test2015.zip
gs://images.cocodataset.org/zips/test2017.zip
gs://images.cocodataset.org/zips/train2014.zip
gs://images.cocodataset.org/zips/train2017.zip
gs://images.cocodataset.org/zips/unlabeled2017.zip
gs://images.cocodataset.org/zips/val2014.zip
gs://images.cocodataset.org/zips/val2017.zip

Jeonghun (James) Lee

10/24/2019

CoCoData SET 관련내용

7/15/2019

DataSet 관련자료 수집