Elgato_dark/inference
1
0
Fork 0
mirror of https://github.com/mlcommons/inference synced 2026-05-24 09:50:02 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
inference/SubmissionExample.ipynb

599 lines
29 KiB
Text
Executable file
Raw Permalink Blame History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# End to End MLPerf Submission example\n",
"\n",
"This is following the [General MLPerf Submission Rules](https://github.com/mlcommons/policies/blob/master/submission_rules.adoc).\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Get the MLPerf Inference Benchmark Suite source code\n",
"\n",
"You run this notebook from the root of the 'mlcommons/inference' repo that you cloned with\n",
"```\n",
"git clone --recurse-submodules https://github.com/mlcommons/inference.git --depth 1\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Build loadgen"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# build loadgen\n",
"!pip install pybind11\n",
"!cd loadgen; CFLAGS=\"-std=c++14 -O3\" python setup.py develop"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"!cd vision/classification_and_detection; python setup.py develop"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Set Working Directory"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%cd vision/classification_and_detection"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Download data\n",
"\n",
"For this example, the ImageNet and/or COCO validation data should already be on the host system. See the [MLPerf Image Classification task](https://github.com/mlcommons/inference/tree/master/vision/classification_and_detection#datasets) for more details on obtaining this. For the following step each validation dataset is stored in /workspace/data/. You should change this to the location in your setup."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"mkdir data\n",
"ln -s /workspace/data/imagenet2012 data/\n",
"ln -s /workspace/data/coco data/"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Download models"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"mkdir models\n",
"\n",
"# resnet50\n",
"wget -q https://zenodo.org/record/2535873/files/resnet50_v1.pb -O models/resnet50_v1.pb \n",
"wget -q https://zenodo.org/record/2592612/files/resnet50_v1.onnx -O models/resnet50_v1.onnx\n",
"\n",
"# ssd-mobilenet\n",
"wget -q http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_2018_01_28.tar.gz -O models/ssd_mobilenet_v1_coco_2018_01_28.tar.gz\n",
"tar zxvf ./models/ssd_mobilenet_v1_coco_2018_01_28.tar.gz -C ./models; mv models/ssd_mobilenet_v1_coco_2018_01_28/frozen_inference_graph.pb ./models/ssd_mobilenet_v1_coco_2018_01_28.pb\n",
"wget -q https://zenodo.org/record/3163026/files/ssd_mobilenet_v1_coco_2018_01_28.onnx -O models/ssd_mobilenet_v1_coco_2018_01_28.onnx \n",
"\n",
"# ssd-resnet34\n",
"wget -q https://zenodo.org/record/3345892/files/tf_ssd_resnet34_22.1.zip -O models/tf_ssd_resnet34_22.1.zip\n",
"unzip ./models/tf_ssd_resnet34_22.1.zip -d ./models; mv models/tf_ssd_resnet34_22.1/resnet34_tf.22.1.pb ./models\n",
"wget -q https://zenodo.org/record/3228411/files/resnet34-ssd1200.onnx -O models/resnet34-ssd1200.onnx"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Run benchmarks using the reference implementation\n",
"\n",
"Lets prepare a submission for ResNet-50 on a cloud datacenter server with a NVIDIA T4 GPU using TensorFlow. \n",
"\n",
"The following script will run those combinations and prepare a submission directory, following the general submission rules documented [here](https://github.com/mlcommons/policies/blob/master/submission_rules.adoc)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import logging\n",
"import os\n",
"logger = logging.getLogger()\n",
"logger.setLevel(logging.CRITICAL)\n",
"\n",
"os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' \n",
"os.environ['CUDA_VISIBLE_DEVICES'] = \"0\"\n",
"\n",
"# final results go here\n",
"ORG = \"mlperf-org\"\n",
"DIVISION = \"closed\"\n",
"SUBMISSION_ROOT = \"/tmp/mlperf-submission\"\n",
"SUBMISSION_DIR = os.path.join(SUBMISSION_ROOT, DIVISION, ORG)\n",
"os.environ['SUBMISSION_ROOT'] = SUBMISSION_ROOT\n",
"os.environ['SUBMISSION_DIR'] = SUBMISSION_DIR\n",
"os.makedirs(SUBMISSION_DIR, exist_ok=True)\n",
"os.makedirs(os.path.join(SUBMISSION_DIR, \"measurements\"), exist_ok=True)\n",
"os.makedirs(os.path.join(SUBMISSION_DIR, \"code\"), exist_ok=True)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"====== resnet50/SingleStream =====\n",
"TestScenario.SingleStream qps=7322.28, mean=0.0078, time=6.828, acc=76.456%, queries=50000, tiles=50.0:0.0077,80.0:0.0078,90.0:0.0078,95.0:0.0079,99.0:0.0131,99.9:0.0135\n",
"accuracy=76.456%, good=38228, total=50000\n",
"TestScenario.SingleStream qps=125.88, mean=0.0079, time=600.138, queries=75546, tiles=50.0:0.0079,80.0:0.0080,90.0:0.0080,95.0:0.0081,99.0:0.0081,99.9:0.0082\n",
"====== resnet50/Server =====\n",
"TestScenario.Server qps=7528.79, mean=0.0832, time=6.641, acc=76.456%, queries=50000, tiles=50.0:0.0809,80.0:0.0922,90.0:0.0932,95.0:0.0941,99.0:0.0963,99.9:0.1022\n",
"accuracy=76.456%, good=38228, total=50000\n",
"TestScenario.Server qps=128.84, mean=116.7138, time=2098.285, queries=270336, tiles=50.0:115.9511,80.0:185.2868,90.0:209.0362,95.0:220.8464,99.0:230.0520,99.9:231.5965\n",
"====== resnet50/Offline =====\n",
"TestScenario.Offline qps=2008.52, mean=0.3050, time=3.112, acc=76.456%, queries=6250, tiles=50.0:0.3017,80.0:0.3416,90.0:0.3465,95.0:0.3525,99.0:0.3646,99.9:1.2464\n",
"accuracy=76.456%, good=38228, total=50000\n",
"TestScenario.Offline qps=285.33, mean=1157.2775, time=2313.086, queries=660000, tiles=50.0:1157.2701,80.0:1850.5871,90.0:2081.7068,95.0:2197.3040,99.0:2289.7431,99.9:2310.5646\n",
"====== resnet50/MultiStream =====\n",
"TestScenario.MultiStream qps=1891.35, mean=0.0879, time=3.357, acc=76.447%, queries=6350, tiles=50.0:0.1002,80.0:0.1265,90.0:0.1311,95.0:0.1321,99.0:0.1356,99.9:0.1422\n",
"accuracy=76.456%, good=38228, total=50000\n",
"TestScenario.MultiStream qps=266.63, mean=0.0904, time=40555.550, queries=10813440, tiles=50.0:0.1050,80.0:0.1289,90.0:0.1369,95.0:0.1376,99.0:0.1386,99.9:0.1399\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:main:Namespace(accuracy=True, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/workspace/inference/vision/classification_and_detection/data/imagenet2012', find_peak_performance=False, inputs=['input_tensor:0'], max_batchsize=8, max_latency=0.0005, mlperf_conf='../../mlperf.conf', model='/workspace/inference/vision/classification_and_detection/models/resnet50_v1.pb', model_name='resnet50', output='/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/accuracy', outputs=['ArgMax:0'], profile='resnet50-tf', qps=145, samples_per_query=40, scenario='SingleStream', threads=2, time=None, user_conf='user.conf')\n",
"INFO:imagenet:loaded 50000 images, cache=0, took=419.7sec\n",
"INFO:main:starting TestScenario.SingleStream\n",
"INFO:main:Namespace(accuracy=False, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/workspace/inference/vision/classification_and_detection/data/imagenet2012', find_peak_performance=False, inputs=['input_tensor:0'], max_batchsize=8, max_latency=0.0005, mlperf_conf='../../mlperf.conf', model='/workspace/inference/vision/classification_and_detection/models/resnet50_v1.pb', model_name='resnet50', output='/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/performance/run_1', outputs=['ArgMax:0'], profile='resnet50-tf', qps=145, samples_per_query=40, scenario='SingleStream', threads=2, time=None, user_conf='user.conf')\n",
"INFO:imagenet:loaded 50000 images, cache=0, took=1.1sec\n",
"INFO:main:starting TestScenario.SingleStream\n",
"INFO:main:Namespace(accuracy=True, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/workspace/inference/vision/classification_and_detection/data/imagenet2012', find_peak_performance=False, inputs=['input_tensor:0'], max_batchsize=8, max_latency=None, mlperf_conf='../../mlperf.conf', model='/workspace/inference/vision/classification_and_detection/models/resnet50_v1.pb', model_name='resnet50', output='/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/accuracy', outputs=['ArgMax:0'], profile='resnet50-tf', qps=145, samples_per_query=40, scenario='Server', threads=2, time=None, user_conf='user.conf')\n",
"INFO:imagenet:loaded 50000 images, cache=0, took=1.1sec\n",
"INFO:main:starting TestScenario.Server\n",
"INFO:main:Namespace(accuracy=False, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/workspace/inference/vision/classification_and_detection/data/imagenet2012', find_peak_performance=False, inputs=['input_tensor:0'], max_batchsize=8, max_latency=None, mlperf_conf='../../mlperf.conf', model='/workspace/inference/vision/classification_and_detection/models/resnet50_v1.pb', model_name='resnet50', output='/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/performance/run_1', outputs=['ArgMax:0'], profile='resnet50-tf', qps=145, samples_per_query=40, scenario='Server', threads=2, time=None, user_conf='user.conf')\n",
"INFO:imagenet:loaded 50000 images, cache=0, took=1.1sec\n",
"INFO:main:starting TestScenario.Server\n",
"INFO:main:Namespace(accuracy=True, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/workspace/inference/vision/classification_and_detection/data/imagenet2012', find_peak_performance=False, inputs=['input_tensor:0'], max_batchsize=8, max_latency=None, mlperf_conf='../../mlperf.conf', model='/workspace/inference/vision/classification_and_detection/models/resnet50_v1.pb', model_name='resnet50', output='/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/accuracy', outputs=['ArgMax:0'], profile='resnet50-tf', qps=1000, samples_per_query=40, scenario='Offline', threads=2, time=None, user_conf='user.conf')\n",
"INFO:imagenet:loaded 50000 images, cache=0, took=1.1sec\n",
"INFO:main:starting TestScenario.Offline\n",
"INFO:main:Namespace(accuracy=False, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/workspace/inference/vision/classification_and_detection/data/imagenet2012', find_peak_performance=False, inputs=['input_tensor:0'], max_batchsize=8, max_latency=None, mlperf_conf='../../mlperf.conf', model='/workspace/inference/vision/classification_and_detection/models/resnet50_v1.pb', model_name='resnet50', output='/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/performance/run_1', outputs=['ArgMax:0'], profile='resnet50-tf', qps=1000, samples_per_query=40, scenario='Offline', threads=2, time=None, user_conf='user.conf')\n",
"INFO:imagenet:loaded 50000 images, cache=0, took=1.1sec\n",
"INFO:main:starting TestScenario.Offline\n",
"INFO:main:Namespace(accuracy=True, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/workspace/inference/vision/classification_and_detection/data/imagenet2012', find_peak_performance=False, inputs=['input_tensor:0'], max_batchsize=8, max_latency=None, mlperf_conf='../../mlperf.conf', model='/workspace/inference/vision/classification_and_detection/models/resnet50_v1.pb', model_name='resnet50', output='/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/accuracy', outputs=['ArgMax:0'], profile='resnet50-tf', qps=145, samples_per_query=40, scenario='MultiStream', threads=2, time=None, user_conf='user.conf')\n",
"INFO:imagenet:loaded 50000 images, cache=0, took=1.1sec\n",
"INFO:main:starting TestScenario.MultiStream\n",
"INFO:main:Namespace(accuracy=False, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/workspace/inference/vision/classification_and_detection/data/imagenet2012', find_peak_performance=False, inputs=['input_tensor:0'], max_batchsize=8, max_latency=None, mlperf_conf='../../mlperf.conf', model='/workspace/inference/vision/classification_and_detection/models/resnet50_v1.pb', model_name='resnet50', output='/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/performance/run_1', outputs=['ArgMax:0'], profile='resnet50-tf', qps=145, samples_per_query=40, scenario='MultiStream', threads=2, time=None, user_conf='user.conf')\n",
"INFO:imagenet:loaded 50000 images, cache=0, took=1.1sec\n",
"INFO:main:starting TestScenario.MultiStream\n"
]
}
],
"source": [
"%%bash\n",
"\n",
"# where to find stuff\n",
"export DATA_ROOT=`pwd`/data\n",
"export MODEL_DIR=`pwd`/models\n",
"\n",
"# options for official runs\n",
"gopt=\"--max-batchsize 8 --samples-per-query 40 --threads 2 --qps 145\"\n",
"\n",
"\n",
"function one_run {\n",
" # args: mode count framework device model ...\n",
" scenario=$1; shift\n",
" count=$1; shift\n",
" framework=$1\n",
" device=$2\n",
" model=$3\n",
" system_id=$framework-$device\n",
" echo \"====== $model/$scenario =====\"\n",
"\n",
" case $model in \n",
" resnet50) \n",
" cmd=\"tools/accuracy-imagenet.py --imagenet-val-file $DATA_ROOT/imagenet2012/val_map.txt\"\n",
" offical_name=\"resnet\";;\n",
" ssd-mobilenet) \n",
" cmd=\"tools/accuracy-coco.py --coco-dir $DATA_ROOT/coco\"\n",
" offical_name=\"ssd-small\";;\n",
" ssd-resnet34) \n",
" cmd=\"tools/accuracy-coco.py --coco-dir $DATA_ROOT/coco\"\n",
" offical_name=\"ssd-large\";;\n",
" esac\n",
" output_dir=$SUBMISSION_DIR/results/$system_id/$offical_name\n",
" \n",
" # accuracy run\n",
" ./run_local.sh $@ --scenario $scenario --accuracy --output $output_dir/$scenario/accuracy\n",
" python $cmd --mlperf-accuracy-file $output_dir/$scenario/accuracy/mlperf_log_accuracy.json \\\n",
" > $output_dir/$scenario/accuracy/accuracy.txt\n",
" cat $output_dir/$scenario/accuracy/accuracy.txt\n",
"\n",
" # performance run\n",
" cnt=0\n",
" while [ $cnt -lt $count ]; do\n",
" let cnt=cnt+1\n",
" ./run_local.sh $@ --scenario $scenario --output $output_dir/$scenario/performance/run_$cnt\n",
" done\n",
" \n",
" # setup the measurements directory\n",
" mdir=$SUBMISSION_DIR/measurements/$system_id/$offical_name/$scenario\n",
" mkdir -p $mdir\n",
" cp ../../mlperf.conf $mdir\n",
"\n",
" # reference app uses command line instead of user.conf\n",
" echo \"# empty\" > $mdir/user.conf\n",
" touch $mdir/README.md\n",
" impid=\"reference\"\n",
" cat > $mdir/$system_id\"_\"$impid\"_\"$scenario\".json\" <<EOF\n",
" {\n",
" \"input_data_types\": \"fp32\",\n",
" \"retraining\": \"none\",\n",
" \"starting_weights_filename\": \"https://zenodo.org/record/2535873/files/resnet50_v1.pb\",\n",
" \"weight_data_types\": \"fp32\",\n",
" \"weight_transformations\": \"none\"\n",
" }\n",
"EOF\n",
"}\n",
"\n",
"function one_model {\n",
" # args: framework device model ...\n",
" one_run SingleStream 1 $@ --max-latency 0.0005\n",
" one_run Server 1 $@\n",
" one_run Offline 1 $@ --qps 1000\n",
" one_run MultiStream 1 $@\n",
"}\n",
"\n",
"\n",
"# run image classifier benchmarks \n",
"export DATA_DIR=$DATA_ROOT/imagenet2012\n",
"one_model tf gpu resnet50 $gopt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There might be large trace files in the submission directory - we can delete them."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"!find {SUBMISSION_DIR}/ -name mlperf_log_trace.json -delete"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Complete submission directory\n",
"\n",
"Add the required meta data to the submission."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"#\n",
"# setup systems directory\n",
"#\n",
"if [ ! -d ${SUBMISSION_DIR}/systems ]; then\n",
" mkdir ${SUBMISSION_DIR}/systems\n",
"fi\n",
"\n",
"cat > ${SUBMISSION_DIR}/systems/tf-gpu.json <<EOF\n",
"{\n",
" \"division\": \"closed\",\n",
" \"status\": \"available\",\n",
" \"submitter\": \"mlperf-org\",\n",
" \"system_name\": \"tf-gpu\",\n",
" \"system_type\": \"datacenter\",\n",
" \n",
" \"number_of_nodes\": 1,\n",
" \"host_memory_capacity\": \"32GB\",\n",
" \"host_processor_core_count\": 1,\n",
" \"host_processor_frequency\": \"3.50GHz\",\n",
" \"host_processor_model_name\": \"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz\",\n",
" \"host_processors_per_node\": 1,\n",
" \"host_storage_capacity\": \"512GB\",\n",
" \"host_storage_type\": \"SSD\",\n",
" \n",
" \"accelerator_frequency\": \"-\",\n",
" \"accelerator_host_interconnect\": \"-\",\n",
" \"accelerator_interconnect\": \"-\",\n",
" \"accelerator_interconnect_topology\": \"-\",\n",
" \"accelerator_memory_capacity\": \"16GB\",\n",
" \"accelerator_memory_configuration\": \"none\",\n",
" \"accelerator_model_name\": \"T4\",\n",
" \"accelerator_on-chip_memories\": \"-\",\n",
" \"accelerators_per_node\": 1,\n",
"\n",
" \"framework\": \"v1.14.0-rc1-22-gaf24dc9\",\n",
" \"operating_system\": \"ubuntu-18.04\",\n",
" \"other_software_stack\": \"cuda-11.2\",\n",
" \"sw_notes\": \"\"\n",
"}\n",
"EOF"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"#\n",
"# setup code directory\n",
"#\n",
"dir=${SUBMISSION_DIR}/code/resnet/reference\n",
"mkdir -p $dir\n",
"echo \"git clone https://github.com/mlcommons/inference.git\" > $dir/VERSION.txt\n",
"git rev-parse HEAD >> $dir/VERSION.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### What's in the submission directory now ?\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/tmp/mlperf-submission/closed/mlperf-org/systems/tf-gpu.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/Offline/user.conf\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/Offline/mlperf.conf\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/Offline/README.md\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/Offline/tf-gpu_reference_Offline.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/SingleStream/user.conf\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/SingleStream/mlperf.conf\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/SingleStream/tf-gpu_reference_SingleStream.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/SingleStream/README.md\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/MultiStream/user.conf\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/MultiStream/mlperf.conf\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/MultiStream/tf-gpu_reference_MultiStream.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/MultiStream/README.md\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/Server/user.conf\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/Server/mlperf.conf\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/Server/README.md\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/Server/tf-gpu_reference_Server.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/performance/run_1/results.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/performance/run_1/mlperf_log_summary.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/performance/run_1/mlperf_log_detail.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/performance/run_1/mlperf_log_accuracy.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/accuracy/results.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/accuracy/accuracy.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/accuracy/mlperf_log_summary.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/accuracy/mlperf_log_detail.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/accuracy/mlperf_log_accuracy.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/performance/run_1/results.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/performance/run_1/mlperf_log_summary.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/performance/run_1/mlperf_log_detail.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/performance/run_1/mlperf_log_accuracy.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/accuracy/results.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/accuracy/accuracy.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/accuracy/mlperf_log_summary.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/accuracy/mlperf_log_detail.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/accuracy/mlperf_log_accuracy.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/performance/run_1/results.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/performance/run_1/mlperf_log_summary.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/performance/run_1/mlperf_log_detail.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/performance/run_1/mlperf_log_accuracy.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/accuracy/results.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/accuracy/accuracy.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/accuracy/mlperf_log_summary.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/accuracy/mlperf_log_detail.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/accuracy/mlperf_log_accuracy.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/performance/run_1/results.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/performance/run_1/mlperf_log_summary.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/performance/run_1/mlperf_log_detail.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/performance/run_1/mlperf_log_accuracy.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/accuracy/results.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/accuracy/accuracy.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/accuracy/mlperf_log_summary.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/accuracy/mlperf_log_detail.txt\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/accuracy/mlperf_log_accuracy.json\r\n",
"/tmp/mlperf-submission/closed/mlperf-org/code/resnet/reference/VERSION.txt\r\n"
]
}
],
"source": [
"!find {SUBMISSION_ROOT}/ -type f"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If we look at some files:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"-- SingleStream Accuracy\n",
"accuracy=76.456%, good=38228, total=50000\n",
"\n",
"-- SingleStream Summary\n",
"================================================\n",
"MLPerf Results Summary\n",
"================================================\n",
"SUT name : PySUT\n",
"Scenario : SingleStream\n",
"Mode : PerformanceOnly\n",
"90th percentile latency (ns) : 8030958\n",
"Result is : VALID\n",
" Min duration satisfied : Yes\n",
" Min queries satisfied : Yes\n",
"\n",
"-- Server Summary\n",
"================================================\n",
"MLPerf Results Summary\n",
"================================================\n",
"SUT name : PySUT\n",
"Scenario : Server\n",
"Mode : PerformanceOnly\n",
"Scheduled samples per second : 144.87\n",
"Result is : INVALID\n",
" Performance constraints satisfied : NO\n",
" Min duration satisfied : Yes\n"
]
}
],
"source": [
"!echo \"-- SingleStream Accuracy\"; head {SUBMISSION_DIR}/results/tf-gpu/resnet/SingleStream/accuracy/accuracy.txt\n",
"!echo \"\\n-- SingleStream Summary\"; head {SUBMISSION_DIR}/results/tf-gpu/resnet/SingleStream/performance/run_1/mlperf_log_summary.txt\n",
"!echo \"\\n-- Server Summary\"; head {SUBMISSION_DIR}/results/tf-gpu/resnet/Server/performance/run_1/mlperf_log_summary.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Run the submission checker\n",
"\n",
"Finally, run the submission checker tool that does some sanity checking on your submission.\n",
"We run it at the end and attach the output to the submission."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!python ../../tools/submission/submission-checker.py --input {SUBMISSION_ROOT} > {SUBMISSION_DIR}/submission-checker.log 2>&1 \n",
"!cat {SUBMISSION_DIR}/submission-checker.log"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
},
"varInspector": {
"cols": {
"lenName": 16,
"lenType": 16,
"lenVar": 40
},
"kernels_config": {
"python": {
"delete_cmd_postfix": "",
"delete_cmd_prefix": "del ",
"library": "var_list.py",
"varRefreshCmd": "print(var_dic_list())"
},
"r": {
"delete_cmd_postfix": ") ",
"delete_cmd_prefix": "rm(",
"library": "var_list.r",
"varRefreshCmd": "cat(var_dic_list()) "
}
},
"types_to_exclude": [
"module",
"function",
"builtin_function_or_method",
"instance",
"_Feature"
],
"window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Reference in a new issue View git blame Copy permalink