could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED

[xiaohujecky]

2018-03-30 15:11:20.697888: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2018-03-30 15:11:20.697903: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2018-03-30 15:11:20.697920: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2018-03-30 15:11:20.697924: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2018-03-30 15:11:20.697929: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2018-03-30 15:11:20.835422: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:893] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-03-30 15:11:20.838154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:940] Found device 0 with properties:
name: Graphics Device
major: 6 minor: 1 memoryClockRate (GHz) 1.6325
pciBusID 0000:01:00.0
Total memory: 10.91GiB
Free memory: 10.75GiB
2018-03-30 15:11:20.838181: I tensorflow/core/common_runtime/gpu/gpu_device.cc:961] DMA: 0
2018-03-30 15:11:20.838185: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: Y
2018-03-30 15:11:20.838192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Graphics Device, pci bus id: 0000:01:00.0)
2018-03-30 15:11:20,870 root INFO loading data
2018-03-30 15:11:20,876 root INFO phase: train
2018-03-30 15:11:20,877 root INFO model_dir: model_01_16
2018-03-30 15:11:20,877 root INFO load_model: False
2018-03-30 15:11:20,877 root INFO output_dir: results
2018-03-30 15:11:20,877 root INFO steps_per_checkpoint: 2000
2018-03-30 15:11:20,877 root INFO batch_size: 64
2018-03-30 15:11:20,877 root INFO num_epoch: 3
2018-03-30 15:11:20,877 root INFO learning_rate: 1
2018-03-30 15:11:20,877 root INFO reg_val: 0
2018-03-30 15:11:20,877 root INFO max_gradient_norm: 5.000000
2018-03-30 15:11:20,877 root INFO clip_gradients: True
2018-03-30 15:11:20,877 root INFO valid_target_length inf
2018-03-30 15:11:20,877 root INFO target_vocab_size: 39
2018-03-30 15:11:20,877 root INFO target_embedding_size: 10.000000
2018-03-30 15:11:20,877 root INFO attn_num_hidden: 256
2018-03-30 15:11:20,877 root INFO attn_num_layers: 2
2018-03-30 15:11:20,877 root INFO visualize: True
2018-03-30 15:11:20,877 root INFO buckets
2018-03-30 15:11:20,877 root INFO [(16, 11), (27, 17), (35, 19), (64, 22), (80, 32)]
2018-03-30 15:11:20,877 root INFO ues GRU in the decoder.
input_tensor dim: (?, 1, 32, ?)
CNN outdim before squeeze: (?, 1, ?, 512)
CNN outdim: (?, ?, 512)
using GRU CELL in decoder
2018-03-30 15:12:45,581 root INFO Created model with fresh parameters.
2018-03-30 15:14:28,178 root INFO train total size : 156
Train: 0%| | 0/156 [00:00<?, ?it/s]2018-03-30 15:14:28,179 root INFO Generating first batch)
2018-03-30 15:15:19,176 root INFO current_step: 0
2018-03-30 15:15:19,176 root INFO [ 1 31 20 30 27 33 16 17 16 2 0 0 0 0 0 0 0]
[ 1. 1. 1. 1. 1. 1. 1. 1. 1. 0. 0. 0. 0. 0. 0. 0. 0.]
2018-03-30 15:15:20.038637: E tensorflow/stream_executor/cuda/cuda_dnn.cc:359] could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2018-03-30 15:15:20.038678: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:369] driver version file contents: """NVRM version: NVIDIA UNIX x86_64 Kernel Module 378.13 Tue Feb 7 20:10:06 PST 2017
GCC version: gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3)
"""
2018-03-30 15:15:20.038696: I tensorflow/stream_executor/cuda/cuda_dnn.cc:370] running driver version: 378.13.0
2018-03-30 15:15:20.038704: E tensorflow/stream_executor/cuda/cuda_dnn.cc:326] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
2018-03-30 15:15:20.038710: F tensorflow/core/kernels/conv_ops.cc:671] Check failed: stream->parent()->GetConvolveAlgorithms(&algorithms)
Aborted (core dumped)