nvme fails when using "intel_iommu=on" kernel option in linux 3.10.0-229.el7.x86_64

[nithind1988]

When x86_64 linux host boots with "intel_iommu=on pci=realloc" option as boot parameter, tnvme fails as NVMe controller is not able to DMA Identity Struct to mentioned PRP1/PRP2 page.

Below fault is in in host dmesg.

[19739.913986] dmar: DRHD: handling fault status reg 202
[19739.913996] dmar: DMAR:[DMA Write] Request device [03:00.2] fault addr ffffc000
DMAR:[fault reason 05] PTE Write access is not set

A debug log (added) is seen correspondingly on NVMe controller that controller has initiated a DMA of identity struct to address 0xffffc400.

Root Cause:
Issue seems to be because tnvme is using enum DataDir to indicate direction of data for DMA mapping and it is not matching with the enum values used by kernel api's and hence kernel DMA mapping api's are doing a mapping with incorrect direction. Below patch seems to resolve the issue.

diff --git a/Cmds/prpData.h b/Cmds/prpData.h
old mode 100644
new mode 100755
index db4203e..04b9efe
--- a/Cmds/prpData.h
+++ b/Cmds/prpData.h
@@ -22,10 +22,10 @@
#include "../Singletons/memBuffer.h"

typedef enum {

• DATADIR_NONE,
• DATADIR_FROM_DEVICE,
• DATADIR_BIDIRECTIONAL,
  DATADIR_TO_DEVICE,
• DATADIR_BIDIRECTIONAL
• DATADIR_FROM_DEVICE,
• DATADIR_NONE,
  } DataDir;

Below is the snippet of dnvme code where it is passed to api map_user_pg_to_dma().

/* Typecase is only possible because the kernel vs. user space contract
 * states the following which agrees with 'enum dma_data_direction'
 * 0=none; 1=to_device, 2=from_device, 3=bidirectional, others illegal */
kernel_dir = (enum dma_data_direction)nvme_64b_send->data_dir;

/* Mapping user pages to dma memory */
err = map_user_pg_to_dma(nvme_dev, kernel_dir, addr,
    nvme_64b_send->data_buf_size, &sg_list, prps, data_buf_type);
if (err < 0) {

Kernel definitions
/*

• These definitions mirror those in pci.h, so they can be used
• interchangeably with their PCI_ counterparts.
  */
  enum dma_data_direction {
  DMA_BIDIRECTIONAL = 0,
  DMA_TO_DEVICE = 1,
  DMA_FROM_DEVICE = 2,
  DMA_NONE = 3,
  };