_import does not work in v2.3.4

[Maddy7868]

Hi Jörg,

I have ElasticSearch v.0.20.6 with plugin 0.20.6.2 and have another instance on same machine v2.3.4 plugin v2.3.4.0. I ran the following commands as mentioned in the release notes.

Please help me out in this.

ElasticSearch v0.20.6

---

curl -XDELETE localhost:9200/test
curl -XPUT 'localhost:9200/test/test/1' -d '{"key":"value 1"}'
curl -XPUT 'localhost:9200/test/test/2' -d '{"key":"value 2"}'
curl -XPUT 'localhost:9200/test2/foo/1' -d '{"key":"value 1"}'
curl -XPUT 'localhost:9200/test2/bar/1' -d '{"key":"value 1"}'
curl -XPOST 'localhost:9200/test/_export'

CYGWIN LOG

---

XXXX@PBVWWSU1 /cygdrive/c/Users/XXXX/Downloads
$ curl -XDELETE localhost:9200/test
{"ok":true,"acknowledged":true}
XXXX@PBVWWSU1 /cygdrive/c/Users/XXXX/Downloads
$ curl -XPUT localhost:9200/test/test/1 -d '{"key":"value 1"}'
{"ok":true,"_index":"test","_type":"test","_id":"1","_version":1}
XXXX@PBVWWSU1 /cygdrive/c/Users/XXXX/Downloads
$ curl -XPUT localhost:9200/test/test/2 -d '{"key":"value 2"}'
{"ok":true,"_index":"test","_type":"test","_id":"2","_version":1}
XXXX@PBVWWSU1 /cygdrive/c/Users/XXXX/Downloads
$ curl -XPUT 'localhost:9200/test2/foo/1' -d '{"key":"value 1"}'
{"ok":true,"_index":"test2","_type":"foo","_id":"1","_version":2}
XXXX@PBVWWSU1 /cygdrive/c/Users/XXXX/Downloads
$ curl -XPUT 'localhost:9200/test2/bar/1' -d '{"key":"value 1"}'
{"ok":true,"_index":"test2","_type":"bar","_id":"1","_version":2}
XXXX@PBVWWSU1 /cygdrive/c/Users/XXXX/Downloads
$ curl -XPOST 'localhost:9200/test/_export'
{"running":true,"state":{"mode":"export","started":"2017-05-18T19:26:41.001Z","path":"file:///C:/Users/XXXX/Personal/elasticsearch-0.20.6/bin/test.tar.gz","node_name":"Agon"}}

COPIED the tar.gz to C:\Users\XXXX\Personal\elasticsearch-2.3.4\logs

STOPPED ELASTICSEARCH v0.20.6
STARTED ELASTICSEARCH v2.3.4

ElasticSearch v2.3.4

---

curl -XDELETE 'localhost:9200/test'
curl -XPOST 'localhost:9200/test/_import
curl -XPOST 'localhost:9200/test/_search?q=*&pretty'

CYGWIN console

---

XXXX@PBVWWSU1 /cygdrive/c/Users/XXXX/Downloads
$ curl -XPOST 'localhost:9200/test/_import'
{"running":true,"state":{"mode":"import","started":"2017-05-18T19:33:37.576Z","p ath":"file:///C:/Users/XXXX/Personal/elasticsearch-2.3.4/logs/test.tar.gz","node_name":"Spider Doppelganger"}}
XXXX@PBVWWSU1 /cygdrive/c/Users/XXXX/Downloads
$ curl -XPOST 'localhost:9200/test/_search?q=*&pretty'
{
"error" : {
"root_cause" : [ {
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "test",
"index" : "test"
} ],
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "test",
"index" : "test"
},
"status" : 404
}

Elastic Search Console

---

[2017-05-18 14:31:34,367][INFO ][node ] [Spider Doppelganger
] started
[2017-05-18 14:31:34,704][INFO ][cluster.routing.allocation] [Spider Doppelgange
r] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started
[[.knapsack][4], [.knapsack][2], [.knapsack][2], [.knapsack][4]] ...]).
[2017-05-18 14:32:04,077][INFO ][cluster.routing.allocation.decider] [Spider Dop
pelganger] low disk watermark [85%] exceeded on [ep-3aQs6TU6CjXCxQmFP-A][Spider
Doppelganger][C:\Users\XXXX\Personal\elasticsearch-2.3.4\data\elasticsea
rch\nodes\0] free: 8.7gb[14.6%], replicas will not be assigned to this node
[2017-05-18 14:32:34,091][INFO ][cluster.routing.allocation.decider] [Spider Dop
pelganger] low disk watermark [85%] exceeded on [ep-3aQs6TU6CjXCxQmFP-A][Spider
Doppelganger][C:\Users\XXXX\Personal\elasticsearch-2.3.4\data\elasticsea
rch\nodes\0] free: 8.7gb[14.6%], replicas will not be assigned to this node
[2017-05-18 14:33:04,104][INFO ][cluster.routing.allocation.decider] [Spider Dop
pelganger] low disk watermark [85%] exceeded on [ep-3aQs6TU6CjXCxQmFP-A][Spider
Doppelganger][C:\Users\XXXX\Personal\elasticsearch-2.3.4\data\elasticsea
rch\nodes\0] free: 8.7gb[14.6%], replicas will not be assigned to this node
[2017-05-18 14:33:34,115][INFO ][cluster.routing.allocation.decider] [Spider Dop
pelganger] low disk watermark [85%] exceeded on [ep-3aQs6TU6CjXCxQmFP-A][Spider
Doppelganger][C:\Users\XXXX\Personal\elasticsearch-2.3.4\data\elasticsea
rch\nodes\0] free: 8.7gb[14.6%], replicas will not be assigned to this node
[2017-05-18 14:33:37,578][INFO ][KnapsackImportAction ] start of import: {"m
ode":"import","started":"2017-05-18T19:33:37.576Z","path":"file:///C:/Users/
XXXX/Personal/elasticsearch-2.3.4/logs/test.tar.gz","node_name":"Spider Dopp
elganger"}
[2017-05-18 14:33:37,791][INFO ][KnapsackImportAction ] end of import: {"mod
e":"import","started":"2017-05-18T19:33:37.576Z","path":"file:///C:/Users/XXXX/Personal/elasticsearch-2.3.4/logs/test.tar.gz","node_name":"Spider Doppel
ganger"}, count = 4
[2017-05-18 14:34:04,128][INFO ][cluster.routing.allocation.decider] [Spider Dop
pelganger] low disk watermark [85%] exceeded on [ep-3aQs6TU6CjXCxQmFP-A][Spider
Doppelganger][C:\Users\XXXX\Personal\elasticsearch-2.3.4\data\elasticsea
rch\nodes\0] free: 8.7gb[14.6%], replicas will not be assigned to this node
[2017-05-18 14:34:34,142][INFO ][cluster.routing.allocation.decider] [Spider Dop
pelganger] low disk watermark [85%] exceeded on [ep-3aQs6TU6CjXCxQmFP-A][Spider
Doppelganger][C:\Users\XXXX\Personal\elasticsearch-2.3.4\data\elasticsea
rch\nodes\0] free: 8.7gb[14.6%], replicas will not be assigned to this node
[2017-05-18 14:35:04,153][INFO ][cluster.routing.allocation.decider] [Spider Dop
pelganger] low disk watermark [85%] exceeded on [ep-3aQs6TU6CjXCxQmFP-A][Spider
Doppelganger][C:\Users\XXXX\Personal\elasticsearch-2.3.4\data\elasticsea
rch\nodes\0] free: 8.7gb[14.6%], replicas will not be assigned to this node
[2017-05-18 14:35:34,165][INFO ][cluster.routing.allocation.decider] [Spider Dop
pelganger] low disk watermark [85%] exceeded on [ep-3aQs6TU6CjXCxQmFP-A][Spider
Doppelganger][C:\Users\XXXX\Personal\elasticsearch-2.3.4\data\elasticsea
rch\nodes\0] free: 8.7gb[14.6%], replicas will not be assigned to this node
[2017-05-18 14:36:04,179][INFO ][cluster.routing.allocation.decider] [Spider Dop
pelganger] low disk watermark [85%] exceeded on [ep-3aQs6TU6CjXCxQmFP-A][Spider
Doppelganger][C:\Users\XXXX\Personal\elasticsearch-2.3.4\data\elasticsea
rch\nodes\0] free: 8.7gb[14.6%], replicas will not be assigned to this node

Thanks,
Maddy

[jprante]

Please check low disk watermark [85%] exceeded error message, it means that not much data can be imported because of precautions in Elasticsearch 2.3.4

[Maddy7868]

Thanks for you reply. I don't see the low disk watermark error messages now. But I still don't see the data in the import. Please could you help me to identify the issue.

ES 0.20.6

---

Command

---

curl -XDELETE localhost:9200/test
curl -XPUT 'localhost:9200/test/test/1' -d '{"key":"value 1"}'
curl -XPUT 'localhost:9200/test/test/2' -d '{"key":"value 2"}'
curl -XPUT 'localhost:9200/test2/foo/1' -d '{"key":"value 1"}'
curl -XPUT 'localhost:9200/test2/bar/1' -d '{"key":"value 1"}'
curl -XPOST 'localhost:9200/test/_export'
tar zxvf test.tar.gz test/_settings
echo '{"index.number_of_shards":"1","index.number_of_replicas":"0"}' > test/_settings/null/null

Log

---

[2017-05-22 12:07:02,040][INFO ][node ] [Emma Frost] {0.20.6}[9360]: initializing ...
[2017-05-22 12:07:02,071][INFO ][plugins ] [Emma Frost] loaded[knapsack-0.20.6.2-${build], sites []
[2017-05-22 12:07:05,883][INFO ][node ] [Emma Frost] {0.20.6}[9360]: initialized
[2017-05-22 12:07:05,884][INFO ][node ] [Emma Frost] {0.20.6}[9360]: starting ...
[2017-05-22 12:07:06,098][INFO ][transport ] [Emma Frost] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.230.34.132:9300]
}
[2017-05-22 12:07:09,384][INFO ][cluster.service ] [Emma Frost] new_master [Emma Frost][VMfT-GqOQ9yoyrPpPEji-Q][inet[/10.230.34.132:9300]], reason: zen
-disco-join (elected_as_master)
[2017-05-22 12:07:09,416][INFO ][discovery ] [Emma Frost] elasticsearch/VMfT-GqOQ9yoyrPpPEji-Q
[2017-05-22 12:07:09,499][INFO ][http ] [Emma Frost] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.230.34.132:9200]
}
[2017-05-22 12:07:09,499][INFO ][node ] [Emma Frost] {0.20.6}[9360]: started
[2017-05-22 12:07:09,515][INFO ][gateway ] [Emma Frost] recovered [0] indices into cluster_state
[2017-05-22 12:08:54,038][INFO ][cluster.metadata ] [Emma Frost] [test]creating index, cause [auto(index api)], shards [5]/[1], mappings []
[2017-05-22 12:08:54,869][INFO ][cluster.metadata ] [Emma Frost] [test]update_mapping [test] (dynamic)
[2017-05-22 12:08:55,041][INFO ][cluster.metadata ] [Emma Frost] [test2] creating index, cause [auto(index api)], shards [5]/[1], mappings []
[2017-05-22 12:08:55,327][INFO ][cluster.metadata ] [Emma Frost] [test2] update_mapping [foo] (dynamic)
[2017-05-22 12:08:55,382][INFO ][cluster.metadata ] [Emma Frost] [test2] update_mapping [bar] (dynamic)
[2017-05-22 12:08:59,716][INFO ][KnapsackService ] add: plugin.knapsack.export.state -> []
[2017-05-22 12:08:59,722][INFO ][KnapsackExportAction ] start of export: {"mode":"export","started":"2017-05-22T17:08:59.669Z","path":"file:///C:/Users/
XXXX/Personal/elasticsearch-0.20.6/bin/test.tar.gz","node_name":"Emma Frost"
}
[2017-05-22 12:08:59,722][INFO ][KnapsackService ] update cluster settings: plugin.knapsack.export.state -> [{"mode":"export","started":"2017-05-22T17:
08:59.669Z","path":"file:///C:/Users/XXXX/Personal/elasticsearch-0.20.6/bin/test.tar.gz","node_name":"Emma Frost"}]
[2017-05-22 12:08:59,723][INFO ][KnapsackExportAction ] getting settings for indices [test]
[2017-05-22 12:08:59,726][INFO ][KnapsackExportAction ] found indices: [test]
[2017-05-22 12:08:59,747][INFO ][KnapsackExportAction ] getting mappings for index test and types []
[2017-05-22 12:08:59,760][INFO ][KnapsackExportAction ] found mappings: [test]
[2017-05-22 12:08:59,765][INFO ][KnapsackExportAction ] adding mapping: test
[2017-05-22 12:08:59,766][INFO ][KnapsackExportAction ] getting aliases forindex test
[2017-05-22 12:08:59,767][INFO ][KnapsackExportAction ] found 0 aliases
[2017-05-22 12:08:59,872][INFO ][KnapsackExportAction ] end of export: {"mode":"export","started":"2017-05-22T17:08:59.669Z","path":"file:///C:/Users/XXXX/Personal/elasticsearch-0.20.6/bin/test.tar.gz","node_name":"Emma Frost"},
packets = 4, total bytes transferred = 177, rate = 35.400000
[2017-05-22 12:08:59,891][INFO ][KnapsackService ] remove: plugin.knapsack.export.state -> [{"mode":"export","started":"2017-05-22T17:08:59.669Z","path
":"file:///C:/Users/XXXX/Personal/elasticsearch-0.20.6/bin/test.tar.gz",
"node_name":"Emma Frost"}]
[2017-05-22 12:08:59,893][INFO ][KnapsackService ] update cluster settings: plugin.knapsack.export.state -> [{"mode":"export","started":"2017-05-22T17:
08:59.669Z","path":"file:///C:/Users/XXXX/Personal/elasticsearch-0.20.6/
bin/test.tar.gz","node_name":"Emma Frost"}]

ES 0.20.6 cygwin logs
XXXX@PBVWWSU1
$ curl -XDELETE localhost:9200/test
{"error":"IndexMissingException[[test] missing]","status":404}
XXXX@PBVWWSU1
$ curl -XPUT 'localhost:9200/test/test/1' -d '{"key":"value 1"}'
curl -XPUT 'localhost:9200/test2/foo/1' -d '{"key":"value 1"}'
curl -XPUT 'localhost:9200/test2/bar/1' -d '{"key":"value 1"}'
curl -XPOST 'localhost:9200/test/_export'{"ok":true,"_index":"test","_type":"test","_id":"1","_version":1}
XXXX@PBVWWSU1
$ curl -XPUT 'localhost:9200/test/test/2' -d '{"key":"value 2"}'
{"ok":true,"_index":"test","_type":"test","_id":"2","_version":1}
XXXX@PBVWWSU1
$ curl -XPUT 'localhost:9200/test2/foo/1' -d '{"key":"value 1"}'
{"ok":true,"_index":"test2","_type":"foo","_id":"1","_version":1}
XXXX@PBVWWSU1
$ curl -XPUT 'localhost:9200/test2/bar/1' -d '{"key":"value 1"}'
{"ok":true,"_index":"test2","_type":"bar","_id":"1","_version":1}
XXXX@PBVWWSU1
$ curl -XPOST 'localhost:9200/test/_export'
{"running":true,"state":{"mode":"export","started":"2017-05-22T17:08:59.669Z","path":"file:///C:/Users/XXXX/Personal/elasticsearch- 0.20.6/bin/test.tar.gz","node_name":"Emma Frost"}}
XXXX@PBVWWSU1
$ tar zxvf test.tar.gz test/_settings
test/_settings/null/null

XXXX@PBVWWSU1
$ echo '{"index.number_of_shards":"1","index.number_of_replicas":"0"}' > test/_settings/null/null

STOPPED ES 0.20.6
STARTED ES 2.3.4
COPIED THE test.tar.gz under C:\Users\XXXX\Personal\elasticsearch-2.3.4\logs directory

ES 2.3.4
++++++++++++++++++++++++++++++++++++++

Commands

---

curl -XDELETE 'localhost:9200/test'
curl -XPOST 'localhost:9200/test/_import?test_settings=test/_settings/null/null'
curl -XGET 'localhost:9200/test/_settings?pretty'
curl -XPOST 'localhost:9200/test/_search?q=*&pretty'

C:\Users\XXXX\Personal\elasticsearch-2.3.4\bin>elasticsearch.bat
[2017-05-22 12:15:20,823][WARN ][bootstrap ] jvm uses the clientvm, make sure to run java with the server vm for best performance by adding - server to the command line
[2017-05-22 12:15:21,321][INFO ][node ] [Razor Fist] version[2.3.4], pid[8604], build[e455fd0/2016-06-30T11:24:31Z]
[2017-05-22 12:15:21,322][INFO ][node ] [Razor Fist] initializing ...
[2017-05-22 12:15:22,355][INFO ][plugins ] [Razor Fist] modules [reindex, lang-expression, lang-groovy], plugins [knapsack], sites []
[2017-05-22 12:15:22,425][INFO ][env ] [Razor Fist] using [1] data paths, mounts [[SYSTEM (C:)]], net usable_space [9.6gb], net total_space
[59.6gb], spins? [unknown], types [NTFS]
[2017-05-22 12:15:22,425][INFO ][env ] [Razor Fist] heap size [989.8mb], compressed ordinary object pointers [unknown]
[2017-05-22 12:15:25,988][INFO ][node ] [Razor Fist] initialized
[2017-05-22 12:15:25,988][INFO ][node ] [Razor Fist] starting ...
[2017-05-22 12:15:26,291][INFO ][transport ] [Razor Fist] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2017-05-22 12:15:26,297][INFO ][discovery ] [Razor Fist] elasticsearch/Jm_HLWQbQpGl9tpLTFn1fg
[2017-05-22 12:15:30,416][INFO ][cluster.service ] [Razor Fist] new_master {Razor Fist}{Jm_HLWQbQpGl9tpLTFn1fg}{127.0.0.1}{127.0.0.1:9300}, reason: zen
-disco-join(elected_as_master, [0] joins received)
[2017-05-22 12:15:30,462][INFO ][gateway ] [Razor Fist] recovered [0] indices into cluster_state
[2017-05-22 12:15:30,548][INFO ][http ] [Razor Fist] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2017-05-22 12:15:30,551][INFO ][node ] [Razor Fist] started
[2017-05-22 12:17:55,383][INFO ][KnapsackImportAction ] start of import: {"mode":"import","started":"2017-05-22T17:17:55.380Z","path":"file:///C:/Users/
XXXX/Personal/elasticsearch-2.3.4/logs/test.tar.gz","node_name":"Razor Fist"}
[2017-05-22 12:17:55,560][INFO ][cluster.metadata ] [Razor Fist] [.knapsack] creating index, cause [api], templates [], shards [5]/[1], mappings []
[2017-05-22 12:17:55,969][INFO ][cluster.routing.allocation] [Razor Fist] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.knaps
ack][4]] ...]).
[2017-05-22 12:18:25,878][INFO ][cluster.metadata ] [Razor Fist] [.knapsack] create_mapping [knapsack]
[2017-05-22 12:18:26,087][INFO ][KnapsackImportAction ] end of import: {"mode":"import","started":"2017-05-22T17:17:55.380Z","path":"file:///C:/Users/XXXX
/Personal/elasticsearch-2.3.4/logs/test.tar.gz","node_name":"Razor Fist"}, count = 4

Cygwin Logs
+++++++++++++++++++++++++++++++++++++++++
XXXX@PBVWWSU1
$ curl -XDELETE 'localhost:9200/test'
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"test", "index":"test"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"test","index": "test"},"status":404}
XXXX@PBVWWSU1
$ curl -XPOST 'localhost:9200/test/_import?test_settings=test/_settings/null/null'
{"running":true,"state":{"mode":"import","started":"2017-05-22T17:17:55.380Z","path":"file:///C:/Users/XXXX/Personal/elasticsearch- 2.3.4/logs/test.tar.gz","node_name":"Razor Fist"}}
XXXX@PBVWWSU1
$ curl -XGET 'localhost:9200/test/_settings?pretty'
{
"error" : {
"root_cause" : [ {
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "test",
"index" : "test"
} ],
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "test",
"index" : "test"
},
"status" : 404
}

XXXX@PBVWWSU1
$ curl -XPOST 'localhost:9200/test/_search?q=*&pretty'
{
"error" : {
"root_cause" : [ {
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "test",
"index" : "test"
} ],
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "test",
"index" : "test"
},
"status" : 404
}