storm-deploy for "new generation" ec2 hardware

[vanjos]

I'm looking to deploy storm (and zookeeper) into our VPC with the new i2 and c3 hardware that has recently come available to see if it makes sense for our setup.

I know there may be some issues with VPC and having to set up an IPSEC VPN, and I'm fully okay with that, however I cannot find a way (since I'm new to leiningen and storm-deploy) to get these new generation machines up and running.

When I try to deploy, I obviously get these errors:

DEBUG compute - >> searching params({imageId=us-east-1/ami-xxxxxxxx, hardwareId=c3.large})
DEBUG compute - >> searching params({imageId=us-east-1/ami-xxxxxxxx, hardwareId=c3.large})
WARN compute - hardware ids that didn't match: [cc1.4xlarge, cg1.4xlarge, cc2.8xlarge, t1.micro, c1.medium, c1.xlarge, m1.large, m1.small, m1.medium, m1.xlarge, m2.xlarge, m2.2xlarge, m2.4xlarge, hi1.4xlarge]
java.util.NoSuchElementException: hardwareId(c3.large) not found

[gworley3]

This is almost certainly because the version of jclouds storm-deploy is using does not know about those. Unfortunately, I think at this point storm-deploy can't come up to the latest version of jclouds without first upgrading to pallet 0.8.0.

[vanjos]

Sorry to be newb about this - but what would the upgrade of pallet to 0.8.0 entail?

[gworley3]

A lot. Many things changed between 0.7.x and 0.8.x, so it's by no means straightforward.

[tbatchelli]

The pallet-jclouds library that wraps jclouds is compatible with both 0.7 and 0.8 of pallet, so an upgrade to pallet 0.8 is not required, although largely overdue.

Having said this, pallet-jclouds might need an update if the APIs have changed from jclouds 1.5.5.  One quick way to test is to directly upgrade the jclouds dependencies 1.6 or whichever is the jclouds version that has support for the new instances (see below)

We’ll be looking into putting out a newer version of pallet-jclouds soon.

Toni.

The libraries to add into project.clj in storm-deploy in order to force an update to jclouds are:

[org.jclouds/jclouds-all ~jclouds-ver]
[org.jclouds.driver/jclouds-sshj ~jclouds-ver]
[org.jclouds.driver/jclouds-slf4j ~jclouds-ver]
[org.jclouds/jclouds-compute ~jclouds-ver]
[org.jclouds/jclouds-blobstore ~jclouds-ver]

where ~jclouds-ver is the version of jclouds that you want.

Also, you might want to update the pallet-jclouds dependency to 1.5.3 (currently 1.5.1)

tbatchelli
Sent with Airmail

On February 13, 2014 at 9:17:33 AM, G Gordon Worley III ([email protected]) wrote:

A lot. Many things changed between 0.7.x and 0.8.x, so it's by
no means straightforward.

---

Reply to this email directly or view it on GitHub:
#57 (comment)

[vanjos]

Got me really excited --- however, still issues:

--- project.clj ---
(defproject storm-deploy "0.0.6-SNAPSHOT"
:source-paths ["src/clj"]
:test-paths ["test/clj"]
:profiles {:dev {:resource-paths ["conf"]}}
:aliases {"deploy-storm" ["run" "-m" "backtype.storm.provision"]}

:repositories {
"sonatype" "https://oss.sonatype.org/content/repositories/releases"
"jclouds-snapshot" "https://oss.sonatype.org/content/repositories/snapshots"
}

:dependencies [
[storm "0.5.4"]
[commons-codec "1.4"]
[org.cloudhoist/pallet "0.7.3"]
[org.cloudhoist/pallet-jclouds "1.5.3-SNAPSHOT"]
[org.cloudhoist/java "0.5.0"]
[org.cloudhoist/git "0.5.0"]
[org.cloudhoist/ssh-key "0.5.0"]
[org.cloudhoist/automated-admin-user "0.5.0"]
[org.cloudhoist/iptables "0.5.0"]
[org.cloudhoist/maven "0.5.0"]
[org.cloudhoist/zookeeper "0.5.1"]
[org.cloudhoist/nagios-config "0.5.0"]
[org.cloudhoist/crontab "0.5.0"]

             [com.jcraft/jsch.agentproxy.usocket-jna "0.0.5"]
             [com.jcraft/jsch.agentproxy.usocket-nc "0.0.5"]
             [com.jcraft/jsch.agentproxy.sshagent "0.0.5"]
             [com.jcraft/jsch.agentproxy.pageant "0.0.5"]
             [com.jcraft/jsch.agentproxy.core "0.0.5"]
             [com.jcraft/jsch.agentproxy.jsch "0.0.5"]
             [com.jcraft/jsch "0.1.49"]
             [org.jclouds.driver/jclouds-sshj "1.5.2"]
     [org.jclouds/jclouds-all "1.6.0"]
     [org.jclouds.driver/jclouds-sshj "1.6.0"]
     [org.jclouds.driver/jclouds-slf4j "1.6.0"]
     [org.jclouds/jclouds-compute "1.6.0"]
     [org.jclouds/jclouds-blobstore "1.6.0"]
             [org.jclouds.provider/aws-ec2 "1.5.2"]
             [org.jclouds.provider/aws-s3 "1.5.2"]

             [log4j/log4j "1.2.14"]
             [jvyaml "1.0.0"]]

:dev-dependencies [[swank-clojure "1.2.1"]
[org.cloudhoist/pallet-lein "0.5.2"]]
:min-lein-version "2.0.0")

Exception in thread "main" java.io.FileNotFoundException: Could not locate clojure/reflect__init.class or clojure/reflect.clj on classpath: (jclouds.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:5440)
at clojure.lang.Compiler.eval(Compiler.java:5415)
at clojure.lang.Compiler.load(Compiler.java:5857)
at clojure.lang.RT.loadResourceScript(RT.java:340)
at clojure.lang.RT.loadResourceScript(RT.java:331)
at clojure.lang.RT.load(RT.java:409)
at clojure.lang.RT.load(RT.java:381)
at clojure.core$load$fn__4511.invoke(core.clj:4905)
at clojure.core$load.doInvoke(core.clj:4904)
at clojure.lang.RestFn.invoke(RestFn.java:409)
at clojure.core$load_one.invoke(core.clj:4729)
at clojure.core$load_lib.doInvoke(core.clj:4766)
at clojure.lang.RestFn.applyTo(RestFn.java:143)
at clojure.core$apply.invoke(core.clj:542)
at clojure.core$load_libs.doInvoke(core.clj:4800)
at clojure.lang.RestFn.applyTo(RestFn.java:138)
at clojure.core$apply.invoke(core.clj:542)
at clojure.core$require.doInvoke(core.clj:4869)
at clojure.lang.RestFn.invoke(RestFn.java:409)
at backtype.storm.provision$eval20$loading__4410__auto____21.invoke(provision.clj:1)
at backtype.storm.provision$eval20.invoke(provision.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:5424)
at clojure.lang.Compiler.eval(Compiler.java:5415)
at clojure.lang.Compiler.load(Compiler.java:5857)
at clojure.lang.RT.loadResourceScript(RT.java:340)
at clojure.lang.RT.loadResourceScript(RT.java:331)
at clojure.lang.RT.load(RT.java:409)
at clojure.lang.RT.load(RT.java:381)
at clojure.core$load$fn__4511.invoke(core.clj:4905)
at clojure.core$load.doInvoke(core.clj:4904)
at clojure.lang.RestFn.invoke(RestFn.java:409)
at clojure.core$load_one.invoke(core.clj:4729)
at clojure.core$load_lib.doInvoke(core.clj:4766)
at clojure.lang.RestFn.applyTo(RestFn.java:143)
at clojure.core$apply.invoke(core.clj:542)
at clojure.core$load_libs.doInvoke(core.clj:4800)
at clojure.lang.RestFn.applyTo(RestFn.java:138)
at clojure.core$apply.invoke(core.clj:542)
at clojure.core$require.doInvoke(core.clj:4869)
at clojure.lang.RestFn.invoke(RestFn.java:409)
at user$eval5$fn__7.invoke(form-init257033639388524929.clj:1)
at user$eval5.invoke(form-init257033639388524929.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:5424)
at clojure.lang.Compiler.eval(Compiler.java:5415)
at clojure.lang.Compiler.load(Compiler.java:5857)
at clojure.lang.Compiler.loadFile(Compiler.java:5820)
at clojure.main$load_script.invoke(main.clj:221)
at clojure.main$init_opt.invoke(main.clj:226)
at clojure.main$initialize.invoke(main.clj:254)
at clojure.main$null_opt.invoke(main.clj:279)
at clojure.main$main.doInvoke(main.clj:354)
at clojure.lang.RestFn.invoke(RestFn.java:422)
at clojure.lang.Var.invoke(Var.java:369)
at clojure.lang.AFn.applyToHelper(AFn.java:165)
at clojure.lang.Var.applyTo(Var.java:482)
at clojure.main.main(main.java:37)
Caused by: java.io.FileNotFoundException: Could not locate clojure/reflect__init.class or clojure/reflect.clj on classpath:
at clojure.lang.RT.load(RT.java:412)
at clojure.lang.RT.load(RT.java:381)
at clojure.core$load$fn__4511.invoke(core.clj:4905)
at clojure.core$load.doInvoke(core.clj:4904)
at clojure.lang.RestFn.invoke(RestFn.java:409)
at clojure.core$load_one.invoke(core.clj:4729)
at clojure.core$load_lib.doInvoke(core.clj:4766)
at clojure.lang.RestFn.applyTo(RestFn.java:143)
at clojure.core$apply.invoke(core.clj:542)
at clojure.core$load_libs.doInvoke(core.clj:4800)
at clojure.lang.RestFn.applyTo(RestFn.java:138)
at clojure.core$apply.invoke(core.clj:544)
at clojure.core$use.doInvoke(core.clj:4880)
at clojure.lang.RestFn.invoke(RestFn.java:458)
at pallet.compute.jclouds$eval5815$loading__4410__auto____5816.invoke(jclouds.clj:1)
at pallet.compute.jclouds$eval5815.invoke(jclouds.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:5424)
... 55 more

[gworley3]

I was only able to go as high as pallet-jclouds 1.5.1 and jclouds 1.5.2 when I was actively working on my storm-deploy fork. Haven't revisited since then.

[vanjos]

So even with pallet-jclouds 1.5.1 and jclouds 1.5.2 it doesn't work for me...

[tbatchelli]

I have pushed a branch 'update-clojure-jclouds` that uses the latest jclouds version (1.7.1), and this should allow using all the newer hardware in AWS. Give it a try and report back if you find any issues so I can proceed to merge it into master.