Java.lang.NoClassDefFoundError: Org/apache/spark/deploy/worker/Worker

Hi, all
*Spark version: bae07e3 [behind 1] fix different versions of commons-lang
dependency and apache/spark#746 addendum*
I have six worker nodes and four of them have this NoClassDefFoundError when
I use thestart-slaves.sh on my driver node. However, running ./bin/spark-class
org.apache.spark.deploy.worker.Worker spark://MASTER_IP:PORT on the worker
nodes works well.
I compile the /spark directory on driver node and distribute to all the
worker nodes. Paths on different nodes are identical.
Here is the logs from one of four driver nodes.
Spark Command: java -cp
::/home/wanghao/spark/conf:/home/wanghao/spark/assembly/target/scala-2.10/spark-assembly-1.0.0-SNAPSHOT-hadoop2.2.0.jar
-Dspark.akka.logLifecycleEvents=true -Xms512m -Xmx512m
org.apache.spark.deploy.worker.Worker spark://192.168.1.12:7077
========================================
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/spark/deploy/worker/Worker
Caused by: java.lang.ClassNotFoundException:
org.apache.spark.deploy.worker.Worker
        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.apache.spark.deploy.worker.Worker.
Program will exit.
Here is spark-env.sh
export SPARK_WORKER_MEMORY=1g
export SPARK_MASTER_IP=192.168.1.12
export SPARK_MASTER_PORT=7077
export SPARK_WORKER_CORES=1
export SPARK_WORKER_INSTANCES=2
hosts file:
127.0.0.1       localhost
192.168.1.12    sing12
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.1.11 sing11
192.168.1.59 sing59
###################
# failed machines
###################
192.168.1.122 host122
192.168.1.123 host123
192.168.1.124 host124
192.168.1.125 host125
Regards,
Wang Hao(王灏)
CloudTeam | School of Software Engineering
Shanghai Jiao Tong University
Address:800 Dongchuan Road, Minhang District, Shanghai, 200240
Email:[email protected]

I made a mistake, that machines in my cluster run different JDKs. After I
unify all the JDKs, the problem is solved.
Regards,
Wang Hao(王灏)
CloudTeam | School of Software Engineering
Shanghai Jiao Tong University
Address:800 Dongchuan Road, Minhang District, Shanghai, 200240
Email:[email protected]

Hi, I run spark on a cluster with 20 machine, but when I start an application use the spark-shell, there only 4 machine is working , the other with just idle, without memery and cpu used, I watch this through webui. I wonder the other machine maybe busy, so i watch the machines using "top" and "free" command, but this is not。 * So I just wonder why not spark assignment work
SGkgYWxsLA0KDQpJ4oCZbSBzdWJtaXR0aW5nIGEgc2ltcGxlIHRhc2sgdXNpbmcgdGhlIHNwYXJr IHNoZWxsIGFnYWluc3QgYSBjYXNzYW5kcmFSREQgKCBEYXRhc3RheCBFbnZpcm9ubWVudCApLg0K SeKAmW0gZ2V0dGluZyB0aGUgZm9sbG93aW5nIGVjZXB0aW9uIGZyb20gb25lIG9mIHRoZSB3b3Jr ZXJzOg0KDQoNCklORk8gMjAxNC0xMC0yNyAxNDowODowMyBha2thLmV2ZW50LnNsZjRqLlNsZjRq TG9nZ2VyOiBTbGY0akxvZ2dlciBzdGFydGVkDQpJTkZPIDIwMTQtMTAtMjcgMTQ6MDg6MDMgUmVt b3Rpbmc6IFN0YXJ0aW5nIHJlbW90aW5nDQpJTkZPIDIwMTQtMTAtMjcgMTQ6MDg6MDMgUmVtb3Rp
Hi All, My cluster has 5 workers each having 4 cores (So total 20 cores).It is in stand alone mode (not using Mesos or Yarn).I want two programs to run at same time. So I have configured "spark.cores.max=3" , but when i run the program it allocates three cores taking one core from each worker making 3 workers to run the program , How to configure such that it takes 3 cores from 1 worker
Hi All,=0A=A0Why does the Worker MPM module forks child processes ,why is it like this.=0A=0AWhy cant it has just one process which will do the regulate thread count depending on the request received./?=0A=0AI just didnt understand why is it like that , can anyone please suugest=0Awhat is the logic behind creating child processes in worker MPM=0A=A0=0AThanks=0AVicky
Hello, I'm testing apache 2.0.35 right now (mpm=worker) but i can't seem to find goodsettings for the server. I try the ab that comes with apache to perform the benchmarks. My old apache 1.3 config : MinSpareServers 5 MaxSpareServers 10 StartServers 5 MaxClients 2048 MaxRequestsPerChild 0 New 2.0.35 worker config : Startservers 5 MaxClients 25 MinSpareThreads 5 MaxSpareThreads 10 ThreadsPerChild
Hi All, I am using spark-0.9.0 and am able to run my program successfully if spark master and worker are in same machine. If i run the same program in spark master in Machine A and worker in Machine B, I am getting below exception I am running program with java -cp "..." instead of scala command In master UI, i see worked node listed properly Form worker Node, if i do ping "MasterNode
Hi All I have one master and one worker on AWS (amazon web service) and am running spark map reduce code provided on the link https://spark.apache.org/examples.html We are using Spark version 1.0.2 Word Count val file = spark.textFile("hdfs://...") val counts = file.flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _) counts.saveAsTextFile
Hi all, Is there a way for spark-submit to restart the JVM in the worker machines? Thanks. Udo.
What is the difference between a Spark Worker and a Spark Slave?
Hi Folks, We have a spark job that is occasionally running out of memory and hanging (I believe in GC). This is it's own issue we're debugging, but in the meantime, there's another unfortunate side effect. When the job is killed (most often because of GC errors), each worker attempts to kill its respective executor. However, it appears that several of the executors fail to shut themselves