Hi,
I am trying to setup a two node Cassandra cluster
Able to start the first node, but not seeing the following exception while
starting the second node
ERROR 17:31:34,315 Exception encountered during startup
java.lang.IllegalStateException: Unable to contact any seeds!
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:947)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:716)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:554)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:451)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
java.lang.IllegalStateException: Unable to contact any seeds!
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:947)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:716)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:554)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:451)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
Exception encountered during startup: Unable to contact any seeds!
ERROR 17:31:34,322 Exception in thread
Thread[StorageServiceShutdownHook,5,main]
java.lang.NullPointerException
at
org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:321)
at
org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:370)
at
org.apache.cassandra.service.StorageService.access$000(StorageService.java:88)
============================My yaml configuration files have these modified
first node yaml
initial_token: -9223372036854775808 # generated this using tokengen tool
seeds: "10.96.19.207" # which is the IP of first node
listen_address: 10.96.19.207 # which is the IP of first node itself
rpc_address: 0.0.0.0
second node yaml
initial_token: 0
seeds: "10.96.19.207" # which is the IP of first node
listen_address: 10.96.10.223 # which is the IP of second node
rpc_address: 0.0.0.0
================Can anyone please help me what went wrong with my configuration?
Regards
Dinesh
Dinesh 's gravatar image asked Aug 27 2013 at 21:45 in Cassandra-User by Dinesh

9 Answers

You would need to configure rpc_address also with hostname/ips on both the
nodes.
Naresh
Naresh Yadav 's gravatar image answered Aug 27 2013 at 22:10 by Naresh Yadav
In my case rpc_address in both the nodes is set to 0.0.0.0 which means it
listens on all interfaces. it has a larger scope (to listen on all
localhost, ipv4, hostnames, ipv6 addresses) than providing just the
hostname/ipv4 addresses
anyway I initially checked that, but it's the same exception I got in this
case also
Regards
Dinesh
Dinesh 's gravatar image answered Aug 27 2013 at 22:36 by Dinesh
Have you verified that your firewall is configured for the cassandra
traffic. At the very least you need to make certain the following ports are
open between nodes: 7000, 7001, 7199, 9160, 61620 and 61621.
John Pyeatt
Singlewire Software,
John Pyeatt 's gravatar image answered Aug 28 2013 at 14:04 by John Pyeatt
Hi John,
I had my firewall disabled in both the nodes
To make sure. I checked it
# rcSuSEfirewall2 status
Checking the status of SuSEfirewall2
unused
if it's, on it says running
Please suggest the further steps, where to look and troubleshoot, if you
have any idea
Regards
Dinesh
Dinesh 's gravatar image answered Aug 28 2013 at 22:41 by Dinesh
To be sure ports are open try to connect from one node to another:
telnet <node ip> 7000
try all ports.
Andrey
Andrey Ilinykh 's gravatar image answered Aug 28 2013 at 22:53 by Andrey Ilinykh
My first node is running and second node is not running in this case
I tried telnet from second node to first node. Following is the stdout
# telnet 10.96.10.207 7000
Trying 10.96.10.207...
Connected to 10.96.10.207.
Escape character is '^]'.
quit
Connection closed by foreign host.
I see the same output for 7199 & 9160 ports. is this correct. I am not
seeing the telnet session as something like telnet>
But when I tried with 7001. I am seeing the below output
# telnet 10.96.10.207 7001
Trying 10.96.10.207...
telnet: connect to address 10.96.10.207: Connection refused
I see the same output for 61620 & 61621 ports
Please suggest
Regards
Dinesh
Dinesh 's gravatar image answered Aug 28 2013 at 23:47 by Dinesh
Have you tried setting rpc_address to the same as listen_address ?
Andy
My first node is running and second node is not running in this case
I tried telnet from second node to first node. Following is the stdout
# telnet 10.96.10.207 7000
Trying 10.96.10.207...
Connected to 10.96.10.207.
Escape character is '^]'.
quit
Connection closed by foreign host.
I see the same output for 7199 & 9160 ports. is this correct. I am not seeing the telnet session as something like telnet>
But when I tried with 7001. I am seeing the below output
# telnet 10.96.10.207 7001
Trying 10.96.10.207...
telnet: connect to address 10.96.10.207<http://10.96.10.207/>: Connection refused
I see the same output for 61620 & 61621 ports
Please suggest
To be sure ports are open try to connect from one node to another:
telnet <node ip> 7000
try all ports.
Andrey
Hi John,
I had my firewall disabled in both the nodes
To make sure. I checked it
# rcSuSEfirewall2 status
Checking the status of SuSEfirewall2 unused
if it's, on it says running
Please suggest the further steps, where to look and troubleshoot, if you have any idea
Have you verified that your firewall is configured for the cassandra traffic. At the very least you need to make certain the following ports are open between nodes: 7000, 7001, 7199, 9160, 61620 and 61621.
In my case rpc_address in both the nodes is set to 0.0.0.0 which means it listens on all interfaces. it has a larger scope (to listen on all localhost, ipv4, hostnames, ipv6 addresses) than providing just the hostname/ipv4 addresses
anyway I initially checked that, but it's the same exception I got in this case also
You would need to configure rpc_address also with hostname/ips on both the nodes.
Naresh
Hi,
I am trying to setup a two node Cassandra cluster
Able to start the first node, but not seeing the following exception while starting the second node
ERROR 17:31:34,315 Exception encountered during startup
java.lang.IllegalStateException: Unable to contact any seeds!
at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:947)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:716)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:554)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:451)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
java.lang.IllegalStateException: Unable to contact any seeds!
at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:947)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:716)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:554)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:451)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
Exception encountered during startup: Unable to contact any seeds!
ERROR 17:31:34,322 Exception in thread Thread[StorageServiceShutdownHook,5,main]
java.lang.NullPointerException
at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:321)
at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:370)
at org.apache.cassandra.service.StorageService.access$000(StorageService.java:88)
=============================
My yaml configuration files have these modified
first node yaml
initial_token: -9223372036854775808 # generated this using tokengen tool
seeds: "10.96.19.207" # which is the IP of first node
listen_address: 10.96.19.207 # which is the IP of first node itself
rpc_address: 0.0.0.0
second node yaml
initial_token: 0
seeds: "10.96.19.207" # which is the IP of first node
listen_address: 10.96.10.223 # which is the IP of second node
rpc_address: 0.0.0.0
==================
Can anyone please help me what went wrong with my configuration?
Regards
Dinesh
Regards
Dinesh
John Pyeatt
Singlewire Software, University of Dundee is a registered Scottish Charity, No: SC015096
Andrew Cobley 's gravatar image answered Aug 29 2013 at 00:10 by Andrew Cobley
Facing the same issue with rpc_address set to the same as listen address on
both nodes
Checked with having
in first node - rpc_address: 10.96.10.207
in second node - rpc_address: 10.96.10.223
Regards
Dinesh
Dinesh 's gravatar image answered Aug 29 2013 at 01:18 by Dinesh
I enabled the debug level for logging. following is the debug entry for the
same exception from node2. Someone please help me
DEBUG 14:53:58,372 Connection version 6 from /10.96.10.105
DEBUG 14:53:58,374 Upgrading incoming connection to be compressed
DEBUG 14:53:58,378 Max version for /10.96.10.105 is 6
DEBUG 14:53:58,379 Setting version 6 for /10.96.10.105
DEBUG 14:53:58,380 set version for /10.96.10.105 to 6
DEBUG 14:53:58,380 Reseting version for /10.96.10.105
DEBUG 14:53:58,683 error writing to rhlcas1/10.96.10.105
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
at sun.nio.ch.IOUtil.write(IOUtil.java:40)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:336)
at java.nio.channels.Channels.writeFullyImpl(Channels.java:59)
at java.nio.channels.Channels.writeFully(Channels.java:81)
at java.nio.channels.Channels.access$000(Channels.java:47)
at java.nio.channels.Channels$1.write(Channels.java:155)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at
org.xerial.snappy.SnappyOutputStream.flush(SnappyOutputStream.java:95)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:205)
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:149)
Regards
Dinesh
Dinesh 's gravatar image answered Aug 29 2013 at 21:08 by Dinesh

Related Discussions

  • Setting Up Prod Cluster in Cassandra-user

  • Hey all, I've been experimenting with Cassandra on a small scale and in my own sandbox for a while now. I'm pretty used to working with it to get small clusters up and running and gossiping with each other. But I just had a new project at work drop into my lap that requires a NoSQL data store. And the developers have selected... you guessed it! Cassasndra as their back end database. ...

  • Failure When Setting Up Cassandra In Cluster in Cassandra-user

  • Hi all, Sorry if this is a fairly stupid question, but we've all only been exposed to Cassandra very recently. We're trying to configure a 2-node cluster with non-default credentials. Here's what I've been doing so far based on my understanding of the documentation. The platform is RHEL 7: 1. Use an RPM I found with Datastax to perform a basic cassandra installation. 2. Change...

  • Decommissioned Node Shows Up In The Gossip Log in Cassandra-user

  • I decommissioned a seed node from the cluster but now it still shows up in the log files but does not show up in "nodetool status". The steps I took to decommission this seed node was first I added another node to the seed node list and remove this node from the seed node list and then restarted all nodes, and then I used "nodetool decommission" to remove the seed node (which is already not a seed...

  • Migrating Data From 2 Node Cluster To A 3 Node Cluster in Cassandra-user

  • We are planning to move data from a 2 node cluster to a 3 node cluster. We are planning to copy the data from the two nodes (snapshot) to the new 2 nodes and hoping that Cassandra will sync it to the third node. Will this work ? are there any other commands to run after we are done migrating, like nodetool repair. Thanks all....

  • Best Practice To Add A Node In A Cluster in Cassandra-user

  • Hi We have a 2 Cluster Node with RF=2. We are planing to add a new node. Should we change RF to 3 in the schema? OR Just added a new node with the same RF=2? Any other Best Practice that we need to take care? Thanks regards Neha...

  • Fail To Add A Node To A Cluster - Unknown Keyspace System_traces in Cassandra-user

  • I have a dev cluster of two Cassandra 2.12 servers on EC2 When adding a new server, I get a "Streaming error occurred java.lang.AssertionError: Unknown keyspace system_traces" exception on the cluster (not the new) server (full log below). Indeed, when I cqlsh to the cluster server, I see the following: cqlsh> DESCRIBE KEYSPACES; system_traces system cqlsh> use system_traces; code"...

  • Vnodes, Adding A Node ? in Cassandra-user

  • I have small test cluster of 2 nodes. I ran a stress test on it and with nodetool status received the following: /usr/local/bin/apache-cassandra-2.0.0-rc1/log $ ../bin/nodetool status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving UN 192.168.0.11 141.13 MB 256 49.2% 4d281e2e-efd9-4abf-bb70-ebdf8e2b4fc3 rack1 UN ...

  • Read After Write Inconsistent Even On A One Node Cluster in Cassandra-user

  • We're doing development on a single node cluster (and yes of course we're not really deploying that way), and we're getting inconsistent behavior on reads after writes. We write values to our keyspaces and then immediately read the values back (in our Cucumber tests). About 20% of the time we get the old value.....if we wait 1 second and redo the query (within the same java method) we get ...

  • Downtime-Limit For A Node In Network-Topology-Replication-Cluster? in Cassandra-user

  • If a node in the cluster goes down and comes up, the data gets synced up on this downed node. Is there a limit on the interval for which the node can remain down? Or the data will be synced up even if the node remains down for weeks/months/years? Regards, Ajay...

  • Bootstrapping A New Node To A Virtual Node Cluster in Cassandra-user

  • Hello all, I have a cluster of three nodes running 1.2.1 and I'd like to increase the capacity by adding a new node. I'm using virtual nodes with 256 tokens and planning to use the same configuration for the new node as well. My cluster looks like this before adding the new node: Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host...

  • Backup Strategies In A Multi DC Cluster in Cassandra-user

  • Hello, I've been experimenting with cassandra for quite a while now. It's time for me to look at backups but I'm not sure what the best practice is. I want to be able to recover the data to a point in time before any user or software errors. We will have two datacentres with 4 servers and RF=3. Each datacentre will have at most 1.6 TB(includes replication, LZ4 compression, using test data) of ...

  • Socket Timeout Errors In One DC In A Multi DC Cluster in Cassandra-user

  • Experiencing socket timeout errors in one DC in most of the nodes in multi dc cluster. Here is error. Client is having intermittent high response time issues in this DC. DC1 does not experience any timeout issues, but DC2 does though. This error started occurring recently and repeats consecutive days. Any suggestions on cause for it? During this situation, when we try to run queries on CQL3...

  • How To Configure Cassandra In A Multi Cluster Mode? in Cassandra-user

  • Hello, I am little bit confusing about cassandra's configuration. There are 2 parameters which I don't understand: listen_address seeds I have 4 identical nodes: 192.168.0.61 cassandra1 192.168.0.62 cassandra2 192.168.0.63 cassandra3 192.168.0.64 cassandra4 What shell I do to configure those 4 nodes into a single cluster? Sincerely, Alexandr...

  • Replacing A Node Using A Replication in Cassandra-user

  • Hi, We currently have a 28 node C* cluster on m1.XLarge instances using Vnodes and are encountering a Raid issue with one of them. The first solution could be to decommission this node and insert a new one in the cluster, since we use vnodes we need to run 28 cleanup after adding a node, this value will increase as our cluster grow. In theory, I would like to duplicate the defective node...

  • Add A Node With Existing Data To A Cluster in Cassandra-user

  • Hi I am attempting to add a cassandra node which has some existing data on it to an existing clutser. Is this a legit thing to do? And what will happen if the same data with different timestamps exists on the node to be added and the existing cluster? What will happen if auto_bootstrapping property is enabled and also i run a repair as soon as the node is added to the cluster? Best Parth...

  • Is There A Way To Remove A Node With Opscenter? in Cassandra-user

  • I know you can use `nodetool removenode` from the command line but is there a way to remove a node from a cluster using OpsCenter?...

  • Adding A Node To Cluster Keeping 100% Data Replicated On All Nodes in Cassandra-user

  • Hi, Our environment will consist of cluster with size not bigger than 2 to 4 nodes per cluster(all located in the same DC). We want to ensure that every node in the cluster will own 100% of the data. A node adding(or removing) procedure will be automated so we want to ensure we're making the right steps. Lets say we have node 'A' up and running and want to add another node 'B' to make a cluster. ...

  • Is There A Way To Add A New Node To A Cluster But Not Sync Old Data? in Cassandra-user

  • Hi there, I am using C* 2.0.10 and I was trying to add a new node to a cluster(actually replace a dead node). But after added the new node some other nodes in the cluster had a very high work-load and affected the whole performance of the cluster. So I am wondering is there a way to add a new node and this node only afford new data?...

  • Recovering From A Faulty Cassandra Node in Cassandra-user

  • Hello, I am using Cassandra 1.2.2 on a 4 node test cluster with vnodes. I waited for over a week to insert lots of data into the cluster. During the end of the process one of the nodes had a hardware fault. I have fixed the hardware fault but the filing system on that node is corrupt so I'll have to reinstall the OS and cassandra. I can think of two ways of reintegrating the host into the cluster...

  • Blocking While A Node Finishes Joining The Cluster After Restart. in Cassandra-user

  • Say I want to do a rolling restart of Cassandra… I can’t just restart all of them because they need some time to gossip and for that gossip to get to all nodes. What is the best strategy for this. It would be something like: /etc/init.d/cassandra restart && wait-for-cassandra.sh … or something along those lines. Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog...

Facebook Google+ Twitter Linkedin
 Discussion Overview

 Group: Cassandra-user

 asked: Aug 27 2013 at 21:45

 active: Aug 29 2013 at 21:08

 posts: 10

 users: 5