[project-voldemort] Zone Expansion Of RW Cluster With Large Number Of Partitions.

Hi,
I am investigating the zone expansion prospects of a RW cluster with large 
partition list (around 10,000). The config generation 
using rebalance-zone-expansion.sh script takes forever at the rate of 
around 80 iterations every 90 mins. One possible way I can think of to 
reduce the generation time is to reduce the number of re-partitioning 
iterations (1000 by default) it goes through, but I fear the re-partioned 
cluster.xml generated will not be optimal with respect to the primary and 
cross-zone partition moves. Are there other parameters which can be tuned 
to expedite the config generation process?
I understand config generation is a one-time process but I suspect the 
large number of partitions are going to hurt during the actual rebalance as 
well? The only parameter which I am aware of is the parallelism parameter 
which looks like a good candidate to play around with to increase the 
rebalance throughput. Are there any other parameters which can be tuned for 
getting high rebalance throughput with large number of partitions?
Also, piggybacking another question related to rebalance, I observed that 
after a failed rebalance (due to simulated transient failures) a few 
servers will be left in a "REBALANCING_MASTER_SERVER" while others switch 
the state back to "NORMAL_SERVER". Will the proxy bridges be broken after 
the server state changes to NORMAL? 
Thanks in advance for your help,
Chinmay
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

We had a situation where the kafka 0.8.2 broker would not come up
I'm looking to create a topic with about 1400 partitions to allow a high degree of parallel processing. We have 5 brokers so that would be 280 partitions per box. Has anyone done something with this number of partitions before? Thanks.
Hi, I am having an issue with a large number of 4000 partitions (each being very small
I currently have a topic with 1024 partitions. I know it's kind of going past the recommended limits, but I kept it like that because I am moving a legacy system to kafka and it has a 1024 parallel partitions. I wanted to understand the costs of having so many partitions a little bit more though. I understand that the brokers will be doing a lot more random disk IO with that many partitions unless
Hi all, I have a question about adding new nodes. I know that I can add new nodes with zero partition on them and can rebalance the cluster so every node has equal number of partitions. But can I add new nodes with more partitions? Let say I have 10 nodes with 30 partitions (i.e. 3 partitions per node), and I want to add 10 more nodes with 30 more partitions (i.e. 60 partitions in total
So I'm in the process of automating the scaling of our voldemort cluster and I'm running into a few questions: 1 - Does num_partitions mod num_servers always have to be 0? 2 - Can the number of partitions be increased after the fact? Essentially I'm sitting on a preconfigured in-production voldemort store with 2 servers and 14 partitions. As I look to automatically scale that, I may need to
Hi All, I have a hadoop cluster with data present in large number of directories ( > 10,000) To run HIVE queries over this data I created an external partitioned table and pointed each directory as a partition to the external table using 'alter table add partition' command. Is there a better way to create a HIVE external table over large number of directories? Also I am facing the following issues
I am planning to use the current Java API and have the following use case: i) A single topic with about 1024 partitions. ii) A number of processes that want to consume these partitions in a deterministic way. The machine -> partitions assignment is done outside of kafka. During the lifetime of a process it might relinquish ownership of partitions and also gain ownership of partitions. This
I wrote a custom partitioner. But when I work as standalone or pseudo-distributed mode, the number of partitions is always 1. I set the numberOfReducer to 4, but the numOfPartitions parameter of custom partitioner is still 1 and all my four mappers' results are going to 1 reducer. The other reducers yield empty files. How can i set the number of partitions in standalone or pseudo-distributed mode
Hi Impala, I have a table partitioned by yearmonth and the first 3 characters of a certain field. The table is stored as Parquet. CREATE EXTERNAL TABLE `parquet_table`( `keyword` string, `field_1` int, `field_2` int', `field_3` int) PARTITIONED BY ( `yearmonth` int, `keyword_prefix` string) STORED AS PARQUET LOCATION '' Say I already have 24 months, and the keyword