Hi
I have a peculiar problem of moving hive data from one hive environment say prod to another dev. The data runs into crores of rows. How can I do that in a less time consuming efficient manner.
Thanks in advance
Get Outlook for Android<https://aka.ms/ghei36>
rakesh sharma 's gravatar image asked Feb 17 2017 at 08:45 in Hive-User by rakesh sharma

2 Answers

Try Flume.
I hope it will help you moving the data efficiently.
Thanks,
Balajee Venkatesh
Hi
I have a peculiar problem of moving hive data from one hive environment say
prod to another dev. The data runs into crores of rows. How can I do that
in a less time consuming efficient manner.
Thanks in advance
Get Outlook for Android <https://aka.ms/ghei36>
balajee venkatesh 's gravatar image answered Feb 17 2017 at 08:48 by balajee venkatesh
Use BDR job if your using cloudera
Dulam, Naresh 's gravatar image answered Feb 17 2017 at 08:49 by Dulam, Naresh

Related Discussions

  • Hive Double Issues While Moving Around RC Files Between Clusters in Hive-user

  • Hi there, I was copying around RC files from an CDH hadoop 2.0 cluster to a new HDP hadoop 2.6 cluster. After creating a new table with the storage options RC file and LOCATION pointing to the right direction I can query all columns, except for the I tried querying with Hive (via tez and MR), beeline, presto. None of these work. The error from hive is: java.lang.RuntimeException...

  • Hive Queries On S3 Data Not Working After Moving To Hive Metastore On CDH4 in Hive-user

  • Hi, We were running some hive queries off data on amazon S3.In the hive script file we are including the access key and secret access key as below. set fs.s3.awsAccessKeyId=ABCD; set fs.s3.awsSecretAccessKey=XYZ; set fs.s3n.awsAccessKeyId=ABCD; set fs.s3n.awsSecretAccessKey=XYZ; and then we run the script using hive -f Everything was working fine until we changed our hive configuration to use...

  • Hive & Json in Hive-user

  • Hi there, What's the best practice to read json data with Hive? select get_json_object? Or hive-json-serde.jar? Thanks, -ari...

  • Hive + MongoDB in Hive-user

  • Hi every one , I am trying to import data from mongodb to hive . i got some jar files to connect mongo and hive . now how to import the data from mongodb to hive ? Thanks in advance. Sandeep Nemuri...

  • HIVE+MAPREDUCE in Hive-user

  • Hi, Need to load the data into hive table using mapreduce, using java. Please suggest the code related to hive +mapreduce. Thanks in advance Ranjini R...

  • Hive BeeLine in Hive-user

  • Hi Users, I have Hive and HiveServer2 on the same machine. But, when I try to *LOAD DATA LOCAL INPATH* using BeeLine I get "Invalid Path Error" and if I do LOAD DATA INPATH it takes it successfully. Can anyone pls tell me why local inpath does not work?? Thanks, Trainee....

  • Importing Data Into Hive in Hive-user

  • Hi all, I'm interested in knowing how everyone is importing their data into their production Hive clusters. Let me explain a little more. At the moment, I have log files (which are divided into 5 minute chunks, per event type (of which there are around 10), per server (a few 10s) arriving on one of my Hadoop nodes. These log files then get glued together by some custom code, into entity+hour buckets...

  • Hive And Unstructured Data in Hive-user

  • Hello I am curious to know how Hive maps the real-world unstructured data (like Facebook logs) with its own structures(tables). In other words, if there is a concept of building a table over unstructured data in Hive then how does the structure is exactly defined? Thank you in advance for your response. Mahsa...

  • Hive Compression, Piping Data in Hive-user

  • Hi there Hive Groupers, I've got a question regarding hive Architecture in regards to compression, or more how Hive treats compressed tables when it reads from them. Use Case: 1. 2 Compressed tables in HDFS, 1TB each. 2. One table is compressed with a splittable compression while the other isn't. 3. Mapreduce program reads each table and write a new text only table (uncompressed around...

  • Flume Data To Hive in Hive-user

  • Hey guys, I am using flume to directly sink data into my hive table. However, there seems to be some schema inconsistency, and I am not sure how to troubleshoot it. I created a hive table 'targeting' in hive, it use sequence file, snappy compression, partitioned by 'epoch'. After the table is created, I could see a folder called 'targeting' under my folder: /hive/cwang49.db/targeting I...

  • Meta Data Tables - Hive in Hive-user

  • Hi all, In Hive do we have any meta data tables where I can see all the table's / view's / Indexes information. For example, in Oracle, we have *USER_TABLES* meta data table to know about all the tables available in the user. *USER_VIEWS* to know about all the view information available in the user. Like this , do we have any meta data tables ( data dictionary tables) avilable ...

  • Hive Orchestration in Hive-user

  • I'm looking for a framework that manages automatic initiation of our daily data loading and processing, with knowledge of dependencies between tables and "data ready" status flags. I think some people call this "Orchestration" (though there's not a settled definitions of this word). I get the impression there are a lot of home grown solutions for this. But I'd like a generalized solution...

  • Hive Sessionization in Hive-user

  • Is there a better way to use Hive to sessionize my log data ? I'm not sure that I'm doing so, below, in the optimal way: The log data is stored in sequence files; a single log entry is a JSON string; eg: {"source": {"api_key": "app_key_1", "user_id": "user0"}, "events": [{"timestamp": 1330988326, "event_type": "high_score", "event_params": {"score": "1123", "level": "9"}}, {"timestamp": 1330987183...

  • Hive Queries in Hive-user

  • I am looking for a relatively efficient way of transferring data between a remote server and Hive without going through the hassle of storing the data first on memory before loading it to Hive. >From what I have read so far there is no such command but it would not hurt to ask. Is it possible to insert data through an insert query in hive? (The equivalent to insert into table_name values (...) in ...

  • Partitioning Hive in Hive-user

  • I all I have a folder structure like /housing/events/event1/date 150820 stored in HDFS. I wanted to create the partition based on year month and day how can I do this during creating or loading data time? Regards Jeet...

  • Hive JDBC in Hive-user

  • I am using a Hive instance using MySQL as the meta-data store. The cluster works fine in command line mode. The problem I am having is that when I attempt to connect using the Hive JDBC connector I get the following error: java.sql.SQLException: org.jpox.jdo.exceptions.TransactionNotReadableException: Cant read fields outside of transactions. You may want to set 'NontransactionalRead=true...

  • Hive Data in Hive-user

  • Hello, I am interested in testing Hive with a huge sample data. Does Hive read all data types? Should the file be a table? Thank you ******************************* This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorised to distribute, copy or use this e...

  • Populating Xml Data In Hive in Hive-user

  • Hi All, I have a csv file ( separated by |) where data is like id data date 1 apple 24-nov-2011 2 mango 26-nov-2011 3 fruits 28-nov-2011 4 papaya 30-nov...

  • LOAD Data Into Hive Table in Hive-user

  • Hi, The loading of data into hive table currently moves the data if the location is hdfs (NON LOCAL). Is there an option that would allow to copy the data instead ? =20 thanks, Navin...

  • Hive Not Reflecting Hdfs Data in Hive-user

  • Hi,=0A=0AI am a hive newbie.I am managing a setup where data is regularly f= ed into HDFS =0Ausing flume.However, hive does not show the data that is re= cently added to the =0AHDFS.It used to earlier,but somehow its not updating= now.The queries i fire all =0Agive answers to the old HDFS and do not refl= ect the newer data added there.=0AIs there a configuration that is messed u= p?Is there someway...