Handling Special Character While Sqoop Import

Hi,
I am doing a Sqoop import from mysql as source, recently I figured out that
data imported through sqoop from mysql was having some special characters
and even control character which was loosing its meaning while moved to
sqoop data files.
Looking out for a solution as how to handle this case of special character
or if possible pruning the unwanted data out of my target dataset.
Looking out for resolution at the earliest!
Thanks!

Hey there,
Could you explain what you mean by "losing its meaning"? It's possible you
may need to set the character set:
http://dev.mysql.com/doc/connector-j/en/connector-j-reference-charsets.html.
-Abe

Hi Abe,
Well with the above statement I mean to say that the data which is residing
in mysql is different from what is been imported via sqoop.
So let me shoot out an example for the same,
*Data in mysql : *सुरेन्द्र कुमार पाण्डेय
*Data in HDFS(Sqoop import) : * M-`M-$M-8M-`M-%M-
So this is the kind of changes I am landing into which is completely
loosing the meaning of the data.
Any help would be appreciated.
Thanks again!

This could be in 2 places: Loading to HDFS, or extracting from MySQL. Sqoop
should load every thing as UTF-8 by default, which supports Hindi.
What is your default character set in MySQL? Could you copy/paste your
my.cnf? Also, what version of MySQL are you running?

Hi Abe,
Thanks for your mail, well mysql table is defined with utf-8 and even the
data is visible like mentioned below,
*Data in mysql : *सुरेन्द्र कुमार पाण्डेय
but as I move the same through sqoop import of data gets corrupted, as
provided in the last thread of this mail.
Well I even tried to set the parameters
*useUnicode=true&characterEncoding=utf8* and *--direct --
still there's no luck.
Additionally, the data is containing some control character like Ctrl-A
(x001) and Ctrl-M likewise, which is even violating the field delimeter set
to sqoop import precisely as Ctrl-A. Is there a way to keep a possible
delimeter which can handle/work with any special or control character
introduced.
Looking out for quick response.
Thanks!

Well it seems to be the issue with Mysql Client configuration present on
the datanodes where sqoop is invoking the m/r job.
I performed a test on my local machine dumping the same data to mysql and
did a sqoop import to the hdfs and I can clearly see the data boarded to
HDFS.
This clearly indicates that the issue was in mysql client configuration
which I need to rectify and set character-set type to utf-8(I thought the
default character-set would be set to utf-8).
But still the later part of the question remains same, how do I manage the
control character present in the data as I don't know what could be the
part of data(as I have encountered Control characters), setting delimiter
as Control character would not solve the meaning if the data contained that
character itself.
Looking out for the standard solution.
Thanks!

Okay, this is driving me insane.  I'm having a problem importing data into MongoDB using the mongoimport utility.  I'm doing an extra-credit exercise on my own to gain more proficiency with MongoDB. (Mainly because this is so cool and I'm having such a great time! :-) )  The problem seems to be related to the extended ASCII character set with regard to string text.  (I think.) What I'm trying
If the import table is with special character in field name and unsupported data type, what can I do with `sqoop --map-column-java` ? Such as Oracle Table @MABCNO GRAPHIC(12) VECTOR$ RAW I try following command but fail. Is there any suggestions ? sqoop import --connect "jdbc:db2://X.X.X.X:50000/DBSRC" --username jazz -P - Jazz
Hello, I have a problem and I doubt the problem is connecting to HDFS, but unable to figure out how to fix this. My MySQl and HDFS are fine, all services are up and running. To confirm sqoop connects, I ran this. pman@pman-lappy:~/hadoop-1.0.3$ sqoop eval --connect jdbc:mysql://localhost/sqoop --username root --password root --query "select * from sqoop_test" Warning: /usr/lib/hbase
Hi, I'm trying to import data from Postgresql which have column types character varying(64)[] character varying(32)[] character varying(321)[] character varying(32)[] character varying(64)[] varchar[] varchar(64)[] I see an issue in SQOOP Jira which shows not resolved https://issues.apache.org/jira/browse/SQOOP-1000 . Is this issue still unnresolved? Do
Hi, I am using db2jcc4.jar , sqoop 1.4.2,  While the connection is closed by sqoop , it throws the below warning,  14/09/15 16:30:39 INFO mapreduce.ImportJobBase: Transferred 24.4863 KB in 14.0591 seconds (1.7417 KB/sec) 14/09/15 16:30:39 INFO mapreduce.ImportJobBase: Retrieved 578 records. 14/09/15 16:30:39 INFO util.AppendUtils: Appending to directory db2_test 14/09/15 16:30:39 INFO util
Hi, I'm just getting started with php/mysql and I have a big question: how do I handle the special chars? when I import an excel sheet into my db, all special chars are displayed wrongly. If a change them to html entities befor the import, it works, but it's ugly to have all the & and whatnot in the db. Is there some way i can administrate thise special chars? also, the function htmlentities
Hi All, Sqoop while used with HCatalog import not able to remove new line (\n) from column data even after using --hive-drop-import-delims option in the command when running Apache Sqoop with Oracle. Sqoop Query: sqoop import --connect jdbc:oracle:thin:@ORA_IP:ORA_PORT:ORA_SID \--username user123 --password passwd123 -table SCHEMA.TBL_2 \ "Sqoop with HCAT" \--null-string "" Data
Hi I am having sqoop import issue with OID (String) as splitby parameter. *Sqoop Command : * $SQOOP_HOME/bin/sqoop import --connect jdbc:oracle:thin:@HOST:POST/ORACLE_DB *Logs : * ..... ..... ..... 15/07/15 16:01:23 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/share/hadoop_echosystem/sqoop-1.4.5/lib/tdgssconfig.jar 15/07/15 16:01:26 DEBUG db.DBConfiguration: Fetching
Hi, I have been trying to do sqoop import using oozie workflow job but oozie job fails. I can see in logs that Launcher is unable to find the sqoop jars. I have checked the shared lib in hdfs. The jars are there. Details of job.properties : workflowAppUri=hdfs://hadoop-namenode:9000/user/rakesh/ladapapps/ 32/295 oozie.wf.application.path=hdfs://hadoop-namenode:9000/user/rakesh/ ladapapps
I looked at the options provided by sqoop export operation but could not find any options to handle bad records. For example once in a while it is possible that a character is present where a number is expected in a huge set of records. Is there a way to handle these scenarios in sqoop without failing the job and providing the bad records in a file.   Even if this is out of the box functionality