1.install and configure hadoop-2.6.0 ($HADOOP_HOME must be set).
2. download mysql-5.6.22.tar.gz source code from http://dev.mysql.com/downloads/mysql/
#tar xf mysql-5.6.22.tar.gz
#cd mysql-5.6.22
#cmake .
#make
#export MYSQL_DIR=/path/to/mysql-5.6.22
3. download mysql-connector-c-6.1.5-src.tar.gz from http://dev.mysql.com/downloads/connector/c/#downloads
#tar xf mysql-connector-c-6.1.5-src.tar.gz
#cd mysql-connector-c-6.1.5-src
#cmake .
#make
#make install
4.download mysql hadoop applier from http://labs.mysql.com.
#tar xf mysql-hadoop-applier-0.1.0-alpha.tar.gz
#cd mysql-hadoop-applier-0.1.0-alpha
5. download FindHDFS.cmake from https://github.com/cloudera/impala/blob/master/cmake_modules/FindHDFS.cmake
#mv FindHDFS.cmake /path/to/mysql-hadoop-applier-0.1.0-alpha/MyCMake
#cmake . -DENABLE_DOWNLOADS=1
#make
#make install
The library 'libreplication' which is to be used by Hadoop Applier is in lib dir
An otherway is :
#mkdir build
#cd build
#cmake .. -DCMAKE_MODULE_PATH:String=../MyCMake -DENABLE_DOWNLOADS=1
#make
#make install
#export PATH=$HADOOP_HOME/bin:$PATH
#export CLASSPATH=$(hadoop classpath)
#cd build/examples/mysql2hdfs
#make
there is a error
[ 77%] Built target replication_static
Linking CXX executable happlier
/usr/bin/ld: warning: libmawt.so, needed by /opt/jdk1.7.0_51/jre/lib/amd64/libjawt.so, not found (try using -rpath or -rpath-link)
/opt/jdk1.7.0_51/jre/lib/amd64/libjawt.so: undefined reference to `awt_Unlock@SUNWprivate_1.1'
/opt/jdk1.7.0_51/jre/lib/amd64/libjawt.so: undefined reference to `awt_GetComponent@SUNWprivate_1.1'
/opt/jdk1.7.0_51/jre/lib/amd64/libjawt.so: undefined reference to `awt_Lock@SUNWprivate_1.1'
/opt/jdk1.7.0_51/jre/lib/amd64/libjawt.so: undefined reference to `awt_GetDrawingSurface@SUNWprivate_1.1'
/opt/jdk1.7.0_51/jre/lib/amd64/libjawt.so: undefined reference to `awt_FreeDrawingSurface@SUNWprivate_1.1'
collect2: ld returned 1 exit status
make[2]: *** [examples/mysql2hdfs/happlier] Error 1
make[1]: *** [examples/mysql2hdfs/CMakeFiles/happlier.dir/all] Error 2
make: *** [all] Error 2
the suggested solution is
export LD_LIBRARY_DIR=${JAVA_HOME}/jre/lib/amd64/xawt:${LD_LIBRARY_DIR}
but the error still exists.
(I found that the above env variable is error,the correct is
export LD_LIBRARY_PATH=${JAVA_HOME}/jre/lib/amd64/xawt:${LD_LIBRARY_PATH}
)
#./happlier
a error is
[root@dmining05 mysql2hdfs]# ./happlier
The default data warehouse directory in HDFS will be set to /usr/hive/warehouse
Change the default data warehouse directory? (Y or N) n
Enter either Y or N:N
loadFileSystems error:
(unable to get stack trace for java.lang.NoClassDefFoundError exception: ExceptionUtils::getStackTrace error.)
hdfsBuilderConnect(forceNewInstance=0, nn=default, port=0, kerbTicketCachePath=(NULL), userName=(NULL)) error:
(unable to get stack trace for java.lang.NoClassDefFoundError exception: ExceptionUtils::getStackTrace error.)
Couldnot connect to HDFS file system
the error may be due to the following cause
http://stackoverflow.com/questions/21064140/hadoop-c-hdfs-test-running-exception
#echo $CLASSPATH
/root/hadoop-2.6.0/etc/hadoop:
/root/hadoop-2.6.0/share/hadoop/common/lib/*:
/root/hadoop-2.6.0/share/hadoop/common/*:
/root/hadoop-2.6.0/share/hadoop/hdfs:
/root/hadoop-2.6.0/share/hadoop/hdfs/lib/*:
/root/hadoop-2.6.0/share/hadoop/hdfs/*:
/root/hadoop-2.6.0/share/hadoop/yarn/lib/*:
/root/hadoop-2.6.0/share/hadoop/yarn/*:
/root/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:
/root/hadoop-2.6.0/share/hadoop/mapreduce/*:
/root/hadoop-2.6.0/contrib/capacity-scheduler/*.jar
For hadoop versions 2.0.0 and above, the classpath doesn't support wild characters. If you add the jars explicitly to the CLASSPATH, your app will work.
Can JNI be made to honour wildcard expansion in the classpath?
One way is to build a file name setclass.sh
#!/bin/bash
hcp=$(hadoop classpath)
#echo $hcp
arr=(${hcp//:/ })
len=${#arr[@]}
let len-=1
echo $len
j=0
export CLASSPATH=/etc/hadoop/conf
for i in ${arr[@]}
do
# echo $i
if [ $j -eq 0 ]; then
export CLASSPATH=$i
elif [ $j -eq $len ]; then
echo $i
else
export CLASSPATH=$CLASSPATH:$i
fi
let j+=1
done
then source it, not to execute it
#source setclass.sh
#echo $CLASSPATH
This work well in current session, if want to make it work gloably,
#$echo $CLASSPATH >> ~/.bashrc
#source ~/.bashrc
open another consele to check it .
mysql configuration in /etc/my.cnf likes
[mysqld]
basedir = /usr/local/mysql
datadir = /var/lib/mysql
port = 3306
#socket = /var/lib/mysql/mysql.sock
user=mysql
bind_address = ::
#bin log conf
log_bin = masterbin_log
binlog_checksum = NONE
binlog_format = ROW
server-id = 2
#./happlier mysql://root@127.0.0.1:3306 hdfs://localhost:9000
the msg is
the right way is
#./happlier mysql://root:123456@127.0.0.1:3306 hdfs://localhost:9000
(you should provide password to access mysql)
but this warning still exists
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Refrences
http://innovating-technology.blogspot.com/2013/04/mysql-hadoop-applier-part-1.html
http://innovating-technology.blogspot.com/2013/04/mysql-hadoop-applier-part-2.html
http://paddy-w.iteye.com/blog/2023656
http://www.tuicool.com/articles/NfArA3i
- 大小: 33.2 KB
分享到:
相关推荐
Its simple programming model, "code once and deploy at any scale" paradigm, and an ever-growing ecosystem make Hadoop an inclusive platform for programmers with different levels of expertise and ...
This course is included in the Expert Exchange on Hadoop: Using SAS/ACCESS service offering to configure SAS/ACCESS Interface to Hadoop or SAS/ACCESS Interface to Impala to work with your Hadoop ...
Hadoop Data Processing and Modelling 英文azw3 本资源转载自网络,如有侵权,请联系上传者或csdn删除 本资源转载自网络,如有侵权,请联系上传者或csdn删除
Starting with the basics of Apache Hadoop and Solr, this book then dives into advanced topics of optimizing search with some interesting real-world use cases and sample Java code.
Hadoop offers distributed processing of large datasets across clusters and is designed to scale up from a single server to thousands of machines, with a very high degree of fault tolerance. It enables...
Big Data, MapReduce, Hadoop, and Spark with Python: Master Big Data Analytics and Data Wrangling with MapReduce Fundamentals using Hadoop, Spark, and Python by LazyProgrammer English | 15 Aug 2016 | ...
This book covers the process from beginning to end, providing a crash course for professionals needing to learn and apply Hadoop quickly., Configure storage, UE, and in-memory computing Integrate ...
Hadoop、hive 、MySQL
win10下搭建Hadoop(jdk+mysql+hadoop+scala+hive+spark),包括jdk的安装、mysql安装和配置,hadoop安装和配置,scala安装和配置,hive安装和配置,spark安装和配置。
Over 60 recipes showing you how to design, configure, manage, monitor, and tune a Hadoop cluster Overview Hands-on recipes to configure a Hadoop cluster from bare metal hardware nodes Practical and ...
Data-intensive Systems: Principles and Fundamentals using Hadoop and Spark (Advanced Information and Knowledge Processing) By 作者: Tomasz – Wiktorski – Tomasz Wiktorski ISBN-10 书号: 3030046028 ...
Data Algorithms Recipes for Scaling Up with Hadoop and Spark 英文epub 本资源转载自网络,如有侵权,请联系上传者或csdn删除 本资源转载自网络,如有侵权,请联系上传者或csdn删除
This book jumps into the world of Hadoop ecosystem components and its tools in a simplified manner, and provides you with the skills to utilize them effectively for faster and effective development of...
windows运行spark或者hadoop程序报winutils.exe错误,将此文件放到hadoop的bin目录下或者直接将HADOOP_HOME配置为此目录即可解决
Title: Scaling Big Data with Hadoop and Solr, 2nd Edition Author: Hrishikesh Vijay Karambelkar Length: 156 pages Edition: 1 Language: English Publisher: Packt Publishing Publication Date: 2015-03-31 ...
使用kettle(etl工具)将mysql的数据抽取到hadoop的文件系统中。
hadoop_mysql-libs.rar
Hadoop 2.7.3 Windows64位 编译bin(包含winutils.exe, hadoop.dll),自己用的,把压缩包里的winutils.exe, hadoop.dll 放在你的bin 目录 在重启eclipse 就好了
我的推荐算法是利用Hadoop技术写的,我们可以利用Hadoop集群的高吞吐量,一次读取多次写入等特点进行大数据处理,我们也可以通过JDBC编程直接把推荐结果写入到Mysql数据库,等我们用到时再把它读出来。 而且系统首页...