我试图在具有以下配置/版本的三个节点的dev Hadoop集群上安装和配置apache airflow:
Operating System: Red Hat Enterprise Linux Server 7.7 python 3.7.3 anaconda 2 spark 2.45 a)sudo yum install gcc gcc-c++ -y b)sudo yum install libffi-devel mariadb-devel cyrus-sasl-devel -y c)pip install 'apache-airflow[all]' d)airflow initdb -- airflow.cfgfile was created with SQLlite
然后我按照下面的命令集使用mysql配置它
a) rpm -Uvh https://repo.mysql.com/mysql80-community-release-el7-3.noarch.rpm b) sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/mysql-community.repo c) yum --enablerepo=mysql80-community install mysql-community-server d) systemctl start mysqld.service
在mysql之下完成以下操作
a) CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci; b) create user 'airflow'@'localhost' identified by 'Airflow123'; c) grant all privileges on * . * to 'airflow'@'localhost';
这是我airflow.cfg文件中的一些详细信息
airflow.cfg
broker_url = sqla+mysql://airflow:airflow@localhost:3306/airflow result_backend = db+mysql://airflow:airflow@localhost:3306/airflow sql_alchemy_conn = mysql://airflow:Airflow123@localhost:3306/airflow executor = CeleryExecutor
我在执行airflow initdb命令时遇到错误
airflow initdb
ImportError: /home/xyz/anaconda2/envs/python3.7.2/lib/python3.7/site-packages/_mysql.cpython-37m-x86_64-linux-gnu.so: symbol mysql_real_escape_string_quote, version libmysqlclient_18 not defined in file libmysqlclient.so.18 with link time reference
已将.bashrc文件设置为:
export AIRFLOW_HOME=~/airflow
这是我创建的目录:
[xyz@innolx5984 airflow]$ pwd /home/xyz/airflow
当我查找该文件“ libmysqlclient”时,我发现了很多实例。
libmysqlclient
[xyz@innolx5984 airflow]$ find /home/xyz/ -name "*libmysqlclient*" /home/xyz/anaconda2/pkgs/mysql-connector-c-6.1.11-h597af5e_1/lib/libmysqlclient.so /home/xyz/anaconda2/pkgs/mysql-connector-c-6.1.11-h597af5e_1/lib/libmysqlclient.a /home/xyz/anaconda2/pkgs/mysql-connector-c-6.1.11-h597af5e_1/lib/libmysqlclient.so.18 /home/xyz/anaconda2/pkgs/mysql-connector-c-6.1.11-h597af5e_1/lib/libmysqlclient.so.18.4.0 /home/xyz/anaconda2/lib/libmysqlclient.a /home/xyz/anaconda2/lib/libmysqlclient.so /home/xyz/anaconda2/lib/libmysqlclient.so.18 /home/xyz/anaconda2/lib/libmysqlclient.so.18.4.0
仅添加更多细节以防万一。
[xyz@innolx5984 airflow]$ mysql_config Usage: /home/xyz/an aconda2/bin/mysql_config [OPTIONS] Options: --cflags [-I/home/xyz/anaconda2/include ] --cxxflags [-I/home/xyz/anaconda2/include ] --include [-I/home/xyz/anaconda2/include] --libs [-L/home/xyz/anaconda2/lib -lmysqlclient ] --libs_r [-L/home/xyz/anaconda2/lib -lmysqlclient ] --plugindir [/home/xyz`/anaconda2/lib/plugin] --socket [/tmp/mysql.sock] --port [0] --version [6.1.11] --variable=VAR VAR is one of: pkgincludedir [/home/xyz/anaconda2/include] pkglibdir [/home/xyz/anaconda2/lib] plugindir [/home/xyz/anaconda2/lib/plugin] Looking for some help and suggestion to resolve this
问题。我不太确定是否朝正确的方向前进。
请按照以下步骤使用Anaconda3将Apache Airflow与MySQL一起安装
1) 安装先决条件
yum install gcc gcc-c++ -y yum install libffi-devel mariadb-devel cyrus-sasl-devel -y dnf install redhat-rpm-config
2) 安装Anaconda3(Python 3.7.6随附)
yum install libXcomposite libXcursor libXi libXtst libXrandr alsa-lib mesa-libEGL libXdamage mesa-libGL libXScrnSaver wget https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh chmod +x Anaconda3-2020.02-Linux-x86_64.sh ./Anaconda3-2020.02-Linux-x86_64.sh
确保conda initialize在安装过程中出现提示时执行此操作。这将确保在后续步骤中使用正确版本的python和pip。
conda initialize
3) 安装Apache Airflow
pip install apache-airflow[mysql,celery]
您可以根据需要添加其他子包。我只包含了Airflow使用MySQL数据库作为后端所需的内容。
4) 初始化气流
export AIRFLOW_HOME=~/airflow airflow initdb
从这里开始,我模仿了您配置MySQL Server所遵循的步骤
5) 安装MySQL服务器
rpm -Uvh https://repo.mysql.com/mysql80-community-release-el7-3.noarch.rpm sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/mysql-community.repo yum --enablerepo=mysql80-community install mysql-server systemctl start mysqld.service
6) 登录到MySQL并为Airflow配置数据库
mysql> CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci; mysql> CREATE user 'airflow'@'localhost' identified by 'Airflow123'; mysql> GRANT ALL privileges on *.* to 'airflow'@'localhost';
7) 更新Airflow配置文件(〜/ airflow / airflow.cfg)
sql_alchemy_conn = mysql://airflow:Airflow123@localhost:3306/airflow executor = CeleryExecutor
8) 初始化气流