storm本地python开发环境搭建，

文章由Byrx.net分享于2020-05-16 09:05:22评论（312）

storm本地python开发环境搭建，


搭建storm本地python开发环境

需要使用python进行storm的开发，这里记录本地开发环境搭建,过程中使用的资源gitee中下载
  • centos7 普通用户
  • python 3.5.1
  • jdk 1.8.0
  • storm 1.1.0
  • lein+streamparse

本地用户faron, 根目录/home/faron

这次使用普通用户安装

python 3.5.1安装

这里我之前使用pyenv已经部署好 /home/faron目录下的pthon环境为3.5.1, 这里可自行搜索

jdk 1.8.0安装

  • 安装包下载

  • 在～/.bashrc文间尾部追加java环境变量

    # java
export JAVA_HOME=//home/faron/lein/java_storm/jdk1.8.0_161
export JRE_HOME=$JAVA_HOME/jre
# export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

  • 激活环境变量

    source ~/.bashrc

  • 校验java环境

    java versoin

    终端显示

    java version "1.8.0_161"
    
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
    
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

strom 1.1.0安装

  • 安装包下载

  • 在～/.bashrc文间尾部追加storm环境变量

    # storm
export STORM_HOME=/home/faron/lein/java_storm/apache-storm-1.1.0
export PATH=$STORM_HOME/bin:$PATH

  • 激活环境变量

    source ~/.bashrc

  • 校验storm环境

    storm version

    终端显示

    Running: //home/faron/lein/java_storm/jdk1.8.0_161/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/home/faron/lein/java_storm/apache-storm-1.1.0 -Dstorm.log.dir=/home/faron/lein/java_storm/apache-storm-1.1.0/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /home/faron/lein/java_storm/apache-storm-1.1.0/lib/storm-core-1.1.0.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/kryo-3.0.3.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/reflectasm-1.10.1.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/asm-5.0.3.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/minlog-1.3.0.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/objenesis-2.1.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/clojure-1.7.0.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/ring-cors-0.1.5.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/disruptor-3.3.2.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/log4j-api-2.8.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/log4j-core-2.8.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/log4j-slf4j-impl-2.8.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/slf4j-api-1.7.21.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/log4j-over-slf4j-1.6.6.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/servlet-api-2.5.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/storm-rename-hack-1.1.0.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/conf org.apache.storm.utils. VersionInfo
    
Storm 1.1.0
    
URL https://git-wip-us.apache.org/repos/asf/storm.git -r e40d213de7067f7d3aa4d4992b81890d8ed6ff31
    
Branch (no branch)
    
Compiled by ptgoetz on 2017-03-21T17:04Z
    
From source with checksum 4ddc442e8b804654454ab26fbec6348

lein+streamparse安装

  • lein安装

    安装过程中可能会出现lein脚本无法下载,leiningen-2.9.3-standalone.jar无法下载的情况
    

    makdir ~/bin
cd ~/bin
cp ~/lein/lein ./
chmod +x ~/bin/lein
# 运行lein安装脚本
lein

    终端显示(下载速度可能会有点慢)

    WARNING: You have $CLASSPATH set, probably by accident.
    
It is strongly recommended to unset this before proceeding.
    
Downloading Leiningen to /home/faron/.lein/self-installs/leiningen-2.9.3-standalone.jar now...
    
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
    
Dload  Upload   Total   Spent    Left  Speed
    
100   637  100   637    0     0    393      0  0:00:01  0:00:01 --:--:--   393
    
27 13.9M   27 3942k    0     0  34711      0  0:07:02  0:01:56  0:05:06 26468

    如果leiningen-2.9.3-standalone.jar下载失败

    cp ~/lein/java_storm/leiningen-2.9.3-standalone.zip ~/.lein/self-installs/

    lein环境验证

    lein version
# >>>终端显示
# WARNING: You have $CLASSPATH set, probably by accident.
# It is strongly recommended to unset this before proceeding.
# Leiningen 2.9.3 on Java 1.8.0_161 Java HotSpot(TM) 64-Bit Server VM

  • streamparse安装

    • 安装

      pip install -i https://pypi.tuna.tsinghua.edu.cn/simple --upgrade pip
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple streamparse

    • 启动wordcount

      mkdir ~/storm_workplace
cd ~/storm_workplace
sparse quickstart wordcount
cd wordcount
sparse run

    • 异常中心包无法下载

      Could not transfer artifact org.apache.storm:storm-core:jar:1.1.0 from/to central (https://repo1.maven.org/maven2/): Connect to repo1.maven.org:443 [repo1.maven.org/151.101.40.209] failed: 拒绝连接 (Connection refused)
      
Could not find artifact org.apache.storm:storm-core:jar:1.1.0 in clojars (https://repo.clojars.org/)
      
Could not transfer artifact org.clojure:clojure:jar:1.7.0 from/to central (https://repo1.maven.org/maven2/): Connect to repo1.maven.org:443 [repo1.maven.org/151.101.40.209] failed: 拒绝连接 (Connection refused)
      
Could not find artifact org.clojure:clojure:jar:1.7.0 in clojars (https://repo.clojars.org/)
      
This could be due to a typo in :dependencies, file system permissions, or network issues.
      
If you are behind a proxy, try setting the 'http_proxy' environment variable.
      

      # 仅限storm1.1.0，因为我的repository里面是storm-core:1.1.0本地环境
tar -zxvf ~/lein/java_storm/repository.tar.gz -C  ~/.m2/
# 再次运行
sparse run

相关内容

    暂无相关文章

评论关闭

最新python教程

python~HOT