Distributed TensorFlow on Raspberry Pi’s Hadoop 3 Cluster

Preparation

  1. Raspbian OS Upgrade
sudo apt-get install pdsh
export PDSH_RCMD_TYPE=ssh
pdsh -R ssh -w pi@192.168.0.[7,8,9,13,17] 'YOUR_SHELL_COMMAND'
alias runshell="pdsh -R ssh -w pi@192.168.0.[7,8,9,13,17]
runshell 'sudo apt-get install virtualenv'
sudo vim /etc/dphys-swapfile# Update
CONF_SWAPSIZE=1000
/etc/init.d/dphys-swapfile restart

Do the job!

1. Install Hadoop 3.1.1 in your Raspberry Pi cluster

pi@master:~ $ jps
18912 NodeManager
2947 NameNode
3091 DataNode
31577 Jps
3257 SecondaryNameNode
18799 ResourceManager
HDFS Overview
Data nodes conditions
YARN Cluster page
YARN nodes

2. Install TensorFlow

runshell 'virtualenv python3 ~/p3 && source ~/p3/bin/activate && pip3 install tensorflow'

3. Get a copy of TonY jar

4. Kick off your distributed TensorFlow job!

pi@master:~/tf $ ls
src tony-cli-0.1.0-all.jar tony.xml
pi@master:~/tf $ cat tony.xml
<configuration>
<property><name>tony.application.insecure-mode</name><value>true</value></property>
</configuration>
pi@master:~/tf $ ls src/
distributed.py
pi@master:~/tf $ CLASSPATH=$(${HADOOP_HDFS_HOME}/bin/hadoop classpath --glob):/home/pi/tf/:/home/pi/tf/* java com.linkedin.tony.cli.ClusterSubmitter --src_dir src --executes src/distributed.py --python_binary_path /home/pi/p3/bin/python

6. Check your job in resource manager web page

pi@slave-3:~ $ source p3/bin/activate
(p3) pi@slave-3:~ $ tensorboard --logdir /tmp/mnist/1
TensorBoard 1.9.0 at http://slave-3:6006 (Press CTRL+C to quit)

FAQ

  1. My pip is screwed up after the Raspbian Stretch upgrade:
pi@master:/usr/local/bin $ pip
Traceback (most recent call last):
File “/usr/bin/pip”, line 9, in <module>
from pip import main
ImportError: cannot import name main
sudo vim /usr/bin/pip 
from pip import main if __name__ == ‘__main__’: 
sys.exit(main())
from pip import __main__if __name__ == ‘__main__’:   sys.exit(__main__._main())

--

--

--

hacker, lifetime learner

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Ramudroid v6.5 — IoT framework integration

7 Habits That Make You a Bad Developer

Web Scraping With Python Using BeautifulSoup and MongoDB

Monitor Cassandra-Reaper with Prometheus and Grafana

乾貨 | 手把手教你用115行代碼做個數獨解析器!

How fast can a Raspberry Pi Zero 2w boot?

DevOps for Data Platform

How to Control Speed and Quality in Cloud Development

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
oliver hu

oliver hu

hacker, lifetime learner

More from Medium

Top Hadoop HDFS Commands | HDFS Shell Guide

Start Work on Apache Kafka || Understand concepts of apache kafka || Apache kafka configuration ||…

What if I wanted to submit remote PySpark jobs to AWS EMR without worrying about library dependen…

Streaming Data from Kafka to Elasticsearch & Visualization via Kibana — Tutorial