Getting started with Clojure and MXNet on AWS
- 4 minsSetting up a Deep Learning development box is often a tedious task that requires one to install the proper drivers and toolkits for the given Deep Learning Framework.
This post will cover how to setup such a box for MXNet and Clojure so that any Clojurist can start playing with MXNet on GPUs.
MXNet
MXNet is the Deep Learning Framework developped jointly by Amazon and Microsoft. It is an incubating Apache Project.
MXNet and Clojure
Long story short: by leveraging Scala, Clojure can talk to the MXNet bindings. And this is amazing! Here is the Clojure Package in the MXNet repository.
AWS Deep Learning AMI
AWS provides AMIs tailored to Deep Learning frameworks. They are pre-configured environments to quickly build deep learning applications. The AMI we will cover in this post is the AWS Deep Learning AMI (base) image - Find it here.
Installation steps
Below are the steps one needs to follow to start playing with MXNet and Clojure on GPUs
- Launch an EC2 instance
- Connect to the EC2 instance
- Update Java to Java 8
- Update Cuda version
- Install leiningen
- Play with Clojure MXNet Examples
Launch an EC2 instance
Launch an EC2 instance with the AWS Deep Learning AMI base image. In this post I assume you select the Deep Learning Base (Amazon Linux) AMI.
Then pick a GPU instance type. p2.xlarge
shoud be enough to get you started and will cost you around $0.90
per hour. Do not forget to terminate it after you are done with your Neural Network training.
Connect to the EC2 instance
Use the pemfile
or username/password pair that was created to connect to your EC2 instance
ssh -i "pemfilename.pem" ec2-user@ec2-1-42-42-0.compute-1.amazonaws.com
Update Java to Java 8
From the EC2 instance, run the following command and select java 8
that comes with the AMI
sudo update-alternatives --config java
It may not be necessary to run this if the default java version is already set to java 8
.
Update Cuda version
From the EC2 instance, run the following commands to use the proper Cuda version required to run MXNet. For the current MXNet 1.4.0-1.5.0
, one needs cuda-9.2
sudo rm /usr/local/cuda
sudo ln -s /usr/local/cuda-9.2 /usr/local/cuda
Install leiningen
Run the following commands to install the latest leiningen
wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein
chmod a+x lein
mv lein /usr/bin
lein
Play with Clojure MXNet Examples
First, one needs to clone the MXNet codebase that comes with Clojure examples
git clone https://github.com/apache/incubator-mxnet.git
Open project.clj
to uncomment the GPU package and comment out the CI package
(defproject
...
;; Uncomment this line
[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-gpu "1.5.0-SNAPSHOT"]
;; Comment this line used for CI
; [org.apache.mxnet/mxnet-full_2.11 "INTERNAL"]
...
)
You should now be able to test that the clojure package can run on your EC2 instance
lein test
Install the 1.5.0-SNAPSHOT
jar locally
lein install
Navigate the filesystem to get to the Clojure examples
cd incubator-mxnet/contrib/clojure-package/examples
We will run the code from the module
example
cd module
One can monitor the GPU processes with the following command
watch nvidia-smi
Now it is time to run some Clojure code and get these Neural Networks to learn something!
Run the code from the REPL
lein repl
mnist-mlp=> (run-all [(context/gpu)])
Or run the code from a leiningen command
lein run :gpu
Conclusion
Congratulations! If you have followed along, you are now able to harness the power of MXNet with Clojure on GPUs! Again, do not forget to terminate your instance once you are done.
Next time, we will cover what MXNet let us do and we will learn how to train Deep Learning models.