Getting started with Clojure and MXNet on AWS- 4 mins
Setting up a Deep Learning development box is often a tedious task that requires one to install the proper drivers and toolkits for the given Deep Learning Framework.
This post will cover how to setup such a box for MXNet and Clojure so that any Clojurist can start playing with MXNet on GPUs.
MXNet is the Deep Learning Framework developped jointly by Amazon and Microsoft. It is an incubating Apache Project.
MXNet and Clojure
Long story short: by leveraging Scala, Clojure can talk to the MXNet bindings. And this is amazing! Here is the Clojure Package in the MXNet repository.
AWS Deep Learning AMI
AWS provides AMIs tailored to Deep Learning frameworks. They are pre-configured environments to quickly build deep learning applications. The AMI we will cover in this post is the AWS Deep Learning AMI (base) image - Find it here.
Below are the steps one needs to follow to start playing with MXNet and Clojure on GPUs
- Launch an EC2 instance
- Connect to the EC2 instance
- Update Java to Java 8
- Update Cuda version
- Install leiningen
- Play with Clojure MXNet Examples
Launch an EC2 instance
Launch an EC2 instance with the AWS Deep Learning AMI base image. In this post I assume you select the Deep Learning Base (Amazon Linux) AMI.
Then pick a GPU instance type.
p2.xlarge shoud be enough to get you started and will cost you around
$0.90 per hour. Do not forget to terminate it after you are done with your Neural Network training.
Connect to the EC2 instance
pemfile or username/password pair that was created to connect to your EC2 instance
ssh -i "pemfilename.pem" email@example.com
Update Java to Java 8
From the EC2 instance, run the following command and select
java 8 that comes with the AMI
sudo update-alternatives --config java
It may not be necessary to run this if the default java version is already set to
Update Cuda version
From the EC2 instance, run the following commands to use the proper Cuda version required to run MXNet. For the current
MXNet 1.4.0-1.5.0, one needs
sudo rm /usr/local/cuda sudo ln -s /usr/local/cuda-9.2 /usr/local/cuda
Run the following commands to install the latest
wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein chmod a+x lein mv lein /usr/bin lein
Play with Clojure MXNet Examples
First, one needs to clone the MXNet codebase that comes with Clojure examples
git clone https://github.com/apache/incubator-mxnet.git
project.clj to uncomment the GPU package and comment out the CI package
(defproject ... ;; Uncomment this line [org.apache.mxnet/mxnet-full_2.11-linux-x86_64-gpu "1.5.0-SNAPSHOT"] ;; Comment this line used for CI ; [org.apache.mxnet/mxnet-full_2.11 "INTERNAL"] ... )
You should now be able to test that the clojure package can run on your EC2 instance
1.5.0-SNAPSHOT jar locally
Navigate the filesystem to get to the Clojure examples
We will run the code from the
One can monitor the GPU processes with the following command
Now it is time to run some Clojure code and get these Neural Networks to learn something!
Run the code from the REPL
lein repl mnist-mlp=> (run-all [(context/gpu)])
Or run the code from a leiningen command
lein run :gpu
Congratulations! If you have followed along, you are now able to harness the power of MXNet with Clojure on GPUs! Again, do not forget to terminate your instance once you are done.
Next time, we will cover what MXNet let us do and we will learn how to train Deep Learning models.