How to Build Nvidia Caffe Deep Learning on Arch Linux

When compiling Caffe, there are some bugs and problems you might hit. This tutorial is going to lead you to conquer those obstacles.

The first thing, of course, is to install pre-request packages listed on the official website. They are

  • CUDA >= 7
  • ATLAS-BLAS
  • Boost >= 1.55 (Suggest 1.59!!! 1.60 has some minor bugs)
  • protobuf, glog, gflags, hdf5
  • OpenCV >= 2.4
  • lmdb, leveldb, snappy
  • Python 2.7

Then, follow the instructions on the website. I think it’s detailed enough for compiling Caffe.

The only difference for Arch Linux is that CUDA is installed in /opt/cuda/ by default instead of /usr/local/cuda. Set CUDA_DIR := /opt/cuda and remember to set USE_CUDNN := 1 in the Makefile.config which should be copied from Makefile.config.example.
Example:

# cuDNN acceleration switch (comment to build without cuDNN).
USE_CUDNN := 1
# CNMeM memory pool switch (github.com/NVIDIA/cnmem required)
# USE_CNMEM := 1

# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1

# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /opt/cuda

Also, I strongly suggest installing Boost 1.59, just don’t use Boost 1.60 which has many known bugs with CUDA!! The easy way to downgrade Boost are shown in Problem 5.

 

Nvidia Digits:

The code of newest version of DIGITS can be clone here: https://github.com/NVIDIA/DIGITS.git. You can find more documents on Nvidia’s developer page.

After downloading DIGITS, you need to install all the required python packages listed in requirements.txt by

DIGITS$for req in $(cat requirements.txt); do pip2 install $req; done

The DIGITS downloaded from Nvidia’s developer page is different. It is a complete package version packed for Ubuntu. All the binaries and libraries Caffe required is packed in it while the GitHub version only contains the source code of web interface.

By the way, DIGITS only support Nvidia’s aCaffe, which is here: https://github.com/NVIDIA/caffe.git. It’s a fork from BLVC/caffe with some feature enhancements for DIGITS.

As for the tutorial of DIGITS, please refer to here.

 

Problem 1: Missing packages

Well… go install them!

 

Problem 2: ImportError: No module named XXX

Python will list the package name you need to install by pip. Don’t forget to install the dependencies listed in requirements.txt!!!

Typically, you can install the Python packages by

DIGITS$for req in $(cat requirements.txt); do pip2 install $req; done

Remember to use pip2 instead pip!!! You can install pip2 by pacman. pip2 is for P{ython 2 while pip might be Python 3, which is not supported by Caffe yet.

 

Problem 3: NumPy

g++ -shared -o python/caffe/_caffe.so python/caffe/_caffe.cpp \
        .build_release/lib/libcaffe.a -fPIC -DNDEBUG -O2 -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I.build_release/src -I./src -I./include -I/usr/local/cuda/include -Wall -Wno-sign-compare -L/usr/lib -L/usr/local/lib -L/usr/lib -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib -lcudart -lcublas -lcurand -lpthread -lglog -lgflags -lprotobuf -lleveldb -lsnappy -llmdb -lboost_system -lhdf5_hl -lhdf5 -lopencv_core -lopencv_highgui -lopencv_imgproc -lboost_thread -lopenblas -lboost_python -lpython2.7
In file included from python/caffe/_caffe.cpp:12:0:
python/caffe/_caffe.hpp:8:31: fatal error: numpy/arrayobject.h: No such file or directory
 #include <numpy/arrayobject.h>
                               ^
compilation terminated.

This is a very tricky bug. You can solve it by simply installing numpy. If you did, then the problem might due to the include PATH doesn’t contain numpy. You can check if you can find arrayobject.h in /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/.

If that still don’t help, please refer to the issue and see the other solutions.

 

Problem 4:  float128 missing

It’s because the compilation of std11 is not enabled. Please add -std=c++11 to your Makefile. The issue has been discussed. The location might be at line 372 in Nvidia’s Caffe. Just add-std=c++11 to the variable of CXXFLAGS in Makefile.

371 # Automatic dependency generation (nvcc is handled separately)
372 CXXFLAGS += -MMD -MP -std=c++11

 

Problem 5: CUDA backend does not build with Boost 1.60

This is a very weird problem one might face. I suggest downgrading Boost to 1.59. When using Boost 1.60, you might have this error messages while compilation:

/usr/lib/gcc/x86_64-unknown-linux-gnu/5.3.0/include/mwaitxintrin.h(36): error: identifier "__builtin_ia32_monitorx" is undefined

/usr/lib/gcc/x86_64-unknown-linux-gnu/5.3.0/include/mwaitxintrin.h(42): error: identifier "__builtin_ia32_mwaitx" is undefined

You can fix this problem by adding definition __STRICT_ANSI__ when using boost 1.60. This solution is from here. Or, you could fix it by downgrading Boost to 1.59 and recompile (make clean && make) Caffe. Please refer to below for how to downgrade Boost on Arch Linux.

If these ways don’t help, the ultimate solution is to use root privilege comment out “__builtin_ia32_monitorx (__P, __E, __H);” and “__builtin_ia32_mwaitx (__E, __H, __C);” in mwaitxintrin.h.

Also, when you use Nvidia’s Caffe, You might face the following error message:
No to_python (by-value) converter found for C++ type: boost::shared_ptr<caffe::Blob<float> >

Traceback (most recent call last):
  File "/opt/caffe/python/classify.py", line 138, in <module>
    main(sys.argv)
  File "/opt/caffe/python/classify.py", line 110, in main
    channel_swap=channel_swap)
  File "/opt/caffe/python/caffe/classifier.py", line 29, in __init__
    in_ = self.inputs[0]
  File "/opt/caffe/python/caffe/pycaffe.py", line 54, in _Net_inputs
    return [list(self.blobs.keys())[i] for i in self._inputs]
  File "/opt/caffe/python/caffe/pycaffe.py", line 28, in _Net_blobs
    return OrderedDict(zip(self._blob_names, self._blobs))
TypeError: No to_python (by-value) converter found for C++ type: boost::shared_ptr<caffe::Blob<float> >

 

The solution is……. downgrade Boost to 1.59. Please refer to the section below for downgrading Boost.

 

Problem 6: PicklingError in logs

Caught PicklingError while saving job: Can't pickle <class 'caffe_pb2.NetParameter'>: it's not found as caffe_pb2.NetParameter

This error message is from Nvidia’s Digits. It’s just because you didn’t make install the Caffe. If you don’t want to do so, you can simply add caffe protobuf to python path before executing DIGITS.

~$export PYTHONPATH=${CAFFE_HOME}/python/caffe/proto:$PYTHONPATH

You can also use this script to setup the whole library path and execute DIGITS.

#!/bin/bash

CAFFE_HOME=$(pwd)/../caffe-nv
CUDA_LIBS=/opt/cuda
BOOST_LIBS=''
CNMEM_LIBS=''
CAFFE_PYTHON=${CAFFE_HOME}/python/caffe/proto

export PYTHONPATH=$CAFFE_PYTHON:$PYTHONPATH
CAFFE_HOME=$CAFFE_HOME LD_LIBRARY_PATH=$CUDA_LIBS:$BOOST_LIBS:$CNMEM_LIBS \
python digits-devserver $@

Original issue

 

Downgrade to Boost 1.59

I strongly suggest using Boost 1.59 for better compatibility!!! Downgrading Arch package to Boost 1.59 could be done by downloading this file, tar it, and go to directory packages-caf64e51065d8b8b4eaba12d415432ac8bce783b/repos/extra-x86_64.

Then follow the standard procedure of Arch package build system.

~$wget https://projects.archlinux.org/svntogit/packages.git/snapshot/packages-caf64e51065d8b8b4eaba12d415432ac8bce783b.tar.gz

#Rename the file

~$mv packages-caf64e51065d8b8b4eaba12d415432ac8bce783b.tar.gz boost-1.59.tar

~$tar -xf ./boost-1.59.tar.gz

~$cd boost-1.59/repos/extra-x86_64

~/boost-1.59/repos/extra-x86_64$makepkg

~/boost-1.59/repos/extra-x86_64$sudo pacman -U boost-1.59.0-3-x86_64.pkg.tar.xz boost-libs-1.59.0-3-x86_64.pkg.tar.xz

Done!! Now, you need to recompile (make clean && make) Caffe.

 

Sample System Configuration

In the end, here is my system configuration which successfully compile Caffe.

-- ******************* Caffe Configuration Summary *******************
-- General:
--   Version           :   <TODO> (Caffe doesn't declare its version in headers)
--   Git               :   rc2-738-g03a84bf
--   System            :   Linux
--   C++ compiler      :   /usr/bin/c++
--   Release CXX flags :   -O3 -DNDEBUG -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
--   Debug CXX flags   :   -g -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
--   Build type        :   Release
--
--   BUILD_SHARED_LIBS :   ON
--   BUILD_python      :   ON
--   BUILD_matlab      :   OFF
--   BUILD_docs        :   ON
--   CPU_ONLY          :   OFF
--   USE_OPENCV        :   ON
--   USE_LEVELDB       :   ON
--   USE_LMDB          :   ON
--   ALLOW_LMDB_NOLOCK :   OFF
--
-- Dependencies:
--   BLAS              :   Yes (Atlas)
--   Boost             :   Yes (ver. 1.59)
--   glog              :   Yes
--   gflags            :   Yes
--   protobuf          :   Yes (ver. 2.6.1)
--   lmdb              :   Yes (ver. 0.9.17)
--   LevelDB           :   Yes (ver. 1.18)
--   Snappy            :   Yes (ver. 1.1.3)
--   OpenCV            :   Yes (ver. 2.4.12.2)
--   CUDA              :   Yes (ver. 7.5)
--
-- NVIDIA CUDA:
--   Target GPU(s)     :   Auto
--   GPU arch(s)       :   sm_35
--   cuDNN             :   Yes (ver. 3.0.07)
--
-- Python:
--   Interpreter       :   /usr/bin/python2.7 (ver. 2.7.11)
--   Libraries         :   /usr/lib64/libpython2.7.so (ver 2.7.11)
--   NumPy             :   /usr/lib/python2.7/site-packages/numpy/core/include (ver 1.10.2)

9 thoughts on “How to Build Nvidia Caffe Deep Learning on Arch Linux

  1. Hi, may I know if there is a step by step approach to installing caffe? I installed via yaourt and now I’m stuck with this error:
    “CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
    Please set them or make sure they are set and tested correctly in the CMake files:
    HDF5_hdf5_cpp_LIBRARY”

    I can’t seem to fix it anyhow. Please help.

    1. Sorry for missing this comment. I hope you’ve solved this problem. I’m not sure what cause this as well, but you can try to unset the variable by executing ‘unset HDF5_hdf5_cpp_LIBRARY’ in your console. This would unset the environment variable in your shell. You can also try to set this variable to correct value in “CMakeCache.txt.” By the way, Nvidia released the latest DIGITS and cuDNN yesterday. Give it a try. It’s a great release.

  2. hi,i get a wrong,when i run ‘make’.
    LD -o .build_release/lib/libcaffe.so.1.0.0-rc3
    /usr/bin/ld:can’t find -latlas
    collect2:error:ld return 1

    i use the manjaro

    1. Hi. This is easy. It’s because your environment doesn’t have atlas.
      “BLAS: install ATLAS by sudo apt-get install libatlas-base-dev or install OpenBLAS or MKL for better CPU performance.”
      Since your environment is not Ubuntu, you need to go to github to download, compile, and install it.

  3. Could you describe how you resolved the Atlas build issue (if you saw it)? “Aborting due to throttling.” Also, the cudnn install appears to be non-trivial (at least now with cudnn 5.1 up), a few more pointers in that direction would be much appreciated!

    1. Yap, I could help with that. This post is quite old. Things changed a lot. I would update this post at around the end of February.
      As for your problem, “Aborting due to throttling”. It’s because Atlas used software to sample your system performance and change the behavior of scheduling. However, modern systems have dynamic frequency scaling which makes the sampling imprecise. Personally, I feel this is not a “must be precise” feature in my setup so I modify the source code to bypass the checking like this post: http://stackoverflow.com/questions/14592401/atlas-install-really-need-to-get-past-cpu-throttle-check.
      The better way to solve this is to disable frequency scaling: http://unix.stackexchange.com/questions/121410/setting-cpu-governor-to-on-demand-or-conservative

    2. I’m so sorry that I don’t have time to update this. I hope you make it work. BTW, you can jump to the new caffe2 since it’s the successor of the original caffe with better performance.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s