Apple silicon M1 for the many-body physicist
Apple silicon has started to revolutionize computing. However, software incompatibilities are an issue, especially for programmes which are not used by the normal consumer. Here I want to quickly introduce the steps I undertook to create an efficient setup of a scientific software stack based on numpy and scipy. In particular, I will show how to get quspin, for exact diagonalisation, and TeNPy, for matrix product state simulations running. A lot of this is collected from different resources, please let me know if you feel like I stole your splendour. I will try to continuously update this if the process has simplified.
First, we need to install anaconda, which since May 2022 has a native Apple silicon branch:
Add conda-forge as a channel and make it the default channel:
conda config --add channels conda-forge
conda config --set channel_priority strict
Then, create a new environment
conda create -n MY-ENV python=3.9 pybind11 cython=0.29.30
Install numpy with veclib/accelerate. This will give you a massive speedup.
pip3 install --no-binary :all: numpy
Test whether this worked, this should show veclib/accelerate in your numpy config:
You can further benchmark by running the tests discussed in this stackoverflow. I am getting 1.12s for the SVD benchmark and 23s for dario.py. I have an M1 Pro with 10 cores.
Now you need to make sure that whatever package you install does not install numpy again:
conda config --set pip_interop_enabled true
For additional security, you can pin your numpy (replace USERNAME and MY-ENV with your username and environment name, respectively)
echo "numpy=1.23.0=pypi*" >> ~/opt/anaconda3/envs/MY-ENV/conda-meta/pinned
Now you can install scipy:
conda install scipy
Again, check that this won’t install numpy from a different channel.
If you only want to install numpy, but not scipy, you can use the following much easier command to do all of the above in one go. This does not work for scipy because scipy does not support accelerate.
conda create -n np_accelerate python=3.9 numpy “blas=*=*accelerate*
Pinning your numpy version might lead to dependency problems when installing other packages. For example, for me numba created problems. One way to solve this is to remove the line in ~/opt/anaconda3/envs/MY-ENV/conda-meta/pinned you created with the above command, installing the package you just failed to install and then reinstalling numpy via pip with the below command. Here I fixed numpy to 1.22.4 because numba=0.55.2 required numpy<1.23.0.
pip3 install --no-binary :all: numpy==1.22.4 --force-reinstall
If you are not a many-body physicist, you are good to go. As far as I understood from people, you could now install tensorflow and other packages. I have not tested this myself.
Let’s do many-body physics: QuSpin for exact diagonalisation
I use QuSpin almost every day for diagonalising some Hamiltonians. I can highly recommend this amazing package written and maintained by Marin Bukov and Phillip Weinberg. Before you install the below dependencies, make sure you pinned numpy (see above)
conda install joblib six dill gmpy2 numexpr boost llvm-openmp numba=0.55.2 openmpi-mpicc
Git clone https://github.com/weinbe58/QuSpin.git
And compile it. Before executing this, read the comment below.
python setup.py install --omp --record install_file.txt
Executing the above line, I got the following error:
No such file or directory: ‘arm64-apple-darwin20.0.0-clang++’
To solve this, I first looked for the clang command and renamed it.
mv clang arm64-apple-darwin20.0.0-clang
mv clang++ arm64-apple-darwin20.0.0-clang++
Probably good practice to undo this bodge after compilation.
Alternative — compile with g++
In case this doesn’t work for you, you can also compile with g++ instead of clang.
For this, you need to first install home-brew, if you haven’t done so yet.
/bin/bash -c “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
And install compilers
brew install gcc
brew install g++
For compilation, we need to set some flags
And we can finally compile:
python setup.py install — omp — default-compiler-flags — record install_file.txt
Get the matrix product states to work — TeNPy
I do most my MPS simulations with Johannes Hausschild’s amazing Tensor Networks Python package. It turns out that it works relatively straightforward.
Download the package and add its location to bashrc
git clone https://github.com/tenpy/tenpy.git $HOME/TeNPy
To make it run faster, compile the package:
I ran the whole test suite provided by the package and everything seems to work. However, I haven’t yet benchmarked the speed at which things are executed. TeNPy uses MKL as its BLAS. In this post, people have compared MKL to Accelerate/veclib on M1 and found them to be pretty equivalent.