Get Pykafka to work with rdkafka on Linux
My former colleague’s from Parse.ly wrote the fantastic pykafka library with an optional c-backed using rdkafka. I’ve had trouble getting it to work, and here are a few things I’ve learned:
- The version of rdkafka installable with apt-get was out of data, and pykafka couldn’t find the headers it need. I instead used the simple build instructions in the rdkafka README to build it from head.
- I was getting the error
ImportError: librdkafka.so.1: cannot open shared object file: No such file or directorywhen trying to use rdkafka from Pykafka. It could be set in the short term by using
LD_LIBRARY_PATH=/usr/local/lib. However, I fixed it permanently by running
sudo ldconfigafter building rdkafka.
- Pykafka has to be installed after building rdkafka. At the moment, Pykafka tries to build a C-extension to connect to rdkafka, and if that fails, it will install without offering the rdkafka backend. Check the output of
pip install pykafkato see if the rdkafka extension built.