Use standard Python tool to build snowboy

[hongquan]

This issue is to track what I'm doing to improve snowboy's setup.py.

Motivation:

• We want to pre-build snowboy to wheel package, to simplify installation of susi_linux, and to reduce image size (won't have to install a lot of stuff for building snowboy from source).
• snowboy's original setup.py doesn't use native Python tool (distutils, setuptools) to build the C++ source. It's setup.py instead runs make to do the job. The consequence is that the generated wheel package has wrong filename and cannot be distributed.

[hongquan]

I already fork snowboy to https://github.com/fossasia/snowboy and working there.

With the setup.py like this

import os
import sys
from setuptools import setup, find_packages, Extension
from distutils.command.build import build


py_dir = 'Python' if sys.version_info[0] < 3 else 'Python3'

class SnowboyBuild(build):
    # Reorder build commands such that swig generated files are also copied.
    sub_commands = [('build_ext', build.has_ext_modules),
                    ('build_py', build.has_pure_modules),
                    ('build_clib', build.has_c_libraries),
                    ('build_scripts', build.has_scripts)]

ext = Extension(
    '_snowboydetect',
    ['swig/Python3/snowboy-detect-swig.i'],
    swig_opts=['-c++'],
    include_dirs=['.'],
)

setup(
    name='snowboy',
    version='1.3.0',
    description='Snowboy is a customizable hotword detection engine',
    maintainer='KITT.AI',
    maintainer_email='[email protected]',
    license='Apache-2.0',
    url='https://snowboy.kitt.ai',
    ext_modules=[ext],
    packages=find_packages(os.path.join('examples', py_dir)),
    package_dir={'snowboy': os.path.join('examples', py_dir)},
    py_modules=['snowboy.snowboydecoder', 'snowboy.snowboydetect'],
    # include_package_data=True,
    package_data={'snowboy': ['resources/*']},
    zip_safe=False,
    long_description="",
    classifiers=[],
    install_requires=[
        'PyAudio',
    ],
    cmdclass={
        'build': SnowboyBuild
    }
)

I can make setuptools build the C++ source code, without calling make. But the problem now is, setuptools doesn't including data files under resources folder in the result package.

[hongquan]

Found the cause:

The project layout is like this (listing only Python files and resources folder):

+-- examples/
    +-- Python3/
         +-- snowboydecoder.py
         +-- snowboydetect.py
         +-- resources/ (symlink)
+-- swig/
+-- resources/
+-- setup.py

And with the setup() written like this:

setup(
    packages=find_packages(os.path.join('examples', py_dir)),
    package_dir={'snowboy': os.path.join('examples', py_dir)},
    py_modules=['snowboy.snowboydecoder', 'snowboy.snowboydetect'],
    package_data={'snowboy': ['resources/*']},
)

the find_packages() will not able to find the Python package under examples/Python3, and this makes the package_data parameter fail.

My discovery is that, the path passed to find_packages has to be a folder which contains a snowboy package, like this, to make it work:

path/
    +-- snowboy
        +-- __init__.py
        +-- snowboydecoder.py

[hongquan]

Working in this branch: https://github.com/fossasia/snowboy/tree/feature/standard-Python-build

Can produce *.so file and include resources files now, but the *.so file is not not linked enough libraries yet.

[hongquan]

This is the output of make:

swig -I../../ -c++ -python -o snowboy-detect-swig.cc snowboy-detect-swig.i
g++ -I/usr/include/python3.6m -I/usr/include/python3.6m  -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.6-EKG1lX/python3.6-3.6.5=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector-strong -Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I../../ -O3 -fPIC -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++0x -c snowboy-detect-swig.cc
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
g++ -I../../ -O3 -fPIC -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++0x  -shared snowboy-detect-swig.o \
../..//lib/ubuntu64/libsnowboy-detect.a -L/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu -L/usr/lib -lpython3.6m -lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions -lm -ldl -lf77blas -lcblas -llapack -latlas -o _snowboydetect.so

This is output of our current setup.py:

$ python3 setup.py build
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/snowboy
copying examples/Python3/snowboy/__init__.py -> build/lib.linux-x86_64-3.6/snowboy
copying examples/Python3/snowboy/snowboydecoder.py -> build/lib.linux-x86_64-3.6/snowboy
copying examples/Python3/snowboy/snowboydetect.py -> build/lib.linux-x86_64-3.6/snowboy
creating build/lib.linux-x86_64-3.6/snowboy/resources
copying examples/Python3/snowboy/resources/dong.wav -> build/lib.linux-x86_64-3.6/snowboy/resources
copying examples/Python3/snowboy/resources/common.res -> build/lib.linux-x86_64-3.6/snowboy/resources
copying examples/Python3/snowboy/resources/snowboy.raw -> build/lib.linux-x86_64-3.6/snowboy/resources
copying examples/Python3/snowboy/resources/ding.wav -> build/lib.linux-x86_64-3.6/snowboy/resources
copying examples/Python3/snowboy/resources/snowboy.wav -> build/lib.linux-x86_64-3.6/snowboy/resources
running build_ext
building '_snowboydetect' extension
swigging swig/Python3/snowboy-detect-swig.i to swig/Python3/snowboy-detect-swig_wrap.cpp
swig -python -c++ -o swig/Python3/snowboy-detect-swig_wrap.cpp swig/Python3/snowboy-detect-swig.i
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/swig
creating build/temp.linux-x86_64-3.6/swig/Python3
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fdebug-prefix-map=/build/python3.6-EKG1lX/python3.6-3.6.5=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -c swig/Python3/snowboy-detect-swig_wrap.cpp -o build/temp.linux-x86_64-3.6/swig/Python3/snowboy-detect-swig_wrap.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -specs=/usr/share/dpkg/no-pie-link.specs -Wl,-z,relro -Wl,-Bsymbolic-functions -specs=/usr/share/dpkg/no-pie-link.specs -Wl,-z,relro -g -fdebug-prefix-map=/build/python3.6-EKG1lX/python3.6-3.6.5=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.6/swig/Python3/snowboy-detect-swig_wrap.o -o build/lib.linux-x86_64-3.6/_snowboydetect.cpython-36m-x86_64-linux-gnu.so

To produce correct *.so file, our script must produce compiler command with the same parameter as make.

Currently, the content of files generated by swig, from snowboy-detect-swig.i are already the same.

[hongquan]

Successfully fix the setup.py and build wheel files for armv7, x86_64 (uploaded to https://repo.fury.io/fossasia/). Waiting for friend to test on Mac OSX before making a pull request to upstream.

[hongquan]

Made Pull Request to upstream here: Kitt-AI/snowboy#493

[hongquan]

I don't know when the Kiit-AI will review and accept the PR. But because it has been done and the wheel packages have been built, I'm closing this issue now.