How to Use AppVeyor to Build and Deploy Python Wheels from C/C++ Code

AppVeyor is a continuous integration (CI) service used to automatically build code projects and deploy relevant artifacts. It provides build environments for Windows, Linux, and macOS. In this article, I will share how to use AppVeyor to build and deploy Python Wheels (Windows edition) from C/C++ code.

Creating Python Wheels

It takes two steps to build a Python wheel from CPython code:

  1. Build *.pyd extension file from C/C++ code.
  2. Pack *.pyd and dependent *.dll files into a wheel file.

For the past few years, I’ve been maintaining the source code of Python barcode extension based on Dynamsoft Barcode Reader C/C++ SDK. Let’s do some changes based on the repository.

The *.pyd Python module

Get the source code:

git clone https://github.com/yushulx/python.git

Install DynamsoftBarcodeReader7.2.1.exe. Copy *.dll files from Dynamsoft\Barcode Reader 7.2.1\Components\C_C++\Redist\x64 to bin folder and copy DBRx64.lib from Dynamsoft\Barcode Reader 7.2.1\Components\C_C++\Lib\DBRx64.lib to lib folder. Create an empty folder named wheel.

pypi python project structure

Open src\setup.py to set the link directories:

elif sys.platform == "win32":
    # Windows
    dbr_lib_name = 'DBRx64'
    dbr_lib_dir = r'..\lib'
    dbr_dll = r'..\bin'

Build the Python barcode module:

cd src
python setup.py build

The *.whl file

Create a folder named dbr under the wheel folder. Add a setup.py file:

from setuptools import setup, find_packages, Distribution
from codecs import open
from os import path
import sys

here = path.abspath(path.dirname(__file__))

with open(path.join(here, 'README.md'), encoding='utf-8') as f:
    long_description = f.read()

class BinaryDistribution(Distribution):
    """Distribution which always forces a binary package with platform name"""
    def has_ext_modules(foo):
        return True

data_info = {'dbr':['*.pyd', 'vcomp110.dll', 'DynamicPdfx64.dll', 'DynamsoftBarcodeReaderx64.dll', 'DynamsoftLicClientx64.dll']}

setup(
    name='dbr',  
    version='7.2.1',  
    description='Dynamsoft Barcode Reader Python project',  
    long_description=long_description, 
    long_description_content_type='text/markdown',
    url='https://www.dynamsoft.com/Products/Dynamic-Barcode-Reader.aspx', 
    license = 'https://www.dynamsoft.com/Products/barcode-reader-license-agreement.aspx',
    author='Dynamsoft', 
    author_email='support@dynamsoft.com', 
    classifiers=[  
        'Development Status :: 5 - Production/Stable',
        'Intended Audience :: Developers',
        'Topic :: Software Development :: Libraries',
        'License :: Other/Proprietary License',
        'Programming Language :: Python',
        'Programming Language :: Python :: 2',
        'Programming Language :: Python :: 2.7',
        'Programming Language :: Python :: 3',
        'Programming Language :: Python :: 3.5',
        'Programming Language :: Python :: 3.6',
        'Programming Language :: Python :: 3.7',
        'Programming Language :: Python :: 3.8',
        'Programming Language :: C++',
        'Programming Language :: Python :: Implementation :: CPython',
        'Operating System :: POSIX :: Linux ',
        'Operating System :: Microsoft :: Windows :: Windows 10',
        'Topic :: Scientific/Engineering',
        'Topic :: Scientific/Engineering :: Image Recognition',
        'Topic :: Software Development',
    ],
    keywords='barcode DataMatrix QRCode 1D PDF417 MaxiCode Aztec',  
    packages=find_packages(exclude=['contrib', 'docs', 'tests']),  
    install_requires=['numpy', 'opencv-python'],  
    package_data=data_info,
    distclass=BinaryDistribution,
    platforms=['Windows', 'Linux']
)

The data_info contains the *.dll and *.whl files.

Create __init__.py under the dbr folder:

from .dbr import *

Copy *.pyd and *.dll files to the dbr folder from the src folder:

copy ..\bin\*.* ..\wheel\dbr\
cd build\lib.win-*\
copy *.* ..\..\..\wheel\dbr\

Build the wheel package under the wheel folder:

python setup.py bdist_wheel

Configuring YAML file for AppVeyor Build Service

Login https://ci.appveyor.com/ with your GitHub account.

Click Projects to import the target repository:

appveyor new project

To trigger the build, create an appveyor.yml file under the project root directory.

Set the branch to build:

branches:
  only:
    - wheel

Add the Python environments:

environment:
  matrix:
    - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2019
      PYTHON: "C:/Python27-x64"
    - PYTHON: "C:/Python35-x64"
    - PYTHON: "C:/Python36-x64"
    - PYTHON: "C:/Python37-x64"
    - PYTHON: "C:/Python38-x64"

Why do I use APPVEYOR_BUILD_WORKER_IMAGE  for Python 2.7? When building the source code with the default environment, I got the following error message:

Traceback (most recent call last):
  File "setup.py", line 63, in <module>
    cmdclass={'install': CustomInstall}
  File "C:\Python27-x64\lib\distutils\core.py", line 151, in setup
    dist.run_commands()
  File "C:\Python27-x64\lib\distutils\dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "C:\Python27-x64\lib\distutils\dist.py", line 972, in run_command
    cmd_obj.run()
  File "C:\Python27-x64\lib\distutils\command\build.py", line 127, in run
    self.run_command(cmd_name)
  File "C:\Python27-x64\lib\distutils\cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "C:\Python27-x64\lib\distutils\dist.py", line 972, in run_command
    cmd_obj.run()
  File "C:\Python27-x64\lib\distutils\command\build_ext.py", line 340, in run
    self.build_extensions()
  File "C:\Python27-x64\lib\distutils\command\build_ext.py", line 449, in build_extensions
    self.build_extension(ext)
  File "C:\Python27-x64\lib\distutils\command\build_ext.py", line 499, in build_extension
    depends=ext.depends)
  File "C:\Python27-x64\lib\distutils\msvc9compiler.py", line 473, in compile
    self.initialize()
  File "C:\Python27-x64\lib\distutils\msvc9compiler.py", line 383, in initialize
    vc_env = query_vcvarsall(VERSION, plat_spec)
  File "C:\Python27-x64\lib\distutils\msvc9compiler.py", line 299, in query_vcvarsall
    raise ValueError(str(list(result.keys())))
ValueError: [u'path']
Command exited with code 1

The issue seems caused by Visual Studio 2008. According to the official documentation – Windows images software, Visual C++ 2008 Express is not installed in the Visual Studio 2019 image. So I can use the Visual Studio 2019 image and set Visual Studio 2015 as the build tool to make it work:

install:
  - SET VS90COMNTOOLS=%VS140COMNTOOLS%

Add the build script:

build_script:
  - cmd: |
      "%PYTHON%/python.exe" -m pip install --upgrade pip
      "%PYTHON%/python.exe" -m pip install --upgrade setuptools wheel numpy

      cd src
      "%PYTHON%/python.exe" setup.py build

      copy ..\bin\*.* ..\wheel\dbr\
      cd build\lib.win-*\
      copy *.* ..\..\..\wheel\dbr\
      cd ..\..\..\wheel\

      "%PYTHON%/python.exe" setup.py bdist_wheel

Set the artifacts:

artifacts:
  - path: wheel\dist\*.whl
    name: wheels

Add the scripts for PyPi deployment:

deploy_script:
  - ps: |
      if($env:APPVEYOR_REPO_TAG -eq 'true') {
        Write-Output ("Deploying " + $env:APPVEYOR_REPO_TAG_NAME + " to PyPI...")
        &amp;"${Env:PYTHON}/python.exe" -m pip install twine
        &amp;"${Env:PYTHON}/python.exe" -m twine upload -u ${Env:USER} -p ${Env:PASS} --skip-existing dist/*.whl
      } else {
        Write-Output "No tag for deployment"
      }

When adding tags to the GitHub repository, the APPVEYOR_REPO_TAG value will be ‘true’. You can encrypt the username and password of the PyPi account by clicking Account > Encrypt YAML.

appveyor encrypt yaml

When building successfully, the *.whl files for Python 2.7, 3.5, 3.6, 3.7 and 3.8 will be uploaded to the PyPi website.

appveyor python build
appveyor pypi deployment

Reference

Source Code

https://github.com/yushulx/python/tree/wheel