feat: add manifest-driven batch rebuild, numpy cross-compilation, and CI pipeline updates
Some checks failed
EcoArk Wheel Pipeline CI / build (push) Failing after 1m41s
Some checks failed
EcoArk Wheel Pipeline CI / build (push) Failing after 1m41s
- Add pipeline/manifest.py for packages.txt parsing - Implement real numpy cross-compilation (rebuild.py) with OHOS NDK + vendored meson - Extend CLI with --manifest, --build-from-source, --stage-integration modes - Add recipes/, experiments/, and scripts/ for integration staging and artifact upload - Update CI workflow with manifest validation, artifact upload, and release publishing - Update Makefile and README with new targets and documentation
This commit is contained in:
82
experiments/README.md
Normal file
82
experiments/README.md
Normal file
@ -0,0 +1,82 @@
|
||||
# NumPy 真实重编实验记录
|
||||
|
||||
## 实验目标
|
||||
|
||||
验证在 x86_64 主机上,使用 OHOS NDK 交叉编译器,为 `aarch64-linux-ohos` (HarmonyOS arm64-v8a) 目标编译 NumPy 1.26.4 完整 native 扩展的真实阻塞点。
|
||||
|
||||
## 实验环境
|
||||
|
||||
- 主机: x86_64 Debian forky/sid, Python 3.12.13
|
||||
- OHOS NDK: `/home/program/tools/command-line-tools/sdk/default/openharmony/native/llvm`
|
||||
- clang 15.0.4, target aarch64-linux-ohos
|
||||
- sysroot 包含 musl libc + crt 启动文件
|
||||
- 链接器: ld.lld 15.0.4
|
||||
- Python 3.12 交叉编译产物 (来自 EcoArk): `/home/program/EcoArk/sdk/pythonrunner/src/main/cpp/third_party/python/`
|
||||
|
||||
## 实验方法
|
||||
|
||||
1. pip 创建 venv,安装 Cython 0.29.36 + meson-python 0.15.0 + ninja + setuptools
|
||||
2. 下载 numpy 1.26.4 sdist
|
||||
3. 使用 numpy 自带的 vendored-meson(含自定义 features 模块)
|
||||
4. 编写 OHOS aarch64 meson cross-compilation file
|
||||
5. 执行 `meson setup` → `meson compile`
|
||||
|
||||
## 结果总结
|
||||
|
||||
### 流程进度
|
||||
|
||||
| 步骤 | 状态 | 备注 |
|
||||
|------|------|------|
|
||||
| NDK 交叉编译器可用性 | ✅ | `clang --target=aarch64-linux-ohos` 可编译 C 程序 |
|
||||
| meson setup | ✅ | 需 `longdouble_format=IEEE_QUAD_LE` 绕过目标不可执行的问题 |
|
||||
| meson compile | ✅ | 195 个 target 全部通过,产出 19 个 .so |
|
||||
| 产物的正确性 | ✅ | 全部 19 个 .so 均为 ARM aarch64 ELF |
|
||||
| so 动态链接 | ✅ | 仅依赖 libc.so (musl) |
|
||||
|
||||
### 第一个真实阻塞点
|
||||
|
||||
**`numpy/core/meson.build:376` — 无法在交叉编译环境中运行测试程序**
|
||||
|
||||
- 原因: numpy 编译时需要运行 C 程序检测 `long double` 的内存布局格式(IEEE_DOUBLE_LE / INTEL_EXTENDED_16_BYTES_LE / IEEE_QUAD_LE / IBM_DOUBLE_DOUBLE_LE 等)
|
||||
- 交叉编译时跨架构二进制无法在 build machine 运行
|
||||
- 解决: 在 cross file 的 `[properties]` 中预声明 `longdouble_format = 'IEEE_QUAD_LE'`
|
||||
|
||||
### 次要阻塞点
|
||||
|
||||
| 阻塞点 | 位置 | 解决方式 |
|
||||
|--------|------|----------|
|
||||
| Cython 0.29 依赖 distutils (Python 3.12 移除) | Cython → distutils.extension | `pip install setuptools` |
|
||||
| meson 缺少 "features" 模块 | meson_cpu/x86/meson.build:2 | 必须使用 numpy 自带的 vendored-meson |
|
||||
| 交叉编译时 sizeof 类检测跳过 | 多个 meson.build 位置 | 系统 meson 自动处理 |
|
||||
| OpenBLAS 未找到 | numpy/core/meson.build | `-Dallow-noblas=true` 降级 |
|
||||
|
||||
### 待解决(未阻塞编译,但影响最终产物)
|
||||
|
||||
1. **SOABI 标签修正**: .so 文件名当前为 `cpython-312-x86_64-linux-gnu`(来自 build machine Python),需改为 `cpython-312-aarch64-linux-ohos`(对应目标平台的 SOABI)
|
||||
2. **Wheel 打包**: 需将纯 Python 文件(来自 source tree)与交叉编译 .so 组合为 wheel,修改 tag 为 `harmonyos_arm64`
|
||||
3. **OpenBLAS**: numpy 当前使用内置 slow fallback BLAS(lapack_lite),如需性能需交叉编译 OpenBLAS
|
||||
4. **pyconfig.h 校准**: 当前使用 x86_64 原生 configure 生成的 pyconfig.h,需针对 aarch64-linux-ohos + musl 重新生成
|
||||
|
||||
## 最小前置依赖清单
|
||||
|
||||
```
|
||||
pip install Cython>=0.29.34,<3.1 meson-python>=0.15.0,<0.16.0 ninja setuptools
|
||||
# 使用 numpy vendored meson(内置 features 模块)
|
||||
# cross file: ohos-aarch64-cross.ini
|
||||
# meson args: -Dallow-noblas=true -Ddisable-threading=true -Ddisable-optimization=true
|
||||
```
|
||||
|
||||
## 参考命令
|
||||
|
||||
```bash
|
||||
# 完整编译流程(已验证通过)
|
||||
pip install Cython==0.29.36 meson-python==0.15.0 ninja setuptools
|
||||
tar xzf numpy-1.26.4.tar.gz
|
||||
cd numpy-1.26.4
|
||||
# 必须用 vendored meson!
|
||||
python3 vendored-meson/meson/meson.py setup build \
|
||||
--cross-file ../ohos-aarch64-cross.ini \
|
||||
-Dallow-noblas=true -Ddisable-threading=true -Ddisable-optimization=true
|
||||
python3 vendored-meson/meson/meson.py compile -C build -j4
|
||||
# 产出在 build/numpy/ 下
|
||||
```
|
||||
43
experiments/ohos-aarch64-cross.ini
Normal file
43
experiments/ohos-aarch64-cross.ini
Normal file
@ -0,0 +1,43 @@
|
||||
# Meson cross-compilation file for OHOS aarch64 (arm64-v8a)
|
||||
#
|
||||
# Usage:
|
||||
# cd numpy-1.26.4
|
||||
# meson setup build \
|
||||
# --cross-file /path/to/ohos-aarch64-cross.ini \
|
||||
# -Dallow-noblas=true \
|
||||
# -Ddisable-threading=true \
|
||||
# -Ddisable-optimization=true
|
||||
#
|
||||
# NOTE: numpy 1.26.x requires VENDORED meson (not system meson).
|
||||
# Use: python3 vendored-meson/meson/meson.py setup ...
|
||||
#
|
||||
# Toolchain path (adjust as needed):
|
||||
# /home/program/tools/command-line-tools/sdk/default/openharmony/native/llvm
|
||||
#
|
||||
# Sysroot:
|
||||
# /home/program/tools/command-line-tools/sdk/default/openharmony/native/sysroot
|
||||
|
||||
[binaries]
|
||||
c = '/home/program/tools/command-line-tools/sdk/default/openharmony/native/llvm/bin/clang'
|
||||
cpp = '/home/program/tools/command-line-tools/sdk/default/openharmony/native/llvm/bin/clang++'
|
||||
ar = '/home/program/tools/command-line-tools/sdk/default/openharmony/native/llvm/bin/llvm-ar'
|
||||
strip = '/home/program/tools/command-line-tools/sdk/default/openharmony/native/llvm/bin/llvm-strip'
|
||||
ld = '/home/program/tools/command-line-tools/sdk/default/openharmony/native/llvm/bin/ld.lld'
|
||||
|
||||
[built-in options]
|
||||
c_args = ['--target=aarch64-linux-ohos', '--sysroot=/home/program/tools/command-line-tools/sdk/default/openharmony/native/sysroot', '-D__MUSL__']
|
||||
c_link_args = ['--target=aarch64-linux-ohos', '--sysroot=/home/program/tools/command-line-tools/sdk/default/openharmony/native/sysroot']
|
||||
cpp_args = ['--target=aarch64-linux-ohos', '--sysroot=/home/program/tools/command-line-tools/sdk/default/openharmony/native/sysroot', '-D__MUSL__']
|
||||
cpp_link_args = ['--target=aarch64-linux-ohos', '--sysroot=/home/program/tools/command-line-tools/sdk/default/openharmony/native/sysroot']
|
||||
|
||||
[properties]
|
||||
skip_sanity_check = true
|
||||
allow-noblas = true
|
||||
# aarch64-linux-ohos uses MUSL which defines IEEE quad (128-bit) long double
|
||||
longdouble_format = 'IEEE_QUAD_LE'
|
||||
|
||||
[host_machine]
|
||||
system = 'linux'
|
||||
cpu_family = 'aarch64'
|
||||
cpu = 'armv8-a'
|
||||
endian = 'little'
|
||||
Reference in New Issue
Block a user