feat: add manifest-driven batch rebuild, numpy cross-compilation, and CI pipeline updates
Some checks failed
EcoArk Wheel Pipeline CI / build (push) Failing after 1m41s

- Add pipeline/manifest.py for packages.txt parsing
- Implement real numpy cross-compilation (rebuild.py) with OHOS NDK + vendored meson
- Extend CLI with --manifest, --build-from-source, --stage-integration modes
- Add recipes/, experiments/, and scripts/ for integration staging and artifact upload
- Update CI workflow with manifest validation, artifact upload, and release publishing
- Update Makefile and README with new targets and documentation
This commit is contained in:
root
2026-05-13 14:16:24 +08:00
parent cdc816467d
commit 553b83953c
15 changed files with 1310 additions and 99 deletions

82
experiments/README.md Normal file
View File

@ -0,0 +1,82 @@
# NumPy 真实重编实验记录
## 实验目标
验证在 x86_64 主机上,使用 OHOS NDK 交叉编译器,为 `aarch64-linux-ohos` (HarmonyOS arm64-v8a) 目标编译 NumPy 1.26.4 完整 native 扩展的真实阻塞点。
## 实验环境
- 主机: x86_64 Debian forky/sid, Python 3.12.13
- OHOS NDK: `/home/program/tools/command-line-tools/sdk/default/openharmony/native/llvm`
- clang 15.0.4, target aarch64-linux-ohos
- sysroot 包含 musl libc + crt 启动文件
- 链接器: ld.lld 15.0.4
- Python 3.12 交叉编译产物 (来自 EcoArk): `/home/program/EcoArk/sdk/pythonrunner/src/main/cpp/third_party/python/`
## 实验方法
1. pip 创建 venv,安装 Cython 0.29.36 + meson-python 0.15.0 + ninja + setuptools
2. 下载 numpy 1.26.4 sdist
3. 使用 numpy 自带的 vendored-meson(含自定义 features 模块)
4. 编写 OHOS aarch64 meson cross-compilation file
5. 执行 `meson setup` → `meson compile`
## 结果总结
### 流程进度
| 步骤 | 状态 | 备注 |
|------|------|------|
| NDK 交叉编译器可用性 | ✅ | `clang --target=aarch64-linux-ohos` 可编译 C 程序 |
| meson setup | ✅ | 需 `longdouble_format=IEEE_QUAD_LE` 绕过目标不可执行的问题 |
| meson compile | ✅ | 195 个 target 全部通过,产出 19 个 .so |
| 产物的正确性 | ✅ | 全部 19 个 .so 均为 ARM aarch64 ELF |
| so 动态链接 | ✅ | 仅依赖 libc.so (musl) |
### 第一个真实阻塞点
**`numpy/core/meson.build:376` — 无法在交叉编译环境中运行测试程序**
- 原因: numpy 编译时需要运行 C 程序检测 `long double` 的内存布局格式(IEEE_DOUBLE_LE / INTEL_EXTENDED_16_BYTES_LE / IEEE_QUAD_LE / IBM_DOUBLE_DOUBLE_LE 等)
- 交叉编译时跨架构二进制无法在 build machine 运行
- 解决: 在 cross file 的 `[properties]` 中预声明 `longdouble_format = 'IEEE_QUAD_LE'`
### 次要阻塞点
| 阻塞点 | 位置 | 解决方式 |
|--------|------|----------|
| Cython 0.29 依赖 distutils (Python 3.12 移除) | Cython → distutils.extension | `pip install setuptools` |
| meson 缺少 "features" 模块 | meson_cpu/x86/meson.build:2 | 必须使用 numpy 自带的 vendored-meson |
| 交叉编译时 sizeof 类检测跳过 | 多个 meson.build 位置 | 系统 meson 自动处理 |
| OpenBLAS 未找到 | numpy/core/meson.build | `-Dallow-noblas=true` 降级 |
### 待解决(未阻塞编译,但影响最终产物)
1. **SOABI 标签修正**: .so 文件名当前为 `cpython-312-x86_64-linux-gnu`(来自 build machine Python),需改为 `cpython-312-aarch64-linux-ohos`(对应目标平台的 SOABI)
2. **Wheel 打包**: 需将纯 Python 文件(来自 source tree)与交叉编译 .so 组合为 wheel,修改 tag 为 `harmonyos_arm64`
3. **OpenBLAS**: numpy 当前使用内置 slow fallback BLAS(lapack_lite),如需性能需交叉编译 OpenBLAS
4. **pyconfig.h 校准**: 当前使用 x86_64 原生 configure 生成的 pyconfig.h,需针对 aarch64-linux-ohos + musl 重新生成
## 最小前置依赖清单
```
pip install Cython>=0.29.34,<3.1 meson-python>=0.15.0,<0.16.0 ninja setuptools
# 使用 numpy vendored meson(内置 features 模块)
# cross file: ohos-aarch64-cross.ini
# meson args: -Dallow-noblas=true -Ddisable-threading=true -Ddisable-optimization=true
```
## 参考命令
```bash
# 完整编译流程(已验证通过)
pip install Cython==0.29.36 meson-python==0.15.0 ninja setuptools
tar xzf numpy-1.26.4.tar.gz
cd numpy-1.26.4
# 必须用 vendored meson!
python3 vendored-meson/meson/meson.py setup build \
--cross-file ../ohos-aarch64-cross.ini \
-Dallow-noblas=true -Ddisable-threading=true -Ddisable-optimization=true
python3 vendored-meson/meson/meson.py compile -C build -j4
# 产出在 build/numpy/ 下
```

View File

@ -0,0 +1,43 @@
# Meson cross-compilation file for OHOS aarch64 (arm64-v8a)
#
# Usage:
# cd numpy-1.26.4
# meson setup build \
# --cross-file /path/to/ohos-aarch64-cross.ini \
# -Dallow-noblas=true \
# -Ddisable-threading=true \
# -Ddisable-optimization=true
#
# NOTE: numpy 1.26.x requires VENDORED meson (not system meson).
# Use: python3 vendored-meson/meson/meson.py setup ...
#
# Toolchain path (adjust as needed):
# /home/program/tools/command-line-tools/sdk/default/openharmony/native/llvm
#
# Sysroot:
# /home/program/tools/command-line-tools/sdk/default/openharmony/native/sysroot
[binaries]
c = '/home/program/tools/command-line-tools/sdk/default/openharmony/native/llvm/bin/clang'
cpp = '/home/program/tools/command-line-tools/sdk/default/openharmony/native/llvm/bin/clang++'
ar = '/home/program/tools/command-line-tools/sdk/default/openharmony/native/llvm/bin/llvm-ar'
strip = '/home/program/tools/command-line-tools/sdk/default/openharmony/native/llvm/bin/llvm-strip'
ld = '/home/program/tools/command-line-tools/sdk/default/openharmony/native/llvm/bin/ld.lld'
[built-in options]
c_args = ['--target=aarch64-linux-ohos', '--sysroot=/home/program/tools/command-line-tools/sdk/default/openharmony/native/sysroot', '-D__MUSL__']
c_link_args = ['--target=aarch64-linux-ohos', '--sysroot=/home/program/tools/command-line-tools/sdk/default/openharmony/native/sysroot']
cpp_args = ['--target=aarch64-linux-ohos', '--sysroot=/home/program/tools/command-line-tools/sdk/default/openharmony/native/sysroot', '-D__MUSL__']
cpp_link_args = ['--target=aarch64-linux-ohos', '--sysroot=/home/program/tools/command-line-tools/sdk/default/openharmony/native/sysroot']
[properties]
skip_sanity_check = true
allow-noblas = true
# aarch64-linux-ohos uses MUSL which defines IEEE quad (128-bit) long double
longdouble_format = 'IEEE_QUAD_LE'
[host_machine]
system = 'linux'
cpu_family = 'aarch64'
cpu = 'armv8-a'
endian = 'little'