juzi/taobao/vendor/native-face

Local tensor face engine

Runtime adapter: taobao/utils/native-face.js.

The vendor bundle, tiny detector and full 68-point model come from the existing, unchanged h5/vendor/faceapi/ files. The added tiny 68-point model is downloaded only from the same official upstream's weights/ directory; its manifest, 77224-byte shard and source URLs/hashes are in reference/native-face-models/. Upstream is https://github.com/justadudewhohacks/face-api.js. MIT and TensorFlow Apache 2.0 licenses are included; original inline Google/Microsoft notices are retained.

Build or verify from the repository root:

node taobao/scripts/build-native-face.cjs
node taobao/scripts/build-native-face.cjs --check
node --test taobao/test/native-face.test.cjs
node taobao/scripts/benchmark-native-face.cjs --repeats=2
node taobao/scripts/benchmark-native-face.cjs --guided --input-sizes=256,160,128 --repeats=2

provenance.json records every source file's SHA-256 and byte length. The quantized model bytes total 627385 bytes including the optional full model; the default tiny detector plus tiny landmark weights total 270545 bytes. Base64 and manifests are packaged in one CJS module. Runtime decoding uses typed arrays, with no filesystem or Buffer.

The generated face-api copy has a private TFJS global registry and shadows Node process/Buffer. Optional Node-only imports (crypto, node-fetch, util, fs) are replaced with functions that explicitly throw if reached. The tfjs-node help-string reference is also replaced. The build fails if an unreviewed static require() remains, because the IDE resolves unreachable imports too. No Node polyfill or native AI plugin is bundled.

Create an engine with create({defaultPixelFormat:'RGBA', maxSide:320, inputSize:160, landmarkModel:'tiny', scoreThreshold:0.35}); these are the defaults. landmarkModel:'full' remains available for controlled comparisons. The runtime default backend remains cpu for compatibility. Select WebGL explicitly with create({backend:'webgl', platform:my, ...}); no unsupported WebGL request silently falls back to the CPU profile that was too slow on phone. init() returns a Promise with backend, inputSize, maxSide, landmarkModel and cpuOptimization. detect(frame) returns a Promise with original width, height, total faceCount and at most two complete detections. Each detection has detection.{box:{x,y,width,height},score} and landmarks.positions containing 68 {x,y} points. All coordinates refer to original raw frame pixels; no mirroring, rotation or preview crop is applied. Two complete detections suffice for the existing multiple-person rejection gate.

Input must have integer width/height in 1..4096 and tightly packed four-channel data in an ArrayBuffer or typed-array view. RGBA, BGRA, ARGB, ABGR and their 8888 aliases are supported. An explicit format is authoritative: the configured default only fills missing metadata. Unknown formats (including NV/YUV), a stride/bytesPerRow other than width*4, and a nonzero rotation marker are rejected.

detect() synchronously snapshots only the downsampled RGB before returning, so native buffer reuse cannot change a queued inference. All model operations use the CPU backend. The lower-level tensor crop/landmark path avoids the high-level withFaceLandmarks() canvas dependency. Concurrent inference rejects with NATIVE_FACE_BUSY. dispose() is idempotent; pending work rejects with NATIVE_FACE_DISPOSED, and tensors are released after that work settles. The last engine instance releases the shared model tensors. stats() returns only backend, selected configuration, memory counters and lifecycle booleans.

The CPU pad-copy specialization copies contiguous source blocks into the already zero-filled output tensor. It preserves the original pad values exactly; unhandled scalar/complex inputs fall back to the bundled implementation. Disable it with cpuOptimization:false (cpuOptimization:'none' in returned diagnostics). It does not change model arithmetic or relax quality/freshness gates. The trial 1x1 convolution specialization showed no stable benefit and was removed.

detect() additionally returns inputSize, maxSide, landmarkModel, cpuOptimization and timings.{snapshotMs,modelMs,tensorMs,detectorMs,landmarkMs, totalMs} in milliseconds. elapsedMs remains total elapsed wall time; source frame age must still be calculated from the camera's frame arrival time.

Tests run the real models without document, Image, canvas, fetch, process, Buffer or host Node imports. The local face fixture derives from the already present reference SDK example C3_180.jpg; it is never transmitted. Tests check actual positive face detection and 68 points, all supported byte layouts, original-coordinate scaling, concurrent/disposed calls, native buffer reuse, and zero tensors after final disposal. The pad kernel is tested bit-for-bit against original CPU pad across ranks 1–4, dtypes and padding/constants; the real full-model detection and all 68 points are identical with it enabled or disabled. An aspect-preserving two-face composition must reach the actual native-face-quality single-person rejection gate. Tests print timings/counts, never the image or point coordinates.

Desktop no-DOM VM stage measurements on the same 540×960 portrait reference (two samples per profile): old full/256/no specialization took about 934 ms (detector 730, landmarks 189, snapshot 16). Tiny/160/pad-copy took about 302 ms (detector 188, landmarks 99, snapshot 15), about 3.1× faster. Default model tensors occupy 1082180 bytes (47 tensors), versus full's 2200644 bytes (68).

The original small face ratio was 0.208, below the H5 near gate of 0.30. With an aspect-preserving crop targeting a valid larger face, tiny/128 can detect one face and 68 points in about 254 ms; however, 128 misses one person in the two-face regression while 160 detects both. Therefore 160 remains the default. Reducing maxSide from 320 to 192 saved only about 24 ms on the eligible portrait; 320 is retained for landmark detail. These are desktop CPU observations, not phone performance guarantees. Benchmark flags --baseline-cpu, --max-side=192 and --input-sizes=160,128 support reproducing the separate comparisons. A zero face result is explicitly flagged by allSamplesDetected:false, not a speed win.

Explicit WebGL candidate

utils/native-face-webgl.js obtains an owned logical-layer Canvas using the official synchronous my.createOffscreenCanvas(32,32) and getContext('webgl',{enable_flip_y_after_read_pixels:false}). The host app must enable its documented Skia capability. The adapter injects the actual GL1 context into the bundled TFJS backend; it never forges HAS_WEBGL or float-extension ENV flags. Initialization runs a 16×16 matrix product, a four-channel convolution, numerical readback checks and observed real shader draw/readPixels counters. A missing or readonly instrumentable native API fails explicitly; the adapter does not report GPU success from API names alone.

The current device candidate is LOCAL-FACE-20261102-07. The preceding 03 candidate executed four matrix shader draws on the iPhone, then failed the strict numerical check with MATMUL_FAILED; it did not process camera frames. Candidate 04 tried default, packed_byte (disables float downloads), then unpacked_byte (also disables packed shaders). Each attempt runs the same matrix/convolution/readback checks with the original numerical tolerance. A failed attempt releases its backend and tracked GL objects before the next attempt. Only numerical/readback failures permit another profile; missing APIs, context loss and resource errors stop explicitly. There is no CPU fallback.

All three profiles failed on the device in 04. The final unpacked byte attempt observed five draws, one byte read, checksum 0 versus expected 2786, 244 mismatches and maximum absolute error 22. These aggregate values have an all-zero signature; they do not by themselves prove every output value is zero or identify the cause. Real camera frames were never processed.

Candidate 05 disables the host's extra readPixels flip using the documented Taobao context option. A canvas-height-based read-coordinate conversion with a smaller framebuffer can produce zero output in an isolated experiment; that is a testable hypothesis, not a confirmed description of the device implementation. Before TFJS setup, the adapter checks a 2×2 RGBA8 framebuffer with clear/readback, an asymmetric shader pattern, byte-texture sampling and float-texture sampling. Fixed nonzero synthetic inputs and a prefilled destination sentinel distinguish an unchanged typed array from incorrect GPU output. Any failed preflight stops initialization and releases its graphics objects. Passing this preflight does not replace the original matrix/convolution numerical checks, compatibility selection or model warmup.

On the device, 05 passed clear/readback but failed the asymmetric shader with three mismatches: expected checksum 2244, actual 2040. The actual checksum equals the preceding clear color's checksum. This is consistent with no draw coverage, but does not uniquely distinguish coverage, coordinates, typed-buffer uploads or GL state. Real camera inference still did not start.

Candidate 06 explicitly initializes drawing state and clears to black before each draw. It first checks a constant-color drawArrays pass and then an indexed pass, followed by the original asymmetric shader and byte/float texture tests. It verifies native strategies in order: standard, buffer_copy (tightly copy a view's bytes into an ArrayBuffer for standard bufferData upload), framebuffer_y (compensate viewport/scissor Y only for a bound framebuffer), then combined. These are candidates selected by actual numerical checks, not assumptions based on phone model. Only VALUE_MISMATCH permits another native strategy; all other failures stop. All four native strategies share one context and release each probe's objects before the next. The TF profiles still create at most three contexts. A selected native strategy must pass every native test, then the unchanged strict matrix/convolution tests and real-model warmup. The source frame freshness limit remains 1500 ms. Device strategy selection, compatibility, speed and automatic capture still need phone acceptance.

Candidate 06 failed on the device after its standard constant-array draw passed: nativeInitialStage=constant_indexed. The later coordinate candidate's black output is not the original failure. Candidate 07 first rebinds the element buffer immediately before indexed draws. If the indexed result still fails while the array draw passed, it verifies a four-vertex triangle strip as an alternative. Strip probes report constantQuad:passed and leave constantIndexed:not_run; drawStrategy:quad_strip identifies the physical method without claiming the native indexed API passed.

Only the owned quad with uploaded indices [0,1,2,2,1,3] and exact TRIANGLES/6/UNSIGNED_SHORT/offset0 arguments can be converted. Upload bytes are checked within typed-view bounds. Unknown buffers, different signatures, numeric allocations and buffers modified with bufferSubData are refused; deletion invalidates the record. This matches the pinned upstream quad data and retains the original tensor checks and model warmup. Native strategies can each add a verified strip attempt, up to eight probes on the same context. This is an experimental device compatibility path, not an assertion that the Taobao indexed API is generally unsupported. Phone success and speed remain unverified.

After the numerical checks, initialization loads the selected real models and warms both the detector on synthetic portrait black pixels and the landmark network on a synthetic tensor crop. Warmup results are discarded and never count as user face/pose data. Only after the warmup does initialization return backend:'webgl', gpu.selftest:'passed' and gpu.stage:'ready'. The source frame age and 1500 ms capture gate remain the camera's responsibility.

The returned gpu record contains fixed stage/code/profile enums, booleans, counts and metrics computed only from the fixed synthetic selftest arrays: requested, selftest, stage, code, contextLost, profile, attemptCount, matmulDraws, convDraws, pixelReads, floatReads, byteReads, selftestMs and warmupMs. Each synthetic matrix/convolution record reports actual length, nonfinite/mismatch counts, maximum absolute error, first expected/actual values and expected/actual checksums. The camera caches those synthetic metrics, profile, attempts and read counts only during initialization; later detection results cannot replace them in copied diagnostics. Errors use NATIVE_FACE_WEBGL_* codes and the same safe record. Native error strings, renderer strings, image pixels and coordinates do not enter that record. Each detect checks actual context-loss state before and after model work; a lost backend must be retired and rebuilt.

The startup record also includes the fixed-input nativeProbe stage/result, sentinel-change and mismatch counts, expected/actual checksums, read return-kind enum, GL error and framebuffer status. The record also preserves nativeInitialStage/nativeInitialCode, selected nativeStrategy, constantArrays/constantIndexed, optional vertex/index buffer byte counts and pixel0 through pixel3 as big-endian RGBA uint32 values from fixed synthetic input only. TFJS readback state is collected before throwing for a numerical mismatch, so a failing value no longer hides a native GL error. These fields are cached only during initialization and cannot be replaced by real-camera inference results.

A shared live engine group admits one backend kind at a time. Initialization, model loading, warmup and inference are queued; decoded weights are not reused across a CPU/WebGL switch. Disposal waits for pending work, then releases model tensors, backend programs/textures, the externally supplied GPGPUContext and all remaining tracked GL objects including TFJS 1.7's vertex buffer/shader leftovers. Native cleanup exceptions are isolated so they cannot preserve a busy lock or replace the original inference error.

node --test taobao/test/native-face.test.cjs taobao/test/native-face-webgl.test.cjs

The WebGL tests run packaged modules with DOM/Node environment names hidden, explicitly supply real headless WebGL1 and abort all network routes. They cover numerical shader/readback, corrupted FLOAT readback followed by packed byte readback, two failed contexts followed by unpacked byte readback, and all three profiles failing closed. Both compatible paths run the real 68-point model and are compared with CPU face boxes, confidence and landmark geometry. They also cover context loss, initialization failure, shared engines, deferred disposal and rebuilding, with zero final tensors and tracked GL objects. Desktop SwiftShader proves the computation and lifecycle paths; it does not establish native Taobao iPhone compatibility or phone inference speed. Those still require the device's actual diagnostics.