juzi/taobao/vendor/native-face/README.md

# Local tensor face engine

Runtime adapter: `taobao/utils/native-face.js`.

The vendor bundle, tiny detector and full 68-point model come from the existing,
unchanged `h5/vendor/faceapi/` files. The added tiny 68-point model is downloaded
only from the same official upstream's `weights/` directory; its manifest,
77224-byte shard and source URLs/hashes are in `reference/native-face-models/`.
Upstream is
https://github.com/justadudewhohacks/face-api.js. MIT and TensorFlow Apache 2.0
licenses are included; original inline Google/Microsoft notices are retained.

Build or verify from the repository root:

```
node taobao/scripts/build-native-face.cjs
node taobao/scripts/build-native-face.cjs --check
node --test taobao/test/native-face.test.cjs
node taobao/scripts/benchmark-native-face.cjs --repeats=2
node taobao/scripts/benchmark-native-face.cjs --guided --input-sizes=256,160,128 --repeats=2
```

`provenance.json` records every source file's SHA-256 and byte length. The
quantized model bytes total 627385 bytes including the optional full model;
the default tiny detector plus tiny landmark weights total 270545 bytes.
Base64 and manifests are packaged in
one CJS module. Runtime decoding uses typed arrays, with no filesystem or Buffer.

The generated face-api copy has a private TFJS global registry and shadows Node
`process`/`Buffer`. Optional Node-only imports (`crypto`, `node-fetch`, `util`,
`fs`) are replaced with functions that explicitly throw if reached. The
tfjs-node help-string reference is also replaced. The build fails if an
unreviewed static `require()` remains, because the IDE resolves unreachable
imports too. No Node polyfill or native AI plugin is bundled.

Create an engine with `create({defaultPixelFormat:'RGBA', maxSide:320,
inputSize:160, landmarkModel:'tiny', scoreThreshold:0.35})`; these are the
defaults. `landmarkModel:'full'` remains available for controlled comparisons.
The runtime default backend remains `cpu` for compatibility. Select WebGL
explicitly with `create({backend:'webgl', platform:my, ...})`; no unsupported
WebGL request silently falls back to the CPU profile that was too slow on phone.
`init()` returns a Promise with backend, inputSize, maxSide, landmarkModel and
cpuOptimization. `detect(frame)` returns a Promise with original width,
height, total faceCount and at most two complete detections. Each detection has
`detection.{box:{x,y,width,height},score}` and `landmarks.positions` containing
68 `{x,y}` points. All coordinates refer to original raw frame pixels; no
mirroring, rotation or preview crop is applied. Two complete detections suffice
for the existing multiple-person rejection gate.

Input must have integer width/height in 1..4096 and tightly packed four-channel
data in an ArrayBuffer or typed-array view. RGBA, BGRA, ARGB, ABGR and their
8888 aliases are supported. An explicit format is authoritative: the configured
default only fills missing metadata. Unknown formats (including NV/YUV), a
stride/bytesPerRow other than width*4, and a nonzero rotation marker are rejected.

`detect()` synchronously snapshots only the downsampled RGB before returning,
so native buffer reuse cannot change a queued inference. All model operations
use the CPU backend. The lower-level tensor crop/landmark path avoids the
high-level `withFaceLandmarks()` canvas dependency. Concurrent inference rejects
with `NATIVE_FACE_BUSY`. `dispose()` is idempotent; pending work rejects with
`NATIVE_FACE_DISPOSED`, and tensors are released after that work settles. The
last engine instance releases the shared model tensors. `stats()` returns only
backend, selected configuration, memory counters and lifecycle booleans.

The CPU `pad-copy` specialization copies contiguous source blocks into the
already zero-filled output tensor. It preserves the original pad values exactly;
unhandled scalar/complex inputs fall back to the bundled implementation. Disable
it with `cpuOptimization:false` (`cpuOptimization:'none'` in returned diagnostics).
It does not change model arithmetic or relax quality/freshness gates. The trial
1x1 convolution specialization showed no stable benefit and was removed.

`detect()` additionally returns inputSize, maxSide, landmarkModel,
cpuOptimization and `timings.{snapshotMs,modelMs,tensorMs,detectorMs,landmarkMs,
totalMs}` in milliseconds. `elapsedMs` remains total elapsed wall time; source
frame age must still be calculated from the camera's frame arrival time.

Tests run the real models without document, Image, canvas, fetch, process,
Buffer or host Node imports. The local face fixture derives from the already
present reference SDK example `C3_180.jpg`; it is never transmitted. Tests check
actual positive face detection and 68 points, all supported byte layouts,
original-coordinate scaling, concurrent/disposed calls, native buffer reuse,
and zero tensors after final disposal. The pad kernel is tested bit-for-bit
against original CPU pad across ranks 1–4, dtypes and padding/constants; the
real full-model detection and all 68 points are identical with it enabled or
disabled. An aspect-preserving two-face composition must reach the actual
native-face-quality single-person rejection gate. Tests print timings/counts,
never the image or point coordinates.

Desktop no-DOM VM stage measurements on the same 540×960 portrait reference
(two samples per profile): old full/256/no specialization took about 934 ms
(detector 730, landmarks 189, snapshot 16). Tiny/160/pad-copy took about 302 ms
(detector 188, landmarks 99, snapshot 15), about 3.1× faster. Default model
tensors occupy 1082180 bytes (47 tensors), versus full's 2200644 bytes (68).

The original small face ratio was 0.208, below the H5 near gate of 0.30. With
an aspect-preserving crop targeting a valid larger face, tiny/128 can detect
one face and 68 points in about 254 ms; however, 128 misses one person in the
two-face regression while 160 detects both. Therefore 160 remains the default.
Reducing maxSide from 320 to 192 saved only about 24 ms on the eligible portrait;
320 is retained for landmark detail. These are desktop CPU observations, not
phone performance guarantees. Benchmark flags `--baseline-cpu`, `--max-side=192`
and `--input-sizes=160,128` support reproducing the separate comparisons. A zero
face result is explicitly flagged by `allSamplesDetected:false`, not a speed win.

## Explicit WebGL candidate

`utils/native-face-webgl.js` obtains an owned logical-layer Canvas using the
official synchronous `my.createOffscreenCanvas(32,32)` and
`getContext('webgl',{enable_flip_y_after_read_pixels:false})`.
The host app must enable its documented Skia capability. The adapter injects the
actual GL1 context into the bundled TFJS backend; it never forges HAS_WEBGL or
float-extension ENV flags. Initialization runs a 16×16 matrix product, a
four-channel convolution, numerical readback checks and observed real shader
draw/readPixels counters. A missing or readonly instrumentable native API fails
explicitly; the adapter does not report GPU success from API names alone.

The current device candidate is `LOCAL-FACE-20261102-07`. The preceding `03`
candidate executed four matrix shader draws on the iPhone, then failed the
strict numerical check with `MATMUL_FAILED`; it did not process camera frames.
Candidate `04` tried `default`, `packed_byte` (disables float downloads),
then `unpacked_byte` (also disables packed shaders). Each attempt runs the same
matrix/convolution/readback checks with the original numerical tolerance. A
failed attempt releases its backend and tracked GL objects before the next
attempt. Only numerical/readback failures permit another profile; missing APIs,
context loss and resource errors stop explicitly. There is no CPU fallback.

All three profiles failed on the device in `04`. The final unpacked byte
attempt observed five draws, one byte read, checksum 0 versus expected 2786,
244 mismatches and maximum absolute error 22. These aggregate values have an
all-zero signature; they do not by themselves prove every output value is zero
or identify the cause. Real camera frames were never processed.

Candidate `05` disables the host's extra readPixels flip using the documented
[Taobao context option](https://developer.alibaba.com/docs/doc.htm?articleId=121761&docType=1&treeId=829).
A canvas-height-based read-coordinate conversion with a smaller framebuffer can
produce zero output in an isolated experiment; that is a testable hypothesis,
not a confirmed description of the device implementation. Before TFJS setup,
the adapter checks a 2×2 RGBA8 framebuffer with clear/readback, an asymmetric
shader pattern, byte-texture sampling and float-texture sampling. Fixed nonzero
synthetic inputs and a prefilled destination sentinel distinguish an unchanged
typed array from incorrect GPU output. Any failed preflight stops initialization
and releases its graphics objects. Passing this preflight does not replace the
original matrix/convolution numerical checks, compatibility selection or model
warmup.

On the device, `05` passed clear/readback but failed the asymmetric shader with
three mismatches: expected checksum 2244, actual 2040. The actual checksum equals
the preceding clear color's checksum. This is consistent with no draw coverage,
but does not uniquely distinguish coverage, coordinates, typed-buffer uploads
or GL state. Real camera inference still did not start.

Candidate `06` explicitly initializes drawing state and clears to black before
each draw. It first checks a constant-color `drawArrays` pass and then an indexed
pass, followed by the original asymmetric shader and byte/float texture tests.
It verifies native strategies in order: `standard`, `buffer_copy` (tightly copy
a view's bytes into an ArrayBuffer for standard bufferData upload),
`framebuffer_y` (compensate viewport/scissor Y only for a bound framebuffer),
then `combined`. These are candidates selected by actual numerical checks, not
assumptions based on phone model. Only `VALUE_MISMATCH` permits another native
strategy; all other failures stop. All four native strategies share one context
and release each probe's objects before the next. The TF profiles still create
at most three contexts. A selected native strategy must pass every native test,
then the unchanged strict matrix/convolution tests and real-model warmup.
The source frame freshness limit remains 1500 ms. Device strategy selection,
compatibility, speed and automatic capture still need phone acceptance.

Candidate `06` failed on the device after its standard constant-array draw
passed: `nativeInitialStage=constant_indexed`. The later coordinate candidate's
black output is not the original failure. Candidate `07` first rebinds the
element buffer immediately before indexed draws. If the indexed result still
fails while the array draw passed, it verifies a four-vertex triangle strip as
an alternative. Strip probes report `constantQuad:passed` and leave
`constantIndexed:not_run`; `drawStrategy:quad_strip` identifies the physical
method without claiming the native indexed API passed.

Only the owned quad with uploaded indices `[0,1,2,2,1,3]` and exact
`TRIANGLES/6/UNSIGNED_SHORT/offset0` arguments can be converted. Upload bytes
are checked within typed-view bounds. Unknown buffers, different signatures,
numeric allocations and buffers modified with bufferSubData are refused;
deletion invalidates the record. This matches the pinned upstream
[quad data](https://raw.githubusercontent.com/tensorflow/tfjs/tfjs-core-v1.7.0/tfjs-core/src/backends/webgl/gpgpu_util.ts)
and retains the original tensor checks and model warmup. Native strategies can
each add a verified strip attempt, up to eight probes on the same context.
This is an experimental device compatibility path, not an assertion that the
Taobao indexed API is generally unsupported. Phone success and speed remain
unverified.

After the numerical checks, initialization loads the selected real models and
warms both the detector on synthetic portrait black pixels and the landmark
network on a synthetic tensor crop. Warmup results are discarded and never
count as user face/pose data. Only after the warmup does initialization return
`backend:'webgl'`, `gpu.selftest:'passed'` and `gpu.stage:'ready'`. The source
frame age and 1500 ms capture gate remain the camera's responsibility.

The returned `gpu` record contains fixed stage/code/profile enums, booleans,
counts and metrics computed only from the fixed synthetic selftest arrays:
requested, selftest, stage, code, contextLost, profile, attemptCount,
matmulDraws, convDraws, pixelReads, floatReads, byteReads, selftestMs and
warmupMs. Each synthetic matrix/convolution record reports actual length,
nonfinite/mismatch counts, maximum absolute error, first expected/actual values
and expected/actual checksums. The camera caches those synthetic metrics,
profile, attempts and read counts only during initialization; later detection
results cannot replace them in copied diagnostics. Errors use
`NATIVE_FACE_WEBGL_*` codes and the same safe record. Native error strings,
renderer strings, image pixels and coordinates do not enter that record. Each
detect checks actual context-loss state before and after model work; a lost
backend must be retired and rebuilt.

The startup record also includes the fixed-input `nativeProbe` stage/result,
sentinel-change and mismatch counts, expected/actual checksums, read return-kind
enum, GL error and framebuffer status. The record also preserves
`nativeInitialStage`/`nativeInitialCode`, selected `nativeStrategy`,
`constantArrays`/`constantIndexed`, optional vertex/index buffer byte counts and
`pixel0` through `pixel3` as big-endian RGBA uint32 values from fixed synthetic
input only. TFJS readback state is collected before
throwing for a numerical mismatch, so a failing value no longer hides a native
GL error. These fields are cached only during initialization and cannot be
replaced by real-camera inference results.

A shared live engine group admits one backend kind at a time. Initialization,
model loading, warmup and inference are queued; decoded weights are not reused
across a CPU/WebGL switch. Disposal waits for pending work, then releases model
tensors, backend programs/textures, the externally supplied GPGPUContext and all
remaining tracked GL objects including TFJS 1.7's vertex buffer/shader leftovers.
Native cleanup exceptions are isolated so they cannot preserve a busy lock or
replace the original inference error.

```
node --test taobao/test/native-face.test.cjs taobao/test/native-face-webgl.test.cjs
```

The WebGL tests run packaged modules with DOM/Node environment names hidden,
explicitly supply real headless WebGL1 and abort all network routes. They cover
numerical shader/readback, corrupted FLOAT readback followed by packed byte
readback, two failed contexts followed by unpacked byte readback, and all three
profiles failing closed. Both compatible paths run the real 68-point model and
are compared with CPU face boxes, confidence and landmark geometry. They also
cover context loss, initialization failure, shared engines, deferred disposal
and rebuilding, with zero final
tensors and tracked GL objects. Desktop SwiftShader proves the computation and
lifecycle paths; it does not establish native Taobao iPhone compatibility or
phone inference speed. Those still require the device's actual diagnostics.