Alibaba's Qwen Business Unit, together with ShanghaiTech University, HKUST, the Institute of Computing Technology, and Southeast University, published the AVA-Encoder framework: it translates a finished film into a text-based Film Knowledge Graph and reconstructs the video from this graph, with the reconstruction difference serving as the learning signal for the encoder. At the time of publication, the work exists as a paper and a project page; executable code is not yet available.

image
image

What happened

The paper AVA-Encoder: Towards Agent-Native Video Representation Learning (arXiv:2608.12313) was submitted to arXiv on August 12, 2026, with version 2 released on August 18. The system compresses a film into a text knowledge graph with a Story–Event–Shot hierarchy and typed nodes for characters, scenes, objects, style, camera, and audio, then reconstructs the video from the graph using a fixed decoder. The reconstruction difference serves as the optimization signal for a two-cycle scheme: the outer cycle trains the encoder policy for shot and keyframe modes before deployment, while an optional inner cycle refines the graph for a specific video during inference. In the benchmark, AVA-Encoder achieves 49.0% overall reconstruction fidelity versus 28.3% for the strongest external baseline, soap2soap — a gain of 20.7 percentage points, or 73.1% in relative terms.

Context

Video agents have a structural gap: to plan the creation of new content, they need structured representations of finished films, but such a layer between raw video and generation has not existed until now, and there was no objective way to measure how much information about a film a text description actually preserves. The scientific core of the work is a self-learning loop in which the encoder policy is trained on the reconstruction signal before deployment and, if necessary, refined for a specific video during inference, without updating foundation model weights. Against this backdrop, the video–graph–video scheme is proposed as a new product primitive for the video-agent market: the graph becomes a structured intermediate representation that agents can read and pass to one another.

Why this matters for the industry

The main contribution of the work for the industry is the evaluation methodology, not generation quality: the fixed decoder and reconstruction difference act as an intrinsic metric of how much cinematic information the representation preserves, and the automatic metrics were validated by blind evaluation by two experts — agreement in 710 out of 730 comparisons, 97.3%. The pseudo-trained policy yields a small 1.4-percentage-point gain over manual prompt tuning (45.8% versus 44.4%), while the 74.3% token savings in the system prompt in shot mode and 70.1% in keyframe mode are practically significant for agentic pipelines. The authors also announce the release of the Film KG Dataset — structured texts for tens of thousands of shots, the first dataset of its kind for training and evaluating agentic video creation, which, upon actual release, could become a standard training set and benchmark for the field.

Why this matters for users

For readers, the specific video–editable graph–video mechanism is of interest: a film's knowledge graph can be semantically edited — changing a character, style, or lighting — and dependent scene elements update according to the graph's links, and the resulting graph can be used as input for other video agents. At present, the value of the work for readers is conceptual: the scheme should be studied via the arXiv paper and project page; it will serve as a blueprint for products at the intersection of video and agents, but the results cannot yet be applied directly.

What is still unknown / limitations

Several key points remain unconfirmed. In the HBDYW/AVA-Encoder GitHub repository, at the time of checking, only the project site is present (16 commits, 8 stars): tools, system prompts, and code are marked as available but have not actually been added to the repository, so the system cannot be reproduced, run, or integrated, and there is no API or pricing. The absolute result of 49.0% reconstruction fidelity means that most information is lost when a film is compressed into a text graph, and the relative gain is calculated only against the strongest of the listed external baselines, soap2soap, so the set of baselines is limited. Whether the Film KG Dataset and code will actually be released is still unknown.

Sources

Author

Look at AI, editorial team