ShengShu Technology Unveils Vidu S2, Bringing Real-Time Interaction and Editing to AI Video

SINGAPORE, Sept. 16, 2026 /PRNewswire/ -- On September 15, 2026, ShengShu Technology unveiled Vidu S2.

Vidu S2 comprises two models: Vidu S2-Avatar, a real-time interactive digital-character model designed for continuous interaction, and Vidu S2-Editing, a real-time video editing model for incoming video streams. Building on these two models, ShengShu Technology has also explored real-time spatial video generation and editing for VR headsets.

With Vidu S1, ShengShu Technology began exploring continuous voice control for video generation. Vidu S2-Avatar further expands the freedom of interaction: users can introduce a new reference image while a character is speaking, allowing the character to interact freely with the referenced content in real time.

Compared with Vidu S1, Vidu S2-Avatar increases real-time output resolution from 540p to 720p. It supports reference image updates at any point during generation and improves instruction following for large-scale movements such as dancing. Vidu S2-Editing enables real-time changes to the style, outfit, subject and background of an incoming video stream.

The end-to-end development of Vidu S2 was led by Jintao Zhang, a PhD student advised by Professor Jun Zhu and Head of Streaming Video Generation and Inference at ShengShu Technology.

Key Capabilities of Vidu S2

S2-Avatar: Updating Reference Images During Interaction

Users can upload a character image and interact with S2-Avatar through text or voice. In addition to facial expressions and poses during conversation, the model supports a broader range of body movements, including dancing.

Dynamic reference images are an important change in this generation. At any point during a live video stream, users can introduce an image of an object, outfit or background—for example, asking the character to pick up a cup shown in the image, change into a specified outfit, or enter a new scene.

These interactions also involve maintaining states after an action is completed. For example, the instruction "pick up the cup, then smile" requires the character to continue holding the cup while smiling. "Take off the hat and put it back on" requires a coherent sequence of taking off the hat, holding it, and putting it back on.

S2-Avatar uses visual feedback and subsequent prompt adjustments to improve the reliability of continuous actions and state preservation.

S2-Editing: Real-Time Editing That Follows the Input Video

S2-Editing receives a continuous incoming video stream and edits it based on text instructions and optional reference images. When a person raises an arm or turns around, or when the camera moves, the edited output needs to follow these changes.

The model currently supports four types of real-time editing tasks:

Real-Time Style Transfer. Based on a reference image, S2-Editing changes the visual style, including brushwork and color, while preserving the subject's silhouette, pose and scene layout.

Real-Time Outfit Change. S2-Editing applies an outfit from a reference image to the person in the video, while maintaining garment boundaries, material textures and natural occlusions during continuous motion.

Real-Time Subject Replacement. S2-Editing transfers the face, hairstyle, clothing and overall appearance of a reference subject while preserving the pose and motion of the source video.

Real-Time Background Replacement. S2-Editing replaces the environment based on a reference image while maintaining the spatial relationship between the subject and background throughout continuous motion.

Spatial Video: Further Exploration for VR Headsets

Spatial video presents slightly different images to the left and right eyes, allowing viewers to perceive the distance between subjects and backgrounds, object scale and scene depth.

S2-Avatar's real-time output can be connected to a spatial video conversion pipeline to generate synchronized left- and right-eye views. S2-Editing supports two approaches: first editing a standard monocular video and then converting it into spatial video, or directly editing an existing spatial video. Both approaches support the four editing tasks described above.

The resulting content can be streamed to VR headsets, allowing users to experience character interaction and video editing in spatial video.

Technical Approach Behind Vidu S2

Real-time interaction places several simultaneous requirements on video models: new instructions need to take effect promptly, actions need to remain coherent over time, characters and backgrounds need to remain stable, and computation needs to keep pace with continuously incoming video.

To address these requirements, ShengShu Technology made several improvements.

Data Processing: Balancing Motion Diversity and Visual Stability

Dance videos often involve camera movement. To preserve motion diversity, eligible footage with smooth camera motion undergoes background stabilization before being used for training.

Video annotations are also structured to describe actions, their timing and outcomes in chronological order, with segments divided according to event boundaries. This helps the model learn transitions between actions as well as state changes after receiving new instructions. Preference- and reward-based optimization is then applied to further improve motion naturalness, expressiveness and instruction following.

Self-Replay Forcing: Training on the Model's Own Generated History

Streaming video is generated segment by segment. Small errors in one segment can carry over into the next and gradually accumulate, resulting in identity drift, motion discontinuities or background changes.

ShengShu Technology introduces Self-Replay Forcing (SRF). The model first generates a longer video trajectory following the same process used during actual inference. Consecutive segments are then sampled from this self-generated trajectory, re-noised, and used for causal replay training with gradients propagated across segments. During replay, losses from later segments can affect earlier segments without requiring full backpropagation through the original long trajectory.

This brings training closer to the historical states the model encounters during actual inference and enables the model to learn the relationships between errors across adjacent segments. Both S2-Avatar and S2-Editing use this training method.

Visual Feedback: Checking Actions Before Adjusting the Next Step

S2-Avatar's Vision-Language Model Agent (VLM Agent) reads user instructions and images, generates prompts, and checks output video frames in chronological order.

If an action has been completed, subsequent prompts maintain the resulting pose and object state. If an action has only been partially completed or deviates from the instruction, the system adjusts subsequent instructions. When the result is ambiguous, the system does not automatically determine that the action has succeeded.

This feedback mechanism is used to improve the reliability of dynamic references and continuous interaction.

720p Real-Time Output: Separating Temporal Modeling from Detail Refinement

S2-Avatar uses a lightweight, single-step Refiner to restore high-resolution details from the backbone model's lower-resolution output. The backbone focuses on motion and long-range temporal modeling, while the Refiner focuses on local appearance, increasing real-time output resolution to 720p with relatively low additional computational overhead.

Training data is also filtered according to actual visual clarity, taking texture, edges and compression loss into account. Two videos with the same nominal 1080p resolution, for example, may provide significantly different amounts of usable detail depending on the degree of compression.

For inference, Vidu S2 adopts and extends selected techniques from TurboDiffusion and TurboServe, combining hybrid attention, low-bit computation and operator fusion to reduce computational overhead.

In the editing pipeline, the encoder, generation backbone, Refiner and decoder are scheduled along a shared timeline, allowing GPU resources to be reused across different stages to reduce resource idle time and end-to-end latency.

Evaluation Results

ShengShu Technology evaluated real-time digital character generation and video editing separately, supplementing public benchmarks with human preference evaluations.

On StreamAV-Bench, S2-Avatar achieved the best results across all nine metrics reported in Table 1 of the research paper, covering visual and audio quality, audio-visual alignment and synchronization, audio instruction fulfillment, and subject and background consistency.

For video editing, S2-Editing achieved an overall score of 4.26 in the joint OpenVE and RefVIE evaluation, outperforming the highest-scoring offline comparison model reported in the paper. On Sparkle-Bench, S2-Editing achieved an overall score of 3.74, with the best results across all reported metrics.

On the ViViD virtual try-on test set, S2-Editing achieved a VFID_I score of 9.9515. The metric measures the distributional difference between generated results and reference videos, with lower values indicating better performance.

In addition, an internal video editing evaluation used 150 paired cases. Outputs were randomly presented under identical source video, reference image and instruction conditions and compared by professionally trained and calibrated evaluators. S2-Editing achieved an average consistency score of 3.56 and received a higher share of evaluator preferences across overall quality, video quality, temporal consistency and semantic adherence.

These comparison results correspond to the models, datasets and evaluation settings reported in the paper.

Next Steps

Spatial video remains at an exploratory stage. The current approach primarily uses fixed viewpoints. Maintaining clear visual quality in VR headsets while further reducing latency is one of the challenges ShengShu Technology plans to address next. In real-time passthrough scenarios in particular, latency between head movement and the displayed image can directly affect the experience.

ShengShu Technology will also continue exploring panoramic spatial video, allowing users to turn their heads and view a more complete generated scene.

For Vidu S, continuous character interaction, video editing and spatial experiences represent the next concrete directions along this technical path.

Experience Vidu S2: https://vidu.com/vidu-stream

API Platform: http://platform.vidu.com/vidu-stream/doc

Technical Report: https://arxiv.org/pdf/2609.11638

SOURCE ShengShu Technology