Website | Docs | Playground | Benchmarks | Paper (SOSP'25)

Pie runs WebAssembly programs—inferlets—directly next to the model. An inferlet drives the forward pass, owns its state, and compiles custom epilogues to GPU kernels, so decoding, sampling, caching, and agent loops are written per application in Python, JavaScript, or Rust.

Models are programs too, written in Poem across language, vision, image, and video. Everything ships as one ~60 MB binary with no Python or PyTorch runtime, running on CUDA, ROCm, Metal, Vulkan, XLA (TPUs), and WebGPU.

Install

curl -fsSL https://pie-project.org/install.sh | bash        # Linux, macOS, WSL
irm https://pie-project.org/install.ps1 | iex               # Windows PowerShell

Import a model from Hugging Face:

pie config init
pie model import Qwen/Qwen3.5-0.8B

See the install guide for manual archives and GPU flavors, or run Pie in your browser on the Playground.

Write an inferlet

Greedy text completion in Python (examples/quickstart-py), one forward pass per token (also in JavaScript and Rust):

from inferlet import chat, model
from inferlet.eta import ForwardPass, Pipeline, StateGeometry, WorkingSet, intrinsics, reduce_argmax


async def main(input: dict) -> dict:
    tokens = chat.prefix() + model.encode(input.get("prompt", "The capital of France is"))
    max_tokens = int(input.get("max_tokens", 8))
    ws = WorkingSet()  # the sequence's state, whatever the model keeps
    pipe = Pipeline()

    done = 0  # tokens already in the working set
    for _ in range(max_tokens):
        fwd = ForwardPass()
        fwd.embed(tokens[done:])                        # the new tokens
        fwd.state([ws], StateGeometry.folding())        # appended, and final
        out = fwd.epilogue(lambda: reduce_argmax(intrinsics.output(0, "logits")))  # runs on the device
        pipe.submit(fwd)
        done = len(tokens)
        tokens.append(await out)
    pipe.close()

    return {"text": model.decode(tokens[-max_tokens:])}

epilogue is traced once and compiled into a GPU kernel; the host only sees the token it publishes. More in examples/: device loop-carried decode, speculative decoding, beam search, KV eviction and prefix sharing, constrained decoding, custom samplers, watermarking, and media generation.

Run & serve

Run an inferlet one-shot, or submit it to a running pie serve with pie-client:

pie run main.py -- --prompt "The capital of France is"
pie-client submit --path main.py -- --prompt "The capital of France is"

pie serve also speaks the OpenAI, Anthropic, and Gemini APIs on the same port as built-in inferlets:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="unused")
print(client.chat.completions.create(model="default", messages=[{"role": "user", "content": "Hi"}]).choices[0].message.content)

To embed in a process, use pie-server (Python) or @pie-project/server (Node and WebGPU browsers).

Build from source

cargo build --release -p pie --bin pie --features metal   # or: cuda, vulkan, wgpu, rocm, xla

Community & License

Discord · Issues · Discussions · Contributing

Apache License 2.0 (NOTICE).