Orso is a shared DataFrame library for Opteryx and Mabel.

Overview

Orso is not intended to compete with Polars or Pandas (or your favorite bear DataFrame technology), instead it is developed as a common layer for Mabel and Opteryx.

Key Use Cases:

• In Opteryx, Orso provides most of the database Cursor functionality
• In Mabel, Orso provides the data schema and validation functionality

Orso DataFrames are row-based, driven by their initial target use-case as the WAL for Mabel and Cursor for Opteryx. Each row in an Orso DataFrame can be quickly converted to a Tuple of values, a Dictionary, or a byte representation.

Installation

Install Orso from PyPI:

pip install orso

Quick Start

Creating a DataFrame

import orso

# Create from list of dictionaries
df = orso.DataFrame([
    {'name': 'Alice', 'age': 30, 'city': 'New York'},
    {'name': 'Bob', 'age': 25, 'city': 'San Francisco'},
    {'name': 'Charlie', 'age': 35, 'city': 'Chicago'}
])

print(f"Created DataFrame with {df.rowcount} rows and {df.columncount} columns")

Displaying Data

# Display the DataFrame
print(df.display())

# Convert to different formats
arrow_table = df.arrow()  # PyArrow Table
pandas_df = df.pandas()   # Pandas DataFrame

Working with Schema

# Access column names
print("Columns:", df.column_names)

# Access schema information  
print("Schema:", df.schema)

Converting Between Formats

# From PyArrow
import pyarrow as pa
arrow_table = pa.table({'x': [1, 2, 3], 'y': ['a', 'b', 'c']})
orso_df = orso.DataFrame.from_arrow(arrow_table)

# To Pandas
pandas_df = orso_df.pandas()

# To and from draken (opteryx-core's columnar engine), when draken is importable
morsel = orso_df.to_draken()
orso_df = orso.DataFrame.from_draken(morsel)

Features

• Lightweight: Minimal overhead for tabular data operations
• Row-based: Optimized for row-oriented operations
• Interoperable: Easy conversion to/from PyArrow, Pandas
• Schema-aware: Built-in data validation and type checking
• Fast serialization: Efficient conversion to bytes, tuples, and dictionaries

API Reference

DataFrame Class

The main DataFrame class provides the following key methods:

• DataFrame(dictionaries=None, *, rows=None, schema=None) - Constructor
• display(limit=5, colorize=True, show_types=True) - Pretty print the DataFrame
• arrow(size=None) - Convert to PyArrow Table
• pandas(size=None) - Convert to Pandas DataFrame
• from_arrow(tables) - Create DataFrame from PyArrow Table(s)
• fetchall() - Get all rows as list of Row objects
• collect() - Materialize the DataFrame
• append(other) - Append another DataFrame
• distinct() - Get unique rows

Properties

• rowcount - Number of rows
• columncount - Number of columns
• column_names - List of column names
• schema - Schema information

Development

Building from Source

# Clone the repository
git clone https://github.com/mabel-dev/orso.git
cd orso

# Install dependencies
pip install -r requirements.txt
pip install -r tests/requirements.txt

# Build the native extension (needs a C++20 compiler; nanobind comes from pip)
make compile

# Run tests
make test

Contributing

Orso is part of the Mabel ecosystem. Contributions are welcome! Please ensure:

1. All tests pass: make test
2. Code follows the project style: make lint
3. New features include appropriate tests
4. Documentation is updated for API changes

Performance Benchmarking

Orso has a micro-benchmark harness in benchmarks/ covering construction, append, Row, Arrow conversion, iteration, display and type parsing. Each case is warmed up and timed over several rounds; compare the minimum against a saved baseline:

# Run the full suite (or a subset with -k <substring>)
python benchmarks/bench.py

# Record a baseline, change code, rebuild the extensions, then compare
python benchmarks/bench.py -o benchmarks/baseline_0.0.246.json
python setup.py build_ext --inplace
python benchmarks/bench.py -c benchmarks/baseline_0.0.246.json

make bench and make bench-baseline wrap the same commands.

Building from source

make compile builds the native layer (orso._native, a nanobind extension over draken headers vendored into native/). There is no Cython any more. The vendored headers are copied from an opteryx-core checkout beside this one with make sync-draken and verified with make check-draken; never edit them in place. See DESIGN.md.

License

Orso is licensed under Apache 2.0 unless explicitly indicated otherwise.

Status

Orso is in beta. Beta means different things to different people, to us, being beta means:

• Interfaces are generally stable but may still have breaking changes
• Unit tests are not reliable enough to capture breaks to functionality
• Bugs are likely to exist in edge cases
• Code may not be tuned for performance

As such, we really don't recommend using Orso in critical applications.

Related Projects

• Opteryx - SQL query engine for data files
• Mabel - Data processing framework