Working with Files and URLs in Daft#

Daft provides powerful capabilities for working with URLs, file paths, and remote storage systems.

Whether you're loading data from local files, cloud storage, or the web, Daft's URL and file handling makes it seamless to work with distributed data sources. Daft supports working with:

Local file paths: file:///path/to/file, /path/to/file
S3: s3://bucket/path, s3a://bucket/path, s3n://bucket/path
GCS: gs://bucket/path
Azure: az://container/path, abfs://container/path, abfss://container/path
HTTP/HTTPS URLs: http://example.com/path, https://example.com/path
Hugging Face datasets: hf://dataset/name
Unity Catalog volumes: vol+dbfs:/Volumes/unity/path

Using file discovery with optimized distributed reads#

daft.from_glob_path helps discover and size files, accepting wildcards and lists of paths. When paired with daft.functions.download, the two functions enable optimized distributed reads of binary data from storage. This is ideal when your data will fit into memory or when you need the entire file content at once.

🐍 Python⚙️ SQL

df = daft.from_pydict({
    "urls": [
        "https://www.google.com",
        "s3://daft-oss-public-data/open-images/validation-images/0001eeaf4aed83f9.jpg",
    ],
})
df = df.with_column("data", df["urls"].download())
df.collect()

df = daft.from_pydict({
    "urls": [
        "https://www.google.com",
        "s3://daft-oss-public-data/open-images/validation-images/0001eeaf4aed83f9.jpg",
    ],
})
df = daft.sql("""
    SELECT
        urls,
        url_download(urls) AS data
    FROM df
""")
df.collect()

Output
╭────────────────────────────────┬────────────────────────────────╮
│ urls                           ┆ data                           │
│ ---                            ┆ ---                            │
│ Utf8                           ┆ Binary                         │
╞════════════════════════════════╪════════════════════════════════╡
│ https://www.google.com         ┆ b"<!doctype html><html itemsc… │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ s3://daft-oss-public-data/open-im… ┆ b"\xff\xd8\xff\xe0\x00\x10JFI… │
╰────────────────────────────────┴────────────────────────────────╯

(Showing first 2 of 2 rows)

This works well for URLs which are HTTP paths to non-HTML files (e.g. jpeg), local filepaths or even paths to a file in an object store such as AWS S3 as well!

The `daft.File` Datatype#

daft.File is particularly useful for working with large files that don't fit in memory or when you only need to access specific portions of a file. This is a common use case when working with audio, video, or image data where loading the entire object is prohibitive. The daft.File Type is subclassed by the daft.AudioFile, daft.ImageFile, and daft.VideoFile types which streamline common operations. It provides a pythonic file-like interface with random access capabilities:

🐍 Python

import daft
from daft.functions import file
from daft.io import IOConfig, S3Config

io_config = IOConfig(s3=S3Config(anonymous=True))

df = daft.from_pydict(
    {
        "urls": [
            "https://www.google.com",
            "s3://daft-oss-public-data/open-images/validation-images/0001eeaf4aed83f9.jpg",
        ],
    }
)

@daft.func
def detect_file_type(file: daft.File) -> str:
    # Read just the first 12 bytes to identify file type
    with file.open() as f:
        header = f.read(12)

    # Common file signatures (magic numbers)
    if header.startswith(b"\xff\xd8\xff"):
        return "JPEG"
    elif header.startswith(b"\x89PNG\r\n\x1a\n"):
        return "PNG"
    elif header.startswith(b"GIF87a") or header.startswith(b"GIF89a"):
        return "GIF"
    elif header.startswith(b"<!") or header.startswith(b"<html"):
        return "HTML"
    elif header.startswith(b"HTTP/"):
        return "HTTP"
    else:
        return None

df = df.with_column(
    "file_type",
    detect_file_type(file(df["urls"], io_config=io_config))
)

df.collect()

Output
╭────────────────────────────────┬───────────╮
│ urls                           ┆ file_type │
│ ---                            ┆ ---       │
│ Utf8                           ┆ Utf8      │
╞════════════════════════════════╪═══════════╡
│ https://www.google.com         ┆ HTML      │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
│ s3://daft-oss-public-data/open-im… ┆ JPEG      │
╰────────────────────────────────┴───────────╯

(Showing first 2 of 2 rows)

The daft.File datatype provides first-class support for handling file data across local and remote storage, enabling seamless file operations in distributed environments.

While the Python classes provide the interface, the actual implementation lives in Rust-based PyDaftFile, which maintains optimized backends for different storage types:

Local filesystem access
Remote object stores with buffered reading

This architecture allows us to implement storage-specific optimizations (like network buffering for S3 or HTTP) while presenting a consistent interface.

Core Design Principles#

daft.File works both within dataframes and as standalone objects
Optimized backend readers for different sources (buffered network access, etc.)
Consistent API regardless of storage location

daft.File mirrors the file interface in Python, but is optimized for distributed computing. Due to its lazy nature, daft.File does not read the file into memory until it is needed. To enforce this pattern, daft.File must be used inside a context manager like with file.open() as f: This works within a daft.func or daft.cls user-defined functions or in native Python code.

Basic Usage#

import daft

@daft.func
def read_header(f: daft.File) -> bytes:
    with f.open() as fh:
        return fh.read(16)

df = (
    daft.from_glob_path("hf://datasets/Eventual-Inc/sample-files/**")
    .with_column("file", daft.functions.file(daft.col("path")))
    .with_column("header", read_header(daft.col("file")))
    .select("path", "size", "file", "header")
)

df.show(5)

Output
╭────────────────────────────────┬─────────┬────────────────────────────────┬────────────────────────────────╮
│ path                           ┆ size    ┆ file                           ┆ header                         │
│ ---                            ┆ ---     ┆ ---                            ┆ ---                            │
│ String                         ┆ Int64   ┆ File[Unknown]                  ┆ Binary                         │
╞════════════════════════════════╪═════════╪════════════════════════════════╪════════════════════════════════╡
│ hf://datasets/Eventual-Inc/sa… ┆ 2534    ┆ Unknown(path: hf://datasets/E… ┆ b"*.7z filter=lfs "            │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ hf://datasets/Eventual-Inc/sa… ┆ 31      ┆ Unknown(path: hf://datasets/E… ┆ b"---\r\nlicense: ap"          │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ hf://datasets/Eventual-Inc/sa… ┆ 822924  ┆ Unknown(path: hf://datasets/E… ┆ b"\xff\xf3\x88\xc4\x00\x00\x0… │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ hf://datasets/Eventual-Inc/sa… ┆ 618408  ┆ Unknown(path: hf://datasets/E… ┆ b"\xff\xf3\x88\xc4\x00\x00\x0… │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ hf://datasets/Eventual-Inc/sa… ┆ 1190736 ┆ Unknown(path: hf://datasets/E… ┆ b"\xff\xf3\x88\xc4\x00\x00\x0… │
╰────────────────────────────────┴─────────┴────────────────────────────────┴────────────────────────────────╯

(Showing first 5 rows)

Byte-Range Reads#

When working with files that pack multiple records into a single blob (e.g. Paimon blob files), you can specify offset and length to read only a specific byte range instead of the entire file. Both parameters must be provided together.

import daft

# Read bytes 1024–2047 from a large blob file
f = daft.File("s3://bucket/data.blob", offset=1024, length=1024)
with f.open() as fh:
    data = fh.read()  # returns exactly 1024 bytes

This also works inside UDFs:

@daft.func
def read_record(file: daft.File) -> bytes:
    with file.open() as f:
        return f.read()

# Construct File references with per-row offsets
df = daft.from_pydict({
    "url": ["s3://bucket/blob"] * 3,
    "offset": [0, 100, 200],
    "length": [100, 100, 50],
})

Using daft.File to read code and walk the AST#

Since daft.File works with any file type with a read method, we can use it to read code and walk the AST for use cases like extracting functions and their signatures for codebase intelligence.

import daft

@daft.func(
    return_dtype=daft.DataType.list(
        daft.DataType.struct(
            {
                "name": daft.DataType.string(),
                "signature": daft.DataType.string(),
                "docstring": daft.DataType.string(),
                "start_line": daft.DataType.int64(),
                "end_line": daft.DataType.int64(),
            }
        )
    )
)
def extract_functions(file: daft.File):
    """Extract all function definitions from a Python file."""
    import ast

    with file.open() as f:
        file_content = f.read().decode("utf-8")

    tree = ast.parse(file_content)
    results = []

    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
            signature = f"def {node.name}({ast.unparse(node.args)})"
            if node.returns:
                signature += f" -> {ast.unparse(node.returns)}"

            results.append({
                "name": node.name,
                "signature": signature,
                "docstring": ast.get_docstring(node),
                "start_line": node.lineno,
                "end_line": node.end_lineno,
            })

    return results


if __name__ == "__main__":
    from daft.functions import file, unnest

    # Discover Python files
    df = (
        daft.from_glob_path("~/git/Daft/daft/functions/**/*.py") # Add your own path here
        .with_column("file", file(daft.col("path")))
        .with_column("functions", extract_functions(daft.col("file")))
        .explode("functions")
        .select(daft.col("path"), unnest(daft.col("functions")))
    )

    df.show(3) # Show the first 3 rows of the dataframe

Output
╭────────────────────────────────┬─────────────────────────────┬────────────────────────────────┬────────────────────────────────┬────────────╮
│ path                           ┆ name                        ┆ signature                      ┆ docstring                      ┆ start_line │
│ ---                            ┆ ---                         ┆ ---                            ┆ ---                            ┆ ---        │
│ String                         ┆ String                      ┆ String                         ┆ String                         ┆ Int64      │
╞════════════════════════════════╪═════════════════════════════╪════════════════════════════════╪════════════════════════════════╪════════════╡
│ file:///Users/myusername007/g… ┆ monotonically_increasing_id ┆ def monotonically_increasing_… ┆ Generates a column of monoton… ┆ 14         │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤
│ file:///Users/myusername007/g… ┆ eq_null_safe                ┆ def eq_null_safe(left: Expre…  ┆ Performs a null-safe equality… ┆ 52         │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤
│ file:///Users/myusername007/g… ┆ cast                        ┆ def cast(expr: Expression, dt… ┆ Casts an expression to the gi… ┆ 68         │
╰────────────────────────────────┴─────────────────────────────┴────────────────────────────────┴────────────────────────────────┴────────────╯

(Showing first 3 rows)