Reading from and Writing to GooseFS#

Daft supports reading and writing data to GooseFS — a distributed caching file system — and understands natively the URL protocol goosefs:// as referring to data that resides in a GooseFS cluster.

Under the hood, Daft speaks GooseFS's native gRPC protocol via OpenDAL's services-goosefs backend. This means that no HDFS/JVM gateway is required — Daft talks to GooseFS masters and workers directly.

URL Format#

URLs to data in GooseFS come in the form:

1

goosefs://{MASTER_HOST}:{MASTER_PORT}/{PATH}

• MASTER_HOST:MASTER_PORT identifies a GooseFS master. If no explicit master_addr is provided in the config, Daft will use this URL authority as the master address.
• PATH is the absolute path to the file or directory inside the GooseFS namespace.

Authorization/Authentication#

GooseFS supports two authentication modes: simple (username-based, the default) and nosasl (no authentication). Anonymous access forces nosasl.

Rely on Environment#

You can configure Daft to automatically discover GooseFS settings from environment variables. Daft supports the following variables when [GooseFSConfig.from_env()][daft.io.GooseFSConfig] is used:

Please be aware that when doing so in a distributed environment such as Ray, Daft will pick these values up from worker machines and thus each worker machine needs to be appropriately provisioned.

If instead you wish to have Daft use configuration from the "driver", you may wish to manually specify your credentials.

Manually specify credentials#

You can pass a [GooseFSConfig][daft.io.GooseFSConfig] into your Daft I/O function calls using a daft.io.IOConfig config object.

daft.set_planning_config is a convenient way to set your daft.io.IOConfig as the default config to use on any subsequent Daft method calls.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

import daft
from daft.io import IOConfig, GooseFSConfig

io_config = IOConfig(
    goosefs=GooseFSConfig(
        master_addr="10.0.0.1:9200",
        auth_type="simple",
        auth_username="alice",
    )
)

# Globally set the default IOConfig for any subsequent I/O calls
daft.set_planning_config(default_io_config=io_config)

# Perform some I/O operation
df = daft.read_parquet("goosefs://10.0.0.1:9200/my_path/**/*")

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

import daft
from daft.io import IOConfig, GooseFSConfig

io_config = IOConfig(
    goosefs=GooseFSConfig(
        # Comma-separated HA master list; the URL authority is ignored.
        master_addr="10.0.0.1:9200,10.0.0.2:9200,10.0.0.3:9200",
        auth_type="simple",
        auth_username="alice",
    )
)

df = daft.read_parquet("goosefs://ha/my_path/**/*", io_config=io_config)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

import daft
from daft.io import IOConfig, GooseFSConfig

# Anonymous mode forces `nosasl` and skips credential forwarding.
io_config = IOConfig(goosefs=GooseFSConfig(anonymous=True))

df = daft.read_parquet(
    "goosefs://10.0.0.1:9200/public/**/*",
    io_config=io_config,
)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

import daft
from daft.io import IOConfig, GooseFSConfig

io_config = IOConfig(
    goosefs=GooseFSConfig(
        master_addr="10.0.0.1:9200",
        root="/warehouse",
        # One of: must_cache, cache_through, through, async_through
        write_type="cache_through",
    )
)

df = daft.read_parquet("goosefs://10.0.0.1:9200/table/**/*", io_config=io_config)

Alternatively, Daft supports overriding the default IOConfig per-operation by passing it into the io_config= keyword argument. This is extremely flexible as you can pass a different [GooseFSConfig][daft.io.GooseFSConfig] per function call if you wish!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

import daft
from daft.io import IOConfig, GooseFSConfig

io_config = IOConfig(
    goosefs=GooseFSConfig(
        master_addr="10.0.0.1:9200",
        auth_username="alice",
    )
)

# Perform some I/O operation but override the IOConfig
df2 = daft.read_csv(
    "goosefs://10.0.0.1:9200/my_other_path/**/*",
    io_config=io_config,
)

Writing Data#

Daft supports writing data to GooseFS using the same GooseFSConfig:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

import daft
from daft.io import IOConfig, GooseFSConfig

io_config = IOConfig(
    goosefs=GooseFSConfig(
        master_addr="10.0.0.1:9200",
        auth_username="alice",
        # Persist writes through to the underlying UFS.
        write_type="cache_through",
    )
)

df = daft.from_pydict({"a": [1, 2, 3], "b": ["x", "y", "z"]})

# Write to GooseFS as Parquet
df.write_parquet("goosefs://10.0.0.1:9200/output/", io_config=io_config)

Configuration Options#

The [GooseFSConfig][daft.io.GooseFSConfig] object supports the following options:

Supported Operations#

Daft supports the following operations with GooseFS:

• Read: read_parquet, read_csv, read_json, and other file readers
• Write: write_parquet, write_csv, write_json
• List: Listing objects with glob pattern matching
• Delete: Deleting objects