File Types
The File DataType provides first-class support for handling file data across local and remote storage, enabling seamless file operations in distributed environments.
File #
File(url: str, io_config: IOConfig | None = None, media_type: MediaType = unknown(), position: int | None = None, size: int | None = None, offset: int | None = None, length: int | None = None)
A file-like object for working with file contents in Daft.
This is an abstract base class that provides a standard file interface compatible with Python's file protocol.
The File object can be used with most Python libraries that accept file-like objects, and implements the standard read/seek/tell interface. Files are read-only in the current implementation.
Examples:
1 2 3 4 5 6 7 8 9 10 11 | |
Methods:
| Name | Description |
|---|---|
as_audio | Convert to AudioFile if this file contains audio data. |
as_hdf5 | Convert to Hdf5File if this file contains HDF5 data. |
as_image | Convert to ImageFile if this file contains image data. |
as_video | Convert to VideoFile if this file contains video data. |
exists | Whether the file exists at its path or URL. |
is_audio | |
is_hdf5 | |
is_image | |
is_video | |
isatty | |
mime_type | Attempts to determine the MIME type of the file. |
open | |
readable | |
seekable | |
size | The size of the file in bytes, derived from the underlying file. |
to_tempfile | Create a temporary file with the contents of this file. |
writable | |
Attributes:
| Name | Type | Description |
|---|---|---|
length | int | None | Deprecated alias for the byte-range read window size, or None for full-file reads. |
name | str | The filename (basename) extracted from the file path or URL. |
offset | int | None | Deprecated alias for |
path | str | The full path or URL of the file. |
position | int | None | The starting byte position for range reads, or None for full-file reads. |
Source code in daft/file/file.py
59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 | |
length #
length: int | None
Deprecated alias for the byte-range read window size, or None for full-file reads.
Note: this returns the requested range size (caller intent), not the derived file size. Use File.size() for the actual file size.
name #
name: str
The filename (basename) extracted from the file path or URL.
Returns:
| Name | Type | Description |
|---|---|---|
str | str | The filename without directory components. |
Example
import daft f = daft.File("s3://bucket/path/to/data.csv") f.name 'data.csv'
offset #
offset: int | None
Deprecated alias for position. The byte offset for range reads, or None for full-file reads.
path #
path: str
The full path or URL of the file.
Returns:
| Name | Type | Description |
|---|---|---|
str | str | The file path or URL. |
Example
import daft f = daft.File("s3://bucket/path/to/data.csv") f.path 's3://bucket/path/to/data.csv'
position #
position: int | None
The starting byte position for range reads, or None for full-file reads.
as_audio #
as_audio() -> AudioFile
Convert to AudioFile if this file contains audio data.
Source code in daft/file/file.py
257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 | |
as_hdf5 #
as_hdf5() -> Hdf5File
Convert to Hdf5File if this file contains HDF5 data.
Source code in daft/file/file.py
293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 | |
as_image #
as_image() -> ImageFile
Convert to ImageFile if this file contains image data.
Source code in daft/file/file.py
276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 | |
as_video #
as_video() -> VideoFile
Convert to VideoFile if this file contains video data.
Source code in daft/file/file.py
241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 | |
exists #
exists() -> bool
Whether the file exists at its path or URL.
Source code in daft/file/file.py
172 173 174 | |
is_audio #
is_audio() -> bool
Source code in daft/file/file.py
225 226 227 228 229 | |
is_hdf5 #
is_hdf5() -> bool
Source code in daft/file/file.py
237 238 239 | |
is_image #
is_image() -> bool
Source code in daft/file/file.py
231 232 233 234 235 | |
is_video #
is_video() -> bool
Source code in daft/file/file.py
219 220 221 222 223 | |
isatty #
isatty() -> bool
Source code in daft/file/file.py
106 107 | |
mime_type #
mime_type() -> str
Attempts to determine the MIME type of the file.
If the MIME type is undetectable, returns 'application/octet-stream'.
Source code in daft/file/file.py
176 177 178 179 180 181 182 183 184 185 186 187 188 189 | |
open #
open(buffer_size: int | None = None) -> PyDaftFile
Source code in daft/file/file.py
89 90 91 92 | |
readable #
readable() -> bool
Source code in daft/file/file.py
97 98 | |
seekable #
seekable() -> bool
Source code in daft/file/file.py
103 104 | |
size #
size() -> int
The size of the file in bytes, derived from the underlying file.
Source code in daft/file/file.py
168 169 170 | |
to_tempfile #
to_tempfile(buffer_size: int = BUFFER_COPY) -> _TemporaryFileWrapper[bytes]
Create a temporary file with the contents of this file.
Returns:
| Type | Description |
|---|---|
_TemporaryFileWrapper[bytes] | _TemporaryFileWrapper[bytes]: The temporary file object. |
The temporary file will be automatically deleted when the returned context manager is closed.
It's important to note that to_tempfile closes the original file object, so it CANNOT be used after calling this method.
Source code in daft/file/file.py
191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 | |
writable #
writable() -> bool
Source code in daft/file/file.py
100 101 | |
ImageFile #
ImageFile(url: str, io_config: IOConfig | None = None)
An image-specific file interface that provides image operations.
Methods:
| Name | Description |
|---|---|
decode | Decode the image file into a PIL Image. |
metadata | Extract basic image metadata from file headers. |
Source code in daft/file/image.py
25 26 27 28 29 30 31 32 33 34 35 36 | |
decode #
decode(mode: str | None = None, buffer_size: int | None = BUFFER_COPY) -> Image
Decode the image file into a PIL Image.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
mode | str | None | Optional image mode to convert to (e.g. "RGB", "RGBA", "L"). | None |
buffer_size | int | None | Read buffer size for full image decode. Defaults to 1 MiB. | BUFFER_COPY |
Returns:
| Type | Description |
|---|---|
Image | PIL.Image.Image: The decoded image. |
Source code in daft/file/image.py
56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | |
metadata #
metadata() -> ImageMetadata
Extract basic image metadata from file headers.
PIL's Image.open() is lazy -- it reads only the file header to determine dimensions, format, and mode without decoding pixel data.
Returns:
| Name | Type | Description |
|---|---|---|
ImageMetadata | ImageMetadata | Image metadata containing width, height, format, mode. |
Source code in daft/file/image.py
38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | |
AudioFile #
AudioFile(url: str, io_config: IOConfig | None = None)
An audio-specific file interface that provides audio operations.
Methods:
| Name | Description |
|---|---|
metadata | Extract basic audio metadata from container headers. |
resample | Resample the audio file to the given sample rate. |
to_numpy | Convert the audio file to a numpy array. |
Source code in daft/file/audio.py
25 26 27 28 29 30 31 32 33 34 | |
metadata #
metadata() -> AudioMetadata
Extract basic audio metadata from container headers.
Returns:
| Name | Type | Description |
|---|---|---|
AudioMetadata | AudioMetadata | Audio metadata object containing: - sample_rate: int - The sample rate of the audio file - channels: int - The number of channels in the audio file - frames: int - The number of frames in the audio file - format: str - The format of the audio file - subtype: str | None - The subtype of the audio file |
Source code in daft/file/audio.py
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | |
resample #
resample(sample_rate: int, buffer_size: int = BUFFER_COPY) -> ndarray[Any, dtype[float64]]
Resample the audio file to the given sample rate.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample_rate | int | The new sample rate. | required |
buffer_size | int | The buffer size to use for the temporary file. | BUFFER_COPY |
Returns:
| Name | Type | Description |
|---|---|---|
AudioFile | ndarray[Any, dtype[float64]] | The resampled audio file. |
Source code in daft/file/audio.py
70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 | |
to_numpy #
to_numpy(buffer_size: int = BUFFER_COPY) -> ndarray[Any, dtype[float64]]
Convert the audio file to a numpy array.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
buffer_size | int | The buffer size to use for the temporary file. | BUFFER_COPY |
Returns:
| Type | Description |
|---|---|
ndarray[Any, dtype[float64]] | np.ndarray[Any, Any]: The audio data as a numpy array. |
Source code in daft/file/audio.py
56 57 58 59 60 61 62 63 64 65 66 67 68 | |
VideoFile #
VideoFile(url: str, io_config: IOConfig | None = None)
A video-specific file interface that provides video operations.
Methods:
| Name | Description |
|---|---|
frames | Lazy iterator of all decoded frames with metadata within time range. |
get_frame_by_idx | |
keyframes | Lazy iterator of keyframes as PIL Images within time range. |
metadata | Extract basic video metadata from container headers. |
Source code in daft/file/video.py
29 30 31 32 33 34 35 36 37 38 39 | |
frames #
frames(start_time: float = 0, end_time: float | None = None, width: int | None = None, height: int | None = None, is_key_frame: bool | None = None, sample_interval_seconds: float | None = None, buffer_size: int = BUFFER_COPY) -> Iterator[VideoFrameData]
Lazy iterator of all decoded frames with metadata within time range.
Mirrors the per-frame schema of daft.read_video_frames().
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
start_time | float | Start of the time range in seconds. Defaults to 0. | 0 |
end_time | float | None | End of the time range in seconds. Defaults to None (end of video). | None |
width | int | None | Optional target width for resizing frames. Must be provided with | None |
height | int | None | Optional target height for resizing frames. Must be provided with | None |
is_key_frame | bool | None | If True, emit only keyframes. If False, emit only non-keyframes. If None, emit all decoded frames. | None |
sample_interval_seconds | float | None | If provided and > 0, sample frames at approximately this time interval in seconds based on | None |
Yields:
| Type | Description |
|---|---|
VideoFrameData | VideoFrameData dicts with keys: frame_index, frame_time, frame_time_base, |
VideoFrameData | frame_pts, frame_dts, frame_duration, is_key_frame, data (PIL Image). |
Source code in daft/file/video.py
106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 | |
get_frame_by_idx #
get_frame_by_idx(idx: int, buffer_size: int = BUFFER_COPY) -> Image
Source code in daft/file/video.py
226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 | |
keyframes #
keyframes(start_time: float = 0, end_time: float | None = None) -> Iterator[Image]
Lazy iterator of keyframes as PIL Images within time range.
Source code in daft/file/video.py
101 102 103 104 | |
metadata #
metadata(buffer_size: int = BUFFER_METADATA) -> VideoMetadata
Extract basic video metadata from container headers.
Returns:
| Name | Type | Description |
|---|---|---|
VideoMetadata | VideoMetadata | Video metadata object containing width, height, fps, frame_count, time_base, keyframe_pts, keyframe_indices |
Source code in daft/file/video.py
41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 | |
Hdf5File #
Hdf5File(url: str, io_config: IOConfig | None = None)
Represents an HDF5 file backed by Daft file IO.
This class keeps File.open() as the inherited raw byte-stream API and provides HDF5-specific helpers that mirror common h5py File and Group operations. HDF5 access uses a smaller default file buffer than the generic File type because h5py performs frequent small reads after seeks while traversing metadata and chunk indexes.
Methods:
| Name | Description |
|---|---|
attrs | Return attributes attached to an HDF5 object. |
keys | Return member names directly under an HDF5 group. |
metadata | Collect object metadata below an HDF5 group. |
open | |
read | Read one or more HDF5 datasets into NumPy arrays. |
visit | Recursively visit object names below an HDF5 group. |
Source code in daft/file/hdf5.py
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | |
attrs #
attrs(h5path: str = '/') -> dict[str, Any]
Return attributes attached to an HDF5 object.
Mirrors h5py's <object>.attrs dictionary-style interface and materializes the attributes as a plain Python dictionary.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
h5path | str | Group or dataset path. Defaults to the root group | '/' |
Returns:
| Type | Description |
|---|---|
dict[str, Any] | A dictionary of attribute names to values. Values follow h5py's |
dict[str, Any] | normal conversion rules, such as NumPy scalars or arrays. |
Source code in daft/file/hdf5.py
138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 | |
keys #
keys(group: str = '/') -> list[str]
Return member names directly under an HDF5 group.
Mirrors h5py Group.keys(), but returns a concrete list[str] instead of a view object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
group | str | Group path whose immediate members should be listed. Defaults to the root group | '/' |
Returns:
| Type | Description |
|---|---|
list[str] | Names of child groups and datasets directly under |
Source code in daft/file/hdf5.py
121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 | |
metadata #
metadata(group: str = '/') -> list[Hdf5ObjectMetadata]
Collect object metadata below an HDF5 group.
This is a Daft convenience around the same recursive traversal used by h5py.File.visititems(). It visits groups and datasets under group and returns DataFrame-friendly dictionaries with stable keys.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
group | str | Group path to traverse. Defaults to the root group | '/' |
Returns:
| Type | Description |
|---|---|
list[Hdf5ObjectMetadata] | A list of metadata dictionaries containing |
list[Hdf5ObjectMetadata] |
|
Raises:
| Type | Description |
|---|---|
TypeError | If |
Source code in daft/file/hdf5.py
70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 | |
open #
open(buffer_size: int | None = HDF5_DEFAULT_BUFFER_SIZE) -> PyDaftFile
Source code in daft/file/hdf5.py
62 63 | |
read #
read(dataset: str) -> ndarray[Any, Any]
read(dataset: list[str] | tuple[str, ...]) -> dict[str, ndarray[Any, Any]]
read(dataset: str | Sequence[str]) -> ndarray[Any, Any] | dict[str, ndarray[Any, Any]]
Read one or more HDF5 datasets into NumPy arrays.
For a single dataset path, this is equivalent to opening the file with h5py and evaluating h5[dataset][()]. Passing a sequence reads multiple datasets with one file open.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset | str | Sequence[str] | A dataset path or sequence of dataset paths. | required |
Returns:
| Type | Description |
|---|---|
ndarray[Any, Any] | dict[str, ndarray[Any, Any]] | A NumPy array for one dataset. For multiple datasets, a dictionary |
ndarray[Any, Any] | dict[str, ndarray[Any, Any]] | keyed by dataset path. |
Raises:
| Type | Description |
|---|---|
TypeError | If any requested path resolves to a group instead of a dataset. |
Source code in daft/file/hdf5.py
202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 | |
visit #
visit(*, group: str = '/') -> list[str]
visit(func: Callable[[str], Any], *, group: str = '/') -> Any
visit(func: Callable[[str], Any] | None = None, *, group: str = '/') -> Any
Recursively visit object names below an HDF5 group.
Thin wrapper around h5py Group.visit. When func is provided, it is called once per visited object name. Returning None from func continues traversal; returning any other value stops traversal and returns that value.
If func is omitted, this method collects and returns all visited names as a list. This matches the common h5py pattern of passing names.append as the visitor.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
func | Callable[[str], Any] | None | Optional visitor callable with signature | None |
group | str | Group path where traversal should start. Defaults to | '/' |
Returns:
| Type | Description |
|---|---|
Any | The visitor's first non- |
Any | visitor completed without one. If |
Any |
|
Source code in daft/file/hdf5.py
160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 | |