common.storages.data_item_storage

DataItemStorage Objects

class DataItemStorage(ABC)

[view_source]

write_empty_items_file

def write_empty_items_file(load_id: str, schema_name: str, table_name: str,
                           columns: TTableSchemaColumns) -> DataWriterMetrics

[view_source]

Writes empty file: only header and footer without actual items. Closed the empty file and returns metrics. Mind that header and footer will be written.

import_items_file

def import_items_file(load_id: str, schema_name: str, table_name: str,
                      file_path: str,
                      metrics: DataWriterMetrics) -> DataWriterMetrics

[view_source]

Import a file from file_path into items storage under a new file name. Does not check the imported file format. Uses counts from metrics as a base. Logically closes the imported file

The preferred import method is a hard link to avoid copying the data. If current filesystem does not support it, a regular copy is used.

closed_files

def closed_files(load_id: str) -> List[DataWriterMetrics]

[view_source]

Return metrics for all fully processed (closed) files

remove_closed_files

def remove_closed_files(load_id: str) -> None

[view_source]

Remove metrics for closed files in a given load_id