common.storages.load_package
TLoadPackageState Objects
class TLoadPackageState(TVersionedState)
created_at
Timestamp when the loadpackage was created
destination_state
private space for destinations to store state relevant only to the load package
TLoadPackage Objects
class TLoadPackage(TypedDict)
load_id
Load id
state
State of the load package
ParsedLoadJobFileName Objects
class ParsedLoadJobFileName(NamedTuple)
Represents a file name of a job in load package. The file name contains name of a table, number of times the job was retired, extension and a 5 bytes random string to make job file name unique. The job id does not contain retry count and is immutable during loading of the data
job_id
def job_id() -> str
Unique identifier of the job
file_name
def file_name() -> str
A name of the file with the data to be loaded
with_retry
def with_retry() -> "ParsedLoadJobFileName"
Returns a job with increased retry count
PackageStorage Objects
class PackageStorage()
APPLIED_SCHEMA_UPDATES_FILE_NAME
updates applied to the destination
__init__
def __init__(storage: FileStorage, initial_state: TLoadPackageStatus) -> None
Creates storage that manages load packages with root at storage and initial package state initial_state
list_packages
def list_packages() -> Sequence[str]
Lists all load ids in storage, earliest first
NOTE: Load ids are sorted alphabetically. This class does not store package creation time separately.
list_failed_jobs_infos
def list_failed_jobs_infos(load_id: str) -> Sequence[LoadJobInfo]
List all failed jobs and associated error messages for a load package with load_id
import_job
def import_job(load_id: str,
job_file_path: str,
job_state: TJobState = "new_jobs") -> None
Adds new job by moving the job_file_path into new_jobs of package load_id
complete_loading_package
def complete_loading_package(load_id: str,
load_state: TLoadPackageStatus) -> str
Completes loading the package by writing marker file with`package_state. Returns path to the completed package
remove_completed_jobs
def remove_completed_jobs(load_id: str) -> None
Deletes completed jobs. If package has failed jobs, nothing gets deleted.
schema_name
def schema_name(load_id: str) -> str
Gets schema name associated with the package
get_load_package_info
def get_load_package_info(load_id: str) -> LoadPackageInfo
Gets information on normalized/completed package with given load_id, all jobs and their statuses.
is_package_partially_loaded
@staticmethod
def is_package_partially_loaded(package_info: LoadPackageInfo) -> bool
Checks if package is partially loaded - has jobs that are not new.
load_package
def load_package() -> TLoadPackage
Get full load package state present in current context. Across all threads this will be the same in memory dict.
commit_load_package_state
def commit_load_package_state() -> None
Commit load package state present in current context. This is thread safe.
destination_state
def destination_state() -> DictStrAny
Get segment of load package state that is specific to the current destination.
clear_destination_state
def clear_destination_state(commit: bool = True) -> None
Clear segment of load package state that is specific to the current destination. Optionally commit to load package.