Skip to main content
Version: Next

GitHub

Need help deploying these sources, or figuring out how to run them in your data stack?
Join our Slack community or book a call with a dltHub Solutions Engineer.

This verified source can be used to load data on issues or pull requests from any GitHub repository onto a destination of your choice using GitHub API.

Resources that can be loaded using this verified source are:

NameDescription
github_reactionsRetrieves all issues, pull requests, comments and reactions associated with them
github_repo_eventsGets all the repo events associated with the repository

Setup Guide​

Grab credentials​

To get the API token, sign-in to your GitHub account and follow these steps:

  1. Click on your profile picture in the top right corner.

  2. Choose "Settings".

  3. Select "Developer settings" on the left panel.

  4. Under "Personal access tokens", click on "Generate a personal access token (preferably under Tokens(classic))".

  5. Grant at least the following scopes to the token by checking them.

    ScopeDescription
    public_repoLimits access to public repositories
    read:repo_hookGrants read and ping access to hooks in public or private repositories
    read:orgRead-only access to organization membership, organization projects, and team membership
    read:userGrants access to read a user's profile data
    read:projectGrants read only access to user and organization projects
    read:discussionAllows read access for team discussions

  6. Finally, click "Generate token".

  7. Copy the token and save it. This is to be added later in the dlt configuration.

You can optionally add API access tokens to avoid making requests as an unauthorized user. If you wish to load data using the github_reaction source, the access token is mandatory.

More information you can see in the GitHub authentication and GitHub API token scopes documentations.

Initialize the verified source​

To get started with your data pipeline, follow these steps:

  1. Enter the following command:

    dlt init github duckdb

    This command will initialize the pipeline example with GitHub as the source and duckdb as the destination.

  2. If you'd like to use a different destination, simply replace duckdb with the name of your preferred destination.

  3. After running this command, a new directory will be created with the necessary files and configuration settings to get started.

For more information, read the guide on how to add a verified source.

Add credentials​

  1. In .dlt/secrets.toml, you can securely store your access tokens and other sensitive information. It's important to handle this file with care and keep it safe. Here's what the file looks like:

    # Put your secret values and credentials here
    # Github access token (must be classic for reactions source)
    [sources.github]
    access_token="please set me up!" # use GitHub access token here

  2. Replace the API token value with the previously copied one to ensure secure access to your GitHub resources.

  3. Next, follow the destination documentation instructions to add credentials for your chosen destination, ensuring proper routing of your data to the final destination.

For more information, read the General Usage: Credentials.

Run the pipeline​

  1. Before running the pipeline, ensure that you have installed all the necessary dependencies by running the command:
    pip install -r requirements.txt
  2. You're now ready to run the pipeline! To get started, run the following command:
    python github_pipeline.py
  3. Once the pipeline has finished running, you can verify that everything loaded correctly by using the following command:
    dlt pipeline <pipeline_name> show
    For example, the pipeline_name for the above pipeline example is github_reactions, you may also use any custom name instead.

For more information, read the guide on how to run a pipeline.

Sources and resources​

dlt works on the principle of sources and resources.

Source github_reactions​

This dlt.source function uses GraphQL to fetch DltResource objects: issues and pull requests along with associated reactions, comments, and reactions to comments.

@dlt.source
def github_reactions(
    owner: str,
    name: str,
    access_token: str = dlt.secrets.value,
    items_per_page: int = 100,
    max_items: int = None,
    max_item_age_seconds: float = None,
) -> Sequence[DltResource]:
   ...

owner: Refers to the owner of the repository.

name: Refers to the name of the repository.

access_token: Classic access token should be utilized and is stored in the .dlt/secrets.toml file.

items_per_page: The number of issues/pull requests to retrieve in a single page. Defaults to 100.

max_items: The maximum number of issues/pull requests to retrieve in total. If set to None, it means all items will be retrieved. Defaults to None.

max_item_age_seconds: The feature to restrict retrieval of items older than a specific duration is yet to be implemented. Defaults to None.

Resource _get_reactions_data ("issues")​

The dlt.resource function employs the _get_reactions_data method to retrieve data about issues, their associated comments, and subsequent reactions.

dlt.resource(
    _get_reactions_data(
        "issues",
        owner,
        name,
        access_token,
        items_per_page,
        max_items,
        max_item_age_seconds,
    ),
    name="issues",
    write_disposition="replace",
),

Source github_repo_events​

This dlt.source fetches repository events incrementally, dispatching them to separate tables based on event type. It loads new events only and appends them to tables.

Note: Github allows retrieving up to 300 events for public repositories, so frequent updates are recommended for active repos.

@dlt.source(max_table_nesting=2)
def github_repo_events(
    owner: str, name: str, access_token: str = None
) -> DltResource:
   ...

owner: Refers to the owner of the repository.

name: Denotes the name of the repository.

access_token: Optional classic or fine-grained access token. If not provided, calls are made anonymously.

max_table_nesting=2 sets the maximum nesting level of child tables to 2.

Read more about nesting levels.

Resource repo_events​

This dlt.resource function serves as the resource for the github_repo_events source. It yields repository events as data items.

dlt.resource(primary_key="id", table_name=lambda i: i["type"])  # type: ignore
def repo_events(
    last_created_at: dlt.sources.incremental[str] = dlt.sources.incremental(
        "created_at", initial_value="1970-01-01T00:00:00Z", last_value_func=max
    )
) -> Iterator[TDataItems]:
   ...

primary_key: Serves as the primary key, instrumental in preventing data duplication.

table_name: Routes data to appropriate tables based on the data type.

last_created_at: This parameter determines the initial value for "last_created_at" in dlt.sources.incremental. If no value is given, the default "initial_value" is used. The function "last_value_func" determines the most recent 'created_at' value.

Read more about incremental loading.

Customization​

Create your own pipeline​

If you wish to create your own pipelines, you can leverage source and resource methods from this verified source.

  1. Configure the pipeline by specifying the pipeline name, destination, and dataset as follows:

    pipeline = dlt.pipeline(
        pipeline_name="github_pipeline",  # Use a custom name if desired
        destination="duckdb",  # Choose the appropriate destination (e.g., duckdb, redshift, post)
        dataset_name="github_reaction_data"  # Use a custom name if desired
    )

    To read more about pipeline configuration, please refer to our documentation.

  2. To load all the data from repo on issues, pull requests, their comments and reactions, you can do the following:

    load_data = github_reactions("duckdb", "duckdb")
    load_info = pipeline.run(load_data)
    print(load_info)

    here, "duckdb" is the owner of the repository and the name of the repository.

  3. To load only the first 100 issues, you can do the following:

    load_data = github_reactions("duckdb", "duckdb", max_items=100)
    load_info = pipeline.run(load_data.with_resources("issues"))
    print(load_info)

  4. You can use fetch and process repo events data incrementally. It loads all data during the first run and incrementally in subsequent runs.

    load_data = github_repo_events(
        "duckdb", "duckdb", access_token=os.getenv(ACCESS_TOKEN)
    )
    load_info = pipeline.run(load_data)
    print(load_info)

    It is optional to use access_token or make anonymous API calls.

Additional Setup guides​

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!

DHelp

Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.