Sling with Components

To navigate including Components with Dagster, we will be using the Dagster dg command line. dg is a useful to initializing and navigating Dagster projects. Because we have been working the virtual envionment and project structure of this course we have not needed dg as much.

We used dg before to lanuch the UI to view and execute our assets but now we will use it to scaffold our Component.

Create Sling Component

To initialize our dlt Component, we will run the dg command to scaffold the Sling Component (dagster_sling.SlingReplicationCollectionComponent):

dg scaffold defs 'dagster_sling.SlingReplicationCollectionComponent' ingest_files

There will now be a new directory in defs containing two files:

.
└── src
    └── dagster_and_etl
        └── defs
            └── ingest_files
                ├── defs.yaml
                └── replication.yaml

These are the only two files we will need to interact with in order to configure our Sling assets (no Python). Let's discuss what goes into each.

Configure the YAML

The replication.yaml should look familiar. That is the yaml file that Sling uses to determine the replication characteristics. The replication.yaml that is generated by the dg command is pretty much empty:

source: {}
streams: {}
target: {}

Let's copy and paste all of the contents from our sling_replication.yaml from the previous lesson:

source: MY_POSTGRES
target: MY_DUCKDB

defaults:
  mode: full-refresh

  object: "{stream_schema}_{stream_table}"

streams:
  data.customers:
  data.products:
  data.orders:
    mode: incremental
    primary_key: order_id
    update_key: order_date

The next file is the defs.yaml. This contains two bits of information. Which component we are using, in this case Sling, and the location of the replication.yaml.

type: dagster_sling.SlingReplicationCollectionComponent

attributes:
  replications:
    - path: replication.yaml

The only thing we need to add are the connection details. Before we had defined these within the SlingResource but there is no need for that with Components. Instead we will enter the connection details in the component.yaml:

type: dagster_sling.SlingReplicationCollectionComponent

attributes:
  sling:
    connections:
      - name: MY_POSTGRES
        type: postgres
        host: localhost
        port: 5432
        database: test_db
        user: test_user
        password: test_pass
      - name: MY_DUCKDB
        type: duckdb
        instance: duckdb:///var/tmp/duckdb.db
  replications:
    - path: replication.yaml

Notice that the names of the connection in the defs.yaml still need to match up to the replication.yaml (MY_DUCKDB and MY_POSTGRES).

Viewing the assets

Because we switching our assets over to components, we can cleanup a lot of our Python code. In resources.py you can remove everything related Sling. And in the assets.py file you can remove the assets generated by the sling_assets. We can still keep the assets downstream of the Component assets.

Now if we run dg dev we can see the same asset graph as before.