Census (dagster-census)¶

This library provides an integration with Census.

dagster_census.census_trigger_sync_op OpDefinition[source]¶

Config Schema:

sync_id (Int):

Id of the parent sync.

force_full_sync (Bool, optional):

If this trigger request should be a Full Sync. Note that some sync configurations such as Append do not support full syncs.

Default Value: False

poll_interval (Float, optional):

The time (in seconds) to wait between successive polls.

Default Value: 10

poll_timeout (Union[Float, None], optional):

The maximum time to wait before this operation is timed out. By default, this will never time out.

Default Value: None

yield_materializations (Bool, optional):

If True, materializations corresponding to the results of the Census sync will be yielded when the op executes.

Default Value: True

asset_key_prefix (List[String], optional):

If provided and yield_materializations is True, these components will be used to prefix the generated asset keys.

Default Value: [‘census’]

Executes a Census sync for a given sync_id and polls until that sync completes, raising an error if it is unsuccessful.

It outputs a CensusOutput which contains the details of the Census sync after it successfully completes.

It requires the use of the census_resource, which allows it to communicate with the Census API.

Examples:

from dagster import job
from dagster_census import census_resource, census_sync_op

my_census_resource = census_resource.configured(
    {
        "api_key": {"env": "CENSUS_API_KEY"},
    }
)

sync_foobar = census_sync_op.configured({"sync_id": "foobar"}, name="sync_foobar")

@job(resource_defs={"census": my_census_resource})
def my_simple_census_job():
    sync_foobar()

dagster_census.census_resource ResourceDefinition[source]¶

Config Schema:

api_key (dagster.StringSource):

Census API Key.

request_max_retries (Int, optional):

The maximum number of times requests to the Census API should be retried before failing.

Default Value: 3

request_retry_delay (Float, optional):

Time (in seconds) to wait between each request retry.

Default Value: 0.25

This resource allows users to programatically interface with the Census REST API to launch syncs and monitor their progress. This currently implements only a subset of the functionality exposed by the API.

Examples:

from dagster import job
from dagster_census import census_resource

my_census_resource = census_resource.configured(
    {
        "api_key": {"env": "CENSUS_API_KEY"},
    }
)

@job(resource_defs={"census":my_census_resource})
def my_census_job():
    ...

class dagster_census.CensusResource(api_key, request_max_retries=3, request_retry_delay=0.25, log=<Logger dagster.builtin (DEBUG)>)[source]¶

This class exposes methods on top of the Census REST API.

class dagster_census.CensusOutput(sync_run, source, destination)[source]¶

Contains recorded information about the state of a Census sync after a sync completes.

sync_run¶

The details of the specific sync run.

Type:

Dict[str, Any]

source¶

Information about the source for the Census sync.

Type:

Dict[str, Any]

destination¶

Information about the destination for the Census sync.

Type:

Dict[str, Any]