Skip to content

dbt integration: Add support for data source connection configuration #5875

@YassinNouh21

Description

@YassinNouh21

Context

PR #5827 added dbt integration with basic data source support.

Problem

The generated data sources use default/inferred configurations but don't allow specifying:

  • BigQuery: GCP project, dataset overrides
  • Snowflake: account, warehouse, role, connection parameters
  • File: base path, file format options

This means generated sources may not connect to the actual data.

Current Limitation

# BigQuery example - no way to override project
return BigQuerySource(
    name=f"{model.name}_source",
    table=model.full_table_name,  # Uses dbt's database.schema.table
    # Missing: project override, dataset override
)

Proposed Enhancement

Add CLI options for data source configuration:

# BigQuery
feast dbt import -m manifest.json -e driver_id \
  --data-source-type bigquery \
  --bigquery-project my-gcp-project

# Snowflake  
feast dbt import -m manifest.json -e driver_id \
  --data-source-type snowflake \
  --snowflake-account my-account \
  --snowflake-warehouse my-warehouse \
  --snowflake-role my-role

# File
feast dbt import -m manifest.json -e driver_id \
  --data-source-type file \
  --file-format parquet \
  --file-base-path /data/warehouse

Or support reading from feature_store.yaml:

offline_store:
  type: bigquery
  project: my-gcp-project
  
dbt:
  inherit_offline_store_config: true

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions