Skip to content

dbt integration: FileSource generates invalid hardcoded paths #5874

@YassinNouh21

Description

@YassinNouh21

Context

PR #5827 added support for FileSource data sources in dbt integration.

Problem

The FileSource implementation generates hardcoded paths that likely don't exist:

# In mapper.py:240-242
return FileSource(
    name=f"{model.name}_source",
    path=f"/data/{model.name}.parquet",  # Hardcoded, likely invalid
    # ...
)

This path /data/{model.name}.parquet is unlikely to match actual file locations.

Impact

  • Generated FileSource objects won't work without manual editing
  • Confusing user experience
  • No clear documentation on how to fix the paths

Proposed Solutions

Option 1: Make path configurable

feast dbt import -m manifest.json -e driver_id --data-source-type file --file-path-template "/my/data/{model}.parquet"

Option 2: Read from dbt meta

Support custom path in dbt model meta:

# dbt model
models:
  - name: driver_stats
    meta:
      feast:
        file_path: "/data/warehouse/driver_stats.parquet"

Option 3: Require explicit path mapping file

feast dbt import -m manifest.json -e driver_id --file-path-config paths.yaml

Recommendation

Implement Option 1 (path template) as it's simplest and covers most use cases. Consider Option 2 for advanced scenarios.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions