feat(bigquery): Support DATE-type event timestamp columns#6362
Merged
Conversation
Jwrede
requested review from
HaoXuAI,
nquinn408 and
shuchu
and removed request for
a team
franciscojavierarceo
approved these changes
May 3, 2026
Contributor
Author
|
Friendly ping @sudohainguyen -- this and #6365 have been open ~10 days. Happy to address feedback. |
Member
|
@Jwrede please resolve the conflicts and fix linting |
Jwrede
force-pushed
the
feat/bq-date-timestamp-type
branch
from
d8ccd7e to
aed2012
Compare
Contributor
Author
|
@ntkathole Rebased onto master and verified ruff check/format passes. All conflicts resolved. |
Member
|
@Jwrede can you please regenerate protos using correct mypy-protobuf version pinned in CI ? |
Contributor
Author
|
@ntkathole Done -- reset all non-DataSource generated files to match master, then hand-edited DataSource_pb2.pyi to add only the timestamp_field_type field in the existing import style. Only DataSource_pb2.py and DataSource_pb2.pyi differ from master now. |
When the event_timestamp column in BigQuery is a DATE type, the generated SQL wraps comparison values in TIMESTAMP(), causing a type mismatch error. This adds a timestamp_field_type parameter to BigQuerySource that, when set to "DATE", generates DATE() comparisons instead. Closes feast-dev#2530 (part 2) Signed-off-by: Jonathan Wrede <[email protected]>
The proto files were regenerated with protobuf 6.31.1 / grpcio-tools 1.80.0, which imports runtime_version -- a module that does not exist in protobuf 4.25.x used by the project. Revert generated code to 4.25.1 format while keeping the new timestamp_field_type field. Signed-off-by: Jonathan Wrede <[email protected]>
Mypy infers str from the ternary expression; annotate with the exact Literal union so the call to get_timestamp_filter_sql passes type checking. Signed-off-by: Jonathan Wrede <[email protected]>
…text Callers that do not use DATE-typed timestamp fields (e.g. Spark offline store tests) should not be forced to pass timestamp_field_type. Adding a default keeps the new field backward-compatible. Signed-off-by: Jonathan Wrede <[email protected]>
A default value on timestamp_field_type breaks the SparkFeatureViewQueryContext subclass because its non-default fields (min_date_partition, max_date_partition) would follow a field with a default. Instead, keep it required and update the Spark test to pass it. Signed-off-by: Jonathan Wrede <[email protected]>
Reset all non-DataSource generated files to match master. Only DataSource_pb2.py and DataSource_pb2.pyi contain our timestamp_field_type additions (field 28). The .pyi stub is hand-edited to match the existing import style used on master. Signed-off-by: Jonathan Wrede <[email protected]>
ntkathole
force-pushed
the
feat/bq-date-timestamp-type
branch
from
27579ae to
a28b283
Compare
ntkathole
approved these changes
May 14, 2026
rpathade
pushed a commit
to rpathade/feast
that referenced
this pull request
May 21, 2026
…6362) * feat(bigquery): Support DATE-type event timestamp columns When the event_timestamp column in BigQuery is a DATE type, the generated SQL wraps comparison values in TIMESTAMP(), causing a type mismatch error. This adds a timestamp_field_type parameter to BigQuerySource that, when set to "DATE", generates DATE() comparisons instead. Closes feast-dev#2530 (part 2) Signed-off-by: Jonathan Wrede <[email protected]> * fix(bigquery): Use protobuf 4.25.x compatible generated code The proto files were regenerated with protobuf 6.31.1 / grpcio-tools 1.80.0, which imports runtime_version -- a module that does not exist in protobuf 4.25.x used by the project. Revert generated code to 4.25.1 format while keeping the new timestamp_field_type field. Signed-off-by: Jonathan Wrede <[email protected]> * fix(bigquery): Add Literal type annotation for cast_style Mypy infers str from the ternary expression; annotate with the exact Literal union so the call to get_timestamp_filter_sql passes type checking. Signed-off-by: Jonathan Wrede <[email protected]> * fix: Make timestamp_field_type default to None in FeatureViewQueryContext Callers that do not use DATE-typed timestamp fields (e.g. Spark offline store tests) should not be forced to pass timestamp_field_type. Adding a default keeps the new field backward-compatible. Signed-off-by: Jonathan Wrede <[email protected]> * fix: Keep timestamp_field_type required in FeatureViewQueryContext A default value on timestamp_field_type breaks the SparkFeatureViewQueryContext subclass because its non-default fields (min_date_partition, max_date_partition) would follow a field with a default. Instead, keep it required and update the Spark test to pass it. Signed-off-by: Jonathan Wrede <[email protected]> * fix: regenerate protos matching upstream mypy-protobuf style Reset all non-DataSource generated files to match master. Only DataSource_pb2.py and DataSource_pb2.pyi contain our timestamp_field_type additions (field 28). The .pyi stub is hand-edited to match the existing import style used on master. Signed-off-by: Jonathan Wrede <[email protected]> --------- Signed-off-by: Jonathan Wrede <[email protected]> Signed-off-by: RutujaPathade <[email protected]>
rpathade
pushed a commit
to rpathade/feast
that referenced
this pull request
May 21, 2026
…6362) * feat(bigquery): Support DATE-type event timestamp columns When the event_timestamp column in BigQuery is a DATE type, the generated SQL wraps comparison values in TIMESTAMP(), causing a type mismatch error. This adds a timestamp_field_type parameter to BigQuerySource that, when set to "DATE", generates DATE() comparisons instead. Closes feast-dev#2530 (part 2) Signed-off-by: Jonathan Wrede <[email protected]> * fix(bigquery): Use protobuf 4.25.x compatible generated code The proto files were regenerated with protobuf 6.31.1 / grpcio-tools 1.80.0, which imports runtime_version -- a module that does not exist in protobuf 4.25.x used by the project. Revert generated code to 4.25.1 format while keeping the new timestamp_field_type field. Signed-off-by: Jonathan Wrede <[email protected]> * fix(bigquery): Add Literal type annotation for cast_style Mypy infers str from the ternary expression; annotate with the exact Literal union so the call to get_timestamp_filter_sql passes type checking. Signed-off-by: Jonathan Wrede <[email protected]> * fix: Make timestamp_field_type default to None in FeatureViewQueryContext Callers that do not use DATE-typed timestamp fields (e.g. Spark offline store tests) should not be forced to pass timestamp_field_type. Adding a default keeps the new field backward-compatible. Signed-off-by: Jonathan Wrede <[email protected]> * fix: Keep timestamp_field_type required in FeatureViewQueryContext A default value on timestamp_field_type breaks the SparkFeatureViewQueryContext subclass because its non-default fields (min_date_partition, max_date_partition) would follow a field with a default. Instead, keep it required and update the Spark test to pass it. Signed-off-by: Jonathan Wrede <[email protected]> * fix: regenerate protos matching upstream mypy-protobuf style Reset all non-DataSource generated files to match master. Only DataSource_pb2.py and DataSource_pb2.pyi contain our timestamp_field_type additions (field 28). The .pyi stub is hand-edited to match the existing import style used on master. Signed-off-by: Jonathan Wrede <[email protected]> --------- Signed-off-by: Jonathan Wrede <[email protected]> Signed-off-by: RutujaPathade <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
When the
event_timestampcolumn in BigQuery is aDATEtype (notTIMESTAMP), the generated SQL wraps comparison values inTIMESTAMP(), causing a type mismatch error. This makes DATE-partitioned summary tables unusable without creating views or duplicate tables.This PR adds an optional
timestamp_field_typeparameter toBigQuerySource. When set to"DATE", SQL generation usesDATE('YYYY-MM-DD')comparisons instead ofTIMESTAMP('...'), both in direct queries (pull_latest_from_table_or_query,pull_all_from_table_or_query) and in the point-in-time join Jinja template.Usage:
Changes:
timestamp_field_typestring field (field 28) toDataSourceDataSourcebase class: addtimestamp_field_typeattribute, equality check, and__init__parameterBigQuerySource: wiretimestamp_field_typethrough__init__,from_proto, and_to_proto_implget_timestamp_filter_sql(): add"date_func"cast style that generatesDATE('YYYY-MM-DD')BigQueryOfflineStore: select cast style based ontimestamp_field_typeDATE()comparisons for DATE-type timestamp fieldsFeatureViewQueryContext: propagatetimestamp_field_typeto template contextBackward-compatible: when
timestamp_field_typeis unset, behavior is unchanged.Which issue(s) this PR fixes:
Fixes #2530 (part 2 -- DATE type event_timestamp support; part 1 was addressed by #6076)
How to test:
4 new tests added:
test_pull_latest_date_type_timestamp_field-- verifiesDATE()cast in pull_latesttest_pull_all_date_type_timestamp_field-- verifiesDATE()cast in pull_alltest_pull_latest_date_type_with_partition_column-- DATE type combined with partition pruningtest_bigquery_source_date_type_proto_roundtrip-- proto serialization roundtrip