Skip to content

Added query_config parameter to read_gbq #14742

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
necnec wants to merge 36 commits into from
Closed

Added query_config parameter to read_gbq #14742

Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
55bf05c
Added udf_resource_uri parameter to read_gbq
necnec Nov 25, 2016
dad9288
Change parameter to kwargs
necnec Nov 28, 2016
9a16a8c
Merge branch 'bigquery-udf-resources'
necnec Nov 28, 2016
f9fae0c
Fix formatting
necnec Nov 28, 2016
42dc9e6
Merge remote-tracking branch 'origin/bigquery-udf-resources'
necnec Nov 28, 2016
c66169d
add read_gbq tests: query parameters and cache
necnec Nov 28, 2016
a96811d
add unit tests read_gbq: query parameters, cache
necnec Nov 28, 2016
ad35a43
fix whatsnew text
necnec Nov 28, 2016
ddb4fd1
Merge branch 'bigquery-udf-resources'
necnec Nov 28, 2016
94fa514
test formatting
necnec Nov 29, 2016
d69ed7f
check tests
necnec Nov 29, 2016
834a2ff
Merge branch 'bigquery-udf-resources'
necnec Nov 29, 2016
640be7a
Change whatnew 0.19.0->0.19.2
necnec Nov 30, 2016
b849300
Change whatsnew 0.19.2 -> 0.20.0
necnec Nov 30, 2016
a952710
Move whatsnew BQ Enhancements -> Enhancements
necnec Dec 2, 2016
0b365da
delete newlines
necnec Dec 2, 2016
c199935
Make query configuration more general
necnec Dec 5, 2016
028c8be
Solve formating problems
necnec Dec 5, 2016
ce8ebe4
Merge branch 'bigquery-udf-resources'
necnec Dec 5, 2016
146f0f3
Merge branch 'master' into bigquery-udf-resources
necnec Dec 5, 2016
8fe77b2
Merge branch 'bigquery-udf-resources'
necnec Dec 5, 2016
c21588a
Merge remote-tracking branch 'pandas-dev/master' into bigquery-udf-re…
necnec Dec 12, 2016
395c0e9
fix formatting
necnec Dec 12, 2016
8a38650
Added example configuration & job_configuration refactoring
necnec Dec 12, 2016
929ad1a
formatting: delete whitespace
necnec Dec 13, 2016
86ed96d
Merge branch 'master' into bigquery-udf-resources
necnec Dec 13, 2016
0ac26a2
added pull request number in whitens
necnec Dec 14, 2016
99521aa
Formatting, documentation, new unit test
necnec Dec 21, 2016
df5dec6
configuration->config & formatting
necnec Dec 22, 2016
8720b03
Delete trailing whitespaces
necnec Dec 22, 2016
ec590af
Throw exception if more than 1 job type in config
necnec Dec 29, 2016
2e02d76
Merge remote-tracking branch 'pandas-dev/master' into bigquery-udf-re…
necnec Dec 29, 2016
e2f801f
hotfix
Dec 29, 2016
b97a1be
formatting
necnec Dec 30, 2016
82f4409
Add some documentation & formatting
necnec Jan 2, 2017
3a238a5
config->configuration
necnec Jan 3, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add unit tests read_gbq: query parameters, cache
  • Loading branch information
@necnec
necnec committed Nov 28, 2016
commit a96811d778b2bb965e352d0798a5c3c2fe014552
47 changes: 47 additions & 0 deletions pandas/io/tests/test_gbq.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -711,6 +711,53 @@ def test_invalid_option_for_sql_dialect(self):
gbq.read_gbq(sql_statement, project_id=_get_project_id(),
dialect='standard', private_key=_get_private_key_path())

def test_query_with_parameters(self):
sql_statement = "SELECT @param1 + @param2 as VALID_RESULT"
query_config = {
"useLegacySql":False,
"parameterMode":"named",
"queryParameters": [
{
"name": "param1",
"parameterType": {
"type": "INTEGER"
},
"parameterValue": {
"value": 1
}
},
{
"name": "param2",
"parameterType": {
"type": "INTEGER"
},
"parameterValue": {
"value": 2
}
}
]
}
# Test that an invalid query without query_config
with tm.assertRaises(ValueError):

@jreback jreback Dec 12, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this necessary? I thought configuration is an optional paramter? when is it needed

@necnec necnec Dec 12, 2016

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jreback Yes, configuration is optional. But this unit test is very special. It processes query with parameters. And in this case you must pass parameters values in configuration.

I've made 2 unit tests. So if you think this test if very special I can remove that.

@jreback jreback Dec 12, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its fine to test. is it seems that this tests means its required somehow though.

@necnec necnec Dec 14, 2016

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jreback so I don't need to change anything here?

@parthea parthea Dec 18, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we update the comment to better explain why we expect a failure here?

For example,
# Test that a query that relies on parameters fails when parameters not supplied via configuration.

gbq.read_gbq(sql_statement, project_id=_get_project_id(),
private_key=_get_private_key_path())

# Test that a correct query with query config

@parthea parthea Dec 18, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

# Test that the query is successful because we have supplied the correct query parameters via the 'configuration' option

df = gbq.read_gbq(sql_statement, project_id=_get_project_id(),
private_key=_get_private_key_path(),
query_config=query_config)
tm.assert_frame_equal(df, DataFrame({'VALID_RESULT': [3]}))

def test_query_no_cache(self):
query = 'SELECT "PI" as VALID_STRING'
query_config = {
"useQueryCache":False,
}
df = gbq.read_gbq(query, project_id=_get_project_id(),
private_key=_get_private_key_path(),
query_config=query_config)
tm.assert_frame_equal(df, DataFrame({'VALID_STRING': ['PI']}))


class TestToGBQIntegration(tm.TestCase):
# Changes to BigQuery table schema may take up to 2 minutes as of May 2015
Expand Down