Support Open Source ClickHouse Deployments

[Pipboyguy]

Description

This will improves support for self-managed ClickHouse open source deployments, while maintaining compatibility with ClickHouse Cloud deployments.

• Allow explicitly specifying the desired engine via table_engine_type in clickhouse_adapter. Valid types are merge_tree, replicated_merge_tree, shared_merge_tree, stripe_log, tiny_log
• Default to MergeTree if no engine is specified, which now works for both Cloud and self-managed deployments
• Update tests to check for the appropriate engine based on annotation
• CH cloud has "date_time_input_format" set to "best_effort", while OS deployments don't. This caused some tests to fail for OS deployments. We override setting for clickhouse_connect sessions.
• Clarify some networking details for making dlt work with OS deployments in docs

Related Issues

• Closes ClickHouse MergeTree Support #1387
• Closes Open Source ClickHouse fails on Timestamp with Timezone #1503

Additional Context

Note that this change does not include support for specifying replication, ZooKeeper or shard details for the ReplicatedMergeTree engine. Users requiring those customizations can continue to specify the full engine definition in their configuration.

[netlify]

✅ Deploy Preview for dlt-hub-docs ready!

Name	Link
🔨 Latest commit	409487c
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/667f36efbe836c0008f48dea
😎 Deploy Preview	https://deploy-preview-1496--dlt-hub-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[jorritsandbrink]

Code itself looks good to me. Just two comments regarding test coverage and the use of sentinel tables.

[jorritsandbrink]

Perhaps out of scope for this PR, but have you considered using pipeline state instead of a sentinel table?

[Pipboyguy]

That is a really good suggestion!

I've dropped the sentinel table workaround for more canonical usage of dlt:

dlt/dlt/destinations/impl/clickhouse/clickhouse.py

Lines 310 to 345 in 409487c

	def has_dataset(self) -> bool:
	try:
	container = Container()
	state_ctx = container[LoadPackageStateInjectableContext]
	datasets = cast(List[str], state_ctx.state.get("datasets", []))
	return self.sql_client.dataset_name in datasets
	except KeyError:
	return False
	def create_dataset(self) -> None:
	with contextlib.suppress(KeyError):
	container = Container()
	state_ctx = container[LoadPackageStateInjectableContext]
	datasets = cast(List[str], state_ctx.state.get("datasets", []))
	if self.sql_client.dataset_name not in datasets:
	datasets.append(self.sql_client.dataset_name)
	state: TLoadPackageState = state_ctx.state
	state["datasets"] = datasets # type: ignore[typeddict-unknown-key]
	commit_load_package_state()
	def drop_dataset(self) -> None:
	with contextlib.suppress(KeyError):
	container = Container()
	state_ctx = container[LoadPackageStateInjectableContext]
	datasets = cast(List[str], state_ctx.state.get("datasets", []))
	if self.sql_client.dataset_name in datasets:
	datasets.remove(self.sql_client.dataset_name)
	state: TLoadPackageState = state_ctx.state
	state["datasets"] = datasets # type: ignore[typeddict-unknown-key]
	commit_load_package_state()
	to_drop_results = self.sql_client._list_tables()
	for table in to_drop_results:
	self.sql_client.execute_sql(
	f"""DROP TABLE {self.sql_client.catalog_name()}.{self.capabilities.escape_identifier(table)} SYNC"""
	)

[jorritsandbrink]

Should we run CI both for ClickHouse cloud and ClickHouse local, like Postgres?

Seems like we don't have proper test coverage without local (e.g. for the Log table engines).

[Pipboyguy]

• Added ClickHouse OSS as a local destination test in .github/workflows/test_destination_clickhouse.yml, so CI also runs local tests.
• Removed the log engine family for now. I'm not even sure why I brought it into this ticket. If users ask for it, it will require more work than what can be done here.