Tencent Cloud · Beginner Data Journey

How does one order travel from MySQL to a BI report?

Start as a complete beginner: see how an operational database records a committed change, how CDC reads the source Binlog, and how COS, Iceberg, Parquet, modeling, and BI each play a different role.

Beginner pathChanged-state highlightsClickable snapshotsBilingual

Cart items10Not orders yet

Source orders0Visible after MySQL COMMIT

Trusted revenue¥0After the quality gate

Ingestion modeReal-time CDCRead source MySQL Binlog

Learning progress0%Place an order first

One continuous route

See the whole map, then learn each section from top to bottom

The most important boundaryMySQL is the operational source of truth, and the Binlog belongs to the source MySQL. The COS data lake stores object files; it has no MySQL Binlog.

Source database

Create a real, committed business order first

A cart is not final business data. CDC can see an order only after the order service successfully commits it to MySQL.

Editable source event

Edit the cart, then commit orders

Cart	User	Product	Quantity	Unit price	Region

Items sharing a cart_id become one orders header row, while each item is written to order_items. Edit first, then select “Group and place orders.”

Stage inspector

User places order

01 / Checkout

Extraction & ingestion

CDC reads the source database Binlog, not a data-lake Binlog

Select INSERT, UPDATE, or DELETE to follow the change from source row to Binlog event, CDC message, and lake file.

Business appOrder serviceBEGIN / INSERT / COMMIT

Source systemOperational MySQLorders / order_itemsBinlog is here

Source logMySQL BinlogCommitted INSERT / UPDATE / DELETE

IngestionCDC ConnectorParse before / after

Stream computeCKafka / OceanusDistribute, transform, check

Data lakeCOS + IcebergParquet / Metadata / ManifestNo MySQL Binlog here

Lake write

Write changes to COS; use Iceberg to organize Parquet as a versioned table

File formats store data, Iceberg manages transactions and file indexes, and COS durably stores the objects.

Storage anatomy

What does each lake file do?

Storage choice

How do HDFS and COS differ?

Snapshot → Parquet mapping

Which Parquet files belong to each Snapshot?

Snapshots reference files through Manifests

Not a one-to-one mapping

A Snapshot represents the whole table after one successful commit. It can reuse old Parquet files, add new Parquet files, and apply a Delete file. Select S1, S2, and S3 to inspect the mapping.

AddedChangedReusedDelete rule

Iceberg metadata chain

Follow the Catalog pointer all the way to real data files

Iceberg is a table format, not a required Catalog name

Core relationship

The Catalog points sales.orders to the current metadata JSON. A Snapshot inside that JSON points to a Manifest list, which points to Manifests, which finally list Data/Delete files.

Write Data/Delete filesWrite ManifestWrite Manifest listWrite new metadata JSONCatalog atomically switches the pointer

Terminology: Apache Iceberg Terms · Iceberg Table Spec

Transform & model

Clean, validate, and model the data into Mart / Gold

EMR, DLC, and Oceanus are compute tools. They read lake tables, apply rules, and commit a new Iceberg Snapshot.

Data quality gate

Create bad data and see what cannot enter the trusted layer

Accepted0Enter Clean / Silver

Rejected0Enter quarantine

Place orders first, then inject an issue.

Lakehouse layers

How one order becomes a trusted metric

Product guide

What does each compute or serving product own?

EMR component stack

How should you understand EMR layers?

Availability depends on the release and cluster type; this does not mean one cluster installs every component.

Version reference: Tencent Cloud EMR Component Versions

Serve, BI & report

Turn governed data into reusable operating metrics

EMR/DLC can explore Gold data directly. Production BI commonly publishes stable models to TCHouse, then applies a shared semantic layer.

Dynamic BI dashboard

Generate an operating report from current trusted orders

Only quality-approved orders are used, and order_items are joined by order_id. Edit source data and place orders again to update every metric.

Report region

Revenue¥0Paid and quality-approved

Paid orders0Unique order_id values

Average order value¥0Revenue ÷ paid orders

Items sold0Trusted orders joined to items

Revenue by regionFrom curated_orders

Order share by regionPaid order distribution

Top productsOnly quality-approved order_id values

Freshness and qualityReport not generated

Query lab

Run an analytics query

SELECT region,
       COUNT(*) AS orders,
       SUM(amount) AS revenue
FROM curated_orders
WHERE status = 'PAID'
GROUP BY region
ORDER BY revenue DESC;

Select “Run query” to see the result

Operations & governance

Continuously check freshness, quality, partitions, and lineage

The knowledge quiz has been removed. This section keeps a production-oriented health check instead.

AppSource MySQLBinlog / CKafkaOceanus / SparkCOS / IcebergTCHouse / BI

Freshness SLAWaiting for check

Quality pass rateWaiting for check

Partitions and small filesWaiting for check

Lineage and ownerWaiting for check