Tech
Building an Agent That Turns DataHub Metadata Into Airflow DAGs
Built this with my teammate for DataHub's "Build with DataHub: The Agent Hackathon." The hackathon has wrapped up, but we're continuing to develop it.
The problem
If you're using DataHub, you already have lineage, freshness SLAs, PII tags, and glossary terms sitting there. But turning that into an actual Airflow DAG still means doing it by hand.
What it does
Give it a table name, and it:
Walks the lineage graph backwards to find every upstream dependency
Reads tags (pii, dail...
Read the full discussion on Dev.to
This article was aggregated from Dev.to. Click to join the conversation.
View on Dev.to