COPAI Data Scraper
Feeds public X posts into COPAI’s AI knowledge base.
- Role
- Data engineer (intern)
- Context
- COPAI · MISTI
- Stack
- Python · Playwright · X (Twitter)
COPAI — the Community of Practice on Artificial Intelligence — is MISTI’s Cambodian AI chatbot, running entirely on the ministry’s own HPC infrastructure. To keep its knowledge base current with updates from key public accounts, I built a pipeline that collects posts from X (Twitter), structures them and uploads them to COPAI automatically.
What I did
- 01
Built a Playwright scraper that captures X’s own API responses, so posts are collected accurately instead of guessed from the page.
- 02
Parsed each post’s text, engagement and type — original, repost, quote, reply, poll, article or media.
- 03
Added incremental scraping with metadata tracking, so every run collects only new posts.
- 04
Uploaded posts straight into COPAI’s knowledge base through its API, as one Markdown document per account per day.
- 05
Automated a cleanup that removes knowledge older than three months, so answers stay current.
- 06
Packaged it with Docker, health checks and start-up validation for reliable unattended runs.
Highlights
Accurate capture
Reads X’s API responses, not the page layout.
Incremental
Only new posts on every run.
Auto upload
Straight into COPAI’s knowledge base.
Runs unattended
Dockerised with health checks.
