All projects
● Pipeline 2026

COPAI Data Scraper

Feeds public X posts into COPAI’s AI knowledge base.

Role
Data engineer (intern)
Context
COPAI · MISTI
Stack
Python · Playwright · X (Twitter)

COPAI — the Community of Practice on Artificial Intelligence — is MISTI’s Cambodian AI chatbot, running entirely on the ministry’s own HPC infrastructure. To keep its knowledge base current with updates from key public accounts, I built a pipeline that collects posts from X (Twitter), structures them and uploads them to COPAI automatically.

What I did

  1. 01

    Built a Playwright scraper that captures X’s own API responses, so posts are collected accurately instead of guessed from the page.

  2. 02

    Parsed each post’s text, engagement and type — original, repost, quote, reply, poll, article or media.

  3. 03

    Added incremental scraping with metadata tracking, so every run collects only new posts.

  4. 04

    Uploaded posts straight into COPAI’s knowledge base through its API, as one Markdown document per account per day.

  5. 05

    Automated a cleanup that removes knowledge older than three months, so answers stay current.

  6. 06

    Packaged it with Docker, health checks and start-up validation for reliable unattended runs.

Highlights

Accurate capture

Reads X’s API responses, not the page layout.

Incremental

Only new posts on every run.

Auto upload

Straight into COPAI’s knowledge base.

Runs unattended

Dockerised with health checks.

Built with

PythonPlaywrightX (Twitter)DockerCOPAI APIasyncio
Next projectV-sartSkincare e-commerce with AI face assessment. All projects