Shahmir Bin Asif

Projects

Grouped by where I did them, most recent first. Featured work opens in full; the rest sits one click away.

Civic IQ

July 2026 – present · Data Engineer

Data pipeline: sources are ingested into a warehouse, modelled, and served to dashboards App events CRM Spreadsheets Ingest Warehouse Models Dashboards
lower cost per new contact than the vendor scrapers
17x
of new contacts since launch come from the engine
98%
website platforms covered by configuration
20+

2026 · Owner, design to production

Config-driven contact scraping engine

One engine, configured per website platform instead of coded per website, now finds almost every new government contact the platform adds, at a fraction of what vendor scrapers cost.

  • Python
  • BigQuery
  • Cloud Run
  • Postgres
  • Gemini
  • pytest
  1. 01

    The problem

    Decision-maker contacts came from vendor scrapers that were expensive per contact, slow to extend to a new kind of website, and silent when a site changed its layout. Thousands of agency websites, built on dozens of platforms, were out of reach.

  2. 02

    What I built

    A scraping engine where each website platform is a configuration, not a code path. Pages are fetched and parsed, candidates are matched against existing contacts through a shared deduplication library, and results land in the warehouse with their provenance.

  3. 03

    Failing loudly

    The part I care most about. When a site changes its layout, the engine detects that its configuration is stale and says so, instead of reporting success with no data. Every run shows up on the observability dashboard with its cost.

Data pipeline: sources are ingested into a warehouse, modelled, and served to dashboards App events CRM Spreadsheets Ingest Warehouse Models Dashboards
contact records in scope
3.9M
rows deleted
0
unexplained changes
0

October 2026 · Lead

Production contact data cleanup

A nine-day cleanup of millions of live contact records, run as a sequence of reviewed SQL steps, with nothing deleted and every change traceable to the step that made it.

  • SQL
  • Postgres
  • BigQuery
  • Python
  • LLM as judge
  1. 01

    The problem

    Years of imports from different sources had left duplicates, broken names and stale records in the production contacts table. Sales teams were reading it live, so anything that touched it had to be reversible and explainable.

  2. 02

    The method

    Nineteen SQL steps in four batches over nine days, each with a dry run, a before-and-after count, and a retire-not-delete rule. Merges proposed by an LLM judge were checked by hand on a random sample before any of them was applied.

  3. 03

    The result

    Around 780 thousand stale records retired, around 100 thousand repaired, nothing deleted. The sample audit of the LLM-judged merges found no wrong merge, and the shared deduplication library that came out of it now runs in every ingestion path.

1 more project at Civic IQ

2026 · Sole author

Pipeline observability and cost attribution

One dashboard where the team sees what each pipeline costs and where it fails, backed by a shared failure taxonomy and per-call usage events bridged to cloud billing.

  • Python
  • BigQuery
  • PostHog
  • GCP Billing
  • pytest
Details

Before this, a failed run and a successful run with no output looked the same, and the AI and vendor spend behind each feature was only visible on the monthly invoice. I defined a failure taxonomy the pipelines report against, emitted a usage event for every AI and vendor call, bridged those events to the cloud billing export, and built the dashboard on top, with a test suite around it.

Motive

September 2024 – June 2026 · Solutions Engineer

Data pipeline: sources are ingested into a warehouse, modelled, and served to dashboards App events CRM Spreadsheets Ingest Warehouse Models Dashboards
engineers adopted it within a month
12
sprint throughput, from 8 tickets to 25
3x
faster query turnaround
60%

2026 · Designer and builder

AI workflow platform for the solutions team

A Claude Code based platform of MCP servers, agents and skills, with safety hooks, that 12 engineers adopted within a month. Sprint throughput went from 8 tickets to 25.

  • Claude Code
  • MCP
  • Python
  • Snowflake
  • GitHub Actions
  1. 01

    The problem

    Every bespoke analytics request started with the same manual work. Find the right data model, write the query, check it against the metadata, format the deliverable. The senior engineers who knew where everything lived were the bottleneck.

  2. 02

    What I built

    Seven MCP servers that expose the team's data models, metadata and tooling to the model, three agents and five skills that encode how the team does the work, and safety hooks so that nothing destructive runs without a person approving it.

  3. 03

    The result

    Twelve engineers were using it within a month. Sprint throughput went from 8 tickets to 25, queries came back about 60% faster, and the know-how that had lived in a few heads became something the whole team could run.

Data pipeline: sources are ingested into a warehouse, modelled, and served to dashboards App events CRM Spreadsheets Ingest Warehouse Models Dashboards
faster onboarding, from an hour to five minutes
12x
backlog, down from three to four weeks
0

2025 · Designer and builder

Metadata-driven Integration Marketplace pipeline

Onboarding a customer integration went from an hour of engineering to five minutes of configuration, which cleared a backlog of three to four weeks and let integrations be sold as recurring revenue.

  • Snowflake
  • SQL
  • Python
  • Airflow
  • Tableau
  1. 01

    The problem

    Each customer integration was its own small engineering job. An hour of work per integration does not sound like much until the queue is three to four weeks long and every request lands on the same people.

  2. 02

    What I built

    A pipeline that reads what an integration needs from metadata and builds the rest itself. Adding a customer became a five-minute configuration change that the solutions team could make without touching code.

  3. 03

    The result

    The backlog went to zero, and because an integration no longer cost engineering time, it could be packaged and sold as a recurring product rather than delivered as a favour.

2 more projects at Motive

2025 – 2026 · Data engineer in a three-person squad

Motive Data Bridge data models

Fifteen data models that give enterprise fleet customers their telematics in a warehouse they own, with cloud cost for the product cut by half.

  • Snowflake
  • SQL
  • Fivetran
  • Tableau
Details

Motive Data Bridge delivers a customer’s own fleet data into their warehouse. In a squad of three I designed and shipped 15 data models for its largest accounts and reworked how the product’s loads were scheduled and scoped, which halved its cloud cost.

2026 · Squad lead

Motive Fetch NG

Led a three-person squad building the next generation of Motive's file delivery pipeline on AWS, defined in YAML and deployed through GitHub Actions. In build when I left.

  • AWS
  • Lambda
  • Step Functions
  • Glue
  • DynamoDB
  • S3
  • GitHub Actions

Afiniti

February 2021 – August 2024 · Data Engineer (Associate Data Engineer until February 2022)

Data pipeline: sources are ingested into a warehouse, modelled, and served to dashboards App events CRM Spreadsheets Ingest Warehouse Models Dashboards
shorter nightly run, from over 4 hours to 42 minutes
88%
to recover from an incident, down from 3 hours
15 min
migrated across 90 tables
8 TB

2021 – 2022 · Associate Data Engineer, then Data Engineer

Self-healing pipeline and an 8 TB warehouse migration

Rebuilt an energy client's nightly pipeline to recover on its own and moved its warehouse from MySQL to Greenplum. Incidents went from 3 hours to 15 minutes, and the nightly run from over 4 hours to 42 minutes.

  • Python
  • SQL
  • Greenplum
  • MySQL
  • Airflow
  • Linux
  1. 01

    The problem

    Twenty-five pipelines moved five to six million records a day, and the largest one broke most nights. Every failure meant about three hours of manual recovery, usually at night, before the client's AI could pair calls the next morning.

  2. 02

    Self-healing first

    I rewrote the pipeline to recognise its own failure modes and restart from the last good point. Recovery dropped to 15 minutes per incident and the run from six hours to just over four, before the warehouse itself was touched.

  3. 03

    Then the migration

    Moved the 8 TB, 90-table warehouse from MySQL to Greenplum and re-partitioned the biggest tables, which brought the nightly run to 42 minutes. Ten behavioural features built on the faster warehouse lifted the client's customer retention by about 10%, and the work earned an early promotion.

3 more projects at Afiniti

2022 – 2024 · Subject matter expert

Greenplum migration programme

Led three client warehouse migrations to Greenplum as the programme's subject matter expert. Daily runtimes fell from 8 to 3 hours, 22 to 14 and 13 to 7, and the migration framework became the corporate standard.

  • Greenplum
  • SQL
  • Python
  • Airflow
Details

After the first migration I wrote up the method as a reusable framework and led the next three with it, for an insurer, a broadcaster and a telecom. It was adopted as the company standard for warehouse migrations, and I mentored the two junior engineers who ran the later ones.

2022 – 2024 · Data engineer

Telecom analytics platform

Ran the analytics platform for a UK telecom client, about 1 TB a day across 30 feeds and 50 Airflow jobs. The customer lifetime value analysis fed the contract renewal, and revenue reconciliation caught anomalies worth about £15K.

  • Airflow
  • PySpark
  • Greenplum
  • SQL
  • Talend
Details

I moved the legacy Talend jobs to PySpark behind a metadata-driven ingestion framework, so adding a feed became configuration rather than a new job. On the analysis side, the lifetime value model was used in the client’s contract renewal, and the reconciliation checks I added caught revenue anomalies worth about £15K.

2021 – 2022

Commission governance framework

Designed the governance framework for commission data, tested against 24 client datasets across telecom, energy and finance, and adopted as the corporate standard.

  • SQL
  • Python
  • Greenplum

Personal projects

Ongoing · Side projects

Data pipeline: sources are ingested into a warehouse, modelled, and served to dashboards App events CRM Spreadsheets Ingest Warehouse Models Dashboards
specialised agents in four waves
9
independent mappings between posting and profile
3

2026 · Side project

Job Application Engine

A multi-agent Claude Code tool that turns a job posting into a tailored CV and cover letter, with every claim checked word for word against my profile before anything is rendered.

  • Claude Code
  • Python
  • YAML
  • JSON Schema
  • python-docx
  • Word automation
  1. 01

    The problem

    Tailoring a CV to each posting by hand is slow, and tailoring it with a language model alone invents things. I wanted the speed without the invention.

  2. 02

    Agents with a validator between them

    One agent extracts the posting's keywords, three map them to my profile from different directions, and every claim carries an exact quote. A script rejects any quote that is not in the profile. A fit check decides whether to apply at all before a word is written.

  3. 03

    Fixed layout, checked output

    The writing agents produce structured YAML, not documents. Deterministic code renders it into one fixed layout, checks that every value reached the page, and only then exports the PDF through Word.

1 more project at Personal projects

Ongoing · Side project

Cricket analytics platform

An end-to-end analytics stack for ball-by-ball cricket data, from PostgreSQL through dbt models to Power BI, running in Docker. Early stage.

  • PostgreSQL
  • Python
  • dbt
  • Power BI
  • Docker
Details