---
title: Backfills at Suno
description: Guide to safely running backfills
---

## Before you start your backfills

If you need to backfill missing fields on a row in a table, make sure you're already filling that row in for newly created rows in said table. You don't want to start the backfill, then realize recently created rows in your table are missing the data you need.

## Backfill steps

### 1. Write your backfill script.

Create a Django command to backfill the data you need. Your backfill script should be [idempotent](https://www.prefect.io/blog/the-importance-of-idempotent-data-pipelines-for-resilience)!

### 2. Download the PEM file for accessing the backfill EC2 instance

Go to the [suno-creds S3 bucket](https://us-east-2.console.aws.amazon.com/s3/buckets/suno-creds?bucketType=general&region=us-east-2&tab=objects#) and download the rider-backfill.pem file. Once downloaded, set the file permissions to be RO for the file owner only (`chmod 400 ~/path/to/rider-backfill.pem`).

### 3. Allowlist your IP address

If it's not there already, add your IP address to [this security group in AWS](https://us-east-2.console.aws.amazon.com/ec2/home?region=us-east-2#SecurityGroup:group-id=sg-0d04945259b1869fd). Run `curl ifconfig.me` to find it.

### 4. SSH into the Backfill EC2 instance

`ssh -i ~/path/to/rider-backfill.pem ubuntu@ec2-3-149-7-153.us-east-2.compute.amazonaws.com`

TODO(Rider): Update with new instance once that's spun up.

### 5. Run your script

Keep an eye on Sentry for possible elevated error rates + whatever DD metrics you've instrumented.

Please don't kill threads/processes on this instance. This is a shared EC2 instance with other backfills potentially running in parallel.