00Why this document exists, and what actually happened

The short version, for the days I only have time to reread one section.

I didn't want a repeat of the IDMS frontend recovery, where a migration moved the backend and quietly left the frontend behind until a real visitor hit an error page. That's why every phase had a matching verification step immediately after it, and it's exactly what caught the real problems below before they reached anyone: nothing here got marked done because a script exited zero, only once I'd checked it against the live system myself.

01Source environment, as it stood before the move

Everything below I read straight off the old box while it was still live and serving traffic. It's now the rollback safety net, described in section 10, not the primary environment.

1.1 The EC2 instance

I found the disk completely full, 29 of 29 GB used, while first gathering numbers for this document. It had silently broken the 6-hourly backup cron since around 5 September: database or disk is full on every run, so no successful backup reached S3 for six days. I cleared Docker build cache and dangling images, freed about 5.7 GB, confirmed both containers stayed untouched the whole time, and manually ran the backup script to confirm a fresh upload landed in S3. Closed. The 50 GB disk I'm speccing for the new box in section 06 is partly a direct response to this.

1.2 Security group rules

1.3 The two Docker containers, side by side

Both containers set TZ=Asia/Kolkata and D1_FETCH_DISABLED=1. That second flag matters more than it looks: the server's own attempt to fetch D1 macro data directly is deliberately disabled, because AWS's IP ranges are blocked by the data source. D1 only ever arrives via the ingestion Mac's push, section 05. If I forget this flag on the new server, nothing will look wrong until someone notices D1 has gone quiet.

The one asymmetry that matters for migration planning: since only /home/ubuntu/ritam has a build: directive, that directory is the single source of truth the shared image is built from. Whatever is missing or wrong on disk there is what ships to both environments. That's exactly the shape of the 11 September incident in section 11.

1.4 nginx and TLS

One nginx process serves all five public hostnames, routed by server_name, each proxying to its matching container port on localhost. Two separate Let's Encrypt certificate bundles cover them:

Both renew automatically via certbot through /etc/cron.d/certbot, twice a day, only actually renewing inside the last 30 days before expiry. Neither is close to expiring, which is exactly the condition I want to migrate under: no clock forcing my hand. Full vhost text is reproduced in section 14 so I never have to re-type anything off a live terminal mid-migration.

02Codebase inventory, current state

File and line counts below come from my local ritam-signals-stg repo, which is what actually gets rebuilt into the Docker image. The live server keeps no separate source checkout of its own outside the image: it's a plain rsync target, not a git clone, which is itself part of the story in section 11.

2.1 Full directory tree, every file, expandable

I got burned once already by a tree that only showed folder rollups. It looked tidy right up until it hid the exact nine files that were missing. Every row below is a real file with its real line count, not a summary.

2.2 Runtime dependencies, requirements-server.txt, the file that actually ships

# Server (AWS) backend deps. NO Playwright here, the browser and scraper stay on the Mac.
fastapi==0.111.0
uvicorn[standard]==0.30.1
requests==2.31.0
cloudscraper==1.2.71
yfinance==0.2.40
openpyxl==3.1.2
python-multipart==0.0.9
pytz==2024.1
scipy==1.13.1
scikit-learn>=1.4.0

The scikit-learn line is new as of 11 September. I'd left it out originally. backend/execution/ml_gate.py imports sklearn.linear_model.LogisticRegression only inside a function body, so the gap sat invisible until the code path that touches it actually ran. I've since added it to this file, rebuilt the image, and confirmed sklearn imports clean in the running container. The real story, and the much bigger thing I found while fixing it, is in section 11 under the nine missing execution files. This tree and this file both describe the source exactly as it stands right now, already fixed, not a future state I'm planning toward.

2.3 Dockerfile

FROM python:3.12-slim

ENV TZ=Asia/Kolkata \
    PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1

RUN apt-get update && apt-get install -y --no-install-recommends tzdata \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY requirements-server.txt .
RUN pip install --no-cache-dir -r requirements-server.txt

COPY . .

EXPOSE 8001
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8001"]

This Dockerfile is architecture-neutral: no arch-pinned base image tag. Built on an ARM host it produces an arm64 image; built on x86 it produces amd64. That matters for section 06: as long as the new EC2 is also Graviton, t4g family, the exact same build command produces a compatible image with zero source changes. I'll also note, for the new box, that COPY . . is the line that copies everything not excluded by .dockerignore, including a live database if I ever forget to exclude data/ there, which is exactly what happened on 11 September, see section 11.

03Database reconnaissance

Both databases are SQLite, one file each, bind-mounted from the host so they survive container rebuilds. Row counts below are live, queried directly against the running containers.

Row counts by table

04Network, DNS & SSL, current state and what changes

All five hostnames currently point at the one EC2's public IP. Checked live via dig against Google's resolver. I'm laying out both the before and the after here in full, one row per hostname, rather than writing out one example and waving at the rest. That shortcut is exactly what made an earlier draft of this document thinner than it should have been.

4.1 DNS, before → after, every hostname

Nameservers stayed ns43.domaincontrol.com and ns44.domaincontrol.com, GoDaddy, the same account already used for kkdtech.com and databuddhi.in. My existing Personal Access Token had confirmed read/write access to this domain's zone, so the same automated cutover approach I'd used on those other migrations applied here with no new credential, once I got the auth scheme right, see the correction below. I'd set every TTL to 600 seconds ahead of the cutover so a rollback would be bounded to 10 minutes worst case, and every one of the four A records propagated on the first check, no waiting required.

4.2 TLS certificates, before → after

I never copy a certificate across boxes. Let's Encrypt certs are cheap and instant to reissue, and a copied private key is one more secret I'd have to move and account for. The new box got its own certs, issued by its own certbot, only once DNS had actually propagated to it. Both issuances succeeded on the first attempt, confirmed with curl -vI against all five hostnames afterward.

4.3 The literal commands, every hostname, no shortcuts

This is the exact sequence I ran in section 08, phases 5 and 6. Two corrections from what I originally drafted here, both learned live: the token in vault/other/godaddy/kkdtech-dns-fix-pat.txt is a dated note, not a bare token, the real PAT is on its own line inside the file, and sso-key auth fails on this token, it needs Bearer specifically, which is what's shown below.

curl -X PUT \
  -H "Authorization: Bearer <godaddy-token, section 12>" \
  -H "Content-Type: application/json" \
  -d '[{"data":"13.126.39.168","ttl":600}]' \
  "https://api.godaddy.com/v1/domains/ritamsignals.in/records/A/stg"

curl -X PUT \
  -H "Authorization: Bearer <godaddy-token>" \
  -H "Content-Type: application/json" \
  -d '[{"data":"13.126.39.168","ttl":600}]' \
  "https://api.godaddy.com/v1/domains/ritamsignals.in/records/A/stg.api"

dig +short stg.ritamsignals.in @8.8.8.8
dig +short stg.api.ritamsignals.in @8.8.8.8
# Both must print the new box's IP before the next line runs.

sudo certbot --nginx -d stg.ritamsignals.in -d stg.api.ritamsignals.in

curl -X PUT \
  -H "Authorization: Bearer <godaddy-token>" \
  -H "Content-Type: application/json" \
  -d '[{"data":"13.126.39.168","ttl":600}]' \
  "https://api.godaddy.com/v1/domains/ritamsignals.in/records/A/@"

curl -X PUT \
  -H "Authorization: Bearer <godaddy-token>" \
  -H "Content-Type: application/json" \
  -d '[{"data":"13.126.39.168","ttl":600}]' \
  "https://api.godaddy.com/v1/domains/ritamsignals.in/records/A/api"

# www is a CNAME to @, it moves on its own once the line above runs.
# Confirm rather than assume:
dig +short www.ritamsignals.in @8.8.8.8

dig +short ritamsignals.in @8.8.8.8
dig +short www.ritamsignals.in @8.8.8.8
dig +short api.ritamsignals.in @8.8.8.8

sudo certbot --nginx -d ritamsignals.in -d www.ritamsignals.in -d api.ritamsignals.in

api.ritamsignals.in is the one record I treat with the most suspicion, since the iOS app talks to it directly with no way to redirect except DNS itself. I exercise the app against the new box, over its raw IP with a Host header override if needed, before I ever flip that specific record.

05The ingestion Mac dependency

This is the dependency I'm most likely to forget, precisely because it lives on a different machine and I don't log into it day to day.

What actually runs there

The launcher auto-stops both processes at 15:35 IST and restarts fresh each day via RunAtLoad plus a StartCalendarInterval entry in the plist. It needs the Mac awake on schedule, which is separate, already-documented infrastructure I keep in my general Mac connectivity reference, not repeated here.

06Destination decision

The account move itself, 964222105432 to 605262305394, is the easy part. Where inside that account is the decision that actually matters.

What I'm provisioning instead

07Pre-migration safety checklist

Every item here should show a checkmark before phase 1 of the actual procedure begins. None of these are optional, and none of them get skipped because I'm in a hurry.

• Confirm my current Mac public IP and update both source and destination security groups. This drifts on home ISPs and has already caused a silent SSH lockout once during this document's own research. Check with curl -4 ifconfig.me before every session that touches either box.
• Run a fresh, verified backup of both databases using sqlite3 .backup, never a raw file copy, and confirm both land in S3 with today's timestamp.
• Confirm disk headroom on the source box is above 20 percent free before starting any data transfer, so I don't repeat the section 01 incident mid-migration.
• Confirm both certificates in section 01 still show more than 14 days of runway, so a renewal race isn't competing with the migration for attention.
• Confirm the ingestion Mac's launchd job and its wake schedule are healthy, and pick a window it isn't actively pushing data in.
• Rebuild the image on the source box first, exactly as I'll do it on the new one, and confirm it starts clean before touching anything else. The 11 September incident happened precisely because I skipped straight to trusting a rebuild I hadn't watched closely. Section 11 has the full account.
• Announce the maintenance window, even if only to myself, with an exact start and end time. A migration with no time box tends to expand.
• Staging migrates first, completely, verified, before production migration even begins. Never both in the same session.

08Procedure, phase by phase

Commands are written to be copied exactly. Wherever a path, key name, or credential file is referenced, its exact name is given; section 12 has what each one is and where it lives.

# 1. Dedicated security group, not reused from another project
aws ec2 create-security-group --profile sunnyxetl-cli --region ap-south-1 \
  --group-name ritam-new-sg --description "Ritam Signals, dedicated, migrated 2026-09" \
  --vpc-id vpc-060c5b91a4970d8a1
# -> sg-0f522aa8842b1a427

# 2. Same three ports as the source, section 01.2
aws ec2 authorize-security-group-ingress --profile sunnyxetl-cli --region ap-south-1 \
  --group-id sg-0f522aa8842b1a427 --protocol tcp --port 22 --cidr 182.69.183.56/32
aws ec2 authorize-security-group-ingress --profile sunnyxetl-cli --region ap-south-1 \
  --group-id sg-0f522aa8842b1a427 --protocol tcp --port 80 --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress --profile sunnyxetl-cli --region ap-south-1 \
  --group-id sg-0f522aa8842b1a427 --protocol tcp --port 443 --cidr 0.0.0.0/0

# 3. Launch, t4g.small, 50 GB gp3, matching the source's architecture
aws ec2 run-instances --profile sunnyxetl-cli --region ap-south-1 \
  --image-id ami-07c2efafe814d8860 --instance-type t4g.small \
  --key-name ritam-migrated-key --security-group-ids sg-0f522aa8842b1a427 \
  --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":50,"VolumeType":"gp3"}}]' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=ritam-migrated}]'
# -> i-0233aa6c5dfb48312, public IP 13.126.39.168

# 4. IAM: new role + instance profile, scoped to a NEW bucket, not a copy
# of the source account's role (can't be attached cross-account anyway).
# Bucket: ritam-db-backups-605262305394. Same inline policy shape as the
# source's ritam-ec2-backup, see section 12.

# Push the current repo to the new box. Excludes local-only artifacts.
rsync -avz --exclude data --exclude node_modules --exclude .git \
  ~/Desktop/sunny2702codes/ritam-signals-stg/ \
  ritam-new-ec2:/home/ubuntu/ritam-stg/

# requirements-server.txt already has scikit-learn as of 11 September,
# so there's no extra dependency step here anymore. The rsync above
# carries the fix across as part of the normal source tree.

# Build. Same Dockerfile, ARM host, produces a compatible arm64 image.
ssh ritam-new-ec2 "cd /home/ubuntu/ritam-stg && docker build -t ritam-backend ."

# Before trusting the build, sanity-check every import that has ever
# bitten me, in a throwaway container, before touching a real one.
ssh ritam-new-ec2 'docker run --rm ritam-backend python3 -c "
import pytz, scipy, sklearn
from backend.execution import munshi_v2, munshi_retrospective, munshi_calibration
from backend.execution.ml_gate import check_ml_gate
print(\"ALL IMPORTS OK\")
"'

# WAL-safe snapshot, the same technique my existing backup script uses.
sqlite3 /home/ubuntu/ritam-stg/data/ritamv2.db ".backup '/tmp/migrate-stg.db'"
gzip /tmp/migrate-stg.db

# Don't forget these two, easy to miss, see section 03.
cp /home/ubuntu/ritam-stg/data/dss_journal.sqlite3 /tmp/
cp /home/ubuntu/ritam-stg/data/dss_vwap.sqlite3 /tmp/

# The config directory as it exists live, not the repo's checked-in
# copy, they've drifted before, see section 11.
tar czf /tmp/migrate-stg-config.tar.gz -C /home/ubuntu/ritam-stg config

scp ritam-ec2:/tmp/migrate-stg.db.gz         ./
scp ritam-ec2:/tmp/dss_journal.sqlite3       ./
scp ritam-ec2:/tmp/dss_vwap.sqlite3          ./
scp ritam-ec2:/tmp/migrate-stg-config.tar.gz ./

scp ./migrate-stg.db.gz         ritam-new-ec2:/tmp/
scp ./dss_journal.sqlite3       ritam-new-ec2:/tmp/
scp ./dss_vwap.sqlite3          ritam-new-ec2:/tmp/
scp ./migrate-stg-config.tar.gz ritam-new-ec2:/tmp/

mkdir -p /home/ubuntu/ritam-stg/data
gunzip -c /tmp/migrate-stg.db.gz > /home/ubuntu/ritam-stg/data/ritamv2.db
cp /tmp/dss_journal.sqlite3 /tmp/dss_vwap.sqlite3 /home/ubuntu/ritam-stg/data/
tar xzf /tmp/migrate-stg-config.tar.gz -C /home/ubuntu/ritam-stg/

# Start the container against the migrated data and config
cd /home/ubuntu/ritam-stg && docker compose up -d

# Confirm this is the migrated data, not an empty fresh DB
docker exec ritam-backend-stg python3 -c "import sqlite3; c=sqlite3.connect('/app/data/ritamv2.db'); print(c.execute('SELECT COUNT(*) FROM trades').fetchone())"
# Expect: (28,). If this prints (0,), the migration copied the wrong file.

09Post-migration verification

I run this exact list after every phase-5 completion, staging and production separately. Nothing is done until every line here is confirmed against the live system, not assumed from a command that exited zero.

• curl https://<host>/api/health on every one of the five hostnames returns 200
• Row counts on the new box's database match the source at the moment of the snapshot, not just table existence; the exact query is in phase 4
• The two DSS sqlite files are present and non-empty on the new box
• The live config on the new box, fetched via GET /api/config/munshi, matches the live config on the old box key for key, not the checked-in repo copy, which has drifted before
• The old box's matching container is stopped, not deleted, and its data directory confirmed intact; that's the rollback safety net
• The ingestion Mac's next scheduled push after DNS propagation is confirmed in logs/scraper.log and logs/d1_inject.log to have actually reached the new box, not failed silently against the old IP from a cached DNS entry
• Certificate on the new box is valid, confirmed with curl -vI, not just that certbot exited without an error
• For production only: the iOS app, which talks to api.ritamsignals.in directly, is exercised once against the new box before the old one is stopped
• Both containers still running 20+ seconds after recreation, not caught in a restart loop; docker ps should show a growing uptime, not a repeatedly resetting one, per the lesson in section 11

10Rollback plan

The design made rollback a DNS change, not a data-recovery operation, for as long as the old box stayed up. That window is now closed, on purpose, sooner than planned.

11Known issues and the counters I've prepared in advance

Every one of these I hit, confirmed, or found during real work on this system. Nothing here is theoretical.

The home ISP IP drift

My Mac's public IP changes without warning. Every SSH lockout I've hit on this box traced back to this. Counter: check curl -4 ifconfig.me at the start of any session that will touch either EC2, before assuming SSH is broken for any other reason.

Config drift between the live server and the checked-in repo

The live munshi_config.json has differed from the repo's own checked-in copy more than once, because live tuning changes get made directly on the server and don't always get ported back. Counter: phase 4 explicitly pulls config from the live host, never from my local repo, and section 09 explicitly diffs live config on both boxes rather than trusting a copied file.

Docker build and transfer through a colima VM can silently truncate a saved image

nerdctl save -o <path> through a virtiofs-mounted host path has silently truncated the resulting tar before, missing its trailing manifest, with the error "error copying stream: file already closed". Counter: save to the VM's own native filesystem first, for example /tmp inside the VM, then copy that file out to the host, rather than saving directly to a host-mounted path. Phase 3 of this migration sidesteps the whole problem by building directly on the destination box instead of saving and transferring an image, which is simpler and avoids this class of bug entirely.

Never override the dte_block or near_max_pain_penalty gates to force an entry

Not a migration risk, a standing rule from this system's own trading history: the one time both were overridden to force a trade, the system was proven right to have blocked it and the forced trade lost money. Included here only so a future incident review doesn't mistake it for a new discovery.

A disk that quietly fills up breaks backups before it breaks anything visible

Documented in section 01. The backup cron failed silently for six days before I noticed, because a full disk produces an error in a log file nobody was tailing, not a visible outage. Counter for the new box: the 50 GB sizing in section 06 is deliberately generous, and a periodic disk-usage check should be a standing habit, not something I only discover while researching an unrelated migration.

12Credential and key index

Names and locations only. No actual secret values are written on this page, even behind the pass code gate. Every item below resolves to a real file or a real vault entry that already exists.

13Directory tree, after migration

I'm not repeating all 137 files a second time here. Section 02.1 already lists every one of them, expandable, and it would only drift out of sync with a second copy sitting in this section. What's below is the delta: exactly what changes about the tree's context, not its contents, once phases 3 and 4 are done.

Because I already fixed both the sklearn gap and the nine missing files directly on the live source before writing this section, there is no remaining source change this migration needs to carry. Everything that differs after the move is infrastructure around the code (the account, the IP, the certificates), not the code itself. That's the entire point of having built this as a portable Docker container with bind-mounted state in the first place.

14Appendix, raw configuration files

Reproduced in full so I never have to re-type anything off a live server terminal during the actual work.

Production nginx vhost

# Main site: ritamsignals.in + www -> React UI, /api -> backend
server {
    server_name ritamsignals.in www.ritamsignals.in;
    client_max_body_size 5m;
    root /var/www/ritam;
    index index.html;

    location /api/ {
        proxy_pass http://127.0.0.1:8001;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /assets/ {
        add_header Cache-Control "public, max-age=31536000, immutable";
    }

    location = /index.html {
        add_header Cache-Control "no-cache, no-store, must-revalidate";
    }

    location / {
        try_files $uri $uri/ /index.html;
    }

    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/ritamsignals.in/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ritamsignals.in/privkey.pem;
}

# API host: api.ritamsignals.in -> backend, everything
server {
    server_name api.ritamsignals.in;
    client_max_body_size 5m;

    location / {
        proxy_pass http://127.0.0.1:8001;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/ritamsignals.in/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ritamsignals.in/privkey.pem;
}

server {
    listen 80;
    server_name ritamsignals.in www.ritamsignals.in api.ritamsignals.in;
    return 301 https://$host$request_uri;
}

Staging nginx vhost

server {
    server_name stg.api.ritamsignals.in;
    add_header X-Robots-Tag "noindex, nofollow" always;

    location / {
        proxy_pass         http://127.0.0.1:8002;
        proxy_set_header   Host              $host;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_read_timeout 120s;
    }

    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/stg.ritamsignals.in/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/stg.ritamsignals.in/privkey.pem;
}

server {
    server_name stg.ritamsignals.in;
    root  /var/www/ritam-stg;
    index index.html;
    add_header X-Robots-Tag "noindex, nofollow" always;

    location /api {
        proxy_pass         http://127.0.0.1:8002;
        proxy_set_header   Host              $host;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_read_timeout 120s;
    }

    location / {
        try_files $uri $uri/ /index.html;
    }

    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/stg.ritamsignals.in/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/stg.ritamsignals.in/privkey.pem;
}

The backup script, fixed and running again

#!/bin/bash
set -e
BUCKET=ritam-db-backups-964222105432
DB=/home/ubuntu/ritam/data/ritamv2.db
TS=$(date +%Y%m%d-%H%M%S)
TMP=/tmp/ritamv2-$TS.db
sqlite3 "$DB" ".backup '$TMP'"
gzip -f "$TMP"
/usr/local/bin/aws s3 cp "$TMP.gz" "s3://$BUCKET/backups/ritamv2-$TS.db.gz" --region ap-south-1 --only-show-errors
/usr/local/bin/aws s3 cp "$TMP.gz" "s3://$BUCKET/ritamv2-latest.db.gz" --region ap-south-1 --only-show-errors
rm -f "$TMP.gz"
logger "ritam_backup: uploaded ritamv2-$TS.db.gz"

Runs every 6 hours via /etc/cron.d/ritam-backup as root, logs to /var/log/ritam_backup.log. I install the same script, unchanged, on the new box during phase 1, pointed at the new box's own data path.