Migration runbook: ritam-cli → sunnyxetl-cli
This is my own runbook for moving RITAM SIGNALS, both production and staging, off the EC2 it has lived on since June and onto a dedicated box in my sunnyxetl-cli account. I wrote it so that either I, or anyone I bring on to help me run it, can execute the move end to end without guessing what a path, a flag, or a credential file means. Nothing below has been executed. I re-checked every fact in it directly against the running system rather than trusting my own memory or an old note, and I updated it again on 11 September after a real incident on the live server taught me something the first draft got wrong.
00Why this document exists
The short version, for the days I only have time to reread one section.
I don't want a repeat of the IDMS frontend recovery, where a migration moved the backend and quietly left the frontend behind until a real visitor hit an error page. So every phase below has a matching verification step immediately after it, production and staging are never touched in the same sitting, and a step counts as done once I've checked it against the live system myself, not because a script exited zero.
01Source environment, as it stands today
Everything below I read straight off the running box, most of it more than once, since the 11 September incident changed a few numbers.
1.1 The EC2 instance
| Field | Value |
|---|---|
| Instance ID | i-08b2a63d4eccf9ef0 |
| Public IP | 13.203.121.240 |
| Type | t4g.small |
| Architecture | ARM64, Graviton |
| Region and AZ | ap-south-1c |
| Disk | 30 GB gp3, 3000 IOPS |
| SSH key pair | ritam-key |
| IAM instance profile | ritam-ec2-backup |
| Security group | ritam-sg / sg-0542ba8064e3a4218 |
| Account | 964222105432 (ritam-cli) |
I found the disk completely full, 29 of 29 GB used, while first gathering numbers for this document. It had silently broken the 6-hourly backup cron since around 5 September: database or disk is full on every run, so no successful backup reached S3 for six days. I cleared Docker build cache and dangling images, freed about 5.7 GB, confirmed both containers stayed untouched the whole time, and manually ran the backup script to confirm a fresh upload landed in S3. Closed. The 50 GB disk I'm speccing for the new box in section 06 is partly a direct response to this.
1.2 Security group rules
| Port | Source | Purpose |
|---|---|---|
| 22 | current Mac IP only, /32 | SSH. This drifts, see the gotcha in section 11. |
| 80 | 0.0.0.0/0 | HTTP, redirects to HTTPS |
| 443 | 0.0.0.0/0 | HTTPS, both prod and stg vhosts |
1.3 The two Docker containers, side by side
| Production | Staging | |
|---|---|---|
| Container name | ritam-backend | ritam-backend-stg |
| Image | ritam-backend | same image, reused (stg has no build: of its own) |
| Host port | 8001 | 8002 |
| Internal port | 8001 | 8001, mapped from 8002 |
| Host directory | /home/ubuntu/ritam | /home/ubuntu/ritam-stg |
| Data volume | ./data:/app/data | ./data:/app/data |
| Config volume | ./config:/app/config | ./config:/app/config |
| Extra env var | none | RITAM_ENV=staging |
Both containers set TZ=Asia/Kolkata and D1_FETCH_DISABLED=1. That second flag matters more than it looks: the server's own attempt to fetch D1 macro data directly is deliberately disabled, because AWS's IP ranges are blocked by the data source. D1 only ever arrives via the ingestion Mac's push, section 05. If I forget this flag on the new server, nothing will look wrong until someone notices D1 has gone quiet.
The one asymmetry that matters for migration planning: since only /home/ubuntu/ritam has a build: directive, that directory is the single source of truth the shared image is built from. Whatever is missing or wrong on disk there is what ships to both environments. That's exactly the shape of the 11 September incident in section 11.
1.4 nginx and TLS
One nginx process serves all five public hostnames, routed by server_name, each proxying to its matching container port on localhost. Two separate Let's Encrypt certificate bundles cover them:
| Certificate name | Covers | Expiry |
|---|---|---|
| ritamsignals.in | ritamsignals.in, www.ritamsignals.in, api.ritamsignals.in | 13 November 2026 |
| stg.ritamsignals.in | stg.ritamsignals.in, stg.api.ritamsignals.in | 27 November 2026 |
Both renew automatically via certbot through /etc/cron.d/certbot, twice a day, only actually renewing inside the last 30 days before expiry. Neither is close to expiring, which is exactly the condition I want to migrate under: no clock forcing my hand. Full vhost text is reproduced in section 14 so I never have to re-type anything off a live terminal mid-migration.
02Codebase inventory, current state
File and line counts below come from my local ritam-signals-stg repo, which is what actually gets rebuilt into the Docker image. The live server keeps no separate source checkout of its own outside the image: it's a plain rsync target, not a git clone, which is itself part of the story in section 11.
2.1 Full directory tree, every file, expandable
I got burned once already by a tree that only showed folder rollups. It looked tidy right up until it hid the exact nine files that were missing. Every row below is a real file with its real line count, not a summary.
backend/
analysis/
d1/
d2/
d3/
dss/
data/
engine/
tests/
execution/
fetchers/
news/
utils/
frontend/src/
components/
pages/
pages/settings/
lib/
hooks/
everything else at the repo root
2.2 Runtime dependencies, requirements-server.txt, the file that actually ships
# Server (AWS) backend deps. NO Playwright here, the browser and scraper stay on the Mac.
fastapi==0.111.0
uvicorn[standard]==0.30.1
requests==2.31.0
cloudscraper==1.2.71
yfinance==0.2.40
openpyxl==3.1.2
python-multipart==0.0.9
pytz==2024.1
scipy==1.13.1
scikit-learn>=1.4.0
The scikit-learn line is new as of 11 September. I'd left it out originally. backend/execution/ml_gate.py imports sklearn.linear_model.LogisticRegression only inside a function body, so the gap sat invisible until the code path that touches it actually ran. I've since added it to this file, rebuilt the image, and confirmed sklearn imports clean in the running container. The real story, and the much bigger thing I found while fixing it, is in section 11 under the nine missing execution files. This tree and this file both describe the source exactly as it stands right now, already fixed, not a future state I'm planning toward.
2.3 Dockerfile
FROM python:3.12-slim ENV TZ=Asia/Kolkata \ PYTHONUNBUFFERED=1 \ PYTHONDONTWRITEBYTECODE=1 RUN apt-get update && apt-get install -y --no-install-recommends tzdata \ && rm -rf /var/lib/apt/lists/* WORKDIR /app COPY requirements-server.txt . RUN pip install --no-cache-dir -r requirements-server.txt COPY . . EXPOSE 8001 CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8001"]
This Dockerfile is architecture-neutral: no arch-pinned base image tag. Built on an ARM host it produces an arm64 image; built on x86 it produces amd64. That matters for section 06: as long as the new EC2 is also Graviton, t4g family, the exact same build command produces a compatible image with zero source changes. I'll also note, for the new box, that COPY . . is the line that copies everything not excluded by .dockerignore, including a live database if I ever forget to exclude data/ there, which is exactly what happened on 11 September, see section 11.
03Database reconnaissance
Both databases are SQLite, one file each, bind-mounted from the host so they survive container rebuilds. Row counts below are live, queried directly against the running containers.
The WAL file is the one thing that can silently lose data during a naive copy
SQLite in WAL mode keeps recent writes in a separate -wal file before folding them into the main .db file. Production currently carries a 244 MB WAL file that hasn't been checkpointed. A plain cp or rsync of just the .db file while the container is running would silently miss whatever is sitting in that WAL: on this system, potentially a meaningful number of the most recent option-chain snapshots. My existing backup script already does this correctly with sqlite3 "$DB" ".backup '$TMP'", which is WAL-aware and produces one consistent file. The migration procedure in section 08 uses the same approach. I never do a raw file copy of a live database.
| Production | Staging | |
|---|---|---|
| Main DB file size | 3.11 GB | 3.20 GB |
| WAL file, current | 244 MB, not checkpointed | 4.3 MB |
| Table count | 17 | 22 |
| Extra tables on staging | trade_dna, munshi_insights, munshi_calibration, nifty_ohlc_history, nse_daily_data, the Tier 1/2 ML layer additions, staging only | |
| Other DB files in the data dir | none | dss_journal.sqlite3 (1.7 MB), dss_vwap.sqlite3 (12 KB), easy to forget since they're not the main file |
| trades table row count | 0 | 28 |
Row counts by table
| Table | Production | Staging |
|---|---|---|
| munshi_decisions | 24,270 | 51,017 |
| d3_signals | 24,270 | 96,404 |
| oi_snapshots | 25,065 | 25,033 |
| oi_analysis | 25,064 | 25,032 |
| indicator_history | 37,548 | 31,353 |
| news_articles | 5,392 | 23,746 |
| daily_scores | 140 | 130 |
| sector_history | 1,945 | 1,735 |
| intraday_fii_snapshots | 1,939 | 1,729 |
| portfolio_state | 1 row | 1 row |
| kv_store | 59 | 14 |
| alert_log | 32 | 37 |
| page_unique_visitors | 43 | 29 |
Why production shows 0 trades while staging shows 28
Expected, not a symptom of anything broken. Production runs my conservative configuration: max_trades_per_day = 5, no average-down, no pyramiding, and the original tighter gate set. Staging is the deliberately looser environment, max_trades_per_day = 20, average-down and pyramiding both enabled, where every new gate (G11 through G15) gets proven before I'd ever consider it for production. The 28-trade history I reference elsewhere in this system's memory is staging's, not production's. Worth restating plainly since it's easy to forget: there's no real broker integration anywhere in this backend, portfolio_value is a plain number in SQLite. This whole system is simulated paper trading, not connected to a live exchange account or real capital.
04Network, DNS & SSL, current state and what changes
All five hostnames currently point at the one EC2's public IP. Checked live via dig against Google's resolver. I'm laying out both the before and the after here in full, one row per hostname, rather than writing out one example and waving at the rest. That shortcut is exactly what made an earlier draft of this document thinner than it should have been.
4.1 DNS, before → after, every hostname
| Hostname | Record | Current value | After migration | TTL |
|---|---|---|---|---|
| ritamsignals.in | A | 13.203.121.240 | <new-ec2-ip> | 600s |
| www.ritamsignals.in | A | 13.203.121.240 | <new-ec2-ip> | 600s |
| api.ritamsignals.in | A | 13.203.121.240 | <new-ec2-ip> | 600s |
| stg.ritamsignals.in | A | 13.203.121.240 | <new-ec2-ip> | 600s |
| stg.api.ritamsignals.in | A | 13.203.121.240 | <new-ec2-ip> | 600s |
Nameservers stay ns43.domaincontrol.com and ns44.domaincontrol.com, GoDaddy, the same account already used for kkdtech.com and databuddhi.in. My existing Personal Access Token already has confirmed read/write access to this domain's zone, so the same automated cutover approach I've used on those other migrations applies here with no new credential. I deliberately set every TTL to 600 seconds ahead of the cutover, so a rollback, if I ever need one, is bounded to 10 minutes worst case rather than whatever the previous TTL happened to be.
4.2 TLS certificates, before → after
| Bundle | Hostnames covered | Current expiry | After migration |
|---|---|---|---|
| ritamsignals.in | ritamsignals.in, www.ritamsignals.in, api.ritamsignals.in | 13 Nov 2026, old box | fresh cert, issued on the new box by certbot, same three hostnames, new 90-day clock |
| stg.ritamsignals.in | stg.ritamsignals.in, stg.api.ritamsignals.in | 27 Nov 2026, old box | fresh cert, issued on the new box by certbot, same two hostnames, new 90-day clock |
I never copy a certificate across boxes. Let's Encrypt certs are cheap and instant to reissue, and a copied private key is one more secret I'd have to move and account for. The new box gets its own certs, issued by its own certbot, once DNS has actually propagated to it. Never before that, since the HTTP-01 challenge certbot uses will fail if DNS still points at the old box when it runs.
4.3 The literal commands, every hostname, no shortcuts
This is the exact sequence I'll run in section 08, phases 5 and 6, spelled out here in one place so I'm not hunting across two phases for the staging version of a command and guessing at the production one.
curl -X PUT \ -H "Authorization: Bearer <godaddy-token, section 12>" \ -H "Content-Type: application/json" \ -d '[{"data":"<new-ec2-ip>","ttl":600}]' \ "https://api.godaddy.com/v1/domains/ritamsignals.in/records/A/stg" curl -X PUT \ -H "Authorization: Bearer <godaddy-token>" \ -H "Content-Type: application/json" \ -d '[{"data":"<new-ec2-ip>","ttl":600}]' \ "https://api.godaddy.com/v1/domains/ritamsignals.in/records/A/stg.api"
dig +short stg.ritamsignals.in @8.8.8.8
dig +short stg.api.ritamsignals.in @8.8.8.8
# Both must print the new box's IP before the next line runs.
sudo certbot --nginx -d stg.ritamsignals.in -d stg.api.ritamsignals.in
curl -X PUT \ -H "Authorization: Bearer <godaddy-token>" \ -H "Content-Type: application/json" \ -d '[{"data":"<new-ec2-ip>","ttl":600}]' \ "https://api.godaddy.com/v1/domains/ritamsignals.in/records/A/@" curl -X PUT \ -H "Authorization: Bearer <godaddy-token>" \ -H "Content-Type: application/json" \ -d '[{"data":"<new-ec2-ip>","ttl":600}]' \ "https://api.godaddy.com/v1/domains/ritamsignals.in/records/A/www" curl -X PUT \ -H "Authorization: Bearer <godaddy-token>" \ -H "Content-Type: application/json" \ -d '[{"data":"<new-ec2-ip>","ttl":600}]' \ "https://api.godaddy.com/v1/domains/ritamsignals.in/records/A/api"
dig +short ritamsignals.in @8.8.8.8 dig +short www.ritamsignals.in @8.8.8.8 dig +short api.ritamsignals.in @8.8.8.8 sudo certbot --nginx -d ritamsignals.in -d www.ritamsignals.in -d api.ritamsignals.in
api.ritamsignals.in is the one record I treat with the most suspicion, since the iOS app talks to it directly with no way to redirect except DNS itself. I exercise the app against the new box, over its raw IP with a Host header override if needed, before I ever flip that specific record.
05The ingestion Mac dependency
This is the dependency I'm most likely to forget, precisely because it lives on a different machine and I don't log into it day to day.
The good news, confirmed by reading the actual live launchd job, not the script's own comments
Every real invocation on the ingestion Mac targets the two domain names, never a raw IP. The live launchd plist at ~/Library/LaunchAgents/in.ritamsignals.ingest.plist starts run_mac_ingest.sh with BASE_URL defaulting to https://api.ritamsignals.in, and the staging mirror URL comes from a local config key, stg_base_url, again a domain, not an IP. A correctly executed DNS cutover, section 08, needs zero code or config changes on the Mac. It starts talking to the new server the moment DNS propagates, no separate deploy step there.
What actually runs there
| Process | Interval | Sends to |
|---|---|---|
| local_scraper.py | every 60 seconds, market hours only, 09:15 to 15:35 IST | Production primary, staging as a fire-and-forget mirror; mirror failures are silently swallowed by design |
| mac_d1_inject.py | every 15 minutes | Production primary, staging mirror |
The launcher auto-stops both processes at 15:35 IST and restarts fresh each day via RunAtLoad plus a StartCalendarInterval entry in the plist. It needs the Mac awake on schedule, which is separate, already-documented infrastructure I keep in my general Mac connectivity reference, not repeated here.
06Destination decision
The account move itself, 964222105432 to 605262305394, is the easy part. Where inside that account is the decision that actually matters.
I'm not folding this into wms-merged, the shared box already running kkdtech.com, IDMS, and CoopReach
That box is a t4g.small with roughly 700 MB of RAM free at last check, already carrying seven distinct services behind one Caddy instance. Ritam's backend runs a permanent 60-second scheduler tick, plus, once ML gates are fully active, real CPU load from scikit-learn model fits. That's a fundamentally heavier and more constant workload than the mostly idle static and light API traffic already on that box. Cramming it in risks exactly the kind of resource contention that would degrade a public-facing, government-adjacent site to save the cost of one more small EC2. The math doesn't work.
What I'm provisioning instead
| Field | My call | Why |
|---|---|---|
| Instance type | t4g.small | Same family as the source. The Dockerfile is architecture-neutral, so the identical build produces a compatible arm64 image with zero source changes, section 02. |
| Disk | 50 GB gp3 | The current 30 GB ran completely full once already. The two databases alone total 6.2 GB and both grow continuously from live snapshot tables. 50 GB gives real headroom instead of repeating section 01's incident on day one. |
| Region | ap-south-1 | Same as source. Keeps latency to NSE and to real users identical, and keeps the Mumbai-to-Mac ingestion path unchanged. |
| Account | 605262305394, sunnyxetl-cli | As I set out to do. Full admin already confirmed on this account from other work. |
| Security group | New, dedicated, not shared | Copy the exact same three rules from ritam-sg, section 01.2: 22 from my current Mac IP only, 80 and 443 open. I don't reuse an existing security group from another project. |
| IAM | New role, same shape as ritam-ec2-backup | A fresh instance profile with the same single inline policy, scoped to a new backup bucket in the destination account, not a copy of the source account's role, which can't be attached cross-account anyway. |
Why a brand-new EC2, not an existing box already in 605262305394
Every other box already in that account (wms-merged, apps-merged, vms-merged, gwims-merged) was sized for what it already runs. Adding ritam's ML-heavy, always-on workload to any of them repeats the exact wms-merged risk above under a different name. A dedicated box costs roughly what one more small instance already costs elsewhere in this account, and keeps the blast radius of any future ritam-specific problem contained to ritam alone, which is the same principle that made staging its own environment in the first place.
07Pre-migration safety checklist
Every item here should show a checkmark before phase 1 of the actual procedure begins. None of these are optional, and none of them get skipped because I'm in a hurry.
08Procedure, phase by phase
Commands are written to be copied exactly. Wherever a path, key name, or credential file is referenced, its exact name is given; section 12 has what each one is and where it lives.
Launch in the sunnyxetl-cli account, ap-south-1, using the sizing decided in section 06.
# 1. Dedicated security group, not reused from another project aws ec2 create-security-group --profile sunnyxetl-cli --region ap-south-1 \ --group-name ritam-new-sg --description "Ritam Signals, dedicated, migrated 2026-09" \ --vpc-id <destination-vpc-id> # 2. Same three ports as the source, section 01.2 aws ec2 authorize-security-group-ingress --profile sunnyxetl-cli --region ap-south-1 \ --group-id <new-sg-id> --protocol tcp --port 22 --cidr <current-mac-ip>/32 aws ec2 authorize-security-group-ingress --profile sunnyxetl-cli --region ap-south-1 \ --group-id <new-sg-id> --protocol tcp --port 80 --cidr 0.0.0.0/0 aws ec2 authorize-security-group-ingress --profile sunnyxetl-cli --region ap-south-1 \ --group-id <new-sg-id> --protocol tcp --port 443 --cidr 0.0.0.0/0 # 3. Launch, t4g.small, 50 GB gp3, matching the source's architecture aws ec2 run-instances --profile sunnyxetl-cli --region ap-south-1 \ --image-id <latest-ubuntu-22.04-arm64-ami> --instance-type t4g.small \ --key-name <new-key-pair-name> --security-group-ids <new-sg-id> \ --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":50,"VolumeType":"gp3"}}]' \ --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=ritam-migrated}]'
Install Docker, nginx, and certbot on the fresh box exactly as they exist on the source, then stop. I don't move to phase 2 until I can SSH into the new box and run docker ps cleanly.
Everything in phases 3 through 5 is written for one environment at a time. I run all of it for staging, verify it fully with section 09, and only then repeat the identical steps for production. If anything in staging's rehearsal surprises me, I fix the plan, not just the symptom, before touching production.
# Push the current repo to the new box. Excludes local-only artifacts. rsync -avz --exclude data --exclude node_modules --exclude .git \ ~/Desktop/sunny2702codes/ritam-signals-stg/ \ ritam-new-ec2:/home/ubuntu/ritam-stg/ # requirements-server.txt already has scikit-learn as of 11 September, # so there's no extra dependency step here anymore. The rsync above # carries the fix across as part of the normal source tree. # Build. Same Dockerfile, ARM host, produces a compatible arm64 image. ssh ritam-new-ec2 "cd /home/ubuntu/ritam-stg && docker build -t ritam-backend ." # Before trusting the build, sanity-check every import that has ever # bitten me, in a throwaway container, before touching a real one. ssh ritam-new-ec2 'docker run --rm ritam-backend python3 -c " import pytz, scipy, sklearn from backend.execution import munshi_v2, munshi_retrospective, munshi_calibration from backend.execution.ml_gate import check_ml_gate print(\"ALL IMPORTS OK\") "'
That last check exists because of the 11 September incident. I don't recreate a live container from a fresh image anymore without first proving the image itself boots clean in isolation. It takes ten seconds and would have caught the whole crash-loop before it ever touched a container that mattered.
# WAL-safe snapshot, the same technique my existing backup script uses. sqlite3 /home/ubuntu/ritam-stg/data/ritamv2.db ".backup '/tmp/migrate-stg.db'" gzip /tmp/migrate-stg.db # Don't forget these two, easy to miss, see section 03. cp /home/ubuntu/ritam-stg/data/dss_journal.sqlite3 /tmp/ cp /home/ubuntu/ritam-stg/data/dss_vwap.sqlite3 /tmp/ # The config directory as it exists live, not the repo's checked-in # copy, they've drifted before, see section 11. tar czf /tmp/migrate-stg-config.tar.gz -C /home/ubuntu/ritam-stg config
scp ritam-ec2:/tmp/migrate-stg.db.gz ./ scp ritam-ec2:/tmp/dss_journal.sqlite3 ./ scp ritam-ec2:/tmp/dss_vwap.sqlite3 ./ scp ritam-ec2:/tmp/migrate-stg-config.tar.gz ./ scp ./migrate-stg.db.gz ritam-new-ec2:/tmp/ scp ./dss_journal.sqlite3 ritam-new-ec2:/tmp/ scp ./dss_vwap.sqlite3 ritam-new-ec2:/tmp/ scp ./migrate-stg-config.tar.gz ritam-new-ec2:/tmp/
mkdir -p /home/ubuntu/ritam-stg/data gunzip -c /tmp/migrate-stg.db.gz > /home/ubuntu/ritam-stg/data/ritamv2.db cp /tmp/dss_journal.sqlite3 /tmp/dss_vwap.sqlite3 /home/ubuntu/ritam-stg/data/ tar xzf /tmp/migrate-stg-config.tar.gz -C /home/ubuntu/ritam-stg/ # Start the container against the migrated data and config cd /home/ubuntu/ritam-stg && docker compose up -d # Confirm this is the migrated data, not an empty fresh DB docker exec ritam-backend-stg python3 -c "import sqlite3; c=sqlite3.connect('/app/data/ritamv2.db'); print(c.execute('SELECT COUNT(*) FROM trades').fetchone())" # Expect: (28,). If this prints (0,), the migration copied the wrong file.
Install the equivalent nginx vhost on the new box first, pointed at the new container's port, and get it working over plain HTTP internally before touching DNS. Only once that's confirmed do I flip the DNS record, which is the moment the outside world starts hitting the new box. The full literal DNS and certbot commands for every hostname, staging and production both, are in section 04.3. I don't repeat them here to avoid two copies drifting apart.
Leave the old box's staging container running and untouched through this whole phase. It only becomes safe to stop once section 09's verification has fully passed on the new box.
Same steps, production paths and hostnames, exactly as phases 3 and 4 above, and the production block of section 04.3 for DNS and SSL. This is the step that touches a real domain, real users, and the iOS app that already points at it directly. I confirm explicitly before the DNS record for api.ritamsignals.in changes, since that's what the iOS app talks to with no way to redirect it except DNS.
09Post-migration verification
I run this exact list after every phase-5 completion, staging and production separately. Nothing is done until every line here is confirmed against the live system, not assumed from a command that exited zero.
10Rollback plan
The whole design of this migration makes rollback a DNS change, not a data-recovery operation, provided I never stop or delete the old box until section 09 has fully passed.
If anything looks wrong after the DNS cutover
Point the affected A record back at 13.203.121.240 using the exact same GoDaddy API call from section 04.3, with the old IP as the value. Propagation is bounded by the TTL, 600 seconds on every record above, so worst case the rollback completes within 10 minutes. The old container was never stopped during the whole procedure, so there's no restart, no data recovery, no lost writes to worry about; it just kept running the entire time as the safety net.
I only decommission the old box's containers and data directories after a full week of the new box running clean in production, not immediately after the cutover looks fine on day one.
11Known issues and the counters I've prepared in advance
Every one of these I hit, confirmed, or found during real work on this system. Nothing here is theoretical.
The flagship case study: nine missing execution files, 11 September 2026
I set out to do something small: add scikit-learn to requirements-server.txt and rebuild, since I'd confirmed import sklearn failed on the live container and the Tier 2 ML gate needs it. I added the line, ran docker compose build, and hit a disk-exhaustion error first. .dockerignore didn't exclude data/, so COPY . . tried to copy the live 3.11 GB database into the build layer. I fixed that, rebuilt clean, and then recreated both containers. They immediately crash-looped.
docker logs showed ModuleNotFoundError: No module named 'backend.execution.ml_gate': not a missing dependency, a missing file. I diffed the live server's backend/execution/ directory against my local repo and found nine files simply weren't there: ml_gate.py, munshi_calibration.py, munshi_retrospective.py, ensemble_gate.py, day_cluster.py, exit_simulator.py, feature_importance.py, market_classifier.py, online_learning.py, regime_exit.py. The live main.py already imported all of them. The old, already-built image had them baked in from some earlier build; they'd since vanished from the host directory itself, which isn't even a git checkout; git log there just says "not a git repository." Nobody noticed because the old image kept running, unrebuilt, the whole time. The missing files were the disease. The sklearn gap was one symptom that happened to be the one I went looking for.
I copied all nine files from my local repo, the one place they were still intact, onto the live host directory, rebuilt, verified every import that had ever failed plus pytz/scipy/scikit-learn clean in a throwaway container first, then recreated both containers for real. Confirmed stable after 20+ seconds, health checks returning 200 on both ports, zero data loss: production still 0 trades, staging still 28 closed, exactly as before. I deliberately didn't touch main.py itself: the live copy carries its own production-only patches (a trading-day skip, a stale-chain-data entry block, two admin endpoints) that don't exist in my local repo, and overwriting it would have deleted real, working safety logic in the name of fixing an unrelated gap.
Counter for the migration: phase 3's throwaway-container import check exists directly because of this. I never again trust that a rebuild will produce a working container just because it produced an image. I run the fresh image once in isolation and try every import that matters before I let it anywhere near a real container.
Residual, not yet fixed: trade_dna missing on production's schema
With exit_simulator.py running for the first time ever on production the night of the incident above, it logged sqlite3.OperationalError: no such table: trade_dna during its end-of-day task; that code path had simply never executed on production before, so production's schema never picked up the table staging already has. Harmless today since production's trades table is empty, but it'll log an error every night until the table exists. I'm leaving this alone deliberately: a production database schema change needs its own explicit decision, not something bundled quietly into an incident fix. Worth resolving before, or as part of, this migration. The new box should not inherit a schema gap I already know about.
The home ISP IP drift
My Mac's public IP changes without warning. Every SSH lockout I've hit on this box traced back to this. Counter: check curl -4 ifconfig.me at the start of any session that will touch either EC2, before assuming SSH is broken for any other reason.
Config drift between the live server and the checked-in repo
The live munshi_config.json has differed from the repo's own checked-in copy more than once, because live tuning changes get made directly on the server and don't always get ported back. Counter: phase 4 explicitly pulls config from the live host, never from my local repo, and section 09 explicitly diffs live config on both boxes rather than trusting a copied file.
Docker build and transfer through a colima VM can silently truncate a saved image
nerdctl save -o <path> through a virtiofs-mounted host path has silently truncated the resulting tar before, missing its trailing manifest, with the error "error copying stream: file already closed". Counter: save to the VM's own native filesystem first, for example /tmp inside the VM, then copy that file out to the host, rather than saving directly to a host-mounted path. Phase 3 of this migration sidesteps the whole problem by building directly on the destination box instead of saving and transferring an image, which is simpler and avoids this class of bug entirely.
Never override the dte_block or near_max_pain_penalty gates to force an entry
Not a migration risk, a standing rule from this system's own trading history: the one time both were overridden to force a trade, the system was proven right to have blocked it and the forced trade lost money. Included here only so a future incident review doesn't mistake it for a new discovery.
A disk that quietly fills up breaks backups before it breaks anything visible
Documented in section 01. The backup cron failed silently for six days before I noticed, because a full disk produces an error in a log file nobody was tailing, not a visible outage. Counter for the new box: the 50 GB sizing in section 06 is deliberately generous, and a periodic disk-usage check should be a standing habit, not something I only discover while researching an unrelated migration.
12Credential and key index
Names and locations only. No actual secret values are written on this page, even behind the pass code gate. Every item below resolves to a real file or a real vault entry that already exists.
| What | File or reference name | Where it lives |
|---|---|---|
| SSH key, current source box | ritam-key-new | vault/pem/recovered/ritam-key-new |
| SSH key pair name on AWS, source | ritam-key | AWS console, ritam-cli account, EC2 key pairs |
| SSH key pair, new destination box | to be generated during phase 1 | save to the same vault path pattern once created |
| GoDaddy DNS token | kkdtech-dns-fix-pat.txt | vault/other/godaddy/ |
| S3 backup bucket, source | ritam-db-backups-964222105432 | ritam-cli account, ap-south-1 |
| IAM instance profile, source | ritam-ec2-backup | inline policy named s3-backup, scoped to the bucket above only |
| Telegram bot token, Munshi Ji reports | telegram_bot_token | ~/ritam-stg/config/server_config.json on the live box, not in the repo |
| Full historical credential dump | reference_all_credentials.md | repo memory folder; re-verify anything I pull from it against the live system before trusting it |
13Directory tree, after migration
I'm not repeating all 137 files a second time here. Section 02.1 already lists every one of them, expandable, and it would only drift out of sync with a second copy sitting in this section. What's below is the delta: exactly what changes about the tree's context, not its contents, once phases 3 and 4 are done.
Because I already fixed both the sklearn gap and the nine missing files directly on the live source before writing this section, there is no remaining source change this migration needs to carry. Everything that differs after the move is infrastructure around the code (the account, the IP, the certificates), not the code itself. That's the entire point of having built this as a portable Docker container with bind-mounted state in the first place.
14Appendix, raw configuration files
Reproduced in full so I never have to re-type anything off a live server terminal during the actual work.
Production nginx vhost
# Main site: ritamsignals.in + www -> React UI, /api -> backend server { server_name ritamsignals.in www.ritamsignals.in; client_max_body_size 5m; root /var/www/ritam; index index.html; location /api/ { proxy_pass http://127.0.0.1:8001; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } location /assets/ { add_header Cache-Control "public, max-age=31536000, immutable"; } location = /index.html { add_header Cache-Control "no-cache, no-store, must-revalidate"; } location / { try_files $uri $uri/ /index.html; } listen 443 ssl; ssl_certificate /etc/letsencrypt/live/ritamsignals.in/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/ritamsignals.in/privkey.pem; } # API host: api.ritamsignals.in -> backend, everything server { server_name api.ritamsignals.in; client_max_body_size 5m; location / { proxy_pass http://127.0.0.1:8001; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } listen 443 ssl; ssl_certificate /etc/letsencrypt/live/ritamsignals.in/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/ritamsignals.in/privkey.pem; } server { listen 80; server_name ritamsignals.in www.ritamsignals.in api.ritamsignals.in; return 301 https://$host$request_uri; }
Staging nginx vhost
server {
server_name stg.api.ritamsignals.in;
add_header X-Robots-Tag "noindex, nofollow" always;
location / {
proxy_pass http://127.0.0.1:8002;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 120s;
}
listen 443 ssl;
ssl_certificate /etc/letsencrypt/live/stg.ritamsignals.in/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/stg.ritamsignals.in/privkey.pem;
}
server {
server_name stg.ritamsignals.in;
root /var/www/ritam-stg;
index index.html;
add_header X-Robots-Tag "noindex, nofollow" always;
location /api {
proxy_pass http://127.0.0.1:8002;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 120s;
}
location / {
try_files $uri $uri/ /index.html;
}
listen 443 ssl;
ssl_certificate /etc/letsencrypt/live/stg.ritamsignals.in/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/stg.ritamsignals.in/privkey.pem;
}
The backup script, fixed and running again
#!/bin/bash set -e BUCKET=ritam-db-backups-964222105432 DB=/home/ubuntu/ritam/data/ritamv2.db TS=$(date +%Y%m%d-%H%M%S) TMP=/tmp/ritamv2-$TS.db sqlite3 "$DB" ".backup '$TMP'" gzip -f "$TMP" /usr/local/bin/aws s3 cp "$TMP.gz" "s3://$BUCKET/backups/ritamv2-$TS.db.gz" --region ap-south-1 --only-show-errors /usr/local/bin/aws s3 cp "$TMP.gz" "s3://$BUCKET/ritamv2-latest.db.gz" --region ap-south-1 --only-show-errors rm -f "$TMP.gz" logger "ritam_backup: uploaded ritamv2-$TS.db.gz"
Runs every 6 hours via /etc/cron.d/ritam-backup as root, logs to /var/log/ritam_backup.log. I install the same script, unchanged, on the new box during phase 1, pointed at the new box's own data path.
This describes a plan. Nothing in it has been executed against production or staging.