{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a8358114",
   "metadata": {},
   "source": [
    "# Venus MFSK receiver (simple version)\n",
    "\n",
    "Decodes the Venus MFSK message from raw SigMF recordings.\n",
    "\n",
    "**Inputs.** The recordings are all `*.sigmf-meta` files in `rx_dir`, or the files in `rx_files` (`rx_dir = None`); a listed `chan0` file brings its `chan1` partner. Each has a `core:description` of the form `<tx file>;<tx gain>;<tx unix time>;<target>`. The tx file is looked up in `tx_dir`, and its `mfsk:*` fields give the waveform parameters and the tx index (scrambling). Recordings whose tx file is not an MFSK file (CW, Q65, ZC, ...) are skipped. Simulated echoes from `sim_mfsk` (`sim:*` fields) have no Doppler; their echo starts at the tx unix time. Files whose names differ only in `chan0`/`chan1` are two linear polarisations of one recording; they are combined to circular as `(x0 + exp(j phase) x1) / sqrt(2)`. The receiver site comes from the file name (`stockert_...`, `dwingeloo_...`, `wsrt_...`).\n",
    "\n",
    "**Processing per recording**\n",
    "1. Doppler: SPICE gives the Doppler of the sub-radar point once per second. It is spline-interpolated, integrated to a phase and removed from the samples. SPICE also gives the echo start: the tx time plus the round-trip light time.\n",
    "2. Each dwell (one tone) is split into Hann windows of one frame (1 FFT bin = 1.5 Hz) with 50% overlap. Every spectrum is divided by its noise mean (median over the tone block / ln 2). The energy at each tone is summed per dwell.\n",
    "3. RFI: the tone changes every dwell, but a carrier doesn't. A tone whose energy stays high in the neighbouring dwells is flagged and set to the noise mean.\n",
    "4. Descrambling (`tone = symbol XOR pn`) and accumulation per code symbol into `E` (n x M), together with its noise mean `L` and variance `V`.\n",
    "\n",
    "**Decoding** runs after every recording: ordered-statistics decoding over the RS code, then a CRC check."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "16b511ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "import hashlib\n",
    "import itertools\n",
    "import json\n",
    "import os\n",
    "import re\n",
    "from types import SimpleNamespace\n",
    "\n",
    "import galois\n",
    "import numpy as np\n",
    "import scipy.interpolate\n",
    "import spiceypy as sp"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18531a9f",
   "metadata": {},
   "source": [
    "## Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b73073d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "rx_dir = \"/Users/telkamp/data/eve2026/wsrt_mfsk2\"   # raw recordings (not Doppler-corrected)\n",
    "# rx_dir = \"data\"                                    # sim_mfsk output\n",
    "# rx_dir = None                                      # use rx_files; a chan0 file brings its chan1\n",
    "rx_files = [\n",
    "    \"/Users/telkamp/data/eve2026/wsrt_mfsk2/wsrt_venus_2026_10_18_11_00_45_1299.650MHz_0.25Msps_ci16_le.chan0.sigmf-meta\",\n",
    "]\n",
    "tx_dir = \"/Users/telkamp/data/eve2026/tx_signals_eve/mfsk-v1\"        # tx files named in core:description\n",
    "kernel_dir = \"/Users/telkamp/notebooks/kernels\"              # SPICE kernels\n",
    "tx_site = \"DWINGELOO\"                                        # transmitter (SPICE name)\n",
    "\n",
    "pol_phase_deg = 210.0      # chan0/chan1 pairs: (x0 + exp(j phase) x1) / sqrt(2)\n",
    "rfi_dwells = 2            # RFI flagging: background from this many dwells on each side\n",
    "rfi_threshold = 2       # flag a tone in a dwell if its background exceeds this x the noise mean\n",
    "osd = dict(n_bases=256, list_depth=10, n_check=50, n_free_sets=16, free_depth=5)   # decoder effort"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3df59653",
   "metadata": {},
   "source": [
    "## Waveform and message format\n",
    "\n",
    "The tx file's `mfsk:*` fields define the waveform:\n",
    "- frame = `1 / bin_hz`; a dwell is `dwell_frames` frames on one tone\n",
    "- tone `t` sits on FFT bin `(t - M/2) * tone_spacing_bins`\n",
    "- dwell `d` carries code symbol `d mod n`\n",
    "\n",
    "The message is a base-37 number followed by a CRC, cut into `k` symbols of `log2 M` bits. Those `k` symbols are then encoded with RS(n, k) over GF(M)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7aa9a19f",
   "metadata": {},
   "outputs": [],
   "source": [
    "ALPHABET = \" ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789\"      # space = 0\n",
    "CRC_SPECS = {16: (0x1021, 0xFFFF), 24: (0x864CFB, 0xB704CE)}   # width -> (polynomial, init)\n",
    "\n",
    "\n",
    "def read_meta(path):\n",
    "    with open(path) as f:\n",
    "        return json.load(f)\n",
    "\n",
    "\n",
    "def mfsk_params(tx_meta):\n",
    "    \"\"\"Waveform parameters from the mfsk:* fields of a tx .sigmf-meta file.\"\"\"\n",
    "    g = read_meta(tx_meta)[\"global\"]\n",
    "    p = SimpleNamespace(fs=g[\"mfsk:sample_rate\"], M=g[\"mfsk:M\"], n=g[\"mfsk:rs_n\"], n_chars=g[\"mfsk:n_chars\"],\n",
    "                        crc_bits=g[\"mfsk:crc_bits\"], scramble=g[\"mfsk:scramble\"], seed=g[\"mfsk:scramble_seed\"])\n",
    "    spacing = g[\"mfsk:tone_spacing_bins\"]\n",
    "    p.Nf = int(round(p.fs / g[\"mfsk:bin_hz\"]))                      # frame = FFT length\n",
    "    p.dwell = p.Nf * g[\"mfsk:dwell_frames\"]                         # samples per dwell\n",
    "    p.n_dwells = int(g[\"mfsk:tx_length_s\"] * p.fs // p.dwell)\n",
    "    p.tone_bins = ((np.arange(p.M) - p.M // 2) * spacing) % p.Nf    # FFT bin of every tone\n",
    "    p.band = np.arange(-p.M // 2 * spacing, p.M // 2 * spacing) % p.Nf   # all bins of the tone block\n",
    "    p.bits = p.M.bit_length() - 1                                   # bits per code symbol\n",
    "    p.msg_bits = (len(ALPHABET) ** p.n_chars - 1).bit_length()\n",
    "    p.k = -(-(p.msg_bits + p.crc_bits) // p.bits)                   # information symbols\n",
    "    return p\n",
    "\n",
    "\n",
    "def scramble_pn(p, tx_index):\n",
    "    \"\"\"Scrambling mask per dwell: tone = symbol XOR pn[dwell], from SHA-256 of (seed, tx index, dwell).\"\"\"\n",
    "    if not p.scramble:\n",
    "        return np.zeros(p.n_dwells, dtype=np.int64)\n",
    "    return np.array([int.from_bytes(hashlib.sha256(b\"venus-mfsk:%d:%d:%d\" % (p.seed, tx_index, d)).digest()[:8], \"big\")\n",
    "                     & (p.M - 1) for d in range(p.n_dwells)])\n",
    "\n",
    "\n",
    "def crc(value, nbits, width):\n",
    "    poly, reg = CRC_SPECS[width]\n",
    "    for i in reversed(range(nbits)):\n",
    "        top = (reg >> (width - 1)) & 1\n",
    "        reg = (reg << 1) & ((1 << width) - 1)\n",
    "        if top ^ ((value >> i) & 1):\n",
    "            reg ^= poly\n",
    "    return reg\n",
    "\n",
    "\n",
    "def symbols_to_text(info, p):\n",
    "    \"\"\"k information symbols -> message, or None if the base-37 range or the CRC check fails.\"\"\"\n",
    "    word = 0\n",
    "    for s in info:\n",
    "        word = (word << p.bits) | int(s)\n",
    "    value, check = word >> p.crc_bits, word & ((1 << p.crc_bits) - 1)\n",
    "    if value >= len(ALPHABET) ** p.n_chars or crc(value, p.msg_bits, p.crc_bits) != check:\n",
    "        return None\n",
    "    text = \"\"\n",
    "    for _ in range(p.n_chars):\n",
    "        value, i = divmod(value, len(ALPHABET))\n",
    "        text = ALPHABET[i] + text\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fed267e",
   "metadata": {},
   "source": [
    "## Recordings\n",
    "\n",
    "Every recording in `rx_dir` (or `rx_files`) whose description names an MFSK tx file in `tx_dir`, with `chan0`/`chan1` files paired. Sorted by tx time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5c46fc52",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 recordings, tx set venus-mfsk4096-rs9-sp3-d4-269s\n",
      "M=4096, RS(9,6), frame 166667 samples (1.500 Hz bins), 101 dwells of 666668 samples, scrambling on\n"
     ]
    }
   ],
   "source": [
    "RX_SITES = {\"dwingeloo\": \"DWINGELOO\", \"stockert\": \"STOCKERT\", \"wsrt\": \"WSRT_RT2\"}   # file name prefix -> SPICE site\n",
    "TARGETS = (\"moon\", \"venus\")\n",
    "CHAN_RE = re.compile(r\"chan[01]\")\n",
    "TX_SET_RE = re.compile(r\"-tx\\d+\\.sigmf-meta$\")\n",
    "\n",
    "\n",
    "def rx_meta_files(rx):\n",
    "    \"\"\"*.sigmf-meta files in directory rx, or the files in list rx plus the chan1 (chan0) partner of a chan0 (chan1) file.\"\"\"\n",
    "    if isinstance(rx, str):\n",
    "        return sorted(glob.glob(os.path.join(rx, \"*.sigmf-meta\")))\n",
    "    files = []\n",
    "    for f in rx:\n",
    "        partner = os.path.join(os.path.dirname(f), CHAN_RE.sub(lambda m: \"chan1\" if m.group() == \"chan0\" else \"chan0\",\n",
    "                                                               os.path.basename(f)))\n",
    "        files += [g for g in (f, partner) if g not in files and os.path.exists(g)]\n",
    "    return files\n",
    "\n",
    "\n",
    "def scan_recordings(rx, tx_dir):\n",
    "    recs, n_other = {}, 0\n",
    "    for meta in rx_meta_files(rx):\n",
    "        g = read_meta(meta)[\"global\"]\n",
    "        fields = g.get(\"core:description\", \"\").split(\";\")\n",
    "        tx_file = fields[0] if fields[0].endswith(\".sigmf-meta\") else fields[0] + \".sigmf-meta\"\n",
    "        tx_path = os.path.join(tx_dir, tx_file)\n",
    "        if (\" \" in fields[0] or \"mfsk:M\" in g\n",
    "                or (\"mfsk:M\" not in read_meta(tx_path)[\"global\"] if os.path.exists(tx_path) else \"mfsk\" not in tx_file)):\n",
    "            n_other += 1                                            # other transmission, or a tx file\n",
    "            continue\n",
    "        sim = any(k.startswith(\"sim:\") for k in g)                  # simulated echo: no Doppler\n",
    "        name = os.path.basename(meta).lower().split(\"_\")\n",
    "        if len(fields) < 3 or not os.path.exists(tx_path) or not (sim or name[0] in RX_SITES):\n",
    "            print(\"skipped %s (description %r)\" % (os.path.basename(meta), \";\".join(fields)))\n",
    "            continue\n",
    "        target = next((f for f in name if f in TARGETS), fields[3].lower() if len(fields) > 3 else \"\")\n",
    "        key = CHAN_RE.sub(\"chan0+1\", os.path.basename(meta))\n",
    "        r = recs.setdefault(key, dict(name=key, metas=[], tx_file=tx_file, tx_unix=float(fields[2]),\n",
    "                                      receiver=None if sim else RX_SITES[name[0]], target=target.upper()))\n",
    "        r[\"metas\"].append(meta)\n",
    "    if n_other:\n",
    "        print(\"skipped %d files of other transmissions or tx files\" % n_other)\n",
    "    if not recs:\n",
    "        raise ValueError(\"no MFSK recordings with a tx file in %s\" % tx_dir)\n",
    "    recs = sorted(recs.values(), key=lambda r: r[\"tx_unix\"])\n",
    "    if len({TX_SET_RE.sub(\"\", r[\"tx_file\"]) for r in recs}) != 1:\n",
    "        raise ValueError(\"recordings belong to different tx sets\")\n",
    "    for r in recs:\n",
    "        r[\"tx_index\"] = read_meta(os.path.join(tx_dir, r[\"tx_file\"]))[\"global\"].get(\"mfsk:tx_index\", 0)\n",
    "    return recs\n",
    "\n",
    "\n",
    "class Recording:\n",
    "    \"\"\"A ci16_le SigMF recording, or a chan0/chan1 pair combined to circular polarisation.\"\"\"\n",
    "\n",
    "    def __init__(self, metas, pol_phase_deg):\n",
    "        metas = sorted(metas)                                       # chan0 first\n",
    "        heads = [read_meta(m) for m in metas]\n",
    "        caps = [h[\"captures\"][0] for h in heads]\n",
    "        assert all(h[\"global\"][\"core:datatype\"] == \"ci16_le\" for h in heads)\n",
    "        assert all(c.get(\"core:sample_start\", 0) == 0 and c[\"core:datetime\"] == caps[0][\"core:datetime\"] for c in caps)\n",
    "        self.fs = heads[0][\"global\"][\"core:sample_rate\"]\n",
    "        self.f0 = caps[0][\"core:frequency\"]\n",
    "        self.start = np.datetime64(caps[0][\"core:datetime\"].rstrip(\"Z\"), \"ns\")\n",
    "        self.iq = [np.memmap(m.replace(\".sigmf-meta\", \".sigmf-data\"), dtype=\"<i2\", mode=\"r\") for m in metas]\n",
    "        self.n_samples = min(len(iq) for iq in self.iq) // 2\n",
    "        self.rot = np.exp(1j * np.deg2rad(pol_phase_deg))\n",
    "\n",
    "    def read(self, a, count):\n",
    "        x = [iq[2 * a:2 * (a + count)].astype(np.float32).view(np.complex64) for iq in self.iq]\n",
    "        return x[0] if len(x) == 1 else (x[0] + self.rot * x[1]) / np.sqrt(2)\n",
    "\n",
    "\n",
    "recordings = scan_recordings(rx_dir if rx_dir is not None else rx_files, tx_dir)\n",
    "p = mfsk_params(os.path.join(tx_dir, recordings[0][\"tx_file\"]))\n",
    "print(\"%d recordings, tx set %s\" % (len(recordings), TX_SET_RE.sub(\"\", recordings[0][\"tx_file\"])))\n",
    "print(\"M=%d, RS(%d,%d), frame %d samples (%.3f Hz bins), %d dwells of %d samples, scrambling %s\"\n",
    "      % (p.M, p.n, p.k, p.Nf, p.fs / p.Nf, p.n_dwells, p.dwell, \"on\" if p.scramble else \"off\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80a1c892",
   "metadata": {},
   "source": [
    "## Doppler correction and echo time (SPICE)\n",
    "\n",
    "The round-trip light time `lt` and its rate `dlt/dt` are computed for the sub-radar point: transmitter → point → receiver, at reception time. The Doppler is `-f0 · dlt/dt`. It is computed once per second, interpolated with a cubic spline and integrated to a phase. That phase is then removed from the samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "fd6447c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "for kernel in [\"naif0012.tls\", \"de440s.bsp\", \"pck00011.tpc\", \"earth_latest_high_prec.bpc\", \"antennas.fk\", \"antennas.bsp\"]:\n",
    "    sp.furnsh(os.path.join(kernel_dir, kernel))\n",
    "\n",
    "\n",
    "def radar_srp(target, et_rx, transmitter, receiver):\n",
    "    \"\"\"(d(round-trip light time)/dt, round-trip light time) of the sub-radar point, for reception at et_rx.\"\"\"\n",
    "    frame = \"IAU_\" + target\n",
    "    srp, _, _ = sp.subpnt(\"INTERCEPT/ELLIPSOID\", target, et_rx, frame, \"CN\", receiver)\n",
    "    ez_r, lt_r = sp.spkcpt(srp, target, frame, et_rx, \"J2000\", \"OBSERVER\", \"CN\", receiver)           # point -> rx\n",
    "    ez_d, lt_d = sp.spkcpo(transmitter, et_rx - lt_r, \"J2000\", \"OBSERVER\", \"CN\", srp, target, frame)  # tx -> point\n",
    "    dlt_r = sp.dvnorm(ez_r) / sp.clight()\n",
    "    ez_d[3:] *= 1 - dlt_r                       # the bounce happens lt_r earlier: rates in the point's time\n",
    "    return dlt_r + sp.dvnorm(ez_d) / sp.clight(), lt_r + lt_d\n",
    "\n",
    "\n",
    "class RadarDoppler:\n",
    "    \"\"\"Doppler of one recording, removed from its samples by phase (sample 0 = recording start).\"\"\"\n",
    "\n",
    "    def __init__(self, rec, target, transmitter, receiver):\n",
    "        self.target, self.transmitter, self.receiver, self.fs = target, transmitter, receiver, rec.fs\n",
    "        self.et0 = sp.str2et(str(rec.start))\n",
    "        self.t = np.arange(-2.0, np.ceil(rec.n_samples / rec.fs) + 3.0)\n",
    "        self.doppler = np.array([-radar_srp(target, self.et0 + t, transmitter, receiver)[0] * rec.f0 for t in self.t])\n",
    "        self.cycles = scipy.interpolate.CubicSpline(self.t, self.doppler).antiderivative()\n",
    "\n",
    "    def remove(self, x, a):\n",
    "        \"\"\"x = samples a, a+1, ... with the Doppler removed.\"\"\"\n",
    "        cycles = np.mod(self.cycles((a + np.arange(len(x))) / self.fs) - self.cycles(0.0), 1.0)\n",
    "        return (x * np.exp(-2j * np.pi * cycles)).astype(np.complex64)\n",
    "\n",
    "    def echo_start(self, tx_unix):\n",
    "        \"\"\"Seconds from the recording start to the leading edge of the echo of a transmission at unix time tx_unix.\"\"\"\n",
    "        et_tx = sp.str2et(str(np.datetime64(round(tx_unix * 1e9), \"ns\")))\n",
    "        et_rx = et_tx\n",
    "        for _ in range(10):                     # solve et_rx = et_tx + lt(et_rx)\n",
    "            et_new = et_tx + radar_srp(self.target, et_rx, self.transmitter, self.receiver)[1]\n",
    "            if abs(et_new - et_rx) < 1e-9:\n",
    "                break\n",
    "            et_rx = et_new\n",
    "        return et_new - self.et0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "062f227f",
   "metadata": {},
   "source": [
    "## Demodulation\n",
    "\n",
    "Per dwell: Hann windows of one frame with 50% overlap, only the windows that lie entirely inside the dwell (7 for 4 frames). The windows overlap, so their energies are slightly correlated. `var_factor` is the noise variance of a dwell sum relative to independent windows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "bd425dd7",
   "metadata": {},
   "outputs": [],
   "source": [
    "hop = p.Nf // 2\n",
    "n_win = (p.dwell - p.Nf) // hop + 1                                  # windows per dwell\n",
    "window = (0.5 - 0.5 * np.cos(2 * np.pi * np.arange(p.Nf) / p.Nf)).astype(np.float32)\n",
    "c = (np.sum(window[:p.Nf - hop] * window[hop:]) / np.sum(window**2))**2   # power correlation of adjacent windows\n",
    "var_factor = (n_win + 2 * (n_win - 1) * c) / n_win\n",
    "\n",
    "\n",
    "def dwell_energies(rec, doppler, start):\n",
    "    \"\"\"Energy at every tone in every dwell (n_dwells x M) in units of the noise mean per window, and the number of\n",
    "    windows per dwell (0 for a dwell not entirely inside the recording). start: sample of the echo start.\n",
    "    doppler: RadarDoppler, or None for a simulated echo.\"\"\"\n",
    "    D = np.zeros((p.n_dwells, p.M))\n",
    "    n = np.zeros(p.n_dwells)\n",
    "    for d in range(p.n_dwells):\n",
    "        a = start + d * p.dwell\n",
    "        if a < 0 or a + p.dwell > rec.n_samples:\n",
    "            continue\n",
    "        x = rec.read(a, p.dwell)\n",
    "        if doppler is not None:\n",
    "            x = doppler.remove(x, a)\n",
    "        for j in range(n_win):\n",
    "            P = np.abs(np.fft.fft(x[j * hop:j * hop + p.Nf] * window))**2\n",
    "            D[d] += P[p.tone_bins] / (np.median(P[p.band]) / np.log(2))   # exponential noise: mean = median / ln 2\n",
    "        n[d] = n_win\n",
    "    return D, n\n",
    "\n",
    "\n",
    "def flag_rfi(D, n, W, threshold):\n",
    "    \"\"\"Flag tone d of a dwell when its median energy in the W dwells before, or the W dwells after, exceeds threshold\n",
    "    x the noise mean. Our signal moves to another tone every dwell; a carrier stays.\"\"\"\n",
    "    Dn = D / np.where(n > 0, n, np.nan)[:, None]\n",
    "    flag = np.zeros(D.shape, dtype=bool)\n",
    "    for d in np.flatnonzero(n > 0):\n",
    "        bg = np.zeros(p.M)\n",
    "        for side in (Dn[max(0, d - W):d], Dn[d + 1:d + 1 + W]):\n",
    "            side = side[~np.isnan(side[:, 0])]\n",
    "            if len(side):\n",
    "                bg = np.maximum(bg, np.median(side, axis=0))\n",
    "        flag[d] = bg > threshold\n",
    "    return flag"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dfb71d6",
   "metadata": {},
   "source": [
    "## Decoder\n",
    "\n",
    "Ordered-statistics decoding (OSD). For every code position, keep the strongest tones; a position's reliability is the gap between its best and second-best tone. Any `k` positions of an RS code fix the whole codeword (MDS). So for a set of `k` positions, every combination of their `list_depth` strongest tones is re-encoded to a full codeword. Each codeword is scored by `sum_i E[i, c_i]`, the maximum-likelihood metric for square-law detection at low SNR. The first position set is the `k` most reliable positions; further sets are random, weighted towards reliable positions.\n",
    "\n",
    "If that fails, a second pass needs only `k-1` positions in the lists. It fixes `k-1` positions to their `free_depth` strongest tones and tries all M values of the remaining information symbol.\n",
    "\n",
    "The best candidate that passes the CRC wins. Only the `n_check` best distinct candidates of each pass are checked, to keep the false-decode rate low. Quality figure: `z = (score - sum L) / sqrt(sum V)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f7eb5748",
   "metadata": {},
   "outputs": [],
   "source": [
    "GF = galois.GF(p.M)\n",
    "rs = galois.ReedSolomon(p.M - 1, p.M - 1 - (p.n - p.k), field=GF)    # shortened to RS(n, k) by encoding k symbols\n",
    "G = rs.encode(GF.Identity(p.k))                                      # k x n systematic generator matrix\n",
    "\n",
    "\n",
    "def first_valid(sc, C, n_check):\n",
    "    \"\"\"Index and text of the best of the n_check best distinct candidates that passes the message checks.\"\"\"\n",
    "    seen = set()\n",
    "    for i in np.argsort(-sc):\n",
    "        if C[i].tobytes() in seen:\n",
    "            continue\n",
    "        seen.add(C[i].tobytes())\n",
    "        text = symbols_to_text(C[i, :p.k], p)\n",
    "        if text is not None:\n",
    "            return i, text\n",
    "        if len(seen) >= n_check:\n",
    "            break\n",
    "    return None, None\n",
    "\n",
    "\n",
    "def decode(E, V, n_bases, list_depth, n_check, n_free_sets, free_depth, seed=0):\n",
    "    \"\"\"Returns (text, codeword, score), or (None, None, None).\"\"\"\n",
    "    n, k = p.n, p.k\n",
    "    rows = np.arange(n)\n",
    "    order = np.argsort(-E, axis=1)[:, :max(list_depth, free_depth, 2)]   # strongest tones per position\n",
    "    rel = (E[rows, order[:, 0]] - E[rows, order[:, 1]]) / np.sqrt(V)    # reliability per position\n",
    "    by_rel = np.argsort(-rel)\n",
    "\n",
    "    # pass 1: k positions from the lists\n",
    "    weight = np.empty(n)\n",
    "    weight[by_rel] = np.exp(-np.arange(n) / k)\n",
    "    weight /= weight.sum()\n",
    "    rng = np.random.default_rng(seed)\n",
    "    combos = np.array(list(itertools.product(range(list_depth), repeat=k)))\n",
    "    tried, scores, cands = set(), [], []\n",
    "    for b in range(n_bases):\n",
    "        S = np.sort(by_rel[:k] if b == 0 else rng.choice(n, size=k, replace=False, p=weight))\n",
    "        if tuple(S) in tried:\n",
    "            continue\n",
    "        tried.add(tuple(S))\n",
    "        P = np.linalg.inv(G[:, S]) @ G                                   # symbols at S -> codeword\n",
    "        C = np.asarray(GF(order[S[None, :], combos]) @ P, dtype=np.int64)\n",
    "        sc = E[rows, C].sum(axis=1)\n",
    "        keep = np.argsort(-sc)[:n_check]\n",
    "        scores.append(sc[keep])\n",
    "        cands.append(C[keep])\n",
    "    sc, C = np.concatenate(scores), np.concatenate(cands)\n",
    "    i, text = first_valid(sc, C, n_check)\n",
    "    if text is not None:\n",
    "        return text, C[i], sc[i]\n",
    "\n",
    "    # pass 2: k-1 positions from the lists, the remaining information symbol u over all M values. The codewords\n",
    "    # through k-1 fixed symbols form a line c = base + u * P[-1] over GF(M) (addition = XOR).\n",
    "    sets = sorted(itertools.combinations(range(n), k - 1), key=lambda S: -rel[list(S)].sum())[:n_free_sets]\n",
    "    combos = np.array(list(itertools.product(range(free_depth), repeat=k - 1)))\n",
    "    u = GF(np.arange(p.M))\n",
    "    Ef = E.astype(np.float32)\n",
    "    cands = []\n",
    "    for S in sets:\n",
    "        S = np.array(S)\n",
    "        f, *others = np.setdiff1d(rows, S)                               # free position, the rest\n",
    "        P = np.linalg.inv(G[:, np.r_[S, f]]) @ G\n",
    "        base = np.asarray(GF(order[S[None, :], combos]) @ P[:-1], dtype=np.int64)   # (combos, n), 0 at f\n",
    "        line = np.asarray(u[:, None] * P[-1][None, :], dtype=np.int64)              # (M, n), u at f\n",
    "        sc = Ef[S, base[:, S]].sum(axis=1)[:, None] + Ef[f][None, :]                # (combos, M)\n",
    "        for j in others:\n",
    "            sc += Ef[j][base[:, j, None] ^ line[None, :, j]]\n",
    "        ci, ui = np.unravel_index(np.argpartition(-sc, n_check, axis=None)[:n_check], sc.shape)\n",
    "        cands.append(base[ci] ^ line[ui])\n",
    "    C = np.concatenate(cands)\n",
    "    sc = E[rows, C].sum(axis=1)\n",
    "    i, text = first_valid(sc, C, n_check)\n",
    "    return (text, C[i], sc[i]) if text is not None else (None, None, None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd67d377",
   "metadata": {},
   "source": [
    "## Accumulate and decode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f78919ac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 0  wsrt_venus_2026_10_18_11_00_45_1299.650MHz_0.25Msps_ci16_le.chan0+1.sigmf-meta  tx04  echo 2026-10-18T11:05:24.560  Doppler 28985.1 Hz  101/101 dwells  RFI 0.36%\n",
      "    no decode\n",
      " 1  wsrt_venus_2026_10_18_11_50_54_1299.650MHz_0.25Msps_ci16_le.chan0+1.sigmf-meta  tx09  echo 2026-10-18T11:55:33.494  Doppler 28338.4 Hz  101/101 dwells  RFI 0.37%\n",
      "    'VENUS ECHO'  z = 11.9\n",
      "\n",
      "DECODED: 'VENUS ECHO'\n"
     ]
    }
   ],
   "source": [
    "E = np.zeros((p.n, p.M))     # energy per code symbol and tone value, summed over all dwells and recordings\n",
    "L = np.zeros(p.n)            # its noise mean per code symbol\n",
    "V = np.zeros(p.n)            # its noise variance per code symbol\n",
    "\n",
    "for i, r in enumerate(recordings):\n",
    "    rec = Recording(r[\"metas\"], pol_phase_deg)\n",
    "    assert rec.fs == p.fs, \"sample rate differs from the tx file\"\n",
    "    if r[\"receiver\"] is None:                                             # simulated echo: no Doppler, echo at tx time\n",
    "        doppler = None\n",
    "        t_echo = (np.datetime64(round(r[\"tx_unix\"] * 1e9), \"ns\") - rec.start) / np.timedelta64(1, \"s\")\n",
    "    else:\n",
    "        doppler = RadarDoppler(rec, r[\"target\"], tx_site, r[\"receiver\"])\n",
    "        t_echo = doppler.echo_start(r[\"tx_unix\"])\n",
    "    D, n = dwell_energies(rec, doppler, int(round(t_echo * rec.fs)))\n",
    "\n",
    "    flag = flag_rfi(D, n, rfi_dwells, rfi_threshold)\n",
    "    D = np.where(flag, n[:, None], D)                                     # flagged cells -> noise mean\n",
    "    pn = scramble_pn(p, r[\"tx_index\"])\n",
    "    D = D[np.arange(p.n_dwells)[:, None], np.arange(p.M)[None, :] ^ pn[:, None]]   # column v = symbol value v\n",
    "    symbol = np.arange(p.n_dwells) % p.n\n",
    "    np.add.at(E, symbol, D)\n",
    "    np.add.at(L, symbol, n)\n",
    "    np.add.at(V, symbol, n * var_factor)\n",
    "\n",
    "    print(\"%2d  %s  tx%02d  echo %s  %s  %d/%d dwells  RFI %.2f%%\"\n",
    "          % (i, r[\"name\"], r[\"tx_index\"], str(rec.start + np.timedelta64(round(t_echo * 1e9), \"ns\"))[:23],\n",
    "             \"no Doppler (sim)\" if doppler is None else \"Doppler %.1f Hz\" % np.interp(t_echo, doppler.t, doppler.doppler),\n",
    "             (n > 0).sum(), p.n_dwells, 100 * flag[n > 0].mean()))\n",
    "    text, codeword, score = decode(E, V, **osd)\n",
    "    if text is None:\n",
    "        print(\"    no decode\")\n",
    "    else:\n",
    "        print(\"    '%s'  z = %.1f\" % (text, (score - L.sum()) / np.sqrt(V.sum())))\n",
    "\n",
    "print(\"\\nDECODED: '%s'\" % text if text is not None else \"\\nNOT DECODED\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.14.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
