for Developers

Sync a daily patient list with the huli CLI

Keep a daily mirror of your patients and appointments: schedule a nightly pull of every Patient and Appointment for one organization, write each page to durable storage, and resume from the exact page you left off when the host reboots mid-run. The loop is cursor-based, so a restart re-reads the last unfinished page and continues — it commits each page before advancing the checkpoint, and an id-level dedup at load time collapses any re-read page into a single row per resource.

Audience

You run server-to-server integrations and own a cron host. You have synced a paginated API before, you know what a checkpoint file buys you, and you want a patient-and-appointment mirror that survives a reboot — backed by an id-level dedup at load time, not a hope that the walk never repeats a page.

You'll need

  • The huli CLI on the cron host. huli version should print a 1.x build; confirm it speaks the v1 public API with huli auth status (below), which fails loudly against an older contract.
  • A SMART Backend Services client registered in HuliPractice (Practice Settings → Integrations → API Keys), holding these scopes:
    • system/Patient.rs — read plus search on Patient.
    • system/Appointment.rs — read plus search on Appointment.
  • The client's RS384 private key on the cron host (file mode 0600), plus the client_id issued at registration.
  • A writable checkpoint directory the cron user owns — this recipe uses /var/lib/huli-sync.

End state

A cron job runs nightly at 02:15 America/Mexico_City. Each run walks every page of the Patient searchset, then every page of the Appointment searchset for Doctora María Fernández's organization at Clínica San Rafael, appending each page to a dated NDJSON file under /var/lib/huli-sync. A checkpoint file records the cursor of the last page committed. If the host reboots mid-run, the next start reads the checkpoint and resumes from that cursor — at worst re-reading the one page that was in flight. The step-5 load dedups on resource id, so the final mirror carries one row per Patient and Appointment.

Steps

1. Store the machine-to-machine credentials once

Register the client once. huli auth setup reads the private key, records the client_id and the token endpoint, and persists an encrypted profile under ~/.config/huli/. Run it once as the cron user, not inside the cron job.

Confirm the profile authenticates. huli auth status mints a throwaway access token against /auth/token and prints the resolved org and granted scopes without writing any data.

profile:  roster-sync
client:   roster-sync@clinica-san-rafael
org:      Clínica San Rafael (01965e2a-8c4d-7000-9001-0000000000b0)
scopes:   system/Patient.rs system/Appointment.rs
token:    valid, expires in 4m51s

If the scope line is missing system/Appointment.rs, the key was minted without it — re-mint in Practice Settings before scheduling. A scope gap surfaces at runtime as

403 Forbidden HPB-00104, not at setup.

2. Write the sync script

The script walks one resource at a time. It reads the page cursor from a checkpoint file, calls the CLI for that page, appends the entries, then advances the checkpoint to the next cursor the page returned. Committing the output before advancing the checkpoint is what makes a restart safe: a crash between append and checkpoint-write re-reads the same page, and the loader in step 5 dedupes on resource id.

Save this as /usr/local/bin/huli-roster-sync.sh.

#!/usr/bin/env bash
set -euo pipefail

PROFILE="roster-sync"
STATE_DIR="/var/lib/huli-sync"
RUN_DATE="$(date +%F)"                       # e.g. 2026-06-02
OUT_DIR="${STATE_DIR}/${RUN_DATE}"
mkdir -p "${OUT_DIR}"

# Walk one FHIR resource type to exhaustion, resuming from a per-resource checkpoint.
sync_resource() {
  local resource="$1"
  local ckpt="${STATE_DIR}/${resource}.cursor"
  local out="${OUT_DIR}/${resource}.ndjson"

  # Resume: a non-empty checkpoint means a prior run stopped mid-roster.
  local cursor=""
  if [[ -s "${ckpt}" ]]; then
    cursor="$(cat "${ckpt}")"
    echo "[$(date -Iseconds)] ${resource}: resuming from cursor ${cursor:0:16}…"
  else
    echo "[$(date -Iseconds)] ${resource}: starting fresh"
  fi

  while :; do
    # `huli fhir search` emits one JSON object per line:
    #   {"page":[...entries...], "next":"<cursor>|"}
    # --cursor "" requests the first page. The CLI handles token refresh,
    # 429 Retry-After backoff, and 5xx retries internally.
    local page_json
    page_json="$(huli fhir search "${resource}" \
      --profile "${PROFILE}" \
      --count 100 \
      --cursor "${cursor}" \
      --output ndjson-page)"

    # Append this page's entries, then advance the checkpoint. Output is
    # committed BEFORE the cursor moves, so a crash here re-reads this page.
    jq -c '.page[]' <<<"${page_json}" >>"${out}"

    cursor="$(jq -r '.next' <<<"${page_json}")"
    if [[ -z "${cursor}" || "${cursor}" == "null" ]]; then
      : >"${ckpt}"                            # roster exhausted: clear checkpoint
      echo "[$(date -Iseconds)] ${resource}: complete"
      break
    fi
    printf '%s' "${cursor}" >"${ckpt}"        # durable resume point
  done
}

sync_resource Patient
sync_resource Appointment

Make it executable.

3. Pin the cursor semantics

Three invariants make the resume correct:

  • The cursor is opaque and stable. Treat next as a base64 token — never parse, truncate, or regenerate it. Passing yesterday's cursor into today's run resumes exactly where that token points; passing an empty string starts a fresh full walk.
  • Append, then advance. The script writes the page to NDJSON before it overwrites the checkpoint. A power loss between those two lines costs you one re-read of a single page on restart, never a gap. Reversing the order — advancing the checkpoint first — is the one change that can drop a page; do not do it.
  • The raw NDJSON can repeat a page. A resume re-reads the in-flight page, and because the cursor orders by a server-side sort key, rows created mid-walk can re-surface on a later page. The walk itself does not promise a duplicate-free file — the step-5 id-level dedup is what makes the final mirror one row per resource. Treat the NDJSON as an at-least-once stream, not exactly-once.

4. Schedule the cron entry

Run nightly at 02:15 in the clinic's timezone. The CRON_TZ prefix pins the schedule to America/Mexico_City regardless of the host clock. A flock guard stops a long run from overlapping the next night's trigger.

Install it as the huli-sync user's per-user crontab. A per-user crontab line has no username field — the schedule goes straight to the command:

CRON_TZ=America/Mexico_City
15 2 * * * /usr/bin/flock -n /var/lib/huli-sync/.lock /usr/local/bin/huli-roster-sync.sh >> /var/log/huli-roster-sync.log 2>&1

Save that as /etc/huli/roster-sync.crontab (owned by the cron user) and install it for the huli-sync service user:

5. Load with id-level dedup

A resume re-reads at most the in-flight page, and a mid-walk write can re-surface a row on a later page. Dedup on the FHIR resource id at load time so any repeat collapses to a single row. This sort -u-then-upsert pattern keeps the loader idempotent no matter how many times a page was re-read.

for resource in Patient Appointment; do
  jq -r '[.id, (.|tojson)] | @tsv' \
    "/var/lib/huli-sync/$(date +%F)/${resource}.ndjson" \
  | sort -u -k1,1 \
  | your-loader upsert --table "fhir_${resource,,}" --key id
done

sort -u -k1,1 keeps the first row per id; your-loader upsert --key id makes the database write a no-op when the row already exists. Either layer alone makes the load idempotent — running both is deliberate redundancy for a cron job you will not be watching.

What to verify

  • huli auth status --profile roster-sync resolves the org and lists both system/Patient.rs and system/Appointment.rs. 200 OK
  • After a full run, both Patient.cursor and Appointment.cursor are empty — a non-empty checkpoint means the walk stopped partway and will resume next start.
  • If the first page's searchset Bundle reports total, the NDJSON line count for each resource should approximate it, allowing for one duplicated page from a resume. Treat it as a sanity check, not an exact reconciliation — a searchset total is an estimate the server may omit or revise across pages.
  • After the step-5 load, row counts in fhir_patient / fhir_appointment equal the distinct id count in the NDJSON — no duplicates survived the dedup.
  • Kill the script mid-run (Ctrl-C during a page), re-run it, and confirm the output resumes from the saved cursor rather than restarting from page one.

What can go wrong

Every error is a FHIR OperationOutcome. Branch on the HTTP status and issue[0].code (the FHIR IssueType); the Huli code (HPB-…) is the prefix of issue[0].diagnostics — split on ": " to read it. There is no details object and no coding.

429 Too Many Requests HPB-00105 — rate limited. The CLI handles this for you: it

reads Retry-After, sleeps, and re-issues the same page without advancing the cursor. If you ever drive the loop with raw curl instead, you must replicate that — honor Retry-After and retry the same cursor, never the next one.

{
  "resourceType": "OperationOutcome",
  "issue": [
    {
      "severity": "error",
      "code": "throttled",
      "diagnostics": "HPB-00105: Rate limit exceeded"
    }
  ]
}
403 Forbidden HPB-00104 — insufficient scope. The token authenticated but

lacks system/Appointment.rs (or system/Patient.rs). The CLI exits non-zero immediately rather than retrying. Re-mint the key in Practice Settings with both .rs scopes selected.

401 Unauthorized HPB-00106 — auth failed. The client_assertion was rejected

— usually a client_id/private-key mismatch or a clock skew that pushed the assertion's iat/exp out of tolerance. Re-run huli auth status to isolate setup from the loop, and verify the cron host's clock is NTP-synced.

401 Unauthorized HPB-00107 — auth expired. The 5-minute access token lapsed

mid-walk on a slow page. The CLI refreshes automatically before each request; you only see this if you pinned a stale token by hand. Let the CLI manage the token — never cache the bearer across pages yourself.

400 Bad Request HPB-00101 — validation error. A search parameter or the

_cursor is malformed — most often a checkpoint file that was hand-edited or truncated. Treat the cursor as opaque: if a checkpoint is corrupt, clear it (: > Patient.cursor) to restart that resource's walk from page one rather than patching the token.

Next recipes

  • Authenticate as a SMART Backend Service — the client_credentials + private_key_jwt (RS384) handshake the CLI performs under huli auth setup, end to end, for when you need to drive /auth/token yourself.
  • Incremental sync with _lastUpdated — narrow the nightly walk to records changed since the last run instead of a full pull.
  • Paginate a large patient list — the cursor / link[rel=next] mechanics this loop rides on, walked by hand against a single searchset.
  • Mirror Encounters and Observations — extend the same checkpointed loop to the clinical resources, adding system/Encounter.rs and system/Observation.rs.