—
Start here
Live
Alerts
Card use
A bar runs from a program's first open of a card to its last close, coloured by login; grey hatched spans were not logged, and a blue hatched span under a bracket is a reservation posted in the AI Foundry Discord's #community-lab. Lab time. Hover a bar or a reservation for its details.
Live, last 15 minutes
Last 24 hours
By person and card
| Login | Card | Held | Of 24 h | Runs | Programs | Last use |
|---|
The last 7 days
People
Everyone logged in, running processes or holding a card. “Doing” is a coarse kind of activity: no commands or files leave the machine.
| Login | Status | Doing | Machines | Idle | Card now | Card use, 24 h | Processes |
|---|
Machines and cards
Click a card for every number and its source.
Last 48 hours
One check every 10 minutes. — Hover a bar for its times.
Worklog
tap to open
AI Foundry lab dashboard › Worklog
What was done in the lab, newest day first; every line opens the page it was written up on, and the lab's own day pages link back here. All of it is 2026.
What we want to do next, oldest first, with the day each item was added and who added it: todo:, below the Worklog.
- Power, then heat — which of the governor's two limits sets the matmul's speed on aifoundry2: on a cool die the 65 W power test alone (zeros 13% faster than random data, Horace's effect), at 66 °C the thermal test alone, and in 80 s runs heat wins (zeros 33% faster); the governor's own log names every step (E64)
- Weighing flipped bits with heat — the thermal camera behind aifoundry2's card watches E60's top rung switched against its zero every 10 s: 5.4 W of flipping bits move the back of the board about 0.2 K, a middle rung in proportion, a null of equal watts not at all (E63)
- Verifying the deterministic transistor-flip experiments — an agent-written proposal's seven experiments, pre-registered and run on three cards: its prices and its zero gate do not hold, its sum of an input and an output bit does, changing the read register costs twelve times what it estimated, and the camera sees the work move between two halves of the die
- Heat follows the work: ET-SoC-1 heat movies — the thermal camera behind aifoundry2's card films compute moved round the die's corners, halves and centre at constant power, and the warm patch on the back of the board follows it, with the camera's live view beside the movies (E62)
- This dashboard's pages checked, and a todo: list — the new-user page offers aifoundry2 again; eighteen linked pages brought up to date; the wired network was down overnight with this page reading All clear
- Inside the silicon: four interactive 3D views — the metal stack, a FinFET, its fins and the channel, each opened from its scale of the chip diagram
- The chip diagram — the scale readout keeps one look while the camera moves, smoother moves, Studio 45 drawn from its measured plans, and its four smallest scales open their 3D views
- Anatomy of a memory access, interactively — the same readout and smoother moves as the chip diagram
- New user? Start here — a line for people in more than one tailnet; aifoundry2 back in the machine field, missing from it since 2 October; the footer's link mended
- New user? Start here — your Claude comes back after a restart: step 2 installs lab-sessions, which brings back each person's tmux session and Remote Control server
- The chip diagram — − and + keep their place in the Up bar
- Anatomy of a memory access, interactively — − and + keep their place, as on the chip diagram
Earlier days: 18 September – 9 October
- Why this dashboard froze for 21 hours, and what watches it now — its publishing check locked itself out after aifoundry2's outage; a keeper on the machine, a standby on aifoundry3 and an outside watchdog
- How energy is measured on the ET-SoC-1 — the tools, a 15-minute tutorial and live runs on aifoundry3: random data costs 14 times as much per multiply-add as zeros
- New user? Start here — only the raw steps now; step 0 is a direct message to the lab lead for the Tailscale invite
- Requests for Roman, the 8 October pass — the list for Friday's visit, notes on 20 items
- aifoundry2's card ran away again and is out of service — idle, it heated to 138 °C and fell off the PCIe bus at 16:46; the cooling at the card got worse at 03:55; back in service since 9 October
- Ethernet between the three machines, measured — 929 Mb/s between any two, about 30 times WiFi; the wired ports got their addresses on 11 and 13 October
todo:
tap to open
Worklog › todo:
What we want to do in the lab, oldest first: the day an item was added, and who added it. An item leaves this list when it is done, and the page it was done on gets its line in the Worklog. The longer ledgers stay where they are: Requests for Roman for the lab's problems, and the review's list for the reports.
- Take the CI runners off the cards' backs — they run as root and can open a card in the middle of someone's run: run them as their own user under the card lock, and retire aifoundry3's dead runner (the lab lead's call); added by an agent (the lab problems pass)
- aifoundry3's demo web app and chatbot — they run as root and can use the card outside any lock: stop them, or put them under the lock (the lab lead's call); added by an agent (the lab problems pass)
- An ssh server on aifoundry3 — Tailscale is the only way in, so when it fails nobody can log in (Yaroslav's decision); added by an agent (the lab problems pass)
- Disk quotas on aifoundry1 — its one disk has no backups, and one person's checkout can fill it for everyone (the lab lead's call); added by an agent (the lab problems pass)
- Cap the core dumps — every crash dump is kept on all three machines; set a limit (root); added by an agent (the lab problems pass)
- Bring the accounts notes in line with the new-user page — the notes the login banners point at predate it; added by an agent (the lab problems pass)
- aifoundry2's tmux — a snap held back from upgrades; Ubuntu's own package would end every running session, so switch at a quiet time; added by an agent (the lab problems pass)
- A UPS for the three machines — every restart since September was a crash, the likeliest cause of aifoundry1's damaged disk blocks (the lab lead's call); added by an agent (the lab review)
- A remote console, or an outlet per machine — one remote switch powers all three, and nobody could see aifoundry2 stuck at its start-up screen for 20 hours on 7–8 October; added by an agent (the lab review)
- A second memory stick for aifoundry3 — it copies memory at about half the rate of the other two, which skews host-side timings (a visit); added by an agent (the lab review)
- Install what is staged for the machines (root) — the login banners (aifoundry1's still tells people not to use card 0), the root-login notice and the card-open tracer (both on aifoundry1 only), and the current et-lab-start; added by an agent (the 2 October repository sweep)
- aifoundry2's BIOS — photograph it, set AC BACK to Always On (it stayed off for 20 hours after the 7 October power cut), test a plug-out, and turn on fan Fail Warning (a visit); added by an agent (on the visit list since 2 October)
- Which firmware each card runs — the new-user brief says 1.4.0 on aifoundry3's card and aifoundry1's card 1 since 7 October, while this page and the docs say 1.3.1 and 1.2.0: confirm, then make them agree; added by an agent (the new-user page pass)
- File the energy tutorial's runs — the page and its scripts are in the repository; the runs on aifoundry3 and the 0.16 s refresh finding are not; added by an agent (the energy tutorial)
- Show the keepers' status on this page — which machine publishes, and whether the standby has taken over; added by an agent (the dashboard watcher work)
- Record the 9 October visit — whether tasks 2–6 were done, and the answers to the seven questions for the lab lead; added by an agent (the 8 October requests pass)
- Hear how the new-user steps went — for the people who joined since 2 October; nothing is on file; added by an agent (the new-user page pass)
- See the fans — load it87 at boot, put both bracket fans on headers Linux reads, add a fan alarm to the collector, and turn on Fail Warning in the BIOS: a stalled fan cost aifoundry2's card twice; added by an agent (the fan swap)
- Better fans for aifoundry2's bracket — one of its two is as old as the fan that stalled; added by Yaroslav (at the fan swap)
- An hour at full load on aifoundry2 — the 10-minute run was still creeping up at the end; added by an agent (the fan swap)
- Reboot aifoundry1 into its new kernel — pending since 9 October; a reboot retrains card 0's link and the machine has no console, so do it on a visit; added by the lab's health check
- Drop the aifoundry2 guards in two experiments — the sparse-parity and PCIe scripts refuse aifoundry2 for reasons that have expired (Yaroslav's call); added by an agent (marking the card back in service)
- Reboot aifoundry3 into its new kernel — when the person working there is done, not before; added by the lab's health check
- Before Sat 18 Oct: land the card-use logger fix — aifoundry1's card-use log reaches its 500 MB cap about 18 October and then deletes history without a word; the fix needs its review's blocking issue solved (a joined minute counted as held time) and a second review; added by an agent (the 11 October maintenance sweep)
- The card sample — this page's card sample has not run since 2 October: a different gate blocks each card; added by an agent (the 11 October maintenance sweep)
- aifoundry1's last damaged disk block — 1.7 KB of a departed user's old clone; deleting it clears the pool's error (the lab lead's call); added by an agent (the 11 October maintenance sweep)
- The live monitor's reader crash — about once in 24,000 reads, on exit, inside the card vendor's management library: report it upstream, and clear its core files; added by an agent (the 11 October maintenance sweep)
- A kernel log longer than an hour — audit lines from sandboxed programs fill it, so the health check's per-card error counts reach back under an hour; added by an agent (the 11 October maintenance sweep)
- The bit-flip heat follow-up — about five card-hours on aifoundry3's card 0, a camera session, an OK for the mount, and the scripts written on site; added by Yaroslav
- Find out why the wired network drops — all three ports lost their link at about 22:35 on 13 October and got it back at 13:56 on 14 October, within seconds of each other, and each has lost it twice since its machine last started: look at the cable and the switch on site; added by an agent (the 14 October sweep)
- Show the health check's information lines here — the wired link was down for 15 hours and this page read All clear, because the health check reports it only as information; added by an agent (the 14 October sweep)
- The card-use logger's memory — it is at its 64 MB limit on aifoundry2 and aifoundry3; an out-of-memory stop would end card-use logging; added by an agent (the 14 October sweep)
- Watch the history page — it has no keeper, no standby, no outside watcher and no visibility check, unlike this page; added by an agent (the 14 October sweep)
- Check the links lab pages point at — a dead footer link survived five deploys; and run the mirror check on a schedule; added by an agent (the 14 October sweep)
- Let lab-sessions update itself — a fix reaches people set up earlier only if they run step 2 again; added by an agent (the 14 October sweep)
Thermal camera, live
The back of aifoundry2's card, seen by the Thermal Master P3 on the phone in aifoundry2's case. It plays while this part of the page is on screen; with the camera off it shows the last picture. The full viewer, with temperatures and the optical camera: the live stream. How it was set up: the thermal camera hub.
About this page
A collector on aifoundry2 reads the three machines every 10 minutes, read-only
(/proc, sysfs counters, et-who, et-lab-health, et-usage) and republishes when something
changed, at least every 60 minutes; an open tab reloads itself. It never opens a card's device node.
When aifoundry2 has published nothing for 90 minutes, the same collector on aifoundry3 takes over until aifoundry2 is
back. A keeper on each of the two checks its collector every minute and repairs what it safely can, and a watchdog
outside the lab reads this page every 15 minutes and alerts the maintainer when it is stale.
It names the lab's users. Per person: login, session and process counts, idle time, coarse activity, card holds and use; never commands, files, addresses or where anyone connects from.
Something wrong or stale on this page? Tell the maintainer, who runs the collector.
For the maintainer (the maintainer): updating and checking it
- On the collecting machine (aifoundry2, or aifoundry3 while it stands in; "This version" below names it), in the
checkout:
tools/lab/dashboard/update.sh nowcollects, rebuilds and republishes at once; open tabs update within seconds.update.sh now --card-samplealso takes a card telemetry sample, behind every rule of the card etiquette.update.sh statusshows the last runs and deploys, a halt, and the space's visibility. - From another machine:
ssh aifoundry2 claude/et-soc1-prototyping/tools/lab/dashboard/update.sh now; or ask Claude in the aifoundry2 session to refresh the lab dashboard. - A machine that shows "approval needed": run
ssh aifoundry1 true(or aifoundry3) on aifoundry2, approve the address it prints, thenupdate.sh now. - Silence a known alert:
update.sh ack <alert id> [days] [note](the id is in each alert's detail). - The keeper:
tools/lab/dashboard/keeper.py statusshows what it sees (the cron line, the last run and publish, the page's age, a halt); it reports the same every minute to the live streamaifoundry-dash/<machine>. Why the page froze for 21 hours on 7–8 October, and what changed: the watcher page. - After a halt (the private mode only: the space was, or may have been, readable without signing
in): check the space, then
update.sh resume; nothing resumes deploying by itself.
This version
—
—
Design and code: tools/lab/dashboard/DESIGN.md.