AI Foundry lab

NO DATA Yaroslav’s agents

—

Start here

New to the lab?Create your account, start Claude on a lab machine, give it one line. Known problemsWhat trips people up, and the requests for the lab lead. Chip diagramThe ET-SoC-1, interactively. The reportsThe observability hub.

Live

◆ AI Foundry lab // live telemetry · history: hour / day / week ↗—

Alerts

Card use

A bar runs from a program's first open of a card to its last close, coloured by login; grey hatched spans were not logged, and a blue hatched span under a bracket is a reservation posted in the AI Foundry Discord's #community-lab. Lab time. Hover a bar or a reservation for its details.

Live, last 15 minutes

Last 24 hours

By person and card

LoginCardHeldOf 24 hRunsProgramsLast use

The last 7 days

People

Everyone logged in, running processes or holding a card. “Doing” is a coarse kind of activity: no commands or files leave the machine.

LoginStatusDoingMachinesIdleCard nowCard use, 24 hProcesses

Machines and cards

Click a card for every number and its source.

Last 48 hours

One check every 10 minutes. — Hover a bar for its times.

Sessions per machine

Worklog

tap to open

AI Foundry lab dashboard › Worklog

What was done in the lab, newest day first; every line opens the page it was written up on, and the lab's own day pages link back here. All of it is 2026.

What we want to do next, oldest first, with the day each item was added and who added it: todo:, below the Worklog.

Earlier days: 18 September – 9 October
Wed 23 Sep

todo:

tap to open

Worklog › todo:

What we want to do in the lab, oldest first: the day an item was added, and who added it. An item leaves this list when it is done, and the page it was done on gets its line in the Worklog. The longer ledgers stay where they are: Requests for Roman for the lab's problems, and the review's list for the reports.

Sat 10 Oct
  • Reboot aifoundry3 into its new kernel — when the person working there is done, not before; added by the lab's health check
Sun 11 Oct
Wed 14 Oct

Thermal camera, live

The back of aifoundry2's card, seen by the Thermal Master P3 on the phone in aifoundry2's case. It plays while this part of the page is on screen; with the camera off it shows the last picture. The full viewer, with temperatures and the optical camera: the live stream. How it was set up: the thermal camera hub.

About this page

A collector on aifoundry2 reads the three machines every 10 minutes, read-only (/proc, sysfs counters, et-who, et-lab-health, et-usage) and republishes when something changed, at least every 60 minutes; an open tab reloads itself. It never opens a card's device node. When aifoundry2 has published nothing for 90 minutes, the same collector on aifoundry3 takes over until aifoundry2 is back. A keeper on each of the two checks its collector every minute and repairs what it safely can, and a watchdog outside the lab reads this page every 15 minutes and alerts the maintainer when it is stale.

It names the lab's users. Per person: login, session and process counts, idle time, coarse activity, card holds and use; never commands, files, addresses or where anyone connects from.

Something wrong or stale on this page? Tell the maintainer, who runs the collector.

For the maintainer (the maintainer): updating and checking it
  • On the collecting machine (aifoundry2, or aifoundry3 while it stands in; "This version" below names it), in the checkout: tools/lab/dashboard/update.sh now collects, rebuilds and republishes at once; open tabs update within seconds. update.sh now --card-sample also takes a card telemetry sample, behind every rule of the card etiquette. update.sh status shows the last runs and deploys, a halt, and the space's visibility.
  • From another machine: ssh aifoundry2 claude/et-soc1-prototyping/tools/lab/dashboard/update.sh now; or ask Claude in the aifoundry2 session to refresh the lab dashboard.
  • A machine that shows "approval needed": run ssh aifoundry1 true (or aifoundry3) on aifoundry2, approve the address it prints, then update.sh now.
  • Silence a known alert: update.sh ack <alert id> [days] [note] (the id is in each alert's detail).
  • The keeper: tools/lab/dashboard/keeper.py status shows what it sees (the cron line, the last run and publish, the page's age, a halt); it reports the same every minute to the live stream aifoundry-dash/<machine>. Why the page froze for 21 hours on 7–8 October, and what changed: the watcher page.
  • After a halt (the private mode only: the space was, or may have been, readable without signing in): check the space, then update.sh resume; nothing resumes deploying by itself.

This version

—

—

Design and code: tools/lab/dashboard/DESIGN.md.