How the lab tests an idea

Before any data is examined, the test is written down: the question, the data and period, the exact rules, the costs, and the pass mark. Once results exist, none of these change. A failed idea stays failed, and a retest counts as a new attempt.

A real edge has to beat fake versions of itself. Each idea runs against placebos built to look similar but carry no information: random entry times, shuffled signals, invented price levels. If the placebo does as well, the idea is luck.

Results must hold on a period the idea was never shaped on. Survivors are then logged forward, day by day, before any money is used. Where execution matters, a strategy runs on paper with real broker fills first, because the price on a screen and the price received are not the same.

Test enough variants and something will look good by chance. The lab keeps a running count of every variant tested, more than 11,000 so far, and raises the pass mark accordingly, using multiple-testing corrections such as the deflated Sharpe ratio.

AI research agents build data pipelines, run the tests and draft the write-ups. They work under the same frozen rules, and every result is checked against the written plan before it is accepted.

When the lab finds an error in its own work, it is logged with a number and corrected, and the affected results are re-run. The first, ERR-0001, was a valuation bug that made some early results look better than they were.

Most ideas fail at one of these steps. Only strategies that pass all of them and trade real money appear on this site: the live engine is described on the home page, and its results are published in the record.