ESSAY · Updated September 2, 2026

The research behind the rankings: what I tested, kept, and threw out

In brief: The rejected ideas are the honest part of any model. Here are the signals I tested and threw out, the two that hurt to lose, and the times my own measuring tools turned out to be wrong.

EssayWritten by Bartholomew Chupka Jr. Editorial standards

I'm a measure-twice, cut-once person. That's where the discipline behind this model comes from — not a blow-up I'm still recovering from, just a preference for having more data before I commit to anything.

Applied to a stock model, that preference has one obvious consequence, and it's the rule everything here runs on.

The rule: out of sample, or it doesn't ship

No signal enters the score unless it improves results on data it was not tuned on.

That sounds obvious. It's also where most quantitative work quietly goes wrong. It is trivially easy to find a rule that would have worked beautifully on the past — sweep enough thresholds and something always wins the history you fed it. The only question that matters is whether it keeps working on data the rule never saw.

So when I test a signal, I tune it on one window and judge it on a held-out window the tuning never touched. If it doesn't survive that, it doesn't ship, no matter how good the in-sample chart looks.

Taking that seriously means rejecting far more than I keep. The rest of this is mostly a list of good ideas that didn't make it, because that list is the only real evidence the rule isn't decorative.

What the score actually is

One number, 0 to 100, on every stock the model can grade. It's the classic GARP idea — growth at a reasonable price — built from three pillars:

  • Quality. Returns on capital, margins, leverage, and whether earnings are backed by cash. Is this a genuinely good business?
  • Growth. Revenue and the forward earnings trajectory. Is it getting bigger?
  • Value. Valuation against the stock's sector and its own history — PEG, earnings and cash-flow multiples. Am I paying a fair price for the first two?

The pillars aren't summed; they're multiplied. A glaring weakness on any one leg drags the whole number down, because a business that's cheap and bad, or great and unaffordable, isn't the setup I'm looking for. Valuation and debt are judged against sector peers, since a debt load that's routine for a utility would be alarming for a software company.

Banks, insurers and REITs don't get a quality-growth score at all. Free cash flow and leverage don't mean the same thing for a balance-sheet business, so they're graded on their own card and labeled as such.

What I threw out

Every one of these was a reasonable idea. Every one failed out of sample, so none of them are in the score.

Price momentum and trend overlays. A 6–12 month momentum tilt, a 200-day-average filter, a golden-cross bonus — all of them degraded returns out of sample, some by double digits of annual return, with deeper drawdowns. The reason is structural, not bad luck. The value pillar leans this model toward names trading at a fair or cheap price, and those are often below trend precisely because they're out of favor. A trend filter systematically avoids exactly those names. It doesn't just fail to help; it removes part of where the return comes from.

Gross profitability. This one stung. Gross profitability is a well-known quality factor and it's genuinely strong on its own — a sleeve that simply buys the highest-gross-profitability names beat both the market and my model on risk-adjusted return in testing. So I tried to wire it into the score. It backfired at every weight I tried: lower return, lower risk-adjusted return, deeper drawdowns than leaving it out.

Those two are the ones I was most hopeful about, and they taught the same lesson. They make sense in a vacuum, but like a selfish athlete, they didn't help out the team. A factor can be a real edge on its own and still poison a score that was already capturing the same thing. Gross profitability stays on the page as a data point. It isn't in the number.

Market-timing crash defense. Going to cash when the S&P falls below its 200-day average sounds prudent. Tested across a decade, it gutted returns and deepened the worst drawdown — because for a strategy that buys into weakness, "the market is below trend" is exactly when the best entries appear. Timing out of weakness sells the bottom and misses the rebound.

A bank-friendly quality bonus. The idea was to let strong financials — high ROE at a cheap multiple — compete despite lacking the cash-flow signals the model leans on. It degraded returns by slipping banks ahead of names that did better.

When the measuring tools were the problem

Rejecting signals is the easy half. The rule only means something if it also applies to the instruments doing the measuring, and mine have been wrong more than once.

A backtest that couldn't see one of its own rules. The book limits how much of any one sector it will hold. I tested that limit repeatedly and it always came back neutral — no measurable cost, no measurable benefit. That reading was structurally guaranteed: the historical data the backtest reconstructs was missing the field the limit is based on, so it was never actually applied in any simulation. Every test of it was comparing two identical books. Months of "no harm" that couldn't have said anything else.

Once that was fixed and the same question asked again, the answer was specific. A tighter limit costs about a point of annual return. The current setting lands essentially level with an unconstrained book — it holds real diversification without costing performance. A looser setting isn't a looser limit at all — it's no limit — and it tested slightly worse. So the setting stayed where it was — but now for a measured reason instead of an assumed one.

Judging a change by the wrong number. In that same work, I argued against a setting because the average score barely moved. Score is a ranking device; it was never built to forecast returns, and I used it as a stand-in for one. Measured properly, the change I'd dismissed was worth roughly a point of annual return. The lesson is uncomfortably close to the one about gross profitability: a number can be useful for its own job and misleading the moment you borrow it for a different one.

A warning that flagged a problem for weeks without naming the culprits. A nightly check flags stocks whose figures contradict each other. It reported a count every night and discarded the names behind it, which made the warning unactionable — you knew something was wrong and had no way to look at it. It now records the names.

And one that wasn't broken at all. I looked up Intel, saw no P/E, and assumed the data was missing. It wasn't: Intel lost money over the trailing year, so a P/E has no denominator, and the site had correctly substituted price-to-sales. The problem was that it never said so anywhere you could see without hovering, and hovering does nothing on a phone. A correct number nobody can interpret is its own kind of defect.

Why publish any of this

I'm not asking anyone to take my word for it, and I don't think a track record of wins would earn that anyway. What I can offer is that every part of this is visible — the process, the rejections, the limits, and the results as they accrue rather than after the fact.

I'm sharing a journey and trying to express data in a way that's easy to digest. That's it. The value in the list above isn't that I'm clever; it's that running the data honestly kept overruling what I wanted to be true. Gross profitability should have worked. Momentum should have helped. Neither did, and the model is better because I didn't argue with the result.

Not letting your own opinion get in the way of the facts is easy to say and genuinely hard to do. Publishing the times it didn't go my way is the only way I know to show I actually mean it.

I'm equally direct about the limits. The backtest universe is survivors-only — a bias I've quantified and haircut rather than hidden. Fundamentals run about ten years deep. The live, forward track record is still young. You can read the full methodology and its limitations on the about page, and watch the real, never-back-dated results accrue on the track record.


The Bull Rankings is automated fundamentals research for education, not personalized investment advice and not a recommendation to buy or sell any security. Always do your own research.

Not investment advice. The Bull Rankings publishes a quantitative ranking model and accompanying analysis for general informational purposes only. Nothing on this page is a recommendation to buy, sell, or hold any security; nothing is personalized to your circumstances, risk tolerance, or tax situation. Investing carries the risk of loss — invest at your own risk and consider consulting a licensed financial professional before acting on anything you read here. See terms and methodology for full disclosures.