Doublebill judges a thumbnail test in two halves that are never averaged together — the look, judged on the picture, and the numbers, judged on whether there are enough of them to mean anything.
Nothing opens in this browser — this one takes a job and does the work on our own graphics cards in Oregon. The free allowance below is how you try it: 18 thumbnails, no card and no payment, on the key you are issued.
It will not name a winner on too few impressions, and a solid gap that came from two different weeks gets a verdict of its own.
The problem
Thumbnail A got three clicks, thumbnail B got one, and the conclusion is that A is three times better. It is not — it is four clicks, which is nothing, and the next forty impressions would reverse it as easily as not. Meanwhile the test that actually was decisive ran across two different weeks, so the gap it found is partly the thumbnails and partly the fact that one of those weeks was a holiday.
The look and the numbers, judged separately and never averaged into one score. A beautiful thumbnail with four impressions behind it is not a finding, and a score that mixes those two things hides which one it came from.
A verdict in its own right. Below the threshold Doublebill says there is no winner yet and how many more impressions it would take — rather than naming the leader and letting you act on four clicks.
A solid gap whose two arms ran in different weeks gets its own verdict, because the gap is confounded. That is the finding most A/B tools silently report as a win.
This is the product. Every other thumbnail tool will tell you which one is ahead, because that is what people want to hear and because "ahead" is always true of something. Doublebill has a verdict that says there is no answer yet, and it uses it — and it tells you roughly how many more impressions would settle it, so the verdict is actionable rather than just discouraging.
How it works
Both thumbnails, and the impressions and clicks each one got, with the dates they ran. The dates matter and are not optional — they are what catches the confounded test.
On Tower 1's own graphics cards: what each thumbnail is actually showing, what reads at small size, what is lost. This half does not know the numbers.
In Python: is the gap bigger than the noise, did the two arms run in the same window, and if not, what that does to the conclusion. This half does not know what the pictures look like.
What it will never do
Every refusal here is a refusal to be decisive when the data is not. That is an unusual thing to build a product on and it is the only reason this one is worth buying.
Pricing
A thumbnail is one picture looked at, so a two-way test is two units. Eighteen free on every key is nine complete tests — enough to find out whether the tests you have been running were ever decisive.
Nine complete A/B tests.
A hundred tests across a year of uploads.
A studio testing every release.
Per thumbnail looked at. A not-enough-data verdict is a real verdict and is charged — the work was done and the answer is the valuable one. Your licence key is emailed to you within two working days of payment.
Checkout is our own bitcoin server, not a card processor. The units you buy are added to your key: buy from the Buy more box inside the program and they arrive on their own once the payment confirms, or buy here and paste the invoice number from your receipt into that box.
Questions
Because the alternative is acting on noise, and acting on noise is how a creator ends up convinced of a rule about thumbnails that was four clicks.
It also tells you how many more impressions would settle it, so the answer is run it longer rather than run it again.
The two arms not running in the same window. Different weeks have different traffic, and a holiday week against a normal one will produce a gap all by itself.
This is checked on the dates you supply, in Python, which is why the dates are required rather than optional.
So that neither can lend the other credibility it has not earned.
A combined score out of ten is the most saleable output this product could have had, and it would be the one that hides whether the verdict came from the image or from the sample.
They are looked at and discarded when the job closes. No gallery, no cache, no archive, and they never go to a hosted vision API.
A key. One line of text, issued to you and shown exactly once — only its hash is kept here, so it cannot be looked up later and we could not tell you what yours is. Every key arrives with a free allowance already on it, which is the tier at the top of the pricing above, and the work starts the first time you send a job with that key on it.
Each job is metered in the unit named in the pricing, and a unit that fails — a file that will not read, a row the model held back — is not charged and is named in the job report rather than silently skipped. When an allowance runs out the next job is refused before any work starts, with the number it used printed on the refusal.
More units are bought in bitcoin. The checkout is a bitcoin server we run ourselves. Buy from the Buy more box inside the program and the units are added to your key on their own as soon as the payment confirms; buy from the plan buttons above and paste the invoice number from your receipt into that same box. Each payment adds its units once, to one key.
Nine of them are free. If a decision you made about your thumbnails comes back as not enough data, that is the most useful thing this will ever tell you.