A/B test calculator
Find out whether your variant really beat the control: 130 conversions from 1000 visitors against 100 from 1000 gives a z-score of 2.10274060562 and a two-sided p-value of 0.0354883261439. Enter four counts and read the result. Free in your browser.
- Free
- Runs in your browser
- No sign-up
- Your data stays on your device
Run — free
Worked example
- Control visitors
- 1,000
- Control conversions
- 100
- Variant visitors
- 1,000
- Variant conversions
- 130
- Control rate
- 0.1
- Variant rate
- 0.13
- Rate diff
- 0.03
- Pooled rate
- 0.115
An A/B test compares two conversion rates and asks whether the observed gap is large enough to reject the idea that both groups share the same underlying probability. You enter four non-negative counts: visitors and conversions for the control arm, and visitors and conversions for the variant arm. This calculator runs the classical two-proportion z-test: it forms each sample rate, pools the successes under the null that the rates are equal, builds the standard error of the difference, and returns the z-score together with a two-sided p-value from the standard normal curve.
How the two-proportion test reads your experiment
An A/B test asks whether the gap between two conversion rates is bigger than chance would explain if both groups really shared the same rate. The calculator first works out each rate: 100 conversions from 1000 visitors is 0.1 for the control, and 130 from 1000 is 0.13 for the variant. It then pools both groups into one shared rate, 0.115, which is the best estimate if there were no real difference. From that pooled rate and the two group sizes it builds the standard error of the difference, here 0.0142670950091. Dividing the observed difference of 0.03 by that error gives a z-score of 2.10274060562.
Reading the result without the usual mistakes
The p-value is two-sided: it is the chance of a gap at least this large in either direction if the rates were equal, here 0.0354883261439. A small value is evidence against equal rates, not proof that the variant will keep winning. The sign of the z-score gives the direction, positive when the variant converted better. This is the classical test on fixed counts, so it does not adjust for looking at the results again and again while traffic is still arriving, and it does not tell you how many visitors you need. A group with zero visitors has no rate at all, so it is refused.
What the two-proportion z-test says about an A/B experiment
A conversion is a binary outcome: a visitor either completed the goal or did not. The control conversion rate is control conversions divided by control visitors. The variant conversion rate is variant conversions divided by variant visitors.
Under the null hypothesis that both arms share one unknown success probability, the most efficient estimate of that shared probability is the pooled rate: all conversions from both arms divided by all visitors from both arms. The standard error of the rate difference then follows the textbook two-proportion formula: the square root of the pooled rate times one minus the pooled rate times the sum of the reciprocal sample sizes. The z-score is the observed rate difference, variant minus control, divided by that standard error.
A positive z-score means the variant converted better than the control; a negative z-score means it converted worse. The two-sided p-value is the probability, under a standard normal, of a z at least as extreme in either direction as the one you observed. A small p-value is evidence against equal rates; a large p-value means the gap is ordinary given the sample sizes.
What you can do with it
Landing-page and checkout experiments
Turn control and variant visitor and conversion counts into a z-score and two-sided p-value for a classical A/B significance read.
Dashboards and newsroom fact-checks
Compute “the variant won” language from published experiment counts without hand arithmetic errors or NaN from empty traffic cells.
FAQ
What happens if a group has zero visitors?
A conversion rate is conversions divided by visitors, so a group with zero visitors makes the rate (and the z-test) undefined. These failures are not billed.
Is the p-value one-sided or two-sided?
Two-sided. The p-value is 2(1 − Φ(|z|)), the probability of a difference at least as large as the one you observed in either direction under equal rates.
Does a positive z-score mean the variant won?
A positive z-score means the variant conversion rate was higher than the control rate. Whether that gap is statistically notable is what the p-value answers; this calculator does not declare a winner by an automatic alpha cutoff.
Can conversions be higher than visitors?
No. A group cannot have more successes than people, so conversions must stay between zero and the number of visitors in that group; anything else is rejected before the test runs.
Does the calculator tell me how many visitors I need?
No. It runs the classical test on the counts you already have and does not compute a required sample size, nor does it correct for checking the results again and again while the test is still running.
Related tools
For developers — API access
Everything on this page is available programmatically. This section is for teams who want to wire it into their own systems; everyone else can just use the tool above.
API endpoint
Prefer to automate it? One authenticated POST creates the task; the result comes back by webhook or a signed link. The same capability also runs here on the web, by email and from Telegram — and soon from our app too.
Call it from your stack
curl -X POST https://api.kit.forhosting.com/stat/ab-test \
-H "Authorization: Bearer $KIT_KEY" \
-H "Content-Type: application/json" \
-d '{"control_visitors":1000,"control_conversions":100,"variant_visitors":1000,"variant_conversions":130}'const res = await fetch("https://api.kit.forhosting.com/stat/ab-test", {
method: "POST",
headers: {
"Authorization": `Bearer ${process.env.KIT_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
"control_visitors": 1000,
"control_conversions": 100,
"variant_visitors": 1000,
"variant_conversions": 130
})
});
const { task_id } = await res.json();import os, requests
res = requests.post(
"https://api.kit.forhosting.com/stat/ab-test",
headers={"Authorization": f"Bearer {os.environ['KIT_KEY']}"},
json={
"control_visitors": 1000,
"control_conversions": 100,
"variant_visitors": 1000,
"variant_conversions": 130
},
)
task_id = res.json()["task_id"]<?php
$res = file_get_contents("https://api.kit.forhosting.com/stat/ab-test", false, stream_context_create([
"http" => [
"method" => "POST",
"header" => "Authorization: Bearer " . getenv("KIT_KEY") . "\r\nContent-Type: application/json",
"content" => '{"control_visitors":1000,"control_conversions":100,"variant_visitors":1000,"variant_conversions":130}',
],
]));
$task = json_decode($res, true);body := bytes.NewBufferString(`{"control_visitors":1000,"control_conversions":100,"variant_visitors":1000,"variant_conversions":130}`)
req, _ := http.NewRequest("POST", "https://api.kit.forhosting.com/stat/ab-test", body)
req.Header.Set("Authorization", "Bearer "+os.Getenv("KIT_KEY"))
req.Header.Set("Content-Type", "application/json")
res, _ := http.DefaultClient.Do(req)Example request
{
"control_visitors": 1000,
"control_conversions": 100,
"variant_visitors": 1000,
"variant_conversions": 130
}Example response
{
"task_id": "tsk_a1b2c3d4e5f6a1b2c3d4e5f6",
"type": "stat.ab_test",
"status": "queued",
"_links": {
"result": "/tasks/tsk_…/result"
}
}The API is asynchronous: the call returns a task_id immediately and the result arrives by webhook. Polling is capped at 1 req/s per task.
Pricing
Published price — no tokens, no invented credits. A failed task is never charged.
Errors
| HTTP | Code | Meaning |
|---|---|---|
401 | unauthorized | Missing or invalid API key. |
402 | insufficient_balance | Your balance doesn't cover the task price. |
404 | unknown_type | That task type doesn't exist. |
429 | rate_limited | Too many requests. Use the webhook instead of polling. |
Technical notes
When either arm has zero visitors the conversion rate is undefined, so the task fails with invalid_input instead of dividing by zero.
What the two-proportion z-test says about an A/B experiment
This endpoint does not invent sequential always-valid p-values, does not adjust for peeking, and does not compute required sample size: it is the classical pooled test of two independent binomial proportions, written so that a product manager, a student, and an automation script all receive the same numbers from the same four counts.
Input contract, response shape, and error behaviour
Counts must be greater than or equal to zero. Conversions must not exceed their matching visitor totals, because you cannot observe more successes than people. POST a JSON body with four required finite numbers: control_visitors and control_conversions for the baseline arm, variant_visitors and variant_conversions for the challenger. Each field may arrive as a JSON number or as a numeric string; aliases such as n1, x1, n2, and x2 are accepted if you already speak textbook notation, but you must send only one name per quantity. Non-finite values, missing fields, and empty strings fail with invalid_input naming the field. Both visitor totals must be strictly greater than zero: a group with zero visitors makes the conversion rate undefined, and the error message states that explicitly so graders and API clients can distinguish an empty arm from other bad input. On success the body echoes the four counts, returns control_rate and variant_rate, the signed rate_diff (variant minus control), the pooled_rate used under the null, the standard_error of the difference, the z_score, a two_sided p_value, and a short formula string. When every visitor converted or none converted in both arms, the standard error is zero and both rates are equal; the calculator then returns z_score zero and p_value one instead of a 0/0 NaN. Failed validation does not consume prepaid balance. Pair this endpoint with your own software if you need confidence intervals, power, or sequential testing; this capability is intentionally a pure analytic two-proportion z-test from four counts.
Where an A/B test calculator earns its keep
Growth teams and product analytics desks need a deterministic A/B test calculator that never silently returns NaN when someone pastes a row with zero visitors from an empty experiment cell. Growth-marketing notebooks use the two-proportion z-test to summarise a landing-page or checkout experiment before handing off to a full statistics package for CUPED, sequential tests, or regression adjustment. Teaching platforms and exam generators need an oracle that marks zero-visitor arms as errors rather than empty success payloads a buggy grader might treat as correct. Content sites that rank for ab test calculator gain a live tool with a clear SEO title and keywords, while automation scripts call the same type over the KIT API at the published $0.002 base price with no model tokens and no outbound network.
Integration use cases
Homework and statistics teaching
Let students submit four counts and check the pooled two-proportion z-test against textbook methods, with invalid_input when a group has zero visitors.
Technical questions
How do I run an A/B test from visitor and conversion counts?
Send control_visitors, control_conversions, variant_visitors, and variant_conversions. The API returns each conversion rate, the pooled rate, the standard error, the z-score (variant minus control), and a two-sided p-value.
What does the A/B test calculator cost?
The browser tool is free. API requests use prepaid KIT balance at $0.002 per successful call; invalid input is not billed.