Open four spam protection listings on the Shopify App Store and try to say what differs between them. Each is invisible to customers. Each installs in minutes. Each blocks the bad submissions and lets the real ones through. The screenshots show a dashboard with a number on it.
Plenty differs, and almost none of it is visible from a listing. Which forms an app can actually see, what it does at three in the morning when its own server is unreachable, whether it hands you the score behind a decision or only a count. Those answers vary considerably between products, and a merchant has no way to find them without asking.
So: seven questions worth asking, what a good answer sounds like, and our own answers. We sell one of these products and we chose the criteria, which is a conflict worth holding in mind. Two of the seven we currently fail, and they are marked.
Run the ten-minute triage in our first dispatch first. A meaningful share of merchants who install an app in this category did not need one: their spam was arriving through a form-builder app with its own protection switched off, or through a checkout endpoint no storefront app can reach. Establish which problem you have before paying anyone to solve a different one.
The seven
Shopify has seven places a stranger can submit text: contact, newsletter, blog comments and reviews, login, account registration, password reset, and the order form. Shopify's own built-in protection covers six of them, and only when four conditions hold. An app that also covers six is not adding coverage; it is adding a dashboard.
The second half of the question matters more. Forms built by form-builder apps and page builders do not carry Shopify's native form type, which is why the built-in setting never sees them and why they are the most common source of spam on a store that has "already turned CAPTCHA on".
This is the step at which most DIY implementations are abandoned: create a Google Cloud project, generate a site key and a secret, pick the right protection type, register the domains, paste both into the app. Several apps in this category still require it, which means the merchant is buying a wrapper around work they must do themselves.
Ask a follow-up: what happens when I add a custom domain six months from now? A reCAPTCHA key does not learn about new domains on its own, and the symptom is verification failing on the new domain only, which is a genuinely confusing outage to debug.
The single most important question on this list, and the one least often answered in a listing. Every scoring system depends on a network round trip, and networks fail. When one does, the filter chooses between rejecting a submission it could not verify and allowing it.
A good answer distinguishes two cases. If the vendor's own service is unreachable, the submission should be allowed, because an outage in a third-party app must not stand between a customer and a contact form. If the underlying CAPTCHA provider returns a hard verification error, which usually means a broken key rather than a network blip, blocking with a visible message is defensible. What no merchant should accept is a filter that fails closed silently, rejecting real orders with no error anyone ever sees.
Two figures, and a vendor should be able to give both without checking. The app's own script is usually small. The real weight is Google's reCAPTCHA library, and the question is on which pages it gets fetched.
An app that loads it unconditionally charges every visitor to every page for a library most of them will never use. An app that gates the fetch on a form actually being present charges only the page views where there is something to protect, and a catalogue page with no form makes no request to Google at all. The strongest answer, which we do not currently give, is to wait until the visitor actually touches the form.
A blocked-count is a number an app can show you without you being able to check it. The question that matters after a merchant suspects a real enquiry went missing is what happened to a specific submission and why, and that requires a per-submission log.
Ask whether the log records the threshold that was in force at the time. Without it, past decisions become uninterpretable the moment the setting changes, and the merchant loses the ability to answer the only useful question after a false positive: what would have happened at a different setting?
A filter with no threshold control has made a decision on your behalf about the relative cost of blocked spam and blocked customers, and that trade differs by store. A blocked blog comment costs nothing. A blocked checkout costs a customer.
The better version of this question is whether the threshold is per form. One dial serving every form means the correct setting is whatever your most sensitive enabled form can tolerate, which on most stores drags the whole store down to the leniency that checkout requires.
This is the honesty test, and the answer you want is "nothing, and neither does anything else in this category."
Every mechanism here tests whether a submission came from automation. A person paid per submission to type guest-post pitches and SEO offers into contact forms passes all of it, and on a lot of stores that is a large share of the actual problem. The effective control is a keyword mail rule, which is free and takes fifteen minutes.
A vendor claiming to stop human-written spam is either describing content classification, in which case ask what model, what it costs and what its false-positive rate is on genuine short enquiries, or is overselling.
The scorecard
The seven condensed, for use against any listing including this one. Print it, or paste it into an email to a vendor.
| Question | Spambuster | Their app |
|---|---|---|
| All seven form types, plus third-party forms | ✓Yes | |
| Provisions its own reCAPTCHA keys | ✓Yes | |
| Fails open when the vendor's service is down | ✓Yes | |
| Loads Google's library only on pages that have a form | ✓Yes | |
| Per-submission log with score and threshold | ✓Yes | |
| Threshold under merchant control | ✓Yes | |
| Per-form thresholds | ✕No | |
| Graduated response instead of pass or block | ✕No | |
| Honest that it cannot stop human spam | ✓Yes | |
| Defends the checkout endpoint | ✕No |
Try it for a fortnight and check the answers yourself
Every claim in the scorecard is verifiable from inside the app: the log shows the scores, the setup asks for no keys, and the page source shows when reCAPTCHA loads. Basic $3 a month, Premium $12 for honeypots and the behavioural layer. Fourteen days free, no card, billed through your normal Shopify invoice, cancel from inside the app.
Start the free 14-day trial →Three questions not on the list
Some obvious criteria are missing, and the omissions are deliberate.
Price is not on the list. Everything in this category costs between nothing and about fifteen dollars a month, which is less than the value of one recovered enquiry on most stores. Choosing on price here is optimising the wrong variable.
Star rating is not on the list. We would say that, because ours is poor: 2.0 from seven reviews, weighted heavily by an older version that conflicted with the checkout button in Shopify's Dawn theme on macOS Safari. That was a real fault, it was fixed and verified against Dawn and Safari, and the fix shipped to every store automatically. We mention it here because a merchant evaluating us will find it, and because a seven-review sample on an app this size tells you less than the answers above do. Read the reviews rather than the number, on any app in this category.
"Blocks 99% of spam" is not on the list, because no vendor can substantiate it. The denominator is unknowable: nobody can count the spam that was never sent, and nobody can distinguish spam that a filter stopped from spam that never arrived. Treat any percentage in a listing as decoration.
What a good conversation sounds like
If you take one thing from this, take the follow-up questions rather than the list. The questions above can all be answered yes by a vendor who has not thought about them. The follow-ups cannot.
- You said it covers all the forms. Which hostname does your script run on? If they cannot answer, they have not thought about the checkout gap.
- You said it fails safely. Fails safely which way, and for which failure? There are two failure modes and they deserve different answers.
- You said it is lightweight. What is the gzipped size, and what condition triggers the reCAPTCHA fetch? Both are checkable in a page source in thirty seconds, on one page with a form and one without.
- You said it logs everything. Does the log record the threshold that was in force? This is the one that separates a real event log from a counter.
- You said it catches sophisticated bots. What is the flag threshold, and what is the heaviest single signal worth? Ours are published in dispatch 03. Most vendors will not answer this at all.
A merchant asking those five will get a straight answer from a vendor who built the thing and an evasion from one who did not. That is most of what an evaluation needs to establish, and it takes one email.
We sell a product in this category and we selected the criteria, which is a conflict readers should weigh. We have deliberately not tested, named or scored competing apps, and we make no claims about them; the right-hand column of the scorecard is for merchants to fill in themselves. Our own answers are verifiable from inside the app and from our storefront source. Corrections, and disagreement about the criteria, to contact@jola.com.au