Reverse CAPTCHA ~ Thoughts, Tips, and Questions

[Russ Edwards]

…

Hi All –

First, some background.

I am thrilled with FormMail and have been using it on several forms on several websites, and have successfully integrated reCAPTCHA on those forms.

Recently, with no changes on my end, reCAPTCHA stopped working properly. Correctly entered challenge words still returned a “reCaptcha verification failed” error.

I suspect the issue is a change made at Go Daddy, my web host, but thus far I have not been able to get them to look very deeply into the issue, and they say everything is working OK on their end.

This prompted me to look into alternatives to reCAPTCHA, and to evaluate the entire issue of how to minimize spambot damage / inconvenience.

A starting point is to accept that FM’s built-in security is as good as it gets with php form processors, so the issue here is not one of security per se, but of annoyance.

For me, as a marketing guy, there is also the issue of perceived credibility, which I’ll address in a moment.

By the way, these are just my observations, and they may be totally off. Please feel free to correct me if I get any of this wrong.

As I step back and look at the whole picture, but strictly through the lens of reducing spam replies to our forms and ignoring for the moment the convenience / inconvenience to customers or their perception of how sophisticated the site is, I don’t see a clear winner of the CAPTCHA / Reverse CAPTCHA contest.

They are really both sides of the same coin in terms of effectiveness, presuming each are properly implemented.

Both will cut down, but not totally eliminate, on our forms being spammed by bots.

The sophistication of the bots has more to do with how much spam we’ll get than the differences between CAPTCHA and Reverse CAPTCHA.

That is, the nature of a given bot determines whether CAPTCHA or Reverse CAPTCHA would be more effective in that particular attack. And we do not get to choose our attackers, so it is literally a coin toss as to which approach one takes to reduce spam.

So to decide which approach is better, we need to look beyond pure effectiveness, since that is a tie.

Ease of implementation, footprint, and other technical issues should not be considered since all those issues can be resolved, with assistance if necessary, and it is foolish to accept having to suffer boatloads of spam because you couldn’t get some piece of code to work as it should.

An exception to that is if there is some technical limitation preventing one method or the other from working in a given instance. I am facing this now, as Go Daddy has likely done something inadvertently that is causing reCAPTCHA to fail on the websites I have hosted there.

Barring such a limiting factor, however, the next thing to consider is the User Experience.

And in that regard, CAPCHA sucks. No one likes it. Not only is it an extra hoop for customers to jump through, but annoying as can be to them.

And to us, the loss of customers (or whomever will be filling out our forms) is far more annoying than any extra spam we would receive. I want customers first and foremost. If I have to accept more spam in order to get more customers, so be it.

Yes, there are some cool CAPTCHAs out there – cute kittens, tic-tac-toe games, and simple questions that eliminate those awful distorted challenge words. But to most customers most of the time, no CAPTCHA is better than any CAPTCHA.

So that is a huge point in favor of the invisible but just as effective Reverse CAPTCHA.

If that were all there was to it, I would have to wonder why anyone – ANYONE - would include a CAPTCHA when Reverse CAPTCHA works as well without annoying customers.

Again, if I am missing something here, and I may be, let me know!

But there is another factor – your website’s perceived credibility. Sometimes that matters a great deal, as it does with my commercial sites.

Users have become accustomed to the “big league” sites having a CAPTCHA on contact, registration, and order forms. They assume that CAPTCHA, especially an industrial grade looking one like reCAPTCHA, is only available to serious web players.

That association causes them to equate sites with reCAPTCHA with high credibility, as if they somehow had to qualify to be eligible for what they see as a high-grade security feature.

We know better, but the masses think differently. To them if they see reCAPTCHA, annoying as it is, it means that site has met some standard and therefore the site can be trusted.

Likewise, they see a site with no CAPTCHA as kind of naked, or amateurish, and even worse if they see what appears to them to be a crude, primitive CAPTCHA– you know the type.

Point is, seeing that reasonably elegant implementation of Google’s reCAPTCHA adds a good deal of credibility to a site, even if it is also annoying at the same time.

My educated guess is that credibility goes a long way – but not 100% - to offset the annoyance of having to do the CAPTCHA dance.

So even if the credibility gives reCAPTCHA ¾ of a point in our head to head contest, Reverse CAPTCHA still wins by a quarter point.

You were keeping score, right?

And that gap can be widened, giving the Reverse CAPTCHA its ¾ point back, by some creative wording somewhere explaining the lack of obvious CAPTCHA is because the site is employing a far more sophisticated and stealthy (and much less annoying) system to eliminate spam, the specifics of which shall remain locked away.

The bottom line for me, therefore, is that Reverse CAPTCHA is the way to go.

It is not without some issues, however, and I’ll mention them in a follow up post to this one, as well as details on how I plan to implement it on my sites.

I welcome comments / criticism.

More later…

Russ Edwards

…

[Russ Edwards]

…

Hi Again –

As I described above, I decided Reverse CAPTCHA made more sense when considering all the primary factors – effectiveness, customer annoyance, and perceived credibility.

But two other concerns came to mind and I don’t have the answers to those – accessibility and smart bots (well, smart bot designers.)

Re: Accessibility, how do screen readers handle the honeypots – the spambot bait in the form of hidden fields?

How do screen readers know not to try to fill in those fields? Obviously, if those fields are filled out FM says Gotcha!

And screen readers see those bait fields much the same way that bots do, right?

If that’s true, and if screen readers can see those fields are hidden and ignore them (as they should) can’t the bots do the same, circumventing the who idea?

Likewise, if legitimate screen readers take the bait and fill them in, they will get rejected along with the spam.

So one way or the other, Reverse CAPTCHA is either bad for accessibility, or is less effective against spambots that are as sophisticated as screen readers.

I’m still going with Reverse CAPTCHA, but the accessibility issue is bothersome to me.

Related, if screen readers can learn what to fill out and what not to, can spambots do the same?

Especially if Reverse CAPTCHA becomes more popular. I’m assuming of all the forms out there using some kind of spambot protection, that 90% or more –perhaps way more – use CAPTCHA rather than Reverse CAPTCHA. What do you think?

Consequently, the spambot developers probably concentrate more on how to defeat CAPTCHA. But if more forms start to use a Reverse CAPTCHA, the efforts to defeat that will increase and it shouldn’t be that difficult to identify the honeypot signatures from those of legit fields, and not fill them in.

Still, I’m becoming more convinced that for now Reverse CAPTCHA is the way to go, and FM makes its implementation a no brainer.

Next, I’ll detail how I will implement Reverse CAPTCHA on my commercial sites.

Russ . . .

…

[Russ Edwards]

…

Hi –

The idea behind Reverse CAPTCHA is to include at least two hidden form fields - hidden from the user’s screen but visible to bots who read the HTML – that if filled out will cause the form submission to be rejected on the theory that only a bot would fill out the hidden field. One of these honeypot fields is already filled out and one is not. If the bot fills out the one that is supposed to remain empty, Sayonara.

Three things to consider are how to hide those fields, what to name them, and how many should there be. There has been discussion about all of those, and some ideas are better than others.

Here is my take on each.

The way to hide these fields is through CSS, and optimally through an external CSS.

Three ways to use CSS to hide them are using the display, visibility and position properties.

Two of those are not a good idea.

Using visibility:hidden fails for the following reason.

Yes, it makes that field invisible on your form, but the space that field would have taken is still there (as a blank space.)

So if you had two or more honeypot fields with visibility:hidden, you would have an awkward empty space in your form. If you had several honeypot fields, you might be inviting a usability issue as your users would wonder what they were missing in that huge blank space.

You can make up for the awkward empty space by including negative margins in the same class as the visibility property, but how inelegant.

On the other hand, display:none not only makes the field invisible, but it takes up no space. Dozens of display:none fields take up not a pixel of space.

The display:none property and value is what I use.

To be clear, visibility:hidden hides the field, but leaves it in the flow so it takes the same space on screen as if it were visible; display:none takes the field completely out of the flow, so the next visible field moves up to where the hidden field would have been.

So if you want your form to give no indication on screen that there are hidden fields, you must use display:none.

The third suggestion is to use absolute positioning to position the display of the honeypot field off the screen some thousands of pixels.

So the field isn’t really hidden, but it is displayed too far from the screen to be seen (which effectively renders it invisible.)

Since it is positioned absolutely, it is also out of the flow, and the next visible field moves up to where the absolutely positioned field would have been, so there is no empty space as there is with visibility:hidden.

Problem is, while absolute positioning the field off screen does not display the field on screen, a copy and paste of your form will reveal its content.

Not that many folks will be selecting and copying your form, but if they did your absolutely positioned honeypot fields would now be on their clipboard and pastable / printable elsewhere.

To be clear, absolute positioning the field off screen hides it, but it is still easily see by the copy function.

No such drawback, minor as it may seem, is associated with display:none.

I should mention that I’ve read a few reports that using display:none causes forms submitted using Chrome to fail, but I have not been able to duplicate that myself, and I believe that may have been an issue with older versions of Chrome and/or using a forms processor other than FM.

In any event, you should try all three methods yourself and note any drawbacks.

Next, how to name the fields? Do we make them gibberish (random) names, or do we give them names a bot might expect?

Since arguments can be made supporting each, I simply use both. This also answers the question of how many honeypot fields to use.

I use five.

The first uses a sensible field name with the value blank.

The second has a gibberish field name with the value blank.

So at that point, one or the other or both should trick a bot, depending upon what it is looking for and how smart it is.

The third field has a normal word field name and normal word for its value.

The fourth and fifth have normal word field names with the values blank.

In the context of all the other legitimate fields, these do not jump out, and the classes with the display:none included are on an external style sheet.

To an average bot, then, these fields should look no different than any other.

The $ATTACK_DETECTION_REVERSE_CAPTCHA array includes the name and values of all five of the honeypot fields, obviously.

I can post the actual CSS and HTML if anyone thinks that would be helpful.

But I think this, along with ATTACK_DETECTION_JUNK set to "true,” is sufficient to make FM’s implementation of Reverse CAPTCHA as efficient as can be, and I do not expect it will defeat all spambots forever but should frustrate a good chunk of them today.

Again, I am a marketing guy, not a programmer, and I may have all this totally wrong. But I have spent some time looking into this, and the above are my conclusions at the moment, subject to change upon further enlightenment.

Comments and criticisms welcome, and I hope this helps someone.

Later…

Russ Edwards

…

[Russ Edwards]

…

Hello Again –

I should add to the above that in creating the 5 honeypot fields I took into consideration those rare events where the user might not see the page in its CSS glory, whether due to an old browser, CSS turned off or whatever. I added text to the labels of each of those 5 fields that essentially instructs to leave those fields blank. Humans will understand that but bots won’t.

I also created 5 differently named CSS classes - each to style just one those 5 fields. The class names are different but all have the same style display:none.

Perhaps these steps are overkill, but its only a few minutes invested and it doesn’t make it any easier for bots. Perhaps I am too anal.

I am considering adding a visible honeypot field named Site URL or something, with the label “Spammers: Enter your website’s address here. Non-Spammers: Leave blank or else…” or some such.

There’s no way a bot wouldn’t take that bait.

One other way to make a honeypot field invisible via CSS is to wrap it in a div with a height of, say, 1 px, a top margin of -1px, and overflow:hidden. That would defeat more sophisticated bots trained to associate fields styled with display:none with fields they should ignore.

I haven’t tried that one, but see no potential downside other than the pain and clutter of wrapping each honeypot field in its own div, as well as all the legit fields, too (so they don’t stand out as being different.)

For now, however, I am pretty sure hiding honeypot fields via CSS display:none should be effective for most of us, under the theory of Security Through Minority.

Again, we are defending against mere spambots, not seasoned hackers looking to plunder bank accounts or something.

To see all the source code for the form I am using to test all of this, here is the test website:

http://thecorvettelist.com/corvette-...ette-test.html

Later…

Russ . . .

…

[russellr]

Hi,

Thanks for posting your ideas and techniques for using Reverse Captcha.

I pretty much agree with all you've said. However, it's hard to get actual data on the effectiveness of Reverse Captcha and Captcha techniques, so many conclusions are "gut feel" or guesswork.

Certainly, all forms we've used that have Reverse Captcha on our site, and customers' sites, have been effective in stopping random spam.

I am considering adding a visible honeypot field named Site URL or something, with the label “Spammers: Enter your website’s address here. Non-Spammers: Leave blank or else…” or some such.

This is a good idea, and avoids the problems you suggested for accessibility.

However, I don't think you want to use the word "spam" or "spammer" anywhere. A message such as this should be sufficient:

<div class="revcap">
<p>If you can see these fields, please ignore them.  We've included them to help protect the internet.</p>
Address: <input type="text" ...>
Blah: <input type="text" ...>
</div>

Where "revcap" is a CSS class that does "display:none".

It's useful to groups spammers into two classes:

1. Random spammers - they and their bots look for any form on any site and try to utilize it.
2. Targeting spammers - they attack a specific site.

If your site is *really important* (such as Google, Microsoft, etc.), then you will be hit by type #2, and there's not much you can do to stop it (but you also have plenty of money to deal with it too!) The spammers can write special software, they can fool people or pay people to fill in forms for them, etc.

Most of us only have to deal with type #1. FormMail's automatic anti-spam features are sufficient to deal with most of these attacks.

Reverse Captcha will take out another chunk of these attacks, and in most cases stop it all.

One last thing...

If you think Captcha provides some "kudos" for your site, then you can always implement Catpcha that does nothing. So, the form looks like it has Captcha, but user mistakes are ignored.

[Russ Edwards]

…

Hi Russell –

Thanks for the tip about not using the word “Spammer” or similar in any part of the form, especially near and spam traps! It is reasonable that a bot might look for similar words as a clue to where potential traps may be.

As to the credibility that might be missing if there is no CAPTCHA on the form (and I am convinced that is the case, especially for my customers’ demographics and psychographics) I have developed a comparable graphic that should fill in the void.

Thanks for your input and for an exceptional product and support!

Russ . . .

…

[Russ Edwards]

…

Hi All –

The reason I started looking into alternatives for CAPTCHA was because all my forms using Google’s reCAPTCHA suddenly stopped working, even though I made no changes to anything on my end.

I doubt Google made any changes to reCAPTCHA, either, but my forms are hosted at Go Daddy, and they have a history of breaking configurations on their shared hosting servers, so I assumed that is where the failure was, although they deny it.

This morning all my forms are suddenly working again! Also, my FTP app – Filezilla – is logging in on the first try, which it hadn’t been doing recently.

I am nearly certain that this has been a GD issue, since two broken things that have significant parts on their end are now fixed.

But this has highlighted the dependency we have on our web hosting companies – they need to dance well with the back and forth and back and forth of CAPTCHA processing. Here is a link to that back and forth spaghetti:

http://code.google.com/apis/recaptcha/docs/display.html

Part of my job is to eliminate as many possible points of failure as I can, and this is a biggie.

Simply eliminating the use of CAPTCHA eliminates this problem!

It also eliminates the usability problem of frustrating users, and the percentage of the time where that frustration causes users / customers to simply go away rather than struggle with a CAPTCHA.

I believe Reverse CAPTCHA is a more than adequate replacement for CAPTCHA, as I have outlined above.

Of course, I also believe that customers, at least my customers, see that reCAPTCHA image as some kind of validation that the website has somehow met some threshold of worthiness, safety, security, or whatever, and is eligible to display that reCAPTCHA “badge.”

Absent that “Good Housekeeping Seal” and the site may seem second rate.

Russell has suggested that to overcome the deficiency of not having reCAPTCHA implemented, one could simply offer a neutered reCAPTCHA on the form.

Great minds (or, at least guys named Russell) think alike.

I thought about that and took it one step further.

I created a substitute image addressing the issue of form “security.”

To be clear on that, we see the issue as one of inconvenience – having our forms used to spam us. That is of no concern to our users, because they are not the ones being spammed. What they see in terms of CAPTCHA is that the site / contact form / registration / order form is somehow more secure, and to them the more secure, the better for them and the more credible the site.

That is, what we see as an annoyance, they see as a security issue. And seeing that CAPTCHA thingie suggests to them that the site has higher security.

To address that, I created an image designed to instill the same sense of “security” that the reCAPTCHA image did.

I don't know how to include images in this forum or even if they are permitted, but here is a link to the Google reCAPTCHA image:

http://eggport.com/images/reCAPTCHA_Sample_Red.png

Here is the Eggport (my company is Eggport) "Spam Stinger" substitute:

http://eggport.com/images/eggport-spam-stinger.png

Obviously “Spam Stinger” is our implementation of Reverse CAPTCHA.

I mention this in the event any of you have the same marketing concern. Simply whip up a similar graphic.

Of course, you should do all you can to secure your users’ data and don’t lead them to believe their info is safe with you if it isn’t.

I’ve made the “NO CAPTCHA” image available as a separate graphic if any of you wish to use it:

http://eggport.com/images/eggport-no-captcha.png

If used, I would add some wording suggesting that you have some alternative process in place that is better than CAPTCHA, so the absence of CAPTCHA doesn’t mean your form is less secure.

You can see the Spam Stinger image in use on my test site here:

http://thecorvettelist.com/corvette-...ette-test.html

This all is still a work in progress, and I am so grateful that FM has the ability to be configured to allow me to be flexible enough to accomplish what I need with my forms.

Later…

Russ . . .

…

[teccet10]

Hi I'm new here. I actually found this post by searching the forum for a honeypot solution and I have marketing background too and since this post is only 2 days old, I thought I'd chime in.

At one time I would have agreed that having a reputable looking captcha device puts you in the same league as those big sites. But not anymore. I've been looking at a lot of online forms over the past few days from huge companies (hyatt dotcom for example- look at their web feedback form) and small companies (that I compete with). None of them are using captcha. At least not a visible one.

So I no longer think it makes a site look more reputable or more professional. On the contrary actually, it's becoming more widely known that REcaptcha is a free google service that anyone can add to their site for free and those things overpower the simple form style that most sites have. So it's like "let's stick this free tool thingy here so we aren't inconvenienced ". It's exactly how I feet about auto-attendants for phone systems. At one time they were used exclusively by big companies. But now small companies use them and some relatively big companies have gone back to answering the phone without having to press 1, press 2 etc. You know- service.

Where I find that I have the best online experiences are where I don't have to jump through any proverbial hoops like recaptcha. Gosh, you know, until I registered for this forum, I had to fill that stinkin thing out every time I did a search and I've not only quite come to hate those things, I also noticed that on a couple of occasions, it didn't work properly. When I used the aural option, it didn't accept my answers at all.

I know those things work (at least for now) and I know they're important because on other forms that I've had over the years, they get a lot of spam. On the other hand, I'd rather have customers get the impression that my site is easy to use and keep submitting service requests with my request form (that's how I use it) than instead say to themselves "ugh, it's got one of those things that's a p.i.t.a. to use, I'll just find some other site or I'll just use the phone." (phone calls take more time than do deleting spam)

On my contact form for the online store I'm setting up, there was an available addon that is a slider like mobile phones have to unlock the touch screen. That unlocks the form. How good it is, I don't know but it won't let you submit until the form is unlocked by the slider. Haven't tried it on a mobile yet though so that could be a drawback. On this request form that I'm just now putting the fishing touches on, I was originally going to install a recaptcha but I'm leaning towards a honeypot or something like "What is this a picture of?" or nothing at all because like I said, I'd rather this site generate leads and spam than no spam and fewer leads.

If I was a betting man, I'd say that in a few years, we'll be saying "recaptcha who?". I think the invisible captchas are where it's trending.

[Russ Edwards]

…

Hi Teccet –

Great points, especially re: telephone auto-attendants.

Years and years ago I bought a music-on-hold thingie at Office Depot for $9.95, assuming it would propel my small company’s image up there with the big-leagues. I had it hooked into a cheap radio set to a local classical music station and, man, I was putting people on hold just to show off.

But then, as the shrewd and wise businessman that I was, I realized any small business with $9.95 could do the same, and my bubble was burst.

Kinda the same deal here with reCAPTCHA.

But now, being even older and wiser than in those music-on-hold days, and perhaps more anal and over-thinking, as well, I see the nuance between how we, as tech savvy marketers see things and how our customers perceive those same things.

Yeah, we all see the wizard behind the reCAPTCHA curtain for what it is – a freebie available to anyone, anywhere, for use on any website with no questions asked.

So, when we see one we’re not impressed. Specifically, we don’t feel an iota better about that site. And if we think about it, we might even glean insight to a number of negative things about that site’s developer – his technical sophistication, his understanding of his customers, his selfishness, his business acumen. We can imagine all sorts of things about why someone might decide to use something we see as so flawed in so many ways.

But that’s just us – we know better.

I am not convinced, however, that our users are as knowledgeable as we are. As ubiquitous as CAPTCHAs have become, I still believe that, depending upon the target market, a significant enough chunk of folks today still see reCAPTCHA as a sign of credibility that it is well worth factoring that in when considering the form’s design.

That is, not all users – perhaps just a fraction of users – know it is a freebie add-on available to any webmaster. Even fewer know it is a Google product. If they did, some Google haters would put on their tin foil hats and run away fast, believing Google would catalog their wrong answers in their permanent record or something.

Obviously, it depends upon the user base. A reCAPTCHA on a tech site, yeah, those users would immediately know the score. They would likely ridicule a tech site using reCAPTCHA.

But a less tech-savvy crowd (like many of my customers) still feels a bit better about sending money to a site that is displaying some indications that they have added security in place. The users may not have any idea as to what that security is, how it works, or how effective it is, but the fact that there is something visible makes them feel better.

Kinda like the elderly gent in the security uniform at my bank.

From a customer’s standpoint, he’s a better alternative than having to empty your pockets and go through a metal detector. And certainly better than nothing at all.

With our forms, anything we ask our users to do beyond filling out the form and hitting Submit is akin to asking them to empty their pockets and walk through the scanner.

So for me, making the customer do anything extra - pick out a certain picture, solve a simple math question, or even having to slide a slider - alters the lead:bail ratio in favor of losing more leads.

Consequently, I’ll opt for covert anti-spam measures (honeypots that are arguably as effective as CAPTCHA) and satisfy the users’ need for reassurance by sticking an elderly gent wearing a security uniform by the door (in the form of a “Secure Site” logo or two.)

That way I think I am having my cake and eating it, too.

For sure YMMV depending upon your customer base. And for sure this will be an ever-evolving issue.

I am grateful that FM has the flexibility to accommodate a wide range of options no matter which road to anti-spam we choose to take.

One quick question about the slider – what about keyboard users or those using screen readers?

Thanks for your input! It is sincerely appreciated.

Russ . . .

…

[teccet10]

A security certificate logo or something like that sounds like a good idea. That way you still have the 85 year old in a rent-a-cop uniform but don't make people have to go through any "porno scanners" as I like to call them. You could boast about how the form has 128 bit encryption and is "totally secure".

Of course if you're in a business where bells and whistles on your website makes you look better than you better use the recaptcha. But I thought we had turned a page in the chapter of web design where bells and whistles were favoured over simplicity and accessibility.

The slider that I was telling you about... you have to click and drag it with your mouse to the right about 250 px to get it to unlock and a small padlock icon turns into a green checkmark. I just tried it on an Android phone and it took some effort but I was able to drag the slider on there. If you are just using a keyboard and no mouse, stylus, touch screen or any other device to slide it then you can't unlock the form. For that reason I won't be implementing it on any mission critical forms. It was my only option for that particular form since it was built in to the cart system and they didn't have any other kind of captcha.

I just set up reverse captcha through formmail which I learned was already built in to do what the honeypot is supposed to do. Since I fully expect the spammers will eventually find a way to ignore hidden css elements, I went ahead and made it perfectly visible... except I styled the textboxes so they were the same colour as the page background and had the same colour border as the background and oh.. by the way, they're only 1px by 1px in size.
So to the naked eye, they're not visible unless you're moving from field to field with the tab key and you're looking really really hard. Works brilliantly. I hope it keeps the spam out.